JP4339671B2 - Imaging device - Google Patents

Description

  The present invention relates to an imaging apparatus.

  In general, a CCD (Charge Coupled Device) as an image pickup element used in a digital camera is a single plate, and missing colors other than the color filters arranged in the pixels are created by a complementary process. On the other hand, since the image signal captured by the CCD contains noise, it is necessary to perform some noise removal processing.

  In addition, a digital camera employs a mode called a RAW mode in which an image signal picked up by a CCD is directly recorded on a recording medium without performing any image processing. Even in the RAW mode, processing for removing noise included in the image signal is required.

For example, Japanese Patent Laid-Open No. 2000-134625 discloses a band dividing step for performing band division processing on a target image with an image that is an image to be processed as input, and a band component changing step for rewriting band components with reference to the divided band components. Discloses a noise removal method including an image synthesis step of synthesizing a rewritten band component with an image.
JP 2000-134625 A

  However, in the related art including the above-described Japanese Patent Application Laid-Open No. 2000-134625, since the noise removal processing is performed after the complement processing, the noise component included in the image signal is diffused in the image space by the complement processing, After that, it is not easy to remove the diffused noise.

  The present invention has been made paying attention to such a problem, and provides an imaging apparatus capable of performing noise removal more completely by performing noise removal processing prior to missing color interpolation processing. It is in.

In order to achieve the above object, the invention according to the first aspect of the present invention is an image pickup in which a plurality of pixels are arranged in a matrix and a color filter having a predetermined color arrangement is arranged in each light receiving portion of the pixels. an imaging apparatus having a device, performs image signal captured a / D converting means for converting a / D, a predetermined conversion process for the image signal from the a / D converting means by the image pickup device By this, the image dividing means for dividing the image signal into image signals having a plurality of different bands and having a plurality of color information different from the color information included in the image signal from the image sensor, and the image dividing means Image changing means for changing an image signal by performing noise reduction processing on at least one image signal of the plurality of image signals generated by the division processing, and the plurality of image signals including the image signal from the image changing means An image signal, wherein the predetermined conversion processing in the image dividing unit by an inverse of the transformation process, including a the synthetic image synthesizing means so as to have the image signals same color sequence and from the imaging element.

The invention according to a second aspect of the present invention is the invention according to the first aspect, the synthesized image signal from the image synthesizing unit performs complement missing color determined by CCD color filter array, It further includes color signal generation means for generating a plurality of color signals.

According to a third aspect of the present invention , in the first or second aspect, the image dividing means includes wavelet transform means, and the image composition means includes inverse wavelet transform means.

The invention according to a fourth aspect of the present invention is the invention according to any one of the first to third aspects , wherein the image changing means is a plurality of images generated by the division processing by the image dividing means. Make different changes to the signals.

The invention according to the fifth aspect of the present invention is an imaging apparatus having an imaging element in which a plurality of pixels are arranged in a matrix and a color filter having a predetermined color arrangement is arranged in each light receiving portion of the pixels. A / D conversion means for A / D converting the image signal picked up by the image pickup device, and the image signal from the A / D conversion means by performing a plurality of conversion processes on the image signal. Is divided into image signals having a plurality of different bands and having a plurality of color information different from the color information included in the image signal from the image sensor, and generated by the division processing by the image dividing means Image changing means for performing noise reduction processing on at least one image signal of the plurality of image signals to change the image signal, and the plurality of divided image signals including the image signal from the image changing means On the other hand, the same color as the image signal from the image sensor is obtained by performing the conversion process opposite to the conversion process in the image dividing unit for the same number of times as the conversion process performed in the image dividing unit. Image synthesizing means for synthesizing the image so as to have an array.

Further, an invention according to a sixth aspect of the present invention is the invention according to the fifth aspect , wherein the composite image signal from the image composition means is supplemented with a missing color determined by a CCD color filter array, It further includes color signal generation means for generating a plurality of color signals.

The invention according to the seventh aspect of the present invention is the invention according to the fifth or sixth aspect, wherein the number of processes and the content of the conversion process are changed by ISO sensitivity and sharpness setting.

  According to the present invention, since noise removal processing is performed prior to missing color complement processing, noise components are not diffused in the image space, and noise removal can be performed more completely. Also, multiple images having specific frequency and color information are created from the input image by a predetermined conversion, and noise is efficiently removed by performing inverse conversion by performing processing suitable for each image. Therefore, it is possible to perform noise removal without impairing the continuity of the image.

(First embodiment)
FIG. 1 is a block diagram showing a schematic configuration of an imaging apparatus according to the first embodiment of the present invention. The CPU 1 is a part that controls each part. The CCD 5 as an image sensor is a part that captures an image of a subject and acquires an image signal. The SDRAM 4 is a part for storing an image signal acquired by the CCD 5 or an image signal subjected to image processing to be described later. The image processing unit 10 is a part that performs predetermined image processing on the image signal captured by the CCD 5. The TFT display unit 2 is a liquid crystal display unit that displays an image signal picked up by the CCD 5. The recording medium 8 is a part for recording an image signal picked up by the CCD 5. The strobe device 11 performs a light emission operation under the control of the strobe control unit 12. The power control unit 9 controls the supply of power necessary for the operation of the imaging apparatus. The operation input device 14 is a part where the user inputs data and instructions. The video interface 3 is a part that becomes an interface of the TFT 2. The CCD interface 6 is a part that serves as an interface for the CCD 5. The operation input interface 13 is a part that becomes an interface of the operation input device 14. The media interface 7 is a part that serves as an interface for recording media.

  FIG. 2 is a flowchart for explaining the photographing procedure. First, an image signal picked up by the CCD 5 is A / D converted and then taken into the SDRAM 4 (step S1). Next, the image signal is read and noise reduction processing is performed (step S2). Next, it is determined whether or not the RAW mode is set. If the RAW mode is set, the process immediately proceeds to step S5. If the RAW mode is not set, image processing including interpolation of missing colors determined by the CCD color filter array is performed (step S4). Details of the image processing will be described later. Thereafter, the process proceeds to step S5.

  In step S5, a header is added to the imaged image signal and recorded on a recording medium 8 such as a recording card. The captured image signal is displayed (RecView display) on the TFT display unit 2 for a predetermined time (step S6).

  FIG. 3 is a flowchart for explaining details of the image processing in step S4 of FIG. Here, as image processing normally performed on the image signal, white balance (WB) processing (step S10), synchronization processing (complement processing) (step S11), color correction processing (step S12), gamma correction processing (step) S13) Conversion processing from RGB space to YCrCb space (step S14) and JPEG compression (encoding) processing (step S15) are performed.

  FIG. 4 is a flowchart for explaining the overall flow of noise removal according to the present embodiment. An image signal from the Bayer array is input and A / D converted, and then wavelet transform is performed on the image signal to perform image division (steps S20 and S21). Here, the simplest Harr Wavelet (N = 1) is used as the wavelet transform. When the scaling coefficient (S) when the R signal is first input as in the Bayer array is obtained, R + (Gr + Gb) + B≈luminance component and low frequency component are obtained (step S22). The wavelet coefficients (W) are R + (Gr−Gb) −B, R + (Gr−Gr) −B, R− (Gr + Gb) + B≈color difference signal and high frequency component, respectively (step S23).

  In this way, by image division using wavelet transform, a plurality of (here, three) different bands are obtained in one operation on the image signal from the CCD 5, and a plurality (here, three) of RGB are Image signals having different color components are generated. Here, wavelet transform is used to obtain such an image signal, but other methods may be used as long as a similar image signal can be obtained.

  Next, an image change such as noise removal is performed by performing LPF processing in a predetermined direction on the wavelet coefficients (color difference signal and high frequency component) (step S24). In the LPF process, separate filters are applied to the three wavelet coefficients.

  Note that the LPF processing is not performed on the luminance component and the low frequency component as the scaling coefficients.

  Next, the original Bayer array image signal is restored by performing inverse wavelet transform on the scaling coefficient (S) obtained in step S22 and the wavelet coefficient subjected to LPF processing (step S25). As described above, in the present embodiment, an image signal from a two-dimensional array (here, Bayer array) is input and noise reduction processing is performed, and then an image based on the same color two-dimensional array (here, Bayer array) again. Return to the signal.

  Next, image processing including complement processing is performed on the restored Bayer array image (step S26).

FIGS. 5A to 5C are diagrams for explaining the details of the wavelet transform in step S21 of FIG. FIG. 5A shows an image sensor based on the Bayer array. The image sensor is composed of 6 × 6 pixels arranged two-dimensionally in the horizontal and vertical directions, and each pixel has R (Red), Gr (Green), Gb (Green), B for each 2 × 2 group. (Blue) color is assigned. When performing the wavelet transform, it is desirable to use the Daubechies base, but here the Harr base is used in consideration of the hardware scale.
In the present embodiment, the wavelet transform is divided into a horizontal direction and a vertical direction wavelet transform, and a calculation formula shown in FIG. 6A is used as a calculation formula used for the horizontal wavelet transformation. By this calculation, a horizontal wavelet transform result as shown in FIG. 5B is obtained. In the wavelet transform using the Harr basis, addition / subtraction is performed using specific two pixels of R, Gr, Gb, and B in units of 2 × 2 pixels. As a general formula, Sn = (Rn + Grn) / √2 (Here, the square root of 2 is expressed as √2) and Wn = (Rn−Grn) / √2. For example, for example, in order to obtain the value of S1 in FIG. 5B, the formula of S1 = (R1 + Gr1) / √2 in FIG. 6A is used. Further, in order to obtain the pixel value of s1 in FIG. 5B, the equation of s1 = (Gb1 + B1) / √2 in FIG. 6A is used.

  Next, a vertical wavelet transform is performed on the horizontal wavelet transform result shown in FIG. 5B to obtain a transform result as shown in FIG. In order to obtain each pixel value in FIG. 5C, here, as a general expression, SSn = (Sn + sn) / √2, SWn = (Sn−sn) / √2, WSn as shown in FIG. 6B. = (Wn + wn) / √2, and WWn = (Wn−wn) / √2.

  By the above-described horizontal wavelet transform and vertical wavelet transform processing, three wavelet coefficients and one scaling coefficient different from the RGB signal from the Bayer array as shown in FIG. 6C are obtained. SSn is a scaling factor, WSn is a signal component detected in the vertical direction, SWn is a signal component detected in the horizontal direction, and WWn is a signal component detected in the diagonal direction, each having a different signal band and color component. is doing. The size of each coefficient is 1/4 of the image signal by the Bayer array.

  FIG. 7 is a diagram for explaining the details of the LPF process in step S24 of FIG. Here, LPF processing is performed on wavelet coefficients (SW, WS, WW) obtained by wavelet transformation. Here, in order not to impair the characteristics of the original image, the LPF corresponding to the vertical, horizontal, and diagonal detection directions is applied. That is, for the wavelet coefficient 101-1 in the horizontal direction, SW_fin is obtained as the LPF result by multiplying the LPF coefficient having a value at the horizontal center position 102-1 in the 3 × 3 array. Also, the vertical wavelet coefficient 101-2 is multiplied by an LPF coefficient having a value at the vertical center position 102-2 of the 3 × 3 array to obtain WS_fin as an LPF result. In addition, for the wavelet coefficient 101-3 in the oblique direction, WW_fin is obtained as an LPF result by multiplying the LPF coefficient having a value at an oblique position 102-3 passing through the center of the 3 × 3 array. Further, the LPF process is not performed on the scaling coefficient 100, and the central pixel value SS5 of the 3 × 3 array is output.

  Note that the filter coefficients may be changed according to the characteristics of the image instead of the LPF coefficients fixed in the direction as shown in FIG.

  The inverse wavelet transform unit 105 performs addition / subtraction as shown in FIG. 8 using SW_fin, WS_fin, WW_fin and the scaling coefficient SS5 calculated so far, thereby obtaining R5 ′, Gr5 ′, Gb5 ′ and B5 ′ are obtained. Each inverse wavelet transform result R5 ', Gr5', Gb5 ', B5' corresponds to the center pixel R5, Gr5, Gb5, B5 of the 6x6 Bayer array shown in FIG.

  Next, the calculated value is clipped to any value from 0 to 1023 (in the case of 10-bit input).

  In the first embodiment, wavelet transform is performed on a signal of a 6 × 6 pixel Bayer array, but the present invention is not limited to this, and an image of one frame may be targeted.

  According to the first embodiment described above, the image signal obtained from the pixels in the Bayer array is decomposed into three image signals having different frequency components and colors by one operation using, for example, wavelet transform, and each is specified. After performing the direction filtering process, a Bayer image signal from which noise has been removed by inverse transformation is obtained. Therefore, noise removal processing is performed prior to color complementation processing, and there is no problem that noise components are diffused in the image space, and more complete noise removal is performed. In addition, multiple images having specific frequency bands and color information are created from the input image by a predetermined conversion, and processing appropriate for each image is performed and inverse conversion is performed, so that efficient noise removal according to noise frequency characteristics is achieved. Thus, it is possible to remove noise without impairing the continuity of the image due to the influence of the missing color determined by the color arrangement of the CCD.

  The above-described image processing unit is often realized by a single ASIC. However, since the noise removal processing according to the present embodiment is performed separately prior to image processing, the ASIC is expanded without changing the ASIC configuration itself. A noise removal function can be added by a DSP or the like. In addition, it is possible to perform noise removal processing without interrupting pipeline processing such as WB processing, missing color complement processing, color correction processing, gamma correction processing, resizing processing, and JPEG compression processing.

(Second Embodiment)
FIG. 9 is a flowchart for explaining an imaging method according to the second embodiment of the present invention. First, an image from a Bayer array composed of 8 × 8 pixels is input and A / D converted, and then wavelet converted (steps S30 and S31). Thus, one scaling coefficient S and three wavelet coefficients (vertical wavelet coefficient WS, horizontal wavelet coefficient SW, and oblique wavelet coefficient WW) are obtained (steps S32-1 to S32-4). . At this time, the size of each coefficient is 4 × 4.

  Next, each wavelet coefficient is subjected to wavelet transform (steps S33-1 to S33-3), and one scaling coefficient S and three wavelet coefficients W are obtained in each step. That is, the wavelet coefficient WS is subjected to wavelet transform (step S33-1), thereby obtaining one scaling coefficient S and three wavelet coefficients W (steps S34-1 to S34-4). Similarly, the wavelet coefficient SW is subjected to wavelet transform (step S33-2), thereby obtaining one scaling coefficient S and three wavelet coefficients W (steps S34-5 to S34-8). Similarly, the wavelet coefficient WW is subjected to wavelet transform (step S33-3), thereby obtaining one scaling coefficient S and three wavelet coefficients W (steps S34-9 to S34-12). At this time, the size of each coefficient is 2 × 2.

  Next, LPF in a predetermined direction is applied to the wavelet coefficients (W) obtained in steps S34-2, S34-3, and S34-4 (step S35-1). Similarly, LPF in a predetermined direction is applied to the wavelet coefficient (W) obtained in steps S34-6, S34-7, and S34-8 (step S35-2). Similarly, LPF in a predetermined direction is applied to the wavelet coefficients (W) obtained in steps S34-10, S34-11, and S34-12 (step S35-3).

  10A to 10C are diagrams for explaining the details of the LPF processing (steps S35-1 to 35-3). FIG. 10A is a diagram for explaining LPF processing for the wavelet coefficients in the vertical direction in step S35-1. For the wavelet coefficients in the vertical direction, addition is performed for the set of pixels A and C and the set of B and D. That is, the pixels A and C are obtained by (A + C) / 2, and the pixels B and D are obtained by (B + D) / 2 and are output as wavelet coefficients WS_WS. FIG. 10B is a diagram for explaining LPF processing for the horizontal wavelet coefficients in step S35-2. For the wavelet coefficients in the horizontal direction, addition is performed for the set of pixels A and B and the set of C and D. That is, the pixels A and B are obtained by (A + B) / 2, and the pixels C and D are obtained by (C + D) / 2 and output as wavelet coefficients SW_SW.

  FIG. 10C is a diagram for explaining the LPF process for the wavelet coefficient in the oblique direction in step S35-3. For the wavelet coefficients in the oblique direction, addition is performed for the set of pixels A and D and the set of B and C. That is, the pixels A and D are obtained by (A + D) / 2, and the pixels C and B are obtained by (B + C) / 2 and are output as wavelet coefficients WW_WW.

  Next, the output wavelet coefficients WS_WS, SW_SW, and WW_WW are multiplied by 2 × 2 (steps S36-1 to S36-3), and then the inverse wavelet transform is performed on each result (step S36-1). S37-1 to S37-3). At the same time, in this step, inverse wavelet transform is performed on the scaling coefficient (S) obtained in steps S34-1, S34-5, and S34-9. At this time, the size of each coefficient is 4 × 4.

  The inverse wavelet transform results in steps S37-1 to S37-3 are subjected to inverse wavelet transform together with the scaling coefficient (S) in step S32-1 (step S38). As a result, the size of each coefficient is 8 × 8. Next, image processing including missing color interpolation is performed on the inverse wavelet transform result in step S38 (step S39).

  According to the second embodiment described above, in addition to the effect of the first embodiment, there is an effect that a low-frequency noise component can be suppressed regardless of the color component. In this embodiment, an 8 × 8 image is input and wavelet transform is performed twice. However, it is also possible to input a larger image and perform wavelet transform three or more times.

  FIG. 11 shows an example in which the number of wavelet transforms is determined according to the ISO sensitivity and sharpness setting set in the camera. Here, the ISO sensitivity is 100 → 200 → 400, the sharpness is Lo → Normal → Hi, and the noise is increased, so that the number of wavelet transforms is increased, and the obtained wavelet coefficient W is smoothed by LPF, median filter, etc. (Steps S50 to S54-5). Thereby, there is an effect that the frequency band for noise removal can be optimized.

1 is a block diagram illustrating a schematic configuration of an imaging apparatus according to a first embodiment of the present invention. It is a flowchart for demonstrating an imaging | photography procedure. It is a flowchart for demonstrating the detail of the image process of step S4 of FIG. It is a flowchart for demonstrating the whole flow of the noise removal which concerns on this embodiment. It is a figure for demonstrating the detail of the wavelet transformation of step S21 of FIG. (A) shows the calculation formula used for horizontal wavelet transformation, (b) shows the calculation formula used for vertical wavelet transformation, and (c) shows the wavelet transformation result. It is a figure for demonstrating the detail of the LPF process of FIG.4 S24. It is a figure which shows an inverse wavelet transformation result. It is a flowchart for demonstrating the imaging method which concerns on 2nd Embodiment of this invention. (A)-(c) is a figure for demonstrating the detail of LPF process. It is a figure which shows the example which determines the frequency | count of several wavelet transformation according to the ISO sensitivity and sharpness setting which are set to the camera.

Explanation of symbols

  1 ... CPU, 2 ... TFT, 3 ... Video I / F, 4 ... SDRAM, 5 ... CCD, 6 ... CCD I / F, 7 ... Media I / F, 8 ... Recording media, 9 ... Power control unit, 10 ... Image Processing unit 11. Strobe device 12 Strobe control unit 13 Operation input I / F 14 Operation input device

Claims (7)

An imaging apparatus having an imaging element in which a plurality of pixels are arranged in a matrix and a color filter of a predetermined color arrangement is arranged in each light receiving portion of the pixels,
A / D conversion means for A / D converting an image signal captured by the image sensor;
By performing a predetermined conversion process on the image signal from the A / D conversion means, the image signal has a plurality of different bands, and a plurality of color information different from the color information included in the image signal from the image sensor. Image dividing means for dividing the image signal having color information;
Image changing means for changing an image signal by performing noise reduction processing on the at least one image signal of the plurality of image signals generated by the division processing by the image dividing means;
The plurality of image signals including the image signal from the image changing unit are converted to have the same color arrangement as the image signal from the imaging element by a conversion process opposite to the predetermined conversion process in the image dividing unit. Image synthesizing means to synthesize,
An imaging apparatus comprising: The synthesized image signal from the image synthesizing unit performs missing color complement determined by CCD color filter array, according to claim 1, characterized in that it further comprises a color signal generating means for generating a plurality of color signals Imaging device. The image dividing means includes a wavelet transform unit, the image synthesizing unit is imaging apparatus according to claim 1 or 2 wherein, characterized in that it comprises an inverse wavelet transform unit. The imaging apparatus according to any one of claims 1 to 3 , wherein the image changing unit performs different changes on the plurality of image signals generated by the division processing by the image dividing unit. An imaging apparatus having an imaging element in which a plurality of pixels are arranged in a matrix and a color filter of a predetermined color arrangement is arranged in each light receiving portion of the pixels,
A / D conversion means for A / D converting an image signal captured by the image sensor;
By performing a plurality of conversion processes on the image signal from the A / D conversion means, the image signal has a plurality of different bands and is different from the color information included in the image signal from the image sensor. Image dividing means for dividing the image signal into color image information,
Image changing means for changing an image signal by performing noise reduction processing on the at least one image signal of the plurality of image signals generated by the division processing by the image dividing means;
With respect to the divided plurality of image signals including image signals from said image changing means, the inverse of the conversion processing and conversion processing by the image dividing unit, and the number of conversion processing performed by the image dividing means Image synthesizing means for synthesizing the image signal from the image sensor so as to have the same color arrangement by performing the same number of times ,
An imaging apparatus comprising: For the combined image signal from the image synthesizing unit performs missing color complement determined by CCD color filter array, according to claim 5, characterized by further comprising a color signal generating means for generating a plurality of color signals Imaging device. ISO sensitivity, the sharpness setting, the imaging apparatus according to claim 5 or 6, wherein changing the contents of the conversion processing and processing times.

Images