JP2012227869A - Image processing device, image processing method and digital camera - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate a luminance signal in a small circuit scale while reducing aliasing in an oblique direction component while maintaining excellent resolution in red and blue parts, from an image signal read from an imaging element using a color filter.SOLUTION: Filtering 204 is performed in an oblique direction only on red (R) and blue (B) data in image signal data 201 read from an imaging element using a color filter. Green (G) data is not filtered and output as it is to generate data 205 having two color components, green and a pixel color obtained by the above processing. High-pass filtering is performed on the data 205 to generate a luminance signal with reduced aliasing.

Description

  The present invention relates to an image processing apparatus that performs image processing on RAW image data output from an image sensor, and more particularly to image processing such as a digital still camera and development image processing software executed on a PC.

  In general, in order to generate a color image using an image sensor that can detect the amount of light, such as a CCD image sensor or a CMOS image sensor, a configuration in which light transmitted through a color filter is incident on the image sensor is employed.

  There are various types of color filters depending on the type of color to be used and the color arrangement assigned to each pixel. At present, the color type is the primary color (red, green, blue), and the color arrangement is the Bayer arrangement. Has become the mainstream.

  FIG. 16 is a diagram showing one unit of a color filter in the primary color Bayer arrangement. Actually, the same arrangement is repeated according to the number of pixels of the image sensor. R is red, G1 and G2 are green, and B is blue.

  FIG. 17 illustrates an Out_Of_Green method (hereinafter referred to as an OG method) that generates a luminance signal (OG signal) from only a green (G) signal among conventional methods for generating a luminance signal using the color filters of the primary color Bayer arrangement shown in FIG. A configuration example of a luminance signal generation circuit for realizing

  First, a zero insertion circuit 2201 is applied to the RAW signal 2200 obtained by digitizing the output 2204 of the image sensor to generate a signal 2205 in which values other than the G pixel are set to 0. Next, a low-pass filter (V-LPF) circuit 2202 for limiting the vertical band and a low-pass filter (H-LPF) circuit 2203 for limiting the horizontal band are applied to obtain a luminance signal (OG signal).

  Further, as another example of the conventional method for generating the luminance signal using the color filters of the primary color Bayer array shown in FIG. 17, the SwitchY method (hereinafter referred to as SWY method) for generating the luminance signal using all the RGB pixels. There is.

  FIG. 18 is a diagram illustrating a configuration example of a luminance signal generation circuit that realizes the SWY method. As is clear from comparison with FIG. 17, the SWY method is a method of obtaining a luminance signal without using the Zero insertion circuit 2201 in the OG method. Hereinafter, a luminance signal obtained by the SWY method is referred to as a SWY signal.

  FIG. 19 is a diagram showing a resolvable spatial frequency characteristic of the OG signal and the SWY signal. The horizontal axis indicates the frequency space in the horizontal (H) direction of the subject, the vertical axis indicates the frequency space in the vertical (V) direction, and the spatial frequency increases as the distance from the origin increases. Since the OG signal generates a luminance signal only from the G signal, the resolution limit in the horizontal and vertical directions is equal to the Nyquist frequency (on the axis, π / 2) of the image sensor. However, since there are lines in which no pixels exist in the oblique direction, the limit resolution frequency in the oblique direction is lower than that in the horizontal and vertical directions, and as a result, the spatial frequency region 2400 on the rhombus becomes a resolvable spatial frequency.

  On the other hand, since the SWY signal is generated using all the pixels, when the subject is achromatic, the square area 2401 as shown in FIG. However, for a red subject, for example, a luminance signal is not output from pixels other than the R pixel, so that the resolution is performed only in the spatial frequency range 2402 that is half of both the horizontal and vertical directions compared to an achromatic subject.

  Patent Document 1 proposes a method of replacing the oblique region 2403 of the OG signal in FIG. 19 with the SWY signal. However, since the resolution limit frequency of the SWY signal is lowered for the chromatic subject, the OG signal is replaced with the SWY signal only when the oblique region 2403 is an achromatic subject. Thereafter, an edge enhancement component is detected using the generated luminance signal and added to the luminance signal to generate a final luminance signal.

  Further, in the method described in Patent Document 2, the first adaptive high-frequency signal is generated by using the OG signal, and the angle adaptive SWY method in which not only the horizontal and vertical directions but also the diagonal band is limited from the signals of all the color pixels. A second high-frequency signal is generated using the luminance signal generated in step S1, and a third high-frequency signal is generated by weighted addition of the first high-frequency signal and the second high-frequency signal according to the spatial frequency of the signal, The final luminance signal is generated by adding the OG signal and the third high-frequency signal.

  Furthermore, the method described in Patent Document 3 obtains each luminance signal output from a plurality of LPF characteristics, calculates an index indicating a measure of redness and blueness of the pixel of interest, and based on the index, The high-frequency luminance signal and the high-frequency-suppressed luminance signal are weighted and added so that the higher the redness or blueness of the target pixel is, the higher the ratio of the luminance signal suppressed by the LPF is. The final luminance signal for is generated.

JP 2003-348609 A JP 2008-72377 A JP 2010-41511 A

  However, when the method described in Patent Document 1 is applied to a signal obtained using a primary color Bayer array color filter, the SWY signal generated by the luminance signal generation circuit shown in FIG. However, although the resolution in the oblique direction is improved, there may be a case where the modulation component toward the low frequency side of the color carrier is conspicuous in the switching process portion of the boundary between the achromatic color and the chromatic color.

  FIG. 4 shows an example of an image when luminance signal processing is performed. FIG. 5A shows Bayer array RAW data of an oblique color palette. Regarding the symbols indicating the colors in the figure, R is red, G is green, B is blue, Cy is cyan, Ye is yellow, and Mg is magenta. FIG. 4C shows image processing data obtained by performing SWY luminance signal processing using all pixels in Bayer array RAW data of an oblique color palette. From the image observation in FIG. 5C, it can be seen that the high frequency information of the original image is secured, but the color carrier aliasing component is conspicuous in the color boundary switching processing portion of the chromatic color palette.

  Hereinafter, the aliasing component image of the color carrier is expressed as “zipper noise” in the switching processing portion between the colors of the chromatic color palette.

  Further, since the OG signal is used for the chromatic subject, the resolution in the oblique direction for the chromatic subject is not improved. Further, when the edge emphasis signal is generated by the luminance signal after a part of the OG signal (oblique area 2403) is replaced with the SWY signal, the switching portion between the OG signal and the SWY signal is emphasized, and the color carrier is shifted to the low frequency side. Since the modulation component of the image appears as an unnatural texture, the degree of edge enhancement is limited.

  Further, in Patent Document 2, in order to suppress image quality degradation in a red subject, a red region is detected, and the region generates a final luminance signal using a first high-frequency signal generated from an OG signal. . Therefore, similarly to Patent Document 1, there is a problem that the aliasing distortion (zipper noise) of the red and blue diagonal lines shown in FIG. 4C cannot be removed.

  Furthermore, Patent Document 3 employs a configuration that can suppress the occurrence of aliasing distortion (zipper noise) in a diagonal component of a red or blue subject, but a red detection circuit, a blue detection circuit, and an index calculation. There is a drawback that the circuit scale becomes large due to the need for means. In addition, in the red and blue portions, the luminance signal is generated so that the ratio of the luminance signal that is suppressed in the high frequency region is high. Therefore, the characteristic is less than a half Nyquist frequency, and red and blue are present. There is a drawback in that the resolution of the chromatic portion is greatly suppressed and the clear stereoscopic effect of the chromatic subject is obstructed. Further, in Patent Document 3, when the level of random noise of the sensor output is high due to high sensitivity photography or the like, an erroneous determination processing operation occurs in calculating the redness and blueness indexes, and the original image has a random graininess. It is expected that a high-band luminance signal with a sense of damaged artificial noise will be output. In addition, the determination processing is diverse, and when the circuit is mounted, the scale becomes large and the image quality adjustment setting for obtaining a desired luminance signal characteristic increases. When implemented in a program, its execution time is expected to increase.

  The present invention has been made in view of such circumstances, and its purpose is to reduce false luminance signals by suppressing the generation of zipper noise when performing image processing for generating luminance signals from RAW original images. However, it is to maintain a good resolution.

  Furthermore, in addition to the above object, the present invention realizes an image processing algorithm and architecture for luminance signal generation with a simple configuration to minimize the circuit scale for obtaining desired characteristics, and at the time of program implementation. The second object is to shorten the execution time.

  In order to achieve the above object, the image processing apparatus according to the first aspect of the present invention is a luminance that generates a luminance signal for each pixel from RAW image data read from an image sensor having a periodically arranged color filter. An image processing apparatus including a signal generation device, wherein the luminance signal generation device is configured such that a pixel position of the RAW image data is an odd number in a row direction, an even number in a column direction, and an even number in a row direction. Thus, low-pass filter processing is not performed for pixels with odd pixel positions in the column direction, and low-pass filter processing is performed only for pixels in odd positions in the row direction and column direction and even pixels in row direction and column direction. Pixel position selection filter processing means to be implemented, and luminance signal processing means for performing high-band luminance signal processing with the output data of the pixel position selection filter processing means as an input To.

  An image processing apparatus according to a second aspect of the present invention is provided with a luminance signal generation apparatus that generates a luminance signal for each pixel from RAW image data read from an image sensor having a color filter with a periodic array. The luminance signal generation device includes pixels in which the pixel position of the RAW image data is odd in the row direction and the pixel position in the column direction is even, and the pixel position in the row direction is even and the pixel position in the column direction is odd. Pixel position selection filter processing means that performs low-pass filter processing only on the pixel, and does not perform low-pass filter processing on pixels in odd positions in the row direction and column direction and pixels in even positions in the row direction and column direction, and And luminance signal processing means for performing high-band luminance signal processing with the output data of the pixel position selection filter processing means as an input.

  A third aspect of the present invention is the image processing apparatus according to any one of the first and second aspects, wherein the filter array of the image pickup device including the color filters of the periodic array is a Bayer array. The pixel position selection filter processing means selectively performs low-pass filter processing on each target pixel of red (R) and blue (B) among the original pixel data of three colors of RGB.

  A fourth aspect of the present invention is the image processing apparatus according to any one of the first and second aspects, wherein the filter array of the image pickup device including the color filters of the periodic array is a Bayer array. The pixel position selection filter processing means selectively performs red (R) and blue (B) when performing low-pass filter processing using peripheral pixels for the red (R) and blue (B) target pixels. Filter processing is performed on the pixels in the peripheral pixel area of odd number (2n + 1) × odd number (2n + 1) with each pixel of interest as the center, and the original image of the Bayer array is displayed as periodic green (G) and the result of the calculation processing Data is converted into two types of color pixel data.

  A fifth aspect of the present invention is the image processing apparatus according to any one of the first and second aspects, wherein the filter array of the image pickup device including the color filters of the periodic array is a Bayer array. The pixel position selection filter processing means selectively performs the low-pass filter processing on the red (R) and blue (B) target pixels using peripheral pixels, and the pixel position selection filter processing means Filter operation processing is performed using only red (R) pixels and blue (B) pixels located in the diagonally left direction, and red and blue are applied to each target pixel of red (R) and blue (B). This is characterized in that the original image of the Bayer array is converted into two types of periodic pixel data of green (G) and magenta (M) by replacing the synthesized magenta (M) color component.

  A sixth aspect of the present invention is the image processing apparatus according to any one of the first and second aspects, wherein the filter array of the image pickup device including the color filters of the periodic array is a Bayer array. The pixel position selection filter processing means selectively performs the low-pass filter processing on the red (R) and blue (B) target pixels using the peripheral pixels, and the pixel position selection filter processing unit is positioned in an oblique direction with the target pixel as the origin. In addition to the red (R) and blue (B) pixels to be processed, the filter calculation processing is also performed in which the components of the green (G) pixels adjacent in the horizontal and vertical directions are added, and attention is paid to red (R) and blue (B). The pixel is replaced with a white (W) color component formed by combining red, blue, and green, and the original image in the Bayer array is converted into two types of color pixel data of green (G) and white (W). It is characterized by converting into.

  A seventh aspect of the present invention is the image processing apparatus according to any one of the fourth to sixth aspects, wherein the image data is further horizontal with respect to the image data converted into two types of periodic color pixel data. And color carrier removal means for generating a high-frequency luminance signal by performing periodic color carrier removal processing in the vertical direction.

  The invention according to claim 8 is the image processing apparatus according to any one of claims 4 to 6, wherein the image data converted into two kinds of periodic color pixel data is horizontal and Bandpass filter processing means for removing a high frequency component including a periodic color carrier in the vertical direction and a low frequency component including a DC component, and a low frequency for color-separating and outputting original pixel data of three colors of RGB for each color RGB color separation signal processing means, and low-pass filter processing means for filtering a band corresponding to a low-frequency component including a direct-current component removed by the band-pass filter processing for the RGB data color-separated and output for each color. And signal adding means for performing addition processing of the high-pass data after the band-pass filter processing and the low-pass color separation RGB component after the low-pass filter processing.

  The invention according to claim 9 is the image processing apparatus according to claim 8, wherein the resolution is controlled by controlling the magnitude of the AC component processed by the band-pass filter processing means as a resolution adjustment signal. The high frequency level adjusting means in the alternating current component is adjusted, and the signal adding means is configured to adjust the high frequency data after the band pass filter processing and the low frequency after the low pass filter processing. Addition processing with color separation RGB components is performed, and a high-frequency luminance signal is obtained by the output of the signal adding means.

  A tenth aspect of the present invention is the image processing apparatus according to the ninth aspect, wherein the high frequency level adjusting means is a magnitude of an alternating current component processed by the bandpass filter processing means as a resolution adjustment signal. High-frequency level adjustment data that determines the magnitude of the AC component independently from the green (G) pixel position, the red (R) pixel position, and the blue (B) pixel position of the original image when controlling the height. Pixel-by-pixel position adjustment data storage means, and multiplication means for multiplying the magnitude of the AC component processed by the band-pass filter processing means in accordance with the high-frequency level adjustment data stored by the pixel position. The signal adding means includes a green (G) pixel position, a red (R) pixel position, and a blue (B) pixel position of the original image with respect to the high-pass data after the band-pass filter processing. The size of the AC component is adjusted for each color. And high-frequency data, which comprises carrying out the addition processing of a low pass color separation RGB components after the low pass filtering.

  An image processing method according to an eleventh aspect of the present invention is an image processing method having a luminance signal generation method for generating a luminance signal for each pixel from RAW image data read from an image sensor having a periodic arrangement of color filters. In the luminance signal generation method, the pixel position of the RAW image data is an odd-numbered pixel in the row direction and an even-numbered pixel position in the column direction, and an even-numbered pixel position in the row direction and an odd-numbered pixel position in the column direction. A pixel position selection filter processing step for performing low-pass filter processing only on pixels at odd positions in the row direction and column direction and pixels at even positions in the row direction and column direction without performing low-pass filter processing for A luminance signal processing step of performing high-band luminance signal processing using the output data of the pixel position selection filter processing step as an input.

  An image processing method according to a twelfth aspect of the present invention is an image processing method including a luminance signal generation method for generating a luminance signal for each pixel from RAW image data read from an image sensor having a color filter with a periodic array. In the luminance signal generation method, the pixel position of the RAW image data is an odd number in the row direction, the pixel position in the column direction is an even number, and the pixel position in the row direction is an even number and the pixel position in the column direction is an odd number. A pixel position selection filter processing step that performs low-pass filter processing only on the pixel, and does not perform low-pass filter processing on pixels in odd positions in the row direction and column direction and pixels in even positions in the row direction and column direction; And a luminance signal processing step for performing high-band luminance signal processing using the output data of the pixel position selection filter processing step as an input.

  A thirteenth aspect of the invention is the image processing method according to any one of the eleventh and twelfth aspects of the invention, wherein the filter arrangement of the image pickup device having the periodic arrangement of color filters is a Bayer arrangement. The pixel position selection filter processing step is characterized in that low-pass filter processing is selectively performed on each pixel of interest of red (R) and blue (B) among the original pixel data of three colors of RGB.

  A fourteenth aspect of the invention is the image processing method according to any one of the eleventh and twelfth aspects, wherein the filter arrangement of the image pickup device having the periodic arrangement of color filters is a Bayer arrangement. In the pixel position selection filter processing step, red (R) and blue (B) are selectively used when low-pass filter processing is performed using peripheral pixels on the red (R) and blue (B) target pixels. Filter processing is performed on the pixels in the peripheral pixel area of odd number (2n + 1) × odd number (2n + 1) with each pixel of interest as the center, and the original image of the Bayer array is displayed as periodic green (G) and the result of the calculation processing Data is converted into two types of color pixel data.

  A fifteenth aspect of the invention is the image processing method according to any one of the eleventh and twelfth aspects, wherein the filter arrangement of the image pickup element having the periodic arrangement of color filters is a Bayer arrangement. In the pixel position selection filter processing step, when the low-pass filter processing is selectively performed on the red (R) and blue (B) target pixels using peripheral pixels, the pixel position selection filter processing step Filter operation processing is performed using only red (R) pixels and blue (B) pixels located in the diagonally left direction, and red and blue are applied to each target pixel of red (R) and blue (B). This is characterized in that the original image of the Bayer array is converted into two types of periodic pixel data of green (G) and magenta (M) by replacing the synthesized magenta (M) color component.

  A sixteenth aspect of the invention is the image processing method according to any one of the eleventh and twelfth aspects, wherein the filter arrangement of the image pickup device having the periodic arrangement of color filters is a Bayer arrangement. In the pixel position selection filter processing step, when the low-pass filter processing is performed selectively on the red (R) and blue (B) target pixels using the peripheral pixels, the pixel position selection filter processing step is positioned in an oblique direction with the target pixel as the origin. In addition to the red (R) and blue (B) pixels to be processed, the filter calculation processing is also performed in which the components of the green (G) pixels adjacent in the horizontal and vertical directions are added, and the attention of red (R) and blue (B) The pixel is replaced with a white (W) color component formed by combining red, blue, and green, and the original image in the Bayer array is converted into two types of periodic pixel data of G and W. To do.

  The invention according to claim 17 is the image processing method according to any one of claims 14 to 16, wherein the image data converted into two types of periodic color pixel data is further horizontal. And a color carrier removal step of performing a periodic color carrier removal process in the vertical direction to generate a high-frequency luminance signal.

  The invention according to claim 18 is the image processing method according to any one of claims 14 to 16, wherein the image data converted into two kinds of periodic color pixel data are horizontally and A bandpass filter processing step for removing a high-frequency component including a periodic color carrier in the vertical direction and a low-frequency component including a direct current component, and a low frequency for color-separating and outputting original pixel data of three colors of RGB for each color An RGB color separation signal processing step, and a low-pass filter processing step of filtering a band corresponding to a low-frequency component including a direct-current component removed by the band-pass filter processing with respect to RGB data color-separated and output for each color. And a signal addition step of performing addition processing of the high-pass data after the band-pass filter processing and the low-pass color separation RGB component after the low-pass filter processing.

  The image processing method according to claim 19 is the image processing method according to claim 18, wherein the resolution is controlled by controlling the magnitude of the AC component processed in the bandpass filter processing step as a resolution adjustment signal. A high-frequency level adjusting step in the AC component, and the signal adding step includes a high-frequency data in which the high-frequency level after the band-pass filter processing is adjusted, and a low-frequency level after the low-pass filter processing. Addition processing with color separation RGB components is performed, and a high-frequency luminance signal is obtained by an output of the signal addition step.

  A twentieth aspect of the present invention is the image processing method according to the nineteenth aspect, wherein the high-frequency level adjusting step includes a magnitude of the AC component processed in the band pass filter processing step as a resolution adjustment signal. High-frequency level adjustment data that determines the magnitude of the AC component independently from the green (G) pixel position, the red (R) pixel position, and the blue (B) pixel position of the original image when controlling the height. A pixel-by-pixel position adjustment data storing step, and a multiplication step for multiplying the magnitude of the AC component processed in the bandpass filter processing step in accordance with the high-frequency level adjustment data stored by the pixel position. And the signal adding step includes a green (G) pixel position, a red (R) pixel position, and a blue (B) pixel position of the original image with respect to the high-pass data after the bandpass filter processing. Adjust the AC component size for each color And high-frequency data, which comprises carrying out the addition processing of a low pass color separation RGB components after the low pass filtering.

  According to a twenty-first aspect of the present invention, there is provided a digital camera including a processing function unit including the image processing apparatus according to any one of the first to tenth aspects.

  As described above, according to the image processing apparatus of the present invention, the false luminance signal is reduced by suppressing the occurrence of aliasing distortion (zipper noise) at the boundary between the color in the oblique direction for the red or blue subject. However, good resolution can be maintained.

  In addition, since the simple configuration is such that only low-pass filter processing is performed on specific pixels of RAW image data, the circuit scale for obtaining desired characteristics can be minimized, and a program At the time of implementation, the execution time can be shortened.

  Furthermore, it is possible to generate a luminance signal with high-bandwidth characteristics with a natural and low noise feeling even for the original image containing random noise components, which is a problem at the time of gain increase, and is combined with predetermined color signal processing and post-processing. In addition, it is possible to provide an imaging device that can capture high-quality still images and moving images.

1 is a schematic configuration diagram of an image processing apparatus according to a first embodiment of the present invention. It is a whole block diagram which shows the imaging device containing the image processing apparatus. It is a figure which shows the detail of the 1st algorithm of the color carrier conversion process in the image processing apparatus. (A) is a diagram showing an oblique color palette image as an input for luminance signal processing of the image processing apparatus, (b) is a diagram showing an image of a color carrier conversion processing result in the embodiment, (c) ) Is a diagram showing an image in which zipper noise is present in the conventional image processing, and FIG. 8D is a diagram showing an image after the low-pass filter processing in the present embodiment. (A) is a diagram showing the color boundary of the subject image, (b) is a diagram showing RAW image data from the image sensor when it has the same color boundary, and (c) is a pixel selection process and a filter calculation process. FIG. 4D is a diagram showing RAW data after the filter calculation process. It is a figure which shows the detail of the 2nd algorithm of the color carrier conversion process in the image processing apparatus. It is a figure which shows the modification of the same color carrier conversion process. It is a figure which shows the 1st process block which combined the color carrier conversion process and the luminance signal process in the image processing apparatus. (A) is a figure which shows the filter processing characteristic implemented by the high frequency luminance signal processing (1) process of the said 1st processing block, The figure (b) is a low frequency luminance signal processing process of the said 1st processing block. It is a figure which shows the filter processing characteristic to implement. (A) is a view showing RAW data composed of G pixels and M pixels generated by the color carrier conversion process of the embodiment, and (b) is a high band of each pixel after the band pass filter process of the embodiment. The figure which shows a component and the figure (c) are figures which show the production | generation algorithm of the high region luminance signal data by which the feeling of resolution adjustment was carried out. It is the 2nd processing block diagram which combined color carrier conversion processing and luminance signal processing in the image processing device. It is a figure which shows the filter processing characteristic implemented at the luminance signal processing (2) process of the said 2nd processing block. It is a flowchart figure which shows the principal part of the color carrier conversion process of the image processing apparatus. It is a flowchart figure which shows the whole same color carrier conversion process. It is a flowchart figure which shows the detail of the same color carrier conversion process. It is a figure which shows 1 unit of a primary color Bayer arrangement | sequence. It is a block diagram which shows the structural example of the luminance signal generation circuit by the conventional OG system. It is a block diagram which shows the structural example of the luminance signal generation circuit by the conventional SWY system. It is a figure which shows the resolvable spatial frequency characteristic of OG signal and SWY signal.

(First embodiment)
Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

  The image processing apparatus for generating a luminance signal according to the present embodiment can be suitably realized by a signal processing circuit that performs signal processing such as so-called development processing in an imaging apparatus that uses an imaging device including, for example, a primary color Bayer array color filter. As shown in FIG. 1, the development processing indicates signal processing whose main purpose is to generate Y data 105, CrCb data 106, and RGB data 107 from RAW image data 101 output from the image sensor.

<Imaging device>
FIG. 2 is a configuration diagram of a typical imaging apparatus according to the embodiment of the present invention. The imaging device 10 is a single-plate digital camera that converts an optical image of a subject imaged via the imaging unit 12 into digital image data and records it on a recording medium 912.

  The imaging unit 12 includes an optical lens 901, an optical LPF 902, a color filter 903, an imaging element 904, and an AFE (analog front end) unit 905.

  The image sensor 904 is an image sensor typified by a CCD type or a CMOS type. A large number of photodiodes (photosensitive pixels) are two-dimensionally arranged on the light receiving surface of the image sensor 904, and subject information that has passed through the optical lens 901 is photoelectrically converted. The optical LPF 902 has a function of removing high frequency components equal to or higher than the sampling frequency depending on the pixel pitch of the image sensor 904, and prevents aliasing from occurring in the final image after image reproduction (signal processing). The color filter 903 has a predetermined color arrangement in which any one of R, G, and B is present at a position corresponding to each pixel of the image sensor 904, and is incident on a photodiode as a light receiving element. Select the light color.

  An example of an RGB primary color type color filter array image sensor is shown in FIG. In the Bayer array 101 shown in FIG. 1, the light receiving elements are arranged in a square matrix at a constant pitch in the row direction and the column direction. The present invention is not limited to the Bayer arrangement.

  The light that has passed through the optical lens 901 in FIG. 2 passes through the optical LPF 902 and the color filter 903 and enters the image sensor 904. A subject image formed on the light receiving surface of the image sensor 904 is converted into a signal charge corresponding to the amount of incident light by each photodiode, and a voltage corresponding to the signal charge based on a pulse provided from a driver circuit (not shown). It is sequentially read out as a signal (image signal).

  The image sensor 904 has an electronic shutter function for controlling the charge accumulation time (shutter speed) of each photodiode according to the timing of the shutter gate pulse. The operation (exposure, reading, etc.) of the image sensor 904 is controlled by the CPU 914.

  The image signal output from the image sensor 904 is sent to the AFE unit 905, and after being subjected to processing such as analog gain and CDS (correlated double sampling), it is converted into a digital signal by A / D conversion processing.

  In addition, in an image sensor 904 represented by a CMOS type, as a means for realizing high-speed reading, there is a mode in which a noise processing unit and an A / D converter are mounted in the image sensor 904 and directly output as a digital signal. .

  The image data A / D converted by the AFE unit 905 is recorded on the recording medium 912 through necessary signal processing according to the operation mode of the imaging apparatus 10 or by omitting signal processing. The imaging apparatus 10 according to the present embodiment can record an image in the JPEG format and can record an image immediately after A / D conversion (hereinafter referred to as a RAW image).

  When recording in the JPEG format, the A / D converted image data is sent to the image signal processing unit (processing function unit) 909 via the preprocessing unit 906. The pre-processing unit 906 and the image signal processing unit 909 perform a synchronization process (a process of calculating the color of each point by interpolating a spatial shift of the color signal associated with the color filter array), white balance (WB) adjustment, gamma A processing circuit that performs various processes such as correction, luminance / color difference signal generation, contour enhancement, and scaling (enlargement / reduction) processing using an electronic zoom function, and processes image signals in accordance with commands from the CPU 914. The pre-processing unit 906 and the image signal processing unit 909 can be connected to a memory unit 908 that can temporarily store an image being processed via the memory control unit 907, and process the image signal while using the memory unit 908.

  In the pre-processing unit 906 and the image signal processing unit 909, the image data that has undergone predetermined signal processing is sent to the compression / decompression unit 910 and compressed according to the JPEG compression format. The compression format is not limited to JPEG, MPEG and other moving image compression methods may be adopted, and a compression engine corresponding to the compression format used is used.

  The compressed image data is recorded on the recording medium 912 via the recording medium I / F (interface) unit 911. The recording medium 912 is not limited to a semiconductor memory typified by a memory card, and various media such as a magnetic disk, an optical disk, and a magneto-optical disk can be used. Further, the recording medium (internal memory) built in the imaging apparatus 10 is not limited to a removable medium.

  On the other hand, in the mode for recording a RAW image, the image data digitized by the A / D conversion of the AFE unit 905 is not subjected to the synchronization or other signal processing, and the memory control unit 907 and the recording media I / F unit 911. To the recording medium 912. In other words, a RAW image is an image that has not undergone signal processing such as gamma correction, white balance adjustment, and synchronization, and holds only one color information that differs for each pixel corresponding to the arrangement pattern of the color filter 903. It is a mosaic image.

  The CPU 914 is a control unit that performs overall control of the imaging apparatus 10 according to a predetermined program, and controls the operation of each circuit in the imaging apparatus 10 based on an instruction signal from the operation panel 913. The ROM 915 stores programs executed by the CPU 914 and various data necessary for control, and the RAM 916 is used as a work area for the CPU 914.

  Through the display processing unit 917 and the monitor I / F unit 918, real-time monitoring at the time of shooting is realized, and images and operation modes after shooting are displayed.

  The operation panel 913 is a means for the user to input various instructions to the imaging apparatus 10. For example, a mode selection switch for selecting the operation mode of the imaging apparatus 10, a cross key for inputting instructions such as a menu item selection operation (cursor movement operation) and a frame advance / return of a playback image, and confirmation of a selection item (registration) ) And an execution key for instructing execution of the operation, a cancel key for deleting a desired object such as a selection item, and canceling the instruction, a power switch, a zoom switch, a release switch, and various other operation means. Note that a touch panel integrated with a monitor connected to the monitor I / F unit 918 may be used.

  The CPU 914 controls the image pickup unit 12 such as the image pickup device 904 according to various shooting conditions (exposure conditions, presence / absence of strobe light emission, shooting mode, etc.) according to an instruction signal input from the operation panel 913, and also displays a real-time monitor. Control, automatic exposure (AE) control, automatic focus adjustment (AF) control, auto white balance (AWB) control, lens drive control, image processing control, read / write control of the recording medium 912, and the like are performed.

<Image processing apparatus and image processing method>
Next, an image processing apparatus that handles RAW image data obtained from the image sensor 904 and an image processing method thereof in the imaging apparatus 10 configured as described above will be described.

  The RAW image data is converted into a color component signal by the preprocessing unit 906 and the image signal processing unit 909 in FIG. 2, or is recorded on the recording medium 912 as a RAW image, and is used as a dedicated image processing apparatus or personal computer. The data is transferred to a color component signal and the like.

  The luminance signal generation process of the present embodiment is performed in the image signal processing unit (luminance signal generation device) 909, and generates a luminance signal for each pixel using an image signal for each pixel of the RAW image data. Therefore, although not particularly described, the processing described below is executed for each target pixel while sequentially updating the target pixel.

  FIG. 1 shows an outline of internal processing of the image signal processing unit 909. Raw image data 101 (hereinafter referred to as RAW data) output from an image sensor 904 having a primary color Bayer array color filter is input to an image signal processing unit 909. At this time, there is a case where the input is performed after the white balance processing or gamma processing for performing gain adjustment for each color of RGB pixels is performed in the preprocessing 906.

  In the image signal processing unit 909, the input RAW data 101 is subjected to color carrier conversion processing in the color carrier conversion processing unit 102 which is the greatest feature of the present embodiment (hereinafter, this conversion processing also uses the same symbol 102). After that, the luminance signal processing in the luminance signal processing unit 103 (hereinafter, this luminance signal processing also uses the same reference numeral 103) and the color signal processing in the color signal processing unit 104 (hereinafter, this luminance signal processing is also the same). (Use reference numeral 104).

  Although details will be described later, the first of the creative portions of the present invention is the color carrier conversion processing 102.

  The data processed and output by the image signal processing unit 909 is a luminance signal (Y) 105, two types of color difference signals (Cr), (Cb) 106, red (R) that has undergone color complementation processing from RAW data, Each data is green (G) and blue (B) 107.

<Color Carrier Conversion Processing 102-First Algorithm>
FIG. 3 shows details of a first algorithm for realizing the color carrier conversion processing unit (pixel position selection filter processing means) 102 in the image signal processing unit 909.

  The RAW image data 201 is output data of an image sensor provided with a Bayer array color filter as a periodic array color filter. In the RAW data 201 illustrated in the figure, blue (B) or red (R) color filters are provided at odd pixel positions in both the row direction and the column direction and even pixel positions in both the row direction and the column direction. This is a case where a green (G) color filter is arranged at a pixel position where the row direction is odd, the column direction is even, and the row direction is even and the column direction is odd. Contrary to the above, the green (G) color filter is arranged at the odd-numbered pixel positions in both the row direction and the column direction and the even-numbered pixel positions in both the row direction and the column direction. A blue (B) or red (R) color filter may be arranged at pixel positions where the direction is odd, the column direction is even, the row direction is even, and the column direction is odd. Of course.

  Then, for each pixel of the RAW data 201, pixel selection processing 202 and 203 is performed from the surrounding pixels for each pixel of interest of red (R) and blue (B).

  At this time, the RAW data is obtained by performing a filter calculation process 204 using pixels in the peripheral pixel area of odd number (2n + 1) × odd number (2n + 1) centering on each pixel of interest of red (R) and blue (B). 201 is converted into color pixel data 205 composed of two types of pixels of periodic green (G) and calculation processing result data.

  FIG. 3 shows a state where n = 1 and low-pass filter processing is performed using four pixels adjacent to the target pixel.

  When pixel selection processing 202 using peripheral pixels is performed on red (R) of the target pixel, the target pixel (R) and the target pixel (R) are positioned in the right diagonal direction and the left diagonal direction. Only four blue (B) pixels are selected. Next, with respect to the pixel of interest (R) and the selected four pixels (B), a filter calculation process that sets the ratio of the pixel of interest (R) to 4 and the ratio of each of the four peripheral pixels (B) to 1 Step 204 is performed to generate a magenta (M) component that is a mixed color of red and blue. The absolute value of the M component is adjusted so that M = G when an achromatic subject is photographed. In the present embodiment, the size of green (G) in the RAW data 201 is set to 1, and 1/8 division processing is performed by the filter calculation processing 204 to calculate the pixel of interest (R) (M). replace.

  Similarly, when pixel selection processing 203 using peripheral pixels is performed for blue (B) of the target pixel, the target pixel (B) and the right diagonal direction and the left diagonal direction centering on the target pixel (B) Only the four red (R) pixels located at are selected. Next, with respect to the target pixel (B) and the selected four pixels (R), a filter calculation process is performed in which the ratio of the target pixel (B) is 4, and the ratio of each of the four peripheral pixels (R) is 1. Step 204 is performed to generate a magenta (M) component that is a mixed color of red and blue. The absolute value of the M component is adjusted so that M = G when an achromatic subject is photographed. In the present embodiment, the size of green (G) in the RAW data 201 is set to 1, and 1/8 division processing is performed by the filter calculation processing 204 to calculate the pixel of interest (B) to the pixel (M) subjected to the calculation processing. replace.

  By converting and replacing the two types of pixels of red (R) and blue (B) as the target pixel with M pixels that have been processed as described above, the array of the input RAW data 201 is converted into a periodic green pixel (G). And magenta pixel (M).

  FIG. 4 shows a processing result image of the color carrier conversion processing 102. FIG. 4A shows an image of the input RGB Bayer array RAW data 201 of the diagonal color palette. FIG. 6B shows images of two types of color pixel data 205, that is, a green pixel (G) and a magenta pixel (M) after the color carrier conversion processing 102 of the input RAW data 201 of the oblique color palette. By observing the color pixel data in FIG. 5B, the aliasing noise (zipper noise component) at the boundary between the colors has already been greatly suppressed at the stage where the color carrier conversion processing 102 of the present embodiment is performed. I understand that.

<Explanation that zipper noise is reduced by this image processing>
Here, how the zipper noise at the boundary between colors is reduced by performing the color carrier conversion processing 102 of the present embodiment will be described with reference to the drawings. In this description, an example in which zipper noise is generated in the luminance signal data at the boundary between colors is used.

  As shown in FIG. 5A, when the subject image has an oblique color boundary from upper right to lower left such that the upper left side of the subject image is red (R) and the lower right side is blue (B), The output of the image sensor having a Bayer array as a color filter is RAW image data in which R pixel data and B pixel data protrude as shown in FIG. In the RAW image data shown in FIG. 4B, the R pixel output is 1, the G pixel output is 0, and the B pixel output is 0 in the red subject portion, and the R pixel output is 0 and the G pixel output in the blue subject portion. The brightness of each pixel is expressed with 0, B pixel output as 1, output 1 as white, and output 0 as black.

  Since general luminance signal processing is low-pass filter processing represented by the SWY method that removes the color carrier component in the vicinity of the periodic Nyquist frequency, it is in the vicinity of the Nyquist frequency with respect to the RAW data shown in FIG. When the low-pass filter processing for removing the color carrier component is performed, the color carrier is continuously removed at locations where the data output is periodically repeated as 1, 0, 1, 0. Converted into typical luminance signal data. Since the periodicity of data output 1, 0, 1, 0... Is lost in the vertical, diagonal, and horizontal directions at the boundary between red (R) and blue (B), the high-frequency components that are in the form of black and white dots Remains. As a result, periodic black and white dot noise (zipper noise) is generated at the color boundary portion of the luminance data to be output.

  On the other hand, when the color carrier conversion processing 102 of this embodiment is performed before the low-pass filter processing for removing the color carrier in the vicinity of the Nyquist frequency is performed, the following is performed.

  That is, peripheral pixels as shown in FIG. 6C are selectively used for each pixel of interest of red (R) and blue (B) other than green (G) shown in FIG. Pixel selection processing 202 and 203 and filter calculation processing 204 are performed.

  Specifically, the filter calculation is performed using only the red (R) pixel and the blue (B) pixel located in the right diagonal direction and the left diagonal direction with each target pixel of red (R) and blue (B) as the central pixel. Processes 202 and 203 are performed.

  As a result, each pixel of interest of red (R) and blue (B) is replaced with a magenta (M) color component formed by combining red and blue, and the color as shown in FIG. The size of the M pixel portion other than the boundary is 0.5 (this is denoted as M in FIG. 4D). In the color boundary portion, the size of the M pixel portion is 0.625 (5/8) (denoted as Mh) and the size is 0.475 (3/8) (denoted as Ml). There are two types. The G pixel output is zero.

  When low-pass filter processing for removing color carrier components in the vicinity of the Nyquist frequency is performed on the RAW data converted into blue (G) pixels and magenta color component (M) pixels shown in FIG. The color carrier component where the data output regularly repeats in the horizontal and horizontal directions, 0.5, 0, 0.5, 0... Is completely removed and converted to continuous luminance signal data.

  In the oblique boundary portion between red (R) and blue (B), the M pixel data output in the vertical direction, the diagonal direction, and the horizontal direction is 0.5, 0.625, 0.425 and 0 of the G pixel. Since it repeats almost regularly, the high-frequency components in the form of black and white dots are greatly suppressed. As a result, a smooth image in which periodic black-and-white dot noise (zipper noise) is suppressed at the color boundary portion of the luminance data to be output.

<Color Carrier Conversion Process 102—Second Algorithm>
Next, details of the second algorithm for realizing the color carrier conversion processing 102 will be described with reference to FIG.

  The RAW data 201 is output data of an image sensor having a Bayer array color filter as a periodic array color filter. For each pixel of the RAW data 201, pixel selection processing 302 and 303 is performed from surrounding pixels selectively for each target pixel of red (R) and blue (B).

  At this time, the filter operation processing 304 using the pixels in the peripheral pixel area of the odd number (2n + 1) × odd number (2n + 1) is performed around each pixel of interest of red (R) and blue (B). Conversion into color pixel data 305 composed of two types of pixels, green (G) data and calculation processing result data.

  FIG. 6 shows a state where n = 1 and low-pass filter processing is performed using 8 pixels adjacent to the target pixel.

  When pixel selection processing 302 using peripheral pixels is performed on red (R) of the target pixel, the target pixel (R) and the target pixel (R) are positioned in the right diagonal direction and the left diagonal direction. In addition to the four blue (B) pixels, the components of the four green (G) pixels adjacent in the horizontal and vertical directions are also selected. Next, the ratio of the target pixel (R) is set to 4 for each of the four peripheral pixels (B) with respect to the nine pixels of the selected target pixel (R), blue (B), and green (G). A filter calculation process 304 is performed in which the ratio is 1 and the ratio of the four surrounding pixels (G) is 2, and a white (W) component that is a mixed color of red, blue, and green is generated. The absolute value of the W component is adjusted so that W = G when an achromatic subject is photographed. In the present embodiment, the size of green (G) in the RAW data 301 is set to 1, and 1/16 division processing is performed by the filter calculation processing 304, and the pixel of interest (R) is calculated and processed to the pixel (W). replace.

  Similarly, when pixel selection processing 303 using peripheral pixels is performed on blue (B) of the target pixel, the target pixel (B) and the right diagonal direction and the left diagonal direction centering on the target pixel (B) In addition to the four red (R) pixels located at, the components of the four green (G) pixels adjacent in the horizontal and vertical directions are also selected. Next, for the selected nine pixels of red (R), green (B), and green (G), the ratio of the pixel of interest (B) is set to 4, and the ratio of each of the four peripheral pixels (R) is set. A filter calculation process 304 is performed with a ratio of 1, 4 peripheral pixels (G) being 2, and a white (W) component that is a mixed color of red, blue, and green is generated. The absolute value of the W component is adjusted so that W = G when an achromatic subject is photographed. In the present embodiment, the size of green (G) in the RAW data 301 is set to 1, and 1/16 division processing is performed by the filter calculation processing 304, and the pixel of interest (R) is calculated and processed to the pixel (W). replace.

  By converting and replacing two types of pixels, red (R) and green (G), which are the target pixels, with W pixels that have been subjected to arithmetic processing, the array of input RAW data is changed to periodic green pixels (G) and white pixels ( W) two types of color pixel data 305.

  In FIG. 6, when the filter calculation process 304 is performed, the ratio of the four peripheral pixels (G) is set to 2, but this value can be adjusted as a coefficient α related to the resolution of the final process result image. It is desirable to take.

<Modified example of filter operation processing>
FIG. 7 shows a first algorithm and a second algorithm for realizing the color carrier conversion processing 102, which are odd (2n + 1) × odd centered around each pixel of interest of red (R) and blue (B). As an example of the case where the filter calculation process is performed on the pixels in the (2n + 1) peripheral pixel area, details of the process when using 5 × 5 peripheral pixels when n = 2 are shown.

  The RAW data 201 is output data of an image sensor having a Bayer array color filter as a periodic array color filter. For each target pixel of red (R) and blue (B), pixel selection processing 2020 and 2030 is performed from surrounding pixels selectively for each pixel of the RAW data 201.

  FIG. 7 shows a state where n = 2 and low-pass filter processing is performed using four pixels adjacent to the target pixel.

  At this time, the RAW processing is performed by performing the filter calculation processing 2040 using the pixels in the peripheral pixel area of (odd number: 5) × (odd number 5) centering on each target pixel of red (R) and blue (B). Data 201 is converted into color pixel data 2050 composed of two types of pixels, periodic green (G) and operation processing result data.

  When pixel selection processing 2020 using peripheral pixels is performed on red (R) of the target pixel, the target pixel (R) and the target pixel (R) are positioned in the right diagonal direction and the left diagonal direction. Four blue (B) pixels and five red (R) pixels are selected. Next, the ratio of the pixel of interest (R) to the four pixels of blue (B) and five pixels of red (R) selected as the pixel of interest R is set to 20, and each of the four peripheral pixels (B). A filter calculation process 2040 is performed in which the ratio is 4 and the ratio of each of the four peripheral pixels (R) is −1 to generate a magenta (M) component that is a mixed color of red and blue. The absolute value of the M component is adjusted so that M = G when an achromatic subject is photographed. In the present embodiment, the size of green (G) in the RAW data 201 is set to 1, and the filter calculation processing 2040 performs 1/32 division processing to the pixel (M) that has undergone calculation processing on the target pixel (R). replace.

  Similarly, when pixel selection processing 2030 using peripheral pixels is performed on blue (B) of the target pixel, the target pixel (B) and the right diagonal direction and the left diagonal direction centering on the target pixel (B) 4 red (R) pixels and 5 blue (B) pixels located at. Next, with respect to the target pixel (B), the selected four red (R) pixels and the five blue (B) pixels, the ratio of the target pixel (B) is set to 20, and the four peripheral pixels (R) ) Is set to 4, and the ratio of each of the four peripheral pixels (B) is set to −1 to generate a magenta (M) component that is a mixed color of red and blue. The absolute value of the M component is adjusted so that M = G when an achromatic subject is photographed. In the present embodiment, the size of green (G) in the RAW data 201 is set to 1, and the filter calculation processing 2040 performs 1/32 division processing, and the pixel (M) that has been subjected to the calculation processing on the target pixel (R). Replace with

  By converting the two types of pixels of red (R) and blue (B) that are the target pixel into M pixels that have undergone arithmetic processing, the array of the input RAW data 201 is replaced with periodic green pixels (G). Conversion into two types of color pixel data 205 with magenta pixels (M).

  In the case where the filter calculation process is performed on the pixels in the peripheral pixel area of (odd number: 5) × (odd number: 5) centering on each target pixel of red (R) and blue (B), the filter calculation process 2040 is performed. In the above description, the data other than those on the diagonal line in the right diagonal direction and the left diagonal direction has been described as the coefficient 0. However, a numerical value coefficient other than 0 may be set to perform the filter calculation process.

  Further, the coefficient setting of the filter calculation processing 2040 does not have to be a fixed value, and may be configured to be freely set as a variable. As an example of setting it freely, it may be set in units of frames, or may be set in units of pixels in the frame.

<Luminance signal processing>
Next, details of the algorithm of the luminance signal processing 103 after the color carrier conversion processing 102 is performed will be described.

  FIG. 8 shows a first processing block diagram in which the color carrier conversion processing 102 and the luminance signal processing 103 are combined.

  The algorithm of the luminance signal processing 103 of the present embodiment is based on the color carrier conversion processing step 402 described with reference to FIG. 3 on the assumption that Bayer array RAW data 401 is input, and the luminance using the data after the color carrier conversion. A high-frequency luminance signal processing step (1) means (band-pass filter processing means) 403 for processing high-frequency components of the signal (hereinafter, this processing step also uses the same reference numeral 403), the RAW data A low-frequency RGB color separation signal processing unit 404 for outputting low-frequency RGB data for each color from 401 (hereinafter, this processing step also uses the same reference numeral 404), and this color separation processing A low-frequency luminance signal processing step (hereinafter, referred to as a low-frequency luminance signal processing unit 405) that generates a low-frequency luminance signal from the low-frequency R, G, and B signals for each color separated in step 404. And the signal adding step in the signal adding means 407 for adding the high-frequency luminance signal and the low-frequency luminance signal (hereinafter, this processing step also uses the same symbol 407). Consists of.

  Next, details of each process will be described.

  In the high-frequency luminance signal processing (1) step 403, the data after color carrier conversion is input, and processing is performed with a bandpass filter processing characteristic 601 as shown in FIG.

  In the band pass filter processing characteristic 601 shown in FIG. 9A, the horizontal axis is the spatial frequency and the point N is the Nyquist frequency. The vertical axis represents the output signal level with a linear scale. The curve of the band pass filter characteristic is an example, and the present invention does not uniquely limit the band pass filter processing characteristic.

  Further, the bandpass filter processing characteristic 601 shown in FIG. 9A shows a one-dimensional characteristic. However, in actual processing, the bandpass filter processing characteristic 601 is performed independently in both the horizontal direction and the vertical direction. By synthesizing the high-frequency luminance signal component with the vertical direction, the high-frequency luminance signal component of the two-dimensional image is generated.

  On the other hand, in the low-frequency RGB color separation signal processing step 404, RAW data 401 is input, color interpolation processing is performed on each RGB color pixel to generate RGB three-color plane data, and FIG. Frequency band limiting processing is performed with a low-pass filter processing characteristic 602 that passes only the DC component and its neighboring components as shown in FIG.

  Also in the low-pass filter processing characteristic 602, the horizontal axis is the spatial frequency and the point N is the Nyquist frequency. The vertical axis represents the output signal level with a linear scale. The curve of the present low-pass filter characteristic 602 is also an example, and the present invention does not uniquely limit the low-pass filter processing characteristic.

  The standard setting point when determining the band limiting characteristic by the low-pass filter having the low-pass filter processing characteristic 602 is that the characteristics of the three-color plane data of the color separated RGB are removed by the band-pass filter processing characteristic 601. The characteristic is to have a band component corresponding to a direct current component and its neighboring components.

  In the low-frequency luminance signal processing step 405, the RGB three-color plane data color-separated in the low-frequency RGB color separation signal processing step 404 is input, and the three components of RGB are synthesized to generate a direct current component. And a low-frequency luminance signal component 406 having a band component corresponding to a component in the vicinity thereof. For example, the standard setting is Y = 0.3R + 0.59G + 0.11B, and the level of the low-frequency luminance signal of the color subject image can be adjusted by adjusting each coefficient.

  By combining the low-frequency luminance signal component (Y_low) 406 thus generated and the high-frequency data (Y_high) after the high-frequency luminance signal processing 403 into one signal 408 in the addition processing step 407, a natural noise feeling is obtained. It is possible to generate a high-frequency luminance signal (Y) 408 with good characteristics.

  In the image of the diagonal color palette after the color carrier conversion process 102 is performed, as shown in FIG. 4B, not only the boundary between red (R) and blue (B) but also all color boundaries and edges. Since the aliasing noise (zipper noise) generated in the image is uniformly suppressed in the screen, the low-frequency component including the DC component and the color carrier component existing in the vicinity of the Nyquist frequency are removed using the band-pass filter processing characteristic 601. By doing so, it is possible to extract a high-frequency luminance signal component (Y_high) that has a good feature in the entire frequency band and is easy to handle in subsequent processing.

  As described above, the high-frequency luminance signal data is generated independently of the low-frequency RGB color separation processing 404 after the low-frequency RGB color separation processing 404, so that the resolution can be adjusted more freely according to the shooting scene. can do.

  Further, the image quality of the luminance signal can be easily adjusted by adjusting the level of the high frequency data (Y_high) after the high frequency luminance signal processing 403 for each frequency band and adding it to the low frequency luminance signal component 406.

  Further, regarding processing called normal edge enhancement, the resolution can be freely adjusted by lifting the arbitrary frequency portion of the bandpass filter processing characteristics 601 independently in the horizontal direction and the vertical direction.

  In addition, when controlling the magnitude of the AC component processed by the band-pass filter processing characteristic 601 as a resolution adjustment signal, high-frequency level adjustment is performed at the G position, R position, and B position of the original image. May be adjusted independently to generate high-frequency luminance signal data in which the high-frequency level is adjusted to an arbitrary size at the G pixel position, R pixel position, and B pixel position of the original image. Hereinafter, this specific example will be described.

<Generation of high-frequency luminance signal data with adjusted resolution>
As described above, when the high-frequency component of the luminance signal is extracted from the two types of checkered pattern data including the G pixel and the M pixel by the band pass filter processing characteristic 601, there is a slight difference in the resolution information for each pixel. There is.

  The band pass filter process 601 shown in FIG. 9 is performed on the RAW data shown in FIG. 10A composed of the G pixel and the M pixel generated in the color carrier conversion process 102, and the result shown in FIG. When a high frequency component of such a luminance signal is used (FIG. 10B), the G pixel position with a lot of high frequency information is relatively white, and the M pixel position where the high frequency information is suppressed is relatively black. For the M pixel portion, only the red (R) pixel and the blue (B) pixel located in the right diagonal direction and the left diagonal direction with the respective target pixels of red (R) and blue (B) as the origin are used. Therefore, the high frequency component in the AC component Yh at the M pixel position of the luminance signal in FIG. 10B is relative to the high frequency component in the AC component Yh of the G pixel portion. Has been suppressed.

  The resolution up to the Nyquist frequency can be secured in the horizontal direction and the vertical direction, compared to the high frequency information of the G pixel portion constituting the checkered pattern of the Bayer arrangement, but similarly for the M pixel portion constituting the checkered pattern. Since the sense of resolution is slightly suppressed by the filter processing in the oblique direction, a slight difference occurs in the resolution information for each pixel.

  In response to this, in the present modification, resolution adjustment for each pixel is performed. As a specific method for adjusting the resolution, the sense of resolution can be adjusted by controlling the size of the AC component (Y_high) 450 processed in the bandpass filter processing 601 for each pixel.

  That is, as shown in FIG. 10C, the resolution correction is performed using the high-frequency level adjusting unit 451 that can control the size of each pixel with respect to the AC component (Y_high) 450 of the bandpass filter processing 601. The AC component (Y_high ′) 454 is output. Specifically, the high frequency level adjusting means 451 adjusts the high frequency level adjustment independently between the G pixel position and the M pixel position (R pixel position and B pixel position) of the original image. A memory (adjustment data storage unit for each pixel position) 453 that stores independent gain data (adjustment data) for each pixel position so that the high level of the M pixel position is relatively larger than the G pixel position. The multiplier (multiplier means) 452 multiplies the alternating current component (Y_high) 450 of the bandpass filter processing 601 by a gain corresponding to the position of this pixel to correct the resolution sense (Y_high). ') Output 454.

  Then, an AC component (Y_high ′) 453 in which a high frequency level is adjusted and resolution is corrected for each pixel after the band pass filter processing, and after the RGB low frequency color separation processing 404 as shown in FIG. A signal (Y_low) obtained by performing low-frequency luminance signal processing 405 on the RGB components is synthesized by addition processing 407, and high-frequency luminance signal generation processing for generating a luminance signal (Y) 408 is performed.

<Other examples of luminance signal processing>
FIG. 11 shows a second processing block diagram in which the color carrier conversion processing 102 and the luminance signal processing 103 are combined.

  The algorithm of the luminance signal processing 103 is based on the assumption that the Bayer array RAW data 401 is input. The color carrier conversion processing step 402 described with reference to FIG. The high-frequency luminance signal processing (2) means (color carrier removal means) 503 for processing the high-frequency components (2) step (hereinafter, this step also uses the same reference numeral 503). .

  Next, details of the processing will be described. In the second high-frequency luminance signal processing (2) step 503, processing is performed with a high-band low-pass filter processing characteristic 603 that removes only the color carrier component as shown in FIG. carry out. In this high-band low-pass filter processing characteristic 603, the horizontal axis is the spatial frequency, and is the point N Nyquist frequency. The vertical axis represents the output signal level with a linear scale. The curve of the present low-pass filter characteristic 603 is an example, and the present invention does not uniquely limit the characteristic.

  The low-pass filter characteristic 603 shows a one-dimensional characteristic. In actual processing, the low-pass filter characteristic 603 is performed independently in both the horizontal direction and the vertical direction, so that the high-frequency luminance signal component in the horizontal direction and the vertical direction is obtained. Are combined to generate a high-frequency luminance signal component of a two-dimensional image.

  In the high-frequency luminance signal processing (2) step 503, since a direct current component is also output, the RGB separated color plane data as shown in the first processing block of FIG. The low-frequency luminance signal level of the color subject image is not adjusted by combining and adjusting the three signals. Therefore, when implemented in hardware, it can be realized with a smaller circuit configuration. Of course, in the high frequency characteristics, a high-frequency luminance signal 505 having a natural noise feeling and good characteristics can be generated.

  As shown in FIG. 4B, the image after the color carrier conversion is not limited to red and blue, and zipper noise generated at all color boundaries and edges is uniformly suppressed on the screen. The high-frequency luminance signal component having the same characteristic can be extracted as shown in FIG. 4D by removing the color carrier component existing in the high-frequency low-pass filter processing characteristic 603.

  Also, regarding processing called edge enhancement, the resolution adjustment can be freely adjusted by lifting the arbitrary frequency portion of the high-band low-pass filter processing characteristic 603 independently in the horizontal direction and the vertical direction. it can.

  Further, when controlling the magnitude of the AC component processed by the high-band low-pass filter processing characteristic 603 as a resolution adjustment signal, as described above, the G pixel position and R pixel position of the original image By independently switching and adjusting the high frequency level adjustment with the B pixel position, the high frequency level is adjusted to an arbitrary size between the G pixel position, the R pixel position, and the B pixel position of the original image. High frequency luminance signal data may be generated.

  In this way, as with the first processing block in FIG. 8, the adjustment of resolution according to the shooting scene can be freely adjusted.

  As described above, according to the present embodiment, in a luminance signal generation device that generates a luminance signal from an image signal obtained from an image sensor using a color filter with a primary color Bayer array, an image having a red subject or a blue subject is obtained. On the other hand, it is possible to generate a luminance signal in which the occurrence of aliasing distortion is sufficiently suppressed.

(Second Embodiment)
The first embodiment can also be realized in software by a computer (CPU, MPU, etc.) of a system or apparatus.

  Therefore, in order to realize the above-described embodiment by a computer, the computer program itself supplied to the computer also realizes the present invention. That is, the computer program itself for realizing the functions of the above-described embodiments is also one aspect of the present invention.

  FIG. 13 shows a flowchart when the color carrier conversion processing of the present invention is realized by a program.

  In FIG. 13, step S <b> 1 is a process of capturing original image data of an RGB Bayer array. Step S2 is a process in which only the R pixel and the B pixel are subjected to low-pass filter processing. Step S3 is a process for generating carrier conversion array data including G pixels and pixels after low-pass filter processing. This flowchart is a flowchart corresponding to the color carrier conversion processing algorithm described in FIG. 3 and FIG.

  FIG. 14 shows a flowchart when the color carrier conversion processing of the present invention is realized by a program.

  In FIG. 14, step S <b> 1 is a process of taking original image data of an RGB Bayer array. Step S2 is a process of performing low pass filter processing only on the R pixel and the B pixel. Step S3 is a process of generating color carrier conversion array data composed of G pixels and pixels after low-pass filter processing. Step S4 is a process of processing the high-frequency luminance signal using the color carrier conversion array data.

  This flowchart is a flowchart in which the luminance signal processing step shown in FIG. 11 is added to the color carrier conversion processing algorithm described with reference to FIGS. 3 and 6.

  FIG. 15 shows a flowchart when the color carrier conversion processing of the present invention is realized by a program.

  In the figure, step S1 is a process of taking in original image data of an RGB Bayer array. Step S2 is a process of performing low pass filter processing only on the R pixel and the B pixel. Step S3 is a step of generating carrier conversion array data composed of G pixels and pixels after the low-pass filter processing. Step S14 is a mid- and high-frequency luminance signal processing step for extracting mid-range and high-frequency components using the color carrier conversion array data. Step S5 is a low-frequency RGB color separation signal processing step for generating low-frequency R, G, and B color separation signals for each color from the RGB Bayer array original image data. Step S6 is a low-frequency luminance signal processing step for generating a low-frequency luminance signal from the low-frequency R, G, B color separation signals. Step S7 is an addition processing step of generating a luminance signal by synthesizing the mid-range / high-range luminance signal and the low-frequency luminance signal.

  This flowchart is a flowchart obtained by adding the luminance signal processing step shown in FIG. 8 to the color carrier conversion processing algorithm described in FIG. 3 and FIG.

  The computer program for realizing the above-described embodiment may be in any form as long as it can be read by a computer. For example, it can be composed of object code, a program executed by an interpreter, script data supplied to the OS, but is not limited thereto.

  A computer program for realizing the above-described embodiment is supplied to a computer via a storage medium or wired / wireless communication. Examples of the storage medium for supplying the program include a magnetic storage medium such as a flexible disk, a hard disk, and a magnetic tape, an optical / magneto-optical storage medium such as an MO, CD, and DVD, and a nonvolatile semiconductor memory.

  As a computer program supply method using wired / wireless communication, there is a method of using a server on a computer network. In this case, a data file (program file) that can be a computer program forming the present invention is stored in the server. The program file may be an executable format or a source code.

  As described above, the present invention suppresses the occurrence of aliasing distortion (zipper noise) at the boundary between colors in a diagonal direction with respect to a red or blue subject, while reducing the false luminance signal, while providing a good solution. Since the image can be maintained, the present invention can be applied to processing inside an imaging apparatus such as a silver salt camera, a still image electronic camera, and a video camera. It can also be applied as an internal process of a software application that develops RAW image data.

DESCRIPTION OF SYMBOLS 10 Imaging device 12 Imaging part 101 Bayer arrangement | sequence 102 Color carrier conversion process part (pixel position selection filter process means)
103 luminance signal processing unit 104 color signal processing unit 105 luminance signal (Y)
106 Two kinds of color difference signals (Cr), (Cb)
107 Red (R), Green (G), Blue (B) data 201, 301 RAW data 202, 203 Pixel selection processing 204 Filter calculation processing 205 Color pixel data 302, 303 pixel selection processing 304 consisting of two types of pixels Filter calculation Processing 305 Color pixel data composed of two types of pixels 401 Bayer array RAW data 402 Color carrier conversion processing step 403 High-frequency luminance signal processing (1) means (bandpass filter processing means)
404 Low-frequency RGB color separation signal processing means 405 Low-frequency luminance signal processing means (low-pass filter processing means)
406 Low-frequency luminance signal component 407 Addition processing means 408 High-frequency luminance signal 451 High-frequency level adjusting means 452 Multiplier (multiplication means)
453 memory (pixel position adjustment data storage means)
503 High-frequency luminance signal processing (2) means (color carrier removal means)
505 High-frequency luminance signal 601 Band-pass filter processing characteristic 602 Low-pass filter processing characteristic 603 High-band low-pass filter processing characteristic 10 Imaging device 12 Imaging unit 901 Optical lens 902 Optical LPF (low-pass filter)
903 Color filter 904 Image sensor 905 AFE (analog front end) unit 906 Preprocessing unit 907 Memory control unit 908 Memory unit 909 Image signal processing unit (luminance signal generation device) (processing function unit)
910 Compression / decompression unit 911 Recording medium I / F (interface) unit 912 Recording medium 913 Operation panel 914 CPU
915 ROM
916 RAM
917 Display processing unit 918 Monitor I / F unit 2020, 2030 Pixel selection processing 2040 Filter operation processing 2050 Color pixel data composed of two types of pixels

Claims (21)

An image processing apparatus including a luminance signal generation apparatus that generates a luminance signal for each pixel from RAW image data read from an image sensor having a periodic arrangement of color filters,
The luminance signal generation device includes:
Low-pass filter processing is not performed on pixels in which the pixel position of the RAW image data is an odd number in the row direction, an even number in the column direction, and an even number in the row direction and an odd number in the column direction. Pixel position selection filter processing means for performing low-pass filter processing only on pixels at odd positions in the row direction and column direction and pixels at even positions in the row direction and column direction;
An image processing apparatus comprising: luminance signal processing means for performing high-band luminance signal processing using output data of the pixel position selection filter processing means as input. An image processing apparatus including a luminance signal generation apparatus that generates a luminance signal for each pixel from RAW image data read from an image sensor having a periodic arrangement of color filters,
The luminance signal generation device includes:
Low-pass filter processing is performed only for pixels in which the pixel position of the RAW image data is an odd number in the row direction and an even number in the column direction, and an even number in the row direction and an odd number in the column direction. Pixel position selection filter processing means that does not perform low-pass filter processing on pixels at odd positions in the direction and column direction and pixels at even positions in the row direction and column direction;
An image processing apparatus comprising: luminance signal processing means for performing high-band luminance signal processing using output data of the pixel position selection filter processing means as input. The image processing apparatus according to any one of claims 1 and 2,
The filter array of the image sensor having the periodic array of color filters is a Bayer array,
The pixel position selection filter processing means includes:
An image processing apparatus, wherein low-pass filter processing is selectively performed on each pixel of interest of red (R) and blue (B) among RGB original pixel data. The image processing apparatus according to any one of claims 1 and 2,
The filter array of the image sensor having the periodic array of color filters is a Bayer array,
The pixel position selection filter processing means includes:
When the low-pass filter process is performed selectively on the red (R) and blue (B) target pixels using peripheral pixels, an odd number (2n + 1) centering on each of the red (R) and blue (B) target pixels. ) × odd (2n + 1) pixels in the peripheral pixel area are subjected to filter arithmetic processing, and the Bayer array original image is converted into two types of color pixel data of periodic green (G) and arithmetic processing result data. An image processing apparatus characterized by converting. The image processing apparatus according to any one of claims 1 and 2,
The filter array of the image sensor having the periodic array of color filters is a Bayer array,
The pixel position selection filter processing means includes:
When the low-pass filter processing is performed on the red (R) and blue (B) target pixels selectively using the peripheral pixels, red (R) positioned in the right diagonal direction and the left diagonal direction with the target pixel as the origin. A magenta (M) color component obtained by performing filter calculation processing using only pixels and blue (B) pixels and combining red and blue for each pixel of interest of red (R) and blue (B) An image processing apparatus characterized by converting an original image of a Bayer array into two types of color pixel data of green (G) and magenta (M). The image processing apparatus according to any one of claims 1 and 2,
The filter array of the image sensor having the periodic array of color filters is a Bayer array,
The pixel position selection filter processing means includes:
When the low-pass filter processing is performed on the target pixel of red (R) and blue (B) selectively using peripheral pixels, red (R) and blue (B) that are positioned obliquely with the target pixel as the origin In addition to the above pixels, filter operation processing is also performed by adding the components of the green (G) pixels adjacent in the horizontal and vertical directions, and red, blue, and green are applied to the target pixel of red (R) and blue (B). Image processing characterized by converting the original image of the Bayer array into two types of color pixel data of periodic green (G) and white (W) apparatus. The image processing apparatus according to any one of claims 4 to 6,
Furthermore, color carrier removal means for generating a high-frequency luminance signal by performing periodic color carrier removal processing in the horizontal and vertical directions on the image data converted into two kinds of periodic color pixel data. An image processing apparatus comprising: The image processing apparatus according to any one of claims 4 to 6,
Band pass filter processing for removing high frequency components including periodic color carriers and low frequency components including DC components in horizontal and vertical directions from image data converted into two types of periodic color pixel data Means,
Low-frequency RGB color separation signal processing means for color-separating and outputting original pixel data of three colors of RGB for each color;
Low-pass filter processing means for filtering a band corresponding to a low-frequency component including a direct-current component removed by the band-pass filter processing with respect to the RGB data color-separated and output for each color;
An image processing apparatus comprising: signal addition means for performing addition processing of the high-pass data after the band-pass filter processing and the low-pass color separation RGB component after the low-pass filter processing. The image processing apparatus according to claim 8, wherein
A high frequency level adjusting means in the AC component for adjusting the resolution feeling by controlling the magnitude of the AC component processed by the bandpass filter processing means as a resolution adjustment signal,
The signal adding means includes
Addition processing of the high-frequency data adjusted high-frequency level after the band-pass filter processing and the low-frequency color separation RGB component after the low-pass filter processing,
A high-frequency luminance signal is obtained from the output of the signal adding means. The image processing apparatus according to claim 9, wherein
The high frequency level adjusting means is
When controlling the magnitude of the AC component processed by the bandpass filter processing means as a resolution adjustment signal, the green (G) pixel position, the red (R) pixel position, and the blue (B) of the original image Pixel position-specific adjustment data storage means for storing high-frequency level adjustment data for determining the magnitude of the AC component independently of the pixel position of
Multiplying means for multiplying the magnitude of the alternating current component processed by the bandpass filter processing means according to the high frequency level adjustment data stored for each pixel position,
The signal adding means includes
For the high-frequency data after the bandpass filter processing, the magnitude of the AC component for each color is determined by the green (G) pixel position, the red (R) pixel position, and the blue (B) pixel position of the original image. An image processing apparatus that performs addition processing of the adjusted high-frequency data and the low-frequency color separation RGB component after the low-pass filter processing. An image processing method having a luminance signal generation method for generating a luminance signal for each pixel from RAW image data read from an image sensor having a periodic arrangement of color filters,
The luminance signal generation method includes:
Low-pass filter processing is not performed on pixels in which the pixel position of the RAW image data is an odd number in the row direction, an even number in the column direction, and an even number in the row direction and an odd number in the column direction. A pixel position selection filter processing step for performing low-pass filter processing only on pixels in odd positions in the row direction and column direction and pixels in even positions in the row direction and column direction;
A luminance signal processing step of performing high-band luminance signal processing with the output data of the pixel position selection filter processing step as an input. An image processing method including a luminance signal generation method for generating a luminance signal for each pixel from RAW image data read from an image sensor having a periodic arrangement of color filters,
The luminance signal generation method includes:
Low-pass filter processing is performed only for pixels in which the pixel position of the RAW image data is an odd number in the row direction and an even number in the column direction, and an even number in the row direction and an odd number in the column direction. A pixel position selection filter processing step in which low-pass filter processing is not performed on pixels in odd positions in the direction and column direction and pixels in even positions in the row direction and column direction;
A luminance signal processing step of performing high-band luminance signal processing with the output data of the pixel position selection filter processing step as an input. The image processing method according to any one of claims 11 and 12,
The filter array of the image sensor having the periodic array of color filters is a Bayer array,
The pixel position selection filter processing step includes
An image processing method, wherein low-pass filter processing is selectively performed on each pixel of interest of red (R) and blue (B) among RGB original pixel data. The image processing method according to any one of claims 11 and 12,
The filter array of the image sensor having the periodic array of color filters is a Bayer array,
The pixel position selection filter processing step includes
When the low-pass filter process is performed selectively on the red (R) and blue (B) target pixels using peripheral pixels, an odd number (2n + 1) centering on each of the red (R) and blue (B) target pixels. ) × odd (2n + 1) pixels in the peripheral pixel area are subjected to filter arithmetic processing, and the Bayer array original image is converted into two types of color pixel data of periodic green (G) and arithmetic processing result data. An image processing method characterized by converting. The image processing method according to any one of claims 11 and 12,
The filter array of the image sensor having the periodic array of color filters is a Bayer array,
The pixel position selection filter processing step includes
When the low-pass filter processing is performed on the red (R) and blue (B) target pixels selectively using the peripheral pixels, red (R) positioned in the right diagonal direction and the left diagonal direction with the target pixel as the origin. A magenta (M) color component obtained by performing filter calculation processing using only pixels and blue (B) pixels and combining red and blue for each pixel of interest of red (R) and blue (B) And converting the Bayer array original image into two types of periodic color pixel data of green (G) and magenta (M). The image processing method according to any one of claims 11 and 12,
The filter array of the image sensor having the periodic array of color filters is a Bayer array,
The pixel position selection filter processing step includes
When the low-pass filter processing is performed on the target pixel of red (R) and blue (B) selectively using peripheral pixels, red (R) and blue (B) that are positioned obliquely with the target pixel as the origin In addition to the above pixels, filter operation processing is also performed by adding the components of the green (G) pixels adjacent in the horizontal and vertical directions, and red, blue, and green are applied to the target pixel of red (R) and blue (B). A white (W) color component formed by synthesizing the image and converting the Bayer array original image into two types of periodic G and W color pixel data. The image processing method according to any one of claims 14 to 16, comprising:
Further, the image data converted into two kinds of periodic color pixel data is subjected to a color carrier removal process for performing a periodic color carrier removal process in the horizontal and vertical directions to generate a high-frequency luminance signal. An image processing method. The image processing method according to any one of claims 14 to 16, comprising:
Band pass filter processing for removing high frequency components including periodic color carriers and low frequency components including DC components in horizontal and vertical directions from image data converted into two types of periodic color pixel data Process,
A low-frequency RGB color separation signal processing step for color-separating and outputting original pixel data of three colors of RGB for each color;
A low-pass filter processing step of filtering a band corresponding to a low-frequency component including a direct-current component removed by the band-pass filter processing with respect to the RGB data color-separated and output for each color;
An image processing method comprising: a signal addition step of performing addition processing of the high-pass data after the band-pass filter processing and the low-pass color separation RGB component after the low-pass filter processing. The image processing method according to claim 18, wherein
Having a high frequency level adjustment step in the AC component for adjusting the resolution by controlling the magnitude of the AC component processed in the bandpass filter processing step as a resolution adjustment signal,
The signal adding step includes
Addition processing of the high-frequency data adjusted high-frequency level after the band-pass filter processing and the low-frequency color separation RGB component after the low-pass filter processing,
A high-frequency luminance signal is obtained from the output of the signal adding step. The image processing method according to claim 19, wherein
The high frequency level adjustment step includes:
When controlling the magnitude of the AC component processed in the bandpass filter processing step as a resolution adjustment signal, the green (G) pixel position, the red (R) pixel position, and the blue (B) of the original image A pixel position-specific adjustment data storage step for storing high-frequency level adjustment data for determining the magnitude of the AC component independently of the pixel position of
A multiplication step of multiplying the magnitude of the alternating current component processed in the band pass filter processing step according to the high frequency level adjustment data stored for each pixel position,
The signal adding step includes
For the high-frequency data after the bandpass filter processing, the magnitude of the AC component for each color is determined by the green (G) pixel position, the red (R) pixel position, and the blue (B) pixel position of the original image. An image processing method, comprising: adding the adjusted high-frequency data and the low-frequency color separation RGB component after the low-pass filter processing. A digital camera comprising a processing function unit including the image processing apparatus according to claim 1.