JP2005151549A - Image processing device and method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To improve the reproduction of the high frequency component of luminance, to reduce jaggyness, and to improve a feeling in resolution. <P>SOLUTION: A G strength estimating unit 291 is designed to calculate an estimation value of G strength at a remarkable pixel position by using a pixel in a local domain. An E strength estimating unit 292 is designed to calculate the estimation value of E strength at the remarkable pixel position by using the pixel in the local domain. A summing processing unit 293 is designed to sum the calculated G strength estimation value and the E strength estimation value, and to calculate a (G+E) strength estimation value at the remarkable pixel position. A high frequency image signal cannot be reproduced by only a G or E interpolation process in a mosaic picture by a color filter with primary 4 color arrangements in which signals of G and E exist only in every other one pixel. The generation of jaggyness and noise in an output image can be reduced and image signals having high frequencies higher than each limit thereof can be reproduced by reproducing such signals added with G and E signals and arranged at the position with one pixel shifted horizontally and vertically on all pixel positions. The present invention is applicable to a digital still camera. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program, and more particularly, a demosaic that obtains a color image signal by interpolating a plurality of colors into all pixels from a mosaic image signal obtained using a color solid-state imaging device. The present invention relates to an image processing apparatus, an image processing method, and a program suitable for use when processing (color interpolation processing or synchronization processing) is executed.

  A solid-state imaging device such as a CCD image sensor or a CMOS image sensor usually has a structure in which light receiving elements are arranged in a lattice shape, and the amount of charge generated by photoelectric conversion of each light receiving element can be sequentially read out. . Usually, since these light receiving elements have a single spectral characteristic, the image signal obtained from the solid-state image sensor is one channel (monochrome) with respect to color. Therefore, when it is desired to obtain a color image (for example, a three-channel image such as RGB) with a single solid-state image sensor, a solid-state image sensor having filters with different spectral characteristics (colors) for each light-receiving element is used. An imaging apparatus using such a single color solid-state imaging device is generally called a single-plate color imaging apparatus.

  Since the image signal obtained from the color solid-state image sensor is a one-channel image, only the intensity of the color of the filter of the corresponding light receiving element is obtained for each pixel. That is, the image signal obtained from the color solid-state imaging device becomes a mosaic image with respect to color. In order to obtain a multi-channel image from a mosaic image that is an output image of a color solid-state imaging device of a single-plate color imaging device, the color information of each pixel of the mosaic image is converted to the surrounding pixel positions by appropriate image processing. It is necessary to interpolate. Such image processing is generally referred to as color interpolation processing, demosaic processing, or synchronization processing. As described above, the demosaic process is an essential process for a single-plate color image pickup apparatus using a color solid-state image pickup device, and various techniques have been developed.

  A problem in this demosaic processing is the reproduction characteristics of high-frequency image signals. In solid-state imaging devices, different on-chip color filters are attached to the light receiving elements arranged in a two-dimensional grid, so that pixels having the same color, that is, the same spectral sensitivity, are the same as the original light receiving elements. They are arranged only at a pitch larger than the arrangement pitch. Therefore, the color solid-state imaging device is likely to cause aliasing in the captured image as compared with a black and white solid-state imaging device having no on-chip color filter. When aliasing occurs in the luminance component of an image, it is often observed as jagginess in the subject outline. In demosaic processing, how to suppress Jugginess is an important issue.

  FIG. 1 shows a primary color Bayer array which is the most widely used color array at present. In the Bayer array, three primary color filters of R, G, and B are used, G is arranged in a checkered pattern, and R and B are line-sequentially arranged (hereinafter, the primary color Bayer array is simply referred to as Bayer array). Called). In the Bayer array, G is arranged in a checkered pattern, so that the G signal exists in all horizontal and vertical phases, but R and B are line sequential, so the corresponding signals are horizontal, There is only every other vertical line.

  In the Bayer array, G having the spectral characteristic closest to the human visual sensitivity characteristic among RGB is most closely present. Therefore, according to a conventionally used technique, a luminance component is generated exclusively from G pixel information. The method is mainstream. In this method, the limit frequency of the reproduced luminance component is determined by the sampling frequency of G in the Bayer array, 0.5 cycle / pixel in the horizontal and vertical directions, and 1 / (2 × sqrt in the oblique 45 degree direction. (2)) It will be cycle / pixel. Theoretically, the limit is the highest in the horizontal or vertical direction, but in reality there is only G in every other pixel in the horizontal and vertical directions, so to reproduce a horizontal or vertical wave of 0.5 cycle / pixel It is necessary to apply an optimum interpolation filter to each of the horizontal and vertical waves. To solve this problem, prepare an optimal interpolation filter for each horizontal and vertical wave, determine the direction of the wave in each local area on the image, and the result of the horizontal interpolation filter according to the determined result And a value obtained by synthesizing the result of the interpolation filter in the vertical direction is used as an interpolation value (for example, Patent Document 1 and Non-Patent Document 1). By using these techniques, it is possible to reproduce high frequencies close to the theoretical limit of 0.5 cycle / pixel in the horizontal and vertical directions using the G pixel information of the Bayer array.

JP-A-7-236147

Murata, Mori, Maenaka, Okada and Chihara, “Correlation Discrimination Color Separation Method in PS-CCD”, Journal of the Institute of Image Information and Television Engineers, Vol. 55, no. 1, pp. 120-132,2001

  On the other hand, since it is theoretically impossible to match the spectral characteristics of an on-chip color filter used in a color solid-state imaging device with a human color matching function, for example, for the three primary colors of the Bayer array, Furthermore, by using an image sensor that obtains the intensity of four colors obtained by adding one color, and by applying a linear matrix to the intensity values of the four colors acquired by such an image sensor, the intensity values of the three primary colors are obtained. Thus, there is a method for improving the color reproduction of the imaging apparatus. For example, there is a technique that can improve color reproducibility using a color solid-state imaging device having a four-color arrangement in which a yellow on-chip filter is added to the conventional three primary colors to obtain the intensity of four colors (for example, Patent Document 2).

JP 2002-271804 A

  However, in the prior art using a color filter of a four-color array, what kind of demosaicing process can be performed to obtain a four-color intensity synchronized with each pixel from the four-color array is concrete. Was not shown.

  Complementary color systems (cyan, magenta, yellow, and green) have been used for image sensors such as video cameras, but color filters having primary color system colors are also practical. It has become. In the conventional color arrangement of the four primary colors, half of all pixels cannot be assigned to G as in the Bayer arrangement, so the conventional technique for reproducing the high frequency of the luminance component using the checkered G arrangement can be used. Disappear. For this reason, when the conventional primary color four-color arrangement is used, the color reproduction is improved, but the high-frequency component of the luminance is not reproduced, and the juginess appears remarkably or the image becomes smooth without smoothness. There was a problem.

  The present invention has been made in view of such a situation, and in a single-plate color imaging device using an array composed of four or more colors of primary colors, it is possible to improve reproduction of high frequency components of luminance, It is possible to reduce the jegginess and improve the resolution.

  The image processing apparatus according to the present invention includes at least four types of filters having different spectral sensitivities, and an image acquisition unit that acquires a mosaic image by an image sensor in which any of a plurality of types of filters is used for each pixel. And image processing means for generating, from the mosaic image acquired by the image acquisition means, a color image in which intensity information for each pixel position corresponding to a plurality of colors determined by the spectral sensitivity of the filter is aligned in all pixels, and image acquisition The color filter array of the image sensor of the means has a spectral characteristic in which the filters of the pixels arranged in a checkered pattern in half of all the pixels have a strong correlation with the spectral characteristic of luminance (for example, the relative visibility curve). The image processing means calculates the color intensity estimation value of the pixels arranged in a checkered pattern for each pixel, and each pixel is calculated based on the intensity estimation value. And calculates the luminance value.

  The color filter array of the image sensor of the image acquisition means includes a first filter corresponding to a first color having a spectral characteristic having a strong correlation with a spectral characteristic of luminance (for example, a relative visibility curve) among a plurality of types of filters. The second filter corresponding to the second color that is arranged every other line in the horizontal and vertical directions and has a spectral characteristic having a strong correlation with a spectral characteristic (for example, a relative visibility curve) having a luminance different from that of the first color. Are arranged on a line different from the first filter every other line in the horizontal and vertical directions, and the image processing means is a first color intensity estimation value at the target pixel position in the mosaic image. A first calculation unit that calculates an intensity estimation value of the second color, a second calculation unit that calculates a second intensity estimation value that is an intensity estimation value of the second color at the target pixel position, and a first calculation unit. Calculated first strength And estimates can be made and a synthesizing means for synthesizing the second intensity estimation value calculated by the second calculation means.

  The first calculating means is based on an intensity value corresponding to a known color already obtained at the target pixel position and a correlation between a known color calculated from a pixel near the target pixel and a target color to be estimated. A third calculation unit that calculates an intensity estimation value corresponding to the target color at the target pixel position, and if the target pixel is not the first color, the color of the filter at the target pixel position is a known color; The first intensity estimated value can be calculated by the third calculating means using one color as the target color.

  The third calculating means includes a generating means for generating a plurality of sets of pixel intensities of pixels corresponding to known colors and pixel intensities corresponding to target colors in the vicinity of the target pixel, and a known generated by the generating means. A fourth calculating means for calculating the center of gravity and inclination of the color distribution between the known color and the target color from a plurality of sets of the pixel intensity of the pixel corresponding to the color and the pixel intensity of the pixel corresponding to the target color; Fifth calculating means for calculating an intensity estimated value corresponding to the target color at the target pixel position based on the gravity center and inclination of the color distribution calculated by the calculating means and the intensity value corresponding to the known color at the target pixel position. Can be provided.

  The fifth calculation means can calculate an intensity estimation value corresponding to the target color at the target pixel position using regression calculation.

  The second calculation means is based on the intensity value corresponding to the known color already obtained at the target pixel position and the correlation between the known color calculated from the pixels near the target pixel and the target color to be estimated. Thus, a third calculating means for calculating an intensity estimation value corresponding to the target color at the target pixel position can be included. If the target pixel is not the second color, the filter of the target pixel position The second intensity estimated value can be calculated by the third calculation means with the color as the known color and the second color as the target color.

  The third calculating means includes a generating means for generating a plurality of sets of pixel intensities of pixels corresponding to known colors and pixel intensities corresponding to target colors in the vicinity of the target pixel, and a known generated by the generating means. A fourth calculating means for calculating the center of gravity and inclination of the color distribution between the known color and the target color from a plurality of sets of the pixel intensity of the pixel corresponding to the color and the pixel intensity of the pixel corresponding to the target color; Fifth calculating means for calculating an intensity estimated value corresponding to the target color at the target pixel position based on the gravity center and inclination of the color distribution calculated by the calculating means and the intensity value corresponding to the known color at the target pixel position. Can be provided.

  The fifth calculation means can calculate an intensity estimation value corresponding to the target color at the target pixel position using regression calculation.

  The first calculating means is based on an intensity value corresponding to a known color already obtained at the target pixel position and a correlation between a known color calculated from a pixel near the target pixel and a target color to be estimated. Thus, it is possible to include two or more third calculation means for calculating an intensity estimation value corresponding to the target color at the target pixel position. When the target pixel is not the first color, the first first The calculation unit 3 can calculate an intensity value corresponding to the second color at the target pixel position, with the filter color at the target pixel position being a known color and the second color being a target color. The second third calculation means may be configured to calculate the first intensity estimation value using the second color as a known color and the first color as a target color.

  The third calculating means includes a generating means for generating a plurality of sets of pixel intensities of pixels corresponding to known colors and pixel intensities of pixels corresponding to target colors, and pixels corresponding to known colors generated by the generating means. Calculated by a fourth calculation means and a fourth calculation means for calculating the centroid and inclination of the color distribution between the known color and the target color from a plurality of sets of the pixel intensity and the pixel intensity of the pixel corresponding to the target color. And a fifth calculating means for calculating an estimated intensity value corresponding to the target color at the target pixel position based on the center of gravity and inclination of the color distribution and the intensity value corresponding to the known color at the target pixel position. Can be.

  The fifth calculation means can calculate an intensity estimation value corresponding to the target color at the target pixel position using regression calculation.

  The second calculation means is based on the intensity value corresponding to the known color already obtained at the target pixel position and the correlation between the known color calculated from the pixels near the target pixel and the target color to be estimated. Thus, it is possible to include two or more third calculation means for calculating an intensity estimation value corresponding to the target color at the target pixel position. If the target pixel is not the second color, the first first The calculation unit 3 may calculate an intensity value corresponding to the first color at the target pixel position, with the filter color at the target pixel position being a known color and the first color as a target color. The second and third calculation means may be configured to calculate the second intensity estimation value using the first color as a known color and the second color as a target color.

  The third calculating means includes a generating means for generating a plurality of sets of pixel intensities of pixels corresponding to known colors and pixel intensities of pixels corresponding to target colors, and pixels corresponding to known colors generated by the generating means. Calculated by a fourth calculation means and a fourth calculation means for calculating the centroid and inclination of the color distribution between the known color and the target color from a plurality of sets of the pixel intensity and the pixel intensity of the pixel corresponding to the target color. And a fifth calculating means for calculating an estimated intensity value corresponding to the target color at the target pixel position based on the center of gravity and inclination of the color distribution and the intensity value corresponding to the known color at the target pixel position. Can be.

  The fifth calculation means can calculate an intensity estimation value corresponding to the target color at the target pixel position using regression calculation.

  The first calculating means includes first intensity estimating means for calculating a first estimated value of the first intensity estimated value, and second intensity for calculating a second estimated value of the first intensity estimated value. An estimation means, a discrimination means for discriminating a texture direction in the vicinity of the target pixel, a first estimated value estimated by the first intensity estimation means based on a discrimination result by the discrimination means, and a second intensity estimation means A selection means for selecting a suitable estimated value among the estimated second estimated values can be provided, and the first intensity estimating means has a known already obtained position at the target pixel position. Based on the intensity value corresponding to the color and the correlation between the known color calculated from the pixels near the target pixel and the target color to be estimated, an intensity estimation value corresponding to the target color at the target pixel position is calculated. Include 3 calculation means If the target pixel is not the first color, the third calculation means sets the first intensity estimated value by the third calculation unit using the filter color at the target pixel position as the known color and the first color as the target color. The first estimated value can be calculated, the second intensity estimating means can include two or more third calculating means, and the pixel of interest is in the first color. If not, the first third calculation means uses the filter color at the target pixel position as the known color, the second color as the target color, and the intensity value corresponding to the second color at the target pixel position. The second and third calculation means can use the second color as a known color and the first color as a target color, and a second estimated value of the first intensity estimated value. The selection means includes a determination result by the determination means, and The first estimation estimated by the first intensity estimating unit based on whether the filter corresponding to the first color is positioned in the horizontal direction or the vertical direction with respect to the target pixel position. A suitable estimated value can be selected from the value and the second estimated value estimated by the second intensity estimating means.

  The second calculating means includes a first intensity estimating means for calculating a first estimated value of the second intensity estimated value, and a second intensity for calculating a second estimated value of the second intensity estimated value. An estimation means, a discrimination means for discriminating a texture direction in the vicinity of the target pixel, a first estimated value estimated by the first intensity estimation means based on a discrimination result by the discrimination means, and a second intensity estimation means A selection means for selecting a suitable estimated value among the estimated second estimated values can be provided, and the first intensity estimating means has a known already obtained position at the target pixel position. Based on the intensity value corresponding to the color and the correlation between the known color calculated from the pixels near the target pixel and the target color to be estimated, an intensity estimation value corresponding to the target color at the target pixel position is calculated. Include 3 calculation means If the target pixel is not the second color, the third calculation means calculates the second intensity estimated value by using the filter color at the target pixel position as the known color and the second color as the target color. The first estimated value can be calculated, the second intensity estimating means can include two or more third calculating means, and the target pixel is in the second color. If not, the first third calculation means sets the color of the filter at the target pixel position as the known color, the first color as the target color, and the intensity value corresponding to the first color at the target pixel position. The second and third calculation means may use the first color as a known color and the second color as a target color, and a second estimated value of the second intensity estimated value. The selection means includes a determination result by the determination means, and The first estimation estimated by the first intensity estimating unit based on whether the filter corresponding to the second color is positioned in the horizontal direction or the vertical direction with respect to the target pixel position. A suitable estimated value can be selected from the value and the second estimated value estimated by the second intensity estimating means.

  The filter arrangement of the plurality of types of filters of the image sensor of the image acquisition means is such that the third color filter different from any of the first color and the second color of the filters is one line horizontally and vertically. And a fourth color filter arranged on a horizontal line different from the first filter and different from any of the first color, the second color, and the third color of the filters. However, it is possible to arrange every other line horizontally and vertically and on a different horizontal line from the second filter.

  The filter arrangement of the plurality of types of filters of the image sensor of the image acquisition means is such that the third color and fourth color filters different from any of the first color and the second color of the filters are horizontal. And every other line vertically and alternately on a horizontal line different from the first filter, the first color of the filter, the second color, the third color, and , The fifth and sixth color filters different from any of the fourth color are arranged alternately in horizontal and vertical lines and on different horizontal lines from the second filter. It can be.

  The third color and the fourth color can have spectral characteristics having high correlation with each other, and the fifth color and the sixth color have spectral characteristics with high correlation with each other. be able to.

  The third color and the fourth color may be the same color.

  The fifth color and the sixth color may be the same color.

  The color filter array of the image sensor of the image acquisition means includes a first filter corresponding to a first color having a spectral characteristic having a strong correlation with a spectral characteristic of luminance (for example, a relative visibility curve) among a plurality of types of filters. The image processing means includes an intensity value corresponding to a known color already obtained at the target pixel position, and a known color calculated from pixels near the target pixel. Based on the correlation between the target colors to be estimated, a first calculation unit that calculates an intensity estimation value corresponding to the target color at the target pixel position can be included. The first intensity estimated value can be calculated by the first calculating means using the color as the known color and the first color as the target color.

  The first calculating means includes a generating means for generating a plurality of sets of pixel intensities of pixels corresponding to known colors and pixel intensities of pixels corresponding to target colors, and pixels corresponding to known colors generated by the generating means. Calculated by a second calculating means and a second calculating means for calculating the center of gravity and inclination of the color distribution between the known color and the target color from a plurality of sets of the pixel intensity corresponding to the target color and the pixel intensity of the pixel corresponding to the target color. And third calculating means for calculating an estimated intensity value corresponding to the target color at the target pixel position based on the center of gravity and inclination of the color distribution and the intensity value corresponding to the known color at the target pixel position. Can be.

  The second calculation means may be configured to calculate an intensity estimation value corresponding to the target color at the target pixel position using regression calculation.

  The filter arrangement of the plurality of types of filters of the image sensor of the image acquisition means is such that filters of the second color different from the first color of the filters are arranged every other line in the horizontal and vertical directions. In the filter position where the filters of the first color and the second color of the filters are not arranged, a third different from both the first color and the second color of the filters is provided. The fourth color filter and the fourth color filter may be arranged so as to form a diagonal lattice pattern every other pixel in the diagonal direction.

  The second color may be a filter color having a spectral characteristic that has sensitivity on a longer wavelength side than the first color, and at least one of the third color and the second color is: The filter color may have a spectral characteristic that has sensitivity on a shorter wavelength side than the first color.

  The filter arrangement of the plurality of types of filters of the image sensor of the image acquisition means includes filters of the second color, the third color, the fourth color, and the fifth color that are different from the first color of the filters. The second color and the third color, and the fourth color and the fifth color are different from each other in the horizontal and vertical directions. It can be arranged so as to be located on a vertical line.

  The second color and the third color may have spectral characteristics having high correlation with each other, and the fourth color and the fifth color may have spectral characteristics with high correlation with each other. be able to.

  The second color and the third color may be the same color.

  The fourth color and the fifth color may be the same color.

  The image processing method of the present invention has at least four types of filters having different spectral sensitivities, and an acquisition step of acquiring a mosaic image by an image sensor in which any of a plurality of types of filters is used for each pixel; An image processing step for generating a color image in which intensity information for each pixel position corresponding to a plurality of colors determined by the spectral sensitivity of the filter is aligned in all pixels from the mosaic image acquired by the processing of the acquisition step, and the acquisition step In the above processing, a mosaic image having a color arrangement such that a filter of pixels arranged in a checkered pattern in half of all pixels has a spectral characteristic having a strong correlation with a spectral characteristic of luminance (for example, a relative visibility curve). The image processing step calculates the color intensity estimation value of the pixels arranged in a checkered pattern for each pixel, and calculates the intensity of each pixel. Characterized in that it comprises a calculation step of calculating a luminance value of each pixel on the basis of the value.

  The program of the present invention includes an acquisition step of acquiring a mosaic image by an image sensor having at least four types of filters having different spectral sensitivities, and any of the plurality of types of filters is used for each pixel, and an acquisition step An image processing step for generating intensity images for each pixel position corresponding to a plurality of colors determined by the spectral sensitivity of the filter from the mosaic image acquired by the processing in step (a). Then, a mosaic image having a color arrangement such that the filter of pixels arranged in a checkered pattern in half of all pixels has a spectral characteristic that has a strong correlation with the spectral characteristic of luminance (for example, the relative visibility curve) is acquired. In the image processing step, an estimated color intensity value of pixels arranged in a checkered pattern is calculated for each pixel, and the intensity estimation is performed for each pixel. To execute a process characterized in that it comprises a calculation step of calculating a luminance value of each pixel based on the values in the computer.

  In the image processing apparatus, the image processing method, and the program according to the present invention, the image sensor includes at least four types of filters having different spectral sensitivities, and any of the plurality of types of filters is used for each pixel. An image is acquired, and from the acquired mosaic image, an intensity estimation value corresponding to a color of a filter having a strong correlation with a spectral characteristic of luminance (for example, a relative visibility curve) arranged in a checkered pattern in the mosaic image is obtained. Based on the calculated intensity estimation value, a luminance value is calculated, and further, intensity information for each pixel position corresponding to a plurality of colors determined by the spectral sensitivity of the filter is obtained for all pixels by calculation based on the luminance value. A similar color image is generated.

  According to the present invention, a mosaic image is acquired, the mosaic image is processed, and a color image is generated. In particular, from the acquired mosaic image, an intensity estimation value corresponding to the color of the filter having a strong correlation with the spectral characteristic of luminance (for example, the relative visibility curve) arranged in a checkered pattern in the mosaic image is calculated, A color image in which the luminance value is calculated based on the calculated intensity estimation value, and the intensity information for each pixel position corresponding to a plurality of colors determined by the spectral sensitivity of the filter is obtained by calculation based on the luminance value. In a single-plate color imaging device that uses an array composed of four or more primary colors, the reproduction of high-frequency components of luminance is improved, and jagginess is reduced, and the sense of resolution is improved. Can be made possible.

  Embodiments of the present invention will be described below. The correspondence relationship between the invention described in this specification and the embodiments of the invention is exemplified as follows. This description is intended to confirm that the embodiments supporting the invention described in this specification are described in this specification. Therefore, even if there is an embodiment that is described in the embodiment of the invention but is not described here as corresponding to the invention, the fact that the embodiment is not It does not mean that it does not correspond to the invention. Conversely, even if an embodiment is described herein as corresponding to an invention, that means that the embodiment does not correspond to an invention other than the invention. Absent.

  Further, this description does not mean all the inventions described in this specification. In other words, this description is for the invention described in the present specification, which is not claimed in this application, that is, for the invention that will be applied for in the future or that will appear and be added by amendment. It does not deny existence.

  According to the present invention, an image processing apparatus can be provided. This image processing apparatus (for example, the digital still camera 201 in FIG. 2) has at least four types of filters (for example, color filters corresponding to each color of RGBE) having different spectral sensitivities, and any of the plurality of types of filters. The image acquisition means (for example, the CCD image sensor 213 in FIG. 2) for acquiring a mosaic image by the image sensor used for each pixel and the mosaic image acquired by the image acquisition means are determined by the spectral sensitivity of the filter. Image processing means (for example, demosaic processing unit 253 in FIG. 5) that generates a color image in which intensity information for each pixel position corresponding to a plurality of colors is the same for all pixels, and a color filter of the image sensor of the image acquisition means The array has a filter with pixels arranged in a checkered pattern in half of all pixels, and the spectral characteristics of luminance (for example, The image processing means calculates the color intensity estimation value of the pixels arranged in a checkered pattern at each pixel, and has the intensity estimation value. The luminance value of each pixel is calculated on the basis of the pixel value, and the luminance information is used to generate a color image in which the intensity information for each pixel position corresponding to a plurality of colors determined by the spectral sensitivity of the filter is uniform in all pixels. it can.

  The color filter array of the image sensor of the image acquisition unit corresponds to the first color (for example, G) having a spectral characteristic (for example, a relative luminous sensitivity curve) that has a strong correlation with the spectral characteristic of the luminance among a plurality of types of filters. The first filter is arranged every other line in the horizontal and vertical directions, and has a spectral characteristic (for example, a relative visibility curve) that is different from the first color and has a strong correlation with the spectral characteristic of luminance. The second filter corresponding to the color (for example, E) is arranged on every other line in the horizontal and vertical directions and on a line different from the first filter, and the image processing means A first calculation unit (for example, a G intensity estimation unit 291 in FIG. 7) that calculates a first intensity estimation value that is an intensity estimation value of the first color in the image, and an intensity estimation of the second color at the target pixel position. Second strength estimate The second calculation means for calculating the value (for example, the E intensity estimation unit 292 in FIG. 7), the first intensity estimation value calculated by the first calculation means, and the first calculation value calculated by the second calculation means. A combining unit (for example, the addition processing unit 293 in FIG. 7) that combines the two estimated intensity values can be provided.

  The first calculation means is based on an intensity value corresponding to a known color already obtained at the target pixel position and a correlation between the known color calculated from the pixels near the target pixel and the target color to be estimated. , Third calculating means for calculating an intensity estimated value corresponding to the target color at the target pixel position (for example, the first intensity estimating unit 311, the second intensity estimating unit 321, and the third intensity estimating unit 322 in FIG. 8). ) And the pixel of interest is not the first color, the color of the filter at the pixel of interest is the known color, the first color is the target color, and the first intensity is calculated by the third calculation means. An estimated value can be calculated.

  The third calculating means is a generating means for generating a plurality of sets of pixel intensities corresponding to the known colors and pixel intensities corresponding to the target colors in the vicinity of the target pixel (for example, the rough interpolation processing unit in FIG. 10). 341) and a plurality of sets of pixel intensities of pixels corresponding to the known colors generated by the generating means and pixel intensities of the pixels corresponding to the target colors, the center of gravity and inclination of the color distribution between the known colors and the target colors are calculated. Fourth calculating means (for example, the statistic calculating section 342 in FIG. 10 and the slope calculating section 381 in FIG. 16), the centroid and inclination of the color distribution calculated by the fourth calculating means, and the known pixel position. A fifth calculating unit (for example, a pixel intensity estimating unit 382 in FIG. 16) that calculates an intensity estimated value corresponding to the target color at the target pixel position based on the intensity value corresponding to the color can be provided.

  The fifth calculation means calculates an intensity estimation value corresponding to the target color at the target pixel position using regression calculation (for example, calculation according to Expression (10), Expression (11), or Expression (12)). Can do.

  The second calculation means is based on the intensity value corresponding to the known color already obtained at the target pixel position and the correlation between the known color calculated from the pixels near the target pixel and the target color to be estimated. , Third calculating means for calculating an intensity estimated value corresponding to the target color at the target pixel position (for example, the first intensity estimating unit 311, the second intensity estimating unit 321 in FIG. 8, the third intensity estimating unit 322). ) And the target pixel is not the second color, the filter color at the target pixel position is the known color, the second color is the target color, and the second intensity estimated value is calculated by the third calculation means. Can be calculated.

  The third calculating means is a generating means for generating a plurality of sets of pixel intensities corresponding to the known colors and pixel intensities corresponding to the target colors in the vicinity of the target pixel (for example, the rough interpolation processing unit in FIG. 10). 341) and a plurality of sets of pixel intensities of pixels corresponding to the known colors generated by the generating means and pixel intensities of the pixels corresponding to the target colors, the center of gravity and inclination of the color distribution between the known colors and the target colors are calculated. Fourth calculating means (for example, the statistic calculating section 342 in FIG. 10 and the slope calculating section 381 in FIG. 16), the centroid and inclination of the color distribution calculated by the fourth calculating means, and the known pixel position. A fifth calculating unit (for example, a pixel intensity estimating unit 382 in FIG. 16) that calculates an intensity estimated value corresponding to the target color at the target pixel position based on the intensity value corresponding to the color can be provided.

  The fifth calculation means calculates an intensity estimation value corresponding to the target color at the target pixel position using regression calculation (for example, calculation according to Expression (10), Expression (11), or Expression (12)). Can do.

  The first calculation means is based on an intensity value corresponding to a known color already obtained at the target pixel position and a correlation between the known color calculated from the pixels near the target pixel and the target color to be estimated. If there are two or more third calculation means for calculating the intensity estimation value corresponding to the target color at the target pixel position, and the target pixel is not the first color, the first third calculation means (for example, FIG. 8, the second intensity estimating unit 321 and the third intensity estimating unit 322) use the filter color at the target pixel position as the known color, the second color as the target color, and the second color at the target pixel position. The corresponding intensity value is calculated, and the second and third calculating means (for example, the fourth intensity estimating unit 323 in FIG. 8) sets the second color as the known color and the first color as the target color. A first intensity estimate can be calculated.

  The third calculating means generates generating means (for example, the coarse interpolation processing unit 341 in FIG. 10) that generates a plurality of sets of pixel intensities of pixels corresponding to known colors and pixel intensities corresponding to target colors, Fourth calculation for calculating the gravity center and inclination of the color distribution between the known color and the target color from a plurality of sets of the pixel intensity of the pixel corresponding to the known color and the pixel intensity of the pixel corresponding to the target color generated by the means Means (for example, the statistic calculator 342 in FIG. 10 and the slope calculator 381 in FIG. 16), the gravity center and slope of the color distribution calculated by the fourth calculator, and the intensity corresponding to the known color at the target pixel position. And a fifth calculation unit (for example, a pixel intensity estimation unit 382 in FIG. 16) that calculates an intensity estimation value corresponding to the target color at the target pixel position based on the value.

  The fifth calculation means calculates an intensity estimation value corresponding to the target color at the target pixel position using regression calculation (for example, calculation according to Expression (10), Expression (11), or Expression (12)). Can do.

  The second calculation means is based on the intensity value corresponding to the known color already obtained at the target pixel position and the correlation between the known color calculated from the pixels near the target pixel and the target color to be estimated. When there are two or more third calculation means for calculating an estimated intensity value corresponding to the target color at the target pixel position, and the target pixel is not the second color, the first third calculation means (for example, FIG. 8, the second intensity estimation unit 321 and the third intensity estimation unit 322) use the filter color at the target pixel position as the known color, the first color as the target color, and the first color at the target pixel position. The corresponding intensity value is calculated, and the second and third calculating means (for example, the fourth intensity estimating unit 323 in FIG. 8) uses the first color as the known color and the second color as the target color. A second intensity estimate can be calculated.

  Third calculation means includes generation means for generating a plurality of sets of pixel intensities of pixels corresponding to the known color and pixel intensities of pixels corresponding to the target color (for example, the rough interpolation processing unit 341 in FIG. 10); Centroid and inclination of a color distribution between the known color and the target color from a plurality of sets of pixel intensity of the pixel corresponding to the known color and pixel intensity of the pixel corresponding to the target color generated by the generating unit A fourth calculation unit (for example, the statistic calculation unit 342 in FIG. 10 and the inclination calculation unit 381 in FIG. 16), the centroid and inclination of the color distribution calculated by the fourth calculation unit, Based on the intensity value corresponding to the known color at the target pixel position, fifth calculation means for calculating an intensity estimation value corresponding to the target color at the target pixel position (for example, the pixel intensity estimation unit 382 in FIG. 16). And It can be obtained.

  The fifth calculation means calculates an intensity estimation value corresponding to the target color at the target pixel position using regression calculation (for example, calculation according to Expression (10), Expression (11), or Expression (12)). Can do.

  The first calculation means is first intensity estimation means for calculating the first estimated value of the first intensity estimated value (for example, the second intensity estimation unit 321 in the second processing block 303 in FIG. 8 or , A third intensity estimating unit 322 in the third processing block 304) and second intensity estimating means for calculating a second estimated value of the first estimated intensity value (for example, the second processing block in FIG. 8). The third intensity estimator 322 and the fourth intensity estimator 323 in 303, or the second intensity estimator 321 and the fourth intensity estimator 323 in the third processing block 304), and the texture in the vicinity of the target pixel A first estimation value estimated by the first intensity estimation unit and a second intensity based on the determination unit (for example, the texture direction determination unit 305 in FIG. 8) for determining the direction and the determination result by the determination unit. Guess Selecting means (for example, switch 324 in FIG. 8) for selecting a suitable estimated value from the second estimated values estimated by the means, and the first intensity estimating means may be provided at the target pixel position. The intensity corresponding to the target color at the target pixel position based on the intensity value corresponding to the known color already obtained and the correlation between the known color calculated from the pixels near the target pixel and the target color to be estimated When a third calculation unit (for example, the second intensity estimation unit 321 and the third intensity estimation unit 322 in FIG. 8) that calculates an estimated value can be included, and the pixel of interest is not the first color, A first estimated value of the first intensity estimated value is calculated by the third calculating means using the filter color at the target pixel position as the known color and the first color as the target color, and the second intensity estimating means Including two or more third calculation means; When the pixel is not in the first color, the first third calculating unit (for example, the second intensity estimating unit 321 and the third intensity estimating unit 322 in FIG. 8) uses the filter color at the target pixel position. Is the known color, the second color is the target color, the intensity value corresponding to the second color at the target pixel position is calculated, and the second third calculating means (for example, the fourth intensity estimation in FIG. 8) is calculated. The unit 323) calculates the second estimated value of the first intensity estimated value using the second color as the known color and the first color as the target color, and the selecting unit is configured to determine the determination result by the determining unit, and The first intensity estimated by the first intensity estimating unit based on whether the filter corresponding to the first color is positioned in the horizontal direction or the vertical direction with respect to the target pixel position. Of the estimated value and the second estimated value estimated by the second intensity estimating means, a suitable estimated value Can be selected.

  The second calculating means is first intensity estimating means for calculating a first estimated value of the second intensity estimated value (for example, the second intensity estimating unit 321 in the second processing block 303 in FIG. 8 or , A third intensity estimating unit 322) in the third processing block 304, and second intensity estimating means for calculating a second estimated value of the second estimated intensity value (for example, the second processing block in FIG. 8). The third intensity estimator 322 and the fourth intensity estimator 323 in 303, or the second intensity estimator 321 and the fourth intensity estimator 323 in the third processing block 304), and the texture in the vicinity of the target pixel A first estimation value estimated by the first intensity estimation unit and a second intensity based on the determination unit (for example, the texture direction determination unit 305 in FIG. 8) for determining the direction and the determination result by the determination unit. Guess Selecting means (for example, switch 324 in FIG. 8) for selecting a suitable estimated value from the second estimated values estimated by the means, and the first intensity estimating means may be provided at the target pixel position. The intensity corresponding to the target color at the target pixel position based on the intensity value corresponding to the known color already obtained and the correlation between the known color calculated from the pixels near the target pixel and the target color to be estimated When the third calculation means (for example, the second intensity estimation unit 321 and the third intensity estimation unit 322 in FIG. 8) for calculating the estimated value can be included, and the pixel of interest is not the second color, Using the filter color at the target pixel position as the known color and the second color as the target color, the third calculation means calculates the first estimated value of the second intensity estimated value, and the second intensity estimating means Including two or more third calculation means When the pixel of interest is not the second color, the first third calculating means (for example, the second intensity estimating unit 321 and the third intensity estimating unit 322 in FIG. 8) An intensity value corresponding to the first color at the target pixel position is calculated using the color as the known color and the first color as the target color, and second third calculation means (for example, the fourth intensity in FIG. 8). The estimation unit 323) calculates the second estimated value of the second intensity estimated value using the first color as the known color and the second color as the target color, and the selecting unit is configured to determine the determination result by the determining unit, The first intensity estimated by the first intensity estimating unit is based on whether the filter corresponding to the second color is positioned in the horizontal direction or the vertical direction with respect to the target pixel position. Of the estimated value and the second estimated value estimated by the second intensity estimating means. A constant value can be selected.

  The filter arrangement of the plurality of types of filters of the image sensor of the image acquisition means is such that a filter of a third color (for example, R) different from any of the first color and the second color of the filters is horizontal. The first and second colors are arranged on a horizontal line different from the first filter and are different from any one of the first color, the second color, and the third color. Four color (e.g., B) filters can be arranged on a horizontal line that is different from the second filter every other line in the horizontal and vertical directions.

  The filter array of the plurality of types of filters of the image sensor of the image acquisition means has a third color (for example, R1 in FIG. 53) different from any of the first color and the second color of the filter and the first color. Filters of four colors (for example, R2 in FIG. 53) are alternately arranged on a horizontal line different from the first filter every other line in the horizontal and vertical directions. Filters of a fifth color (for example, B1 in FIG. 53) and a sixth color (for example, B2 in FIG. 53) that are different from any of the second color, the third color, and the fourth color are provided. , Every other line in the horizontal and vertical directions, and can be alternately arranged on a different horizontal line from the second filter.

  The third color and the fourth color can have spectral characteristics having a high correlation with each other, and the fifth color and the sixth color can have spectral characteristics with a high correlation with each other.

  The third color and the fourth color may be the same color.

  The fifth color and the sixth color may be the same color.

  The color filter array of the image sensor of the image acquisition unit corresponds to the first color (for example, G) having a spectral characteristic having a strong correlation with the spectral characteristic of luminance (for example, the relative visibility curve) among the plurality of types of filters. The first filter is arranged in a checkered pattern, and the image processing means includes third calculation means (for example, the first intensity estimation unit 621 and the third intensity estimation unit 622 in FIG. 56). The first intensity estimated value can be calculated by the third calculation means with the filter color at the pixel position as the known color and the first color as the target color.

  The first calculation means includes generation means (for example, a rough interpolation processing unit 631 in FIG. 57) that generates a plurality of sets of pixel intensities of pixels corresponding to known colors and pixel intensities corresponding to target colors, A second calculation unit that calculates a center of gravity and an inclination of a color distribution between the known color and the target color from a plurality of sets of pixel intensity of the pixel corresponding to the known color and pixel intensity of the pixel corresponding to the target color generated by the generation unit The calculation means (for example, the statistic calculation unit 342 in FIG. 57 and the inclination calculation unit 381 in FIG. 16) and the gravity center and inclination of the color distribution calculated by the fourth calculation unit correspond to the known color at the target pixel position. Based on the intensity value, a third calculating unit (for example, a pixel intensity estimating unit 382 in FIG. 57) that calculates an intensity estimated value corresponding to the target color at the target pixel position can be provided.

  The second calculation means calculates an intensity estimation value corresponding to the target color at the target pixel position using regression calculation (for example, calculation according to Expression (10), Expression (11), or Expression (12)). Can do.

  In the filter arrangement of the plurality of types of filters of the image sensor of the image acquisition means, filters of the second color (for example, R) different from the first color of the filters are arranged every other line in the horizontal and vertical directions. A third color different from any of the first color and the second color of the filter is provided at a filter position where the filters of the first color and the second color of the filter are not disposed. The filters of (for example, B) and the fourth color (for example, E) can be arranged so as to form an oblique lattice pattern every other pixel in the oblique direction.

  The second color is a filter color having a spectral characteristic that has a sensitivity on a longer wavelength side than the first color, and at least one of the third color and the fourth color is shorter than the first color. The filter color may have spectral characteristics that have sensitivity on the wavelength side.

  The filter arrangement of the plurality of types of filters of the image sensor of the image acquisition means includes a second color (for example, R1 in FIG. 68) different from the first color, and a third color (for example, in FIG. 68). R2), the fourth color (for example, B1 in FIG. 68), and the fifth color (for example, B2 in FIG. 68) have two horizontal and vertical filters, and one pixel in the diagonal direction. Each of the second and third colors, and the fourth and fifth colors are arranged so as to be located on different horizontal and vertical lines. Can do.

  The second color and the third color may have spectral characteristics having high correlation with each other, and the fourth color and the fifth color may have spectral characteristics with high correlation with each other. be able to.

  The second color and the third color may be the same color.

  The fourth color and the fifth color may be the same color.

  According to the present invention, an image processing method is provided. This image processing method has at least four types of filters (for example, color filters corresponding to each color of RGBE) having different spectral sensitivities, and any one of a plurality of types of filters is used for each pixel. The acquisition step (for example, the process of step S1 in FIG. 23) for acquiring the mosaic image by the above, and the intensity for each pixel position corresponding to a plurality of colors determined by the spectral sensitivity of the filter from the mosaic image acquired by the process of the acquisition step And an image processing step (for example, the processing in step S4 in FIG. 23) for generating a color image in which information is aligned for all pixels. In the processing in the acquisition step, pixels arranged in a checkered pattern in half of all pixels The color filter has a spectral characteristic that has a spectral characteristic that has a strong correlation with the spectral characteristic of luminance (for example, the relative visibility curve). One mosaic image is acquired, and the image processing step calculates the color intensity estimation value of the pixels arranged in the checkered pattern at each pixel, and calculates the luminance value of each pixel based on the intensity estimation value Calculation step (for example, step S23 in FIG. 24 or step S453 in FIG. 62).

  According to the present invention, a program is provided. This program has at least four types of filters (for example, color filters corresponding to each color of RGBE) having different spectral sensitivities, and mosaic is performed by an image sensor in which any one of a plurality of types of filters is used for each pixel. Intensity information for each pixel position corresponding to a plurality of colors determined by the spectral sensitivity of the filter is obtained from the acquisition step (for example, the processing of step S1 in FIG. 23) for acquiring an image and the mosaic image acquired by the processing of the acquisition step. An image processing step (for example, the processing in step S4 in FIG. 23) for generating a color image that matches all pixels, and in the processing of the acquisition step, a filter included in pixels arranged in a checkered pattern in half of all the pixels However, color arrangements that have spectral characteristics that have a strong correlation with the spectral characteristics of luminance (for example, the relative visibility curve) are also included. A mosaic image is acquired, and the image processing step calculates the color intensity estimation value of the pixels arranged in the checkered pattern for each pixel, and calculates the luminance value of each pixel based on the intensity estimation value. A computer is caused to execute a process including a calculation step (for example, the process of step S23 of FIG. 24 or the process of step S453 of FIG. 62).

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 2 is a block diagram showing a configuration of a digital still camera 201 that executes arithmetic processing to which the present invention is applied.

  As shown in FIG. 2, a digital still camera 201 includes a lens 211, an aperture 212, a CCD (Charge Coupled Devices) image sensor 213, a correlated double sampling (CDS) circuit 214, an A / D converter 215, and a DSP. (Digital Signal Processor) From block 216, timing generator 217, D / A converter 218, video encoder 219, display unit 220, codec (CODEC: COmpression / DECompression) processing unit 221, memory 222, CPU 223, and operation input unit 224 Composed.

  The CCD is a semiconductor element that converts light information into an electrical signal (photoelectric conversion). The CCD image sensor 213 has a plurality of light receiving elements (pixels) that convert light into electricity, and changes in light for each pixel. It converts to an electric signal independently. The correlated double sampling circuit 214 samples the reset noise, which is the main component of the noise included in the output signal of the CCD image sensor 213, from the sampled video signal period of the output pixel signals, and the reference period. Is a circuit that removes the signal by subtracting the sampled signal. The A / D converter 215 converts the supplied analog signal after noise removal into a digital signal.

  The DSP block 216 is a block having a signal processing processor and an image RAM, and the signal processing processor performs pre-programmed image processing or hardware arithmetic processing on the image data stored in the image RAM. The image processing configured as follows is performed. The timing generator 217 is a logic circuit that generates various horizontal and vertical driving pulses necessary for driving the CCD and pulses used in the analog front processing in synchronization with the reference clock. The timing clock generated by the timing generator 217 is also supplied to the codec processing unit 221, the memory 222, and the CPU 223 via the bus 225.

  The D / A converter 218 converts the supplied digital signal into an analog signal and outputs the analog signal. The video encoder 219 encodes the supplied analog signal into video data in a format that can be displayed on the display unit 220. The display unit 220 is configured by, for example, an LCD (Liquid Crystal Display) and displays the video signal supplied from the video encoder 219.

  The codec processing unit 221 executes processing based on a compression or expansion algorithm of digital image data such as JPEG (Joint Picture Experts Group). The memory 222 is configured by, for example, a semiconductor memory, a magnetic disk, a magneto-optical disk, or an optical disk, and stores supplied data or outputs stored data based on control of the CPU 223. . Note that the memory 222 may be detachable from the digital still camera 201.

  The CPU 223 controls each unit of the digital still camera 201 based on a user operation input supplied from the operation input unit 224 via the bus 225. The operation input unit 224 includes, for example, a jog dial, a key, a lever, a button, or a touch panel as well as a button for instructing recording, and receives an operation input by a user.

  Light input through the lens 211 and the diaphragm 212 is incident on the CCD image sensor 213, converted into an electrical signal by photoelectric conversion at the light receiving element, and supplied to the correlated double sampling circuit 214. The correlated double sampling circuit 214 removes noise by subtracting the sampled video signal period and the sampled reference period from the pixel signals output from the CCD image sensor 213, and A / D This is supplied to the converter 215. The A / D converter 215 converts the supplied analog signal after noise removal into a digital signal, and temporarily stores it in the image RAM of the DSP block 216.

  The timing generator 217 controls the CCD image sensor 213, the correlated double sampling circuit 214, the A / D converter 215, and the DSP block 216 so as to maintain image capture at a constant frame rate during imaging. .

  The DSP block 216 receives supply of pixel stream data at a constant rate, temporarily stores it in the image RAM, and executes image processing (to be described later) on the temporarily stored image data in the signal processing processor. When the image block is displayed on the display unit 220 after the image processing is completed, the DSP block 216 displays the code data in the D / A converter 218 and the codec processing unit 221 in the case where the image data is stored in the memory 222. Supply image data.

  The D / A converter 218 converts the digital image data supplied from the DSP block 216 into an analog signal and supplies the analog signal to the video encoder 219. The video encoder 219 converts the supplied analog image signal into a video signal, and outputs the video signal to the display unit 220 for display. That is, the display unit 220 plays a role of a camera finder in the digital still camera 201. The codec processing unit 221 performs a predetermined encoding on the image data supplied from the DSP block 216 and supplies the encoded image data to the memory 222 for storage.

  Further, the codec processing unit 221 reads data designated by the user from among the data stored in the memory 222 based on the control of the CPU 223 that has received the user's operation input from the operation input unit 224, and performs predetermined decoding. The decoded signal is output to the DSP block 216. Thus, the decoded signal is supplied to the D / A converter 218 via the DSP block 216, converted into an analog signal, encoded by the video encoder 219, and displayed on the display unit 220.

  Incidentally, the on-chip color filter of the CCD image sensor 213 in FIG. 2 normally uses three or four kinds of colors, and these on-chip color filters have different colors alternately for each light receiving element. They are arranged in a mosaic. For example, FIG. 3 shows an example of an array of four primary colors in which a fourth color (hereinafter referred to as E) is added to three colors RGB (Red, Green, Blue). .

  In the array shown in FIG. 3, one of the G filters that transmits only green (G) light in the Bayer array is a filter that transmits only E light, and only red (R) light is present. One filter of R that transmits light and one filter of B that transmits only blue (B) light, a total of four is configured as a minimum unit. Of RGB, G has a spectral characteristic closest to human visibility characteristics. G and E in the array shown in FIG. 3 have spectral characteristics that have a strong correlation with the spectral characteristics of luminance (for example, the relative visibility curve).

  Here, the four-color arrangement of the on-chip color filter of the CCD image sensor 213 will be further described. The intensity of each pixel of the mosaic image obtained by the on-chip color filter of the CCD image sensor 213 is proportional to the product of the spectral characteristic of the corresponding on-chip filter and the spectral characteristic of incident light. FIG. 4 shows an example of the spectral characteristics of each color in the primary color system four-color arrangement shown in FIG. The spectral characteristics of the filter need to be carefully designed in consideration of the spectral characteristics and noise characteristics of the sensor itself, the coefficients of a linear matrix, which will be described later, and the physical constraints of the material used for the filter. When applying the present invention, the effect is not particularly affected by the spectral sensitivity characteristics of these filters. Therefore, if the spectral sensitivity is designed in consideration of the above-described color reproducibility and noise characteristics. Good. For example, FIG. 4 shows an example in which the spectral characteristic of the fourth color E has a peak on the shorter wavelength side than G, but E has a peak on the longer wavelength side than G (slightly yellow). Spectral characteristics such as (more visible colors) may be used.

  As described above, when the color filter having the color arrangement shown in FIG. 3 is used as the on-chip color filter of the CCD image sensor 213, the image temporarily stored in the image RAM of the DSP block 216 is stored in each pixel. Both have only one color of R, G, B, and E. Therefore, the signal processing processor of the DSP block 216 processes this image by using an image processing program or hardware incorporated in advance, and generates image data having data of all colors in all pixels.

  FIG. 5 is a block diagram showing a more detailed configuration of the DSP block 216 of FIG.

  As described above, the DSP block 216 includes the image RAM 241 and the signal processing processor 242, and the signal processing processor 242 includes the white balance adjustment unit 251, the gamma correction unit 252, the demosaic processing unit 253, and the gamma inverse correction unit. 254, a matrix processing unit 255, a gamma correction unit 256, and a YC conversion unit 257.

  The mosaic image converted into a digital signal by the A / D converter 215 is temporarily stored in the image RAM 241. The mosaic image is an intensity signal corresponding to any color of R, G, B, or E for each pixel, that is, an array determined by the color filter used in the CCD image sensor 213 (for example, FIG. It is composed of periodic pattern intensity signals of the primary color four-color arrangement described above.

  The white balance adjustment unit 251 performs processing (white balance adjustment processing) for applying an appropriate coefficient to the mosaic image according to the color of each pixel intensity so that the color balance of the achromatic subject region becomes an achromatic color. ). The gamma correction unit 252 performs gamma correction on each pixel intensity of the mosaic image whose white balance has been adjusted. A numerical value “gamma (γ)” is used to represent the response characteristics of the gradation of the image. The gamma correction is a correction process for correctly displaying the brightness and color saturation of the image displayed on the display unit 220. The signal output to the display unit 220 reproduces the brightness and color of the image by applying a specific voltage for each pixel. However, due to the characteristics (gamma value) of the display unit 220, the brightness and color of the actually displayed image does not double the brightness of the CRT even if the input voltage is doubled (has non-linearity). For this reason, the gamma correction unit 252 performs a correction process so that the brightness and color saturation of the image displayed on the display unit 220 are correctly displayed.

  The demosaic processing unit 253 statistically calculates the color distribution shape, thereby performing demosaic processing for aligning all the intensities (intensity information) of R, G, B, and E at each pixel position of the mosaic image subjected to gamma correction. Execute. Therefore, the output signals from the demosaic processing unit 253 are four image signals corresponding to the four colors R, G, B, and E. The gamma inverse correction unit 254 performs a process of correcting the linear characteristic by applying the inverse of the gamma characteristic applied to the demosaiced 4-channel image in order to perform matrix processing in the matrix processing unit 255 in the subsequent stage.

  The matrix processing unit 255 applies an intensity value [Rm, Gm] of the three primary colors by applying a 3 × 4 linear matrix based on preset coefficients to the supplied pixels [R, G, B, E]. , Bm]. This matrix processing needs to be applied to linear image intensity. Linear matrix coefficients are important design items for optimal color reproduction, but since matrix processing is applied after demosaicing, the specific values of linear matrix coefficients are designed independently of demosaicing. be able to. The output of the matrix processing unit 255 is three images corresponding to the three colors Rm, Gm, and Bm that have been color corrected.

  After the matrix processing, the gamma correction unit 256 performs gamma correction again on the color-corrected three-channel image. The YC conversion unit 257 generates and outputs a Y image and a C image (YCbCr image signal) by subjecting the three-channel image of R, G, B to band processing for matrix processing and chroma components.

  In the signal processing processor 242 of the DSP block 216, the gamma correction unit 252 performs gamma correction before the demosaic processing by the demosaic processing unit 253. This is because the reliability of the demosaic processing of the demosaic processing unit 253 can be further improved by executing the demosaic operation in the non-linear pixel intensity space subjected to the gamma correction.

  For example, when the input image is a high-contrast contour region, the color distribution covers a very bright intensity region and a very dark intensity region. Physically, the object reflected light is obtained by multiplying the variation of the object surface by the incident light intensity from the illumination. Therefore, in the linear pixel intensity space proportional to the incident light intensity to the camera, the bright intensity region The distribution of the object color in the sparsely spreads (sparsely), and the distribution of the object color in the dark pixel intensity region does not spread so much and tends to be compact.

  In the demosaic processing unit 253, demosaic processing is executed by statistically calculating the color distribution shape. However, in the high-contrast contour region, the pixel intensity variation in the bright region and the pixel intensity variation in the dark region are greatly different, and it becomes difficult to apply statistical linear regression. Therefore, prior to demosaic processing in the demosaic processing unit 253, the input data is subjected to non-linear pixel intensity conversion such as gamma correction to raise the dark pixel intensity region (close to the bright pixel intensity region). ) By suppressing the dispersion of the pixel intensity to some extent, the reliability of the linear regression processing executed in the demosaic processing unit 253 can be improved.

  The nonlinear transformation applied for such a purpose is preferably a power transformation with an exponent smaller than 1 as in gamma correction, but a combination of a power portion and a linear portion as in sRGB gamma usually used in a color profile or the like. Even if it is such a conversion, any non-linear conversion may be used as long as it can be regarded as being substantially the same as a power function. Needless to say, even if the non-linear conversion is omitted, a non-linear conversion process such as gamma correction may be performed after the demosaic process.

  Further, the gamma characteristic of the gamma correction unit 252 applied before the demosaic process and the gamma characteristic of the gamma correction unit 256 applied before the YC conversion need not be the same. Of course, gamma correction before demosaic processing is not essential in applying the present invention.

  Next, FIG. 6 executes a demosaic process that is a process of sequentially interpolating or estimating the intensity of a color that does not exist so that all RGBE colors exist at all pixel positions. It is a block diagram which shows the further detailed structure of the demosaic process part 253 of FIG.

  The demosaic processing unit 253 includes a local region extraction unit 281, a G + E intensity estimation processing unit 282, an R intensity estimation unit 283-1, a G intensity estimation unit 283-2, a B intensity estimation unit 283-3, and an E intensity estimation unit 283. -4.

  The local area extraction unit 281 cuts out a pixel of a local area having a predetermined size around the target pixel position from the gamma-corrected mosaic image. Here, the local area to be cut out is a 9 × 9 pixel rectangular area centered on the target pixel position. The G + E intensity estimation processing unit 282 calculates the G + E intensity at the target pixel position using pixels existing in the local region. The R intensity estimation unit 283-1, the G intensity estimation unit 283-2, the B intensity estimation unit 283-3, and the E intensity estimation unit 283-4 are the pixels existing in the local region and the G + E intensity estimation processing unit 282. Is used to calculate an estimated value of each RGBE intensity. Since the R intensity estimating unit 283-1, the G intensity estimating unit 283-2, the B intensity estimating unit 283-3, and the E intensity estimating unit 283-4 have basically the same configuration, When it is not necessary to individually distinguish the R intensity estimating unit 283-1, the G intensity estimating unit 283-2, the B intensity estimating unit 283-3, and the E intensity estimating unit 283-4, R (G, B, or E) Intensity estimation unit 283, or R intensity estimation unit 283-1, G intensity estimation unit 283-2, B intensity estimation unit 283-3, and E intensity estimation unit 283-4 are all RGBE It is assumed to be called an intensity estimation unit 283.

  FIG. 7 is a block diagram showing a configuration of the G + E intensity estimation processing unit 282 of FIG.

  The G + E intensity estimation processing unit 282 includes a G intensity estimation unit 291, an E intensity estimation unit 292, and an addition processing unit 293.

  The G intensity estimation unit 291 calculates an estimated value of the G intensity at the target pixel position using the pixels in the local region. The E intensity estimation unit 292 calculates an estimated value of the E intensity at the target pixel position using the pixels in the local region. The addition processing unit 293 calculates the (G + E) intensity estimated value of the target pixel position by adding the calculated G intensity estimated value and the E intensity estimated value. In the mosaic image obtained by the color filter of the primary color four-color arrangement described with reference to FIG. 3, since both G and E have signals only at intervals of every other pixel, interpolation processing of only G or only E It is not easy to reproduce high-frequency image signals by itself, and it is easy to generate jagginess and zipper noise. However, by using the fact that they are arranged at a position that is deviated by one pixel horizontally and vertically, by reproducing a signal such as the sum of G and E signals at all pixel positions, it is higher than the respective limits. It becomes possible to reproduce a high-frequency image signal, and it is possible to suppress the occurrence of jagginess and noise in the output image.

  FIG. 8 is a block diagram showing the configuration of the G intensity estimating unit 291 and the E intensity estimating unit 292. First, common points between the G intensity estimating unit 291 and the E intensity estimating unit 292 will be described, and then differences between the G intensity estimating unit 291 and the E intensity estimating unit 292 will be described.

  The G intensity estimator 291 and the E intensity estimator 292 are a pixel-of-interest selector 301, a first processing block 302, a second processing block 303, a third processing block 304, a texture direction determination unit 305, and The switch 306 is configured. The first processing block 302 includes a first intensity estimation unit 311, and the second processing block 303 and the third processing block 304 include a second intensity estimation unit 321 and a third intensity estimation unit. 322, a fourth intensity estimation unit 323, and a switch 324.

  The target pixel selection unit 301 selects a target pixel position, that is, a central pixel from a 9 × 9 pixel local region, and outputs the pixel intensity. The texture direction determination unit 305 examines the texture of the image in the local region and outputs a result of determining whether it is mainly a vertical stripe or a horizontal stripe. The first intensity estimator 311, the second intensity estimator 321, the third intensity estimator 322, and the fourth intensity estimator 323 calculate the intensity estimate values of predetermined colors at the local region pixel and the target pixel position. Based on this, the intensity of the color to be estimated is calculated. Here, the color to be estimated indicates G for the G intensity estimation unit 291 and E for the E intensity estimation unit 292. The four types of intensity estimation units exist because of the color intensity of a given target pixel position and the reason that there are four types of processing depending on the color arrangement of the local region. The switch 324 selects and outputs which of the two input intensity estimation values should be used based on the texture direction determination result output by the texture direction determination unit 305. The switch 306 selects and outputs which of the four input intensity estimation values should be used based on the color of the target pixel position in the original mosaic image. The output of the switch 306 becomes the output of the G intensity estimating unit 291 and the E intensity estimating unit 292.

  As described above, the operations of the G intensity estimation unit 291 and the E intensity estimation unit 292 must be switched depending on the original color of the target pixel position. In the configuration shown in FIG. 8, for the four types of operations based on the color of the pixel of interest, the selected pixel of interest is output as it is, or the first processing block 302, the second processing block 303, and the third In the processing block 304, an estimated value is calculated, and finally, which value is to be output is selected and output by the switch 306.

  Four types of operations will be described. First, if the target pixel position of the mosaic image is the color to be interpolated, no special estimation process is required, and if the intensity of the target pixel of the mosaic image selected by the target pixel selection unit 301 is output as it is. Therefore, the output of the pixel-of-interest selection unit 301 is output from the switch 306 (the input a in the figure is output). Second, when the target pixel position of the mosaic image is G or E but is not a color to be interpolated, the first intensity estimation unit 311 of the first processing block 302 performs G and E pixels in the local region. An intensity estimated value calculated based on the information is output from the switch 306 (input b in the figure is output).

  Third, when the target pixel position of the mosaic image is shifted horizontally by one pixel from the pixel to be interpolated, that is, if the color to be interpolated is G, the target pixel position is B, and the color to be interpolated If E is E, and the target pixel position is R, the second intensity estimating unit 321, the third intensity estimating unit 322, the fourth intensity estimating unit 323, and the switch of the second processing block 303 The estimated intensity value calculated by the process 324 is output from the switch 306 (the input c in the figure is output). Fourth, when the target pixel position of the mosaic image is shifted by one pixel vertically from the pixel of the color to be interpolated, that is, if the color to be interpolated is G, the target pixel position is R and the color to be interpolated If E is E, and the target pixel position is B, the third intensity estimating unit 322, the second intensity estimating unit 321, the fourth intensity estimating unit 323, and the switch 324 of the third processing block 304 are used. The intensity estimated value calculated by the above is output from the switch 306 (the input d in the figure is output).

  Here, the operation of the second processing block 303 will be described. The second intensity estimation unit 321 calculates an intensity estimation value of the color to be interpolated using the color intensity of the target pixel position and the pixel information of the local region. The third intensity estimation unit 322 calculates an intensity estimation value of a color opposite to the color to be interpolated out of G or E using the color intensity of the target pixel position and the pixel information of the local region. Then, based on the intensity estimation value by the third intensity estimation unit 322, the fourth intensity estimation unit 323 calculates the intensity estimation value of the color to be interpolated. Two intensity estimation values are calculated in this way, and one of them is selected and output by the switch 324 based on the texture direction determined by the texture direction determination unit 305.

  As described above, the output of the second processing block 303 is selected by the switch 306 when the target pixel position of the mosaic image is horizontally shifted from the pixel of the color to be interpolated by one pixel. At this time, if the image has a texture such as a horizontal stripe that has a high correlation in the horizontal direction, it can be expected that the color at the target pixel position has a strong correlation with the color to be interpolated. On the other hand, if the image has a texture that is highly correlated in the vertical direction, such as vertical stripes, it cannot be expected that the color at the target pixel position has a strong correlation with the color to be interpolated. Therefore, when the texture is close to horizontal stripes, the second intensity estimation unit 321 estimates the intensity of the color to be directly interpolated from the color at the target pixel position. Conversely, when the texture is close to vertical stripes, The intensity estimation unit 322 estimates the intensity of a color (E for G or G for E) different from the color to be interpolated from the color of the target pixel position, and then the interpolation calculated by the fourth intensity estimation unit 323 is performed. A power color intensity estimate is selected.

  Next, the operation of the third processing block 304 will be described. The third intensity estimating unit 322 calculates an estimated intensity value of the color to be interpolated using the color intensity of the target pixel position and the pixel information of the local region. The second intensity estimation unit 321 calculates an intensity estimation value of a color of G or E opposite to the color to be interpolated using the color intensity of the target pixel position and the pixel information of the local region. Then, based on the intensity estimated value by the second intensity estimating unit 321, the fourth intensity estimating unit 323 calculates the intensity estimated value of the color to be interpolated. In this way, two intensity estimation values are calculated, and one of them is selected and output by the switch 324 based on the texture direction determined by the texture direction determination unit 305.

  As described above, the output of the third processing block 304 is selected by the switch 306 when the target pixel position of the mosaic image is shifted by one pixel vertically from the pixel of the color to be interpolated. At this time, if the image has a texture having a high correlation in the vertical direction such as a vertical stripe, it can be expected that the color at the target pixel position has a strong correlation with the color to be interpolated. On the other hand, if the image has a texture that is highly correlated in the horizontal direction, such as horizontal stripes, it cannot be expected that the color at the target pixel position has a strong correlation with the color to be interpolated. Therefore, when the texture is close to a vertical stripe, the third intensity estimation unit 322 estimates the intensity of the color to be directly interpolated from the color at the target pixel position. Conversely, when the texture is close to a horizontal stripe, The second intensity estimating unit 321 estimates the intensity of a color (E for G or G for E) different from the color to be interpolated from the color of the target pixel position, and then calculated by the fourth intensity estimating unit 323. A color intensity estimation value to be interpolated is selected.

  By doing in this way, the reproducibility with respect to an image signal with a large high frequency component like a fine stripe pattern can be improved.

  The common part of the operations of the G intensity estimation unit 291 and the E intensity estimation unit 292 has been described above. The difference between the G intensity estimation unit 291 and the E intensity estimation unit 292 is that the color to be interpolated is G or E, Thus, the only difference is that the horizontally adjacent color and the vertically adjacent color are R or B, which causes the switch 306 to behave differently. The output of each switch 306 of the G intensity estimation unit 291 and the E intensity estimation unit 292 will be described with reference to FIG.

  As shown in FIG. 9, when the target pixel is R, the switch 306 of the G intensity estimating unit 291 outputs the intensity estimated value calculated in the third processing block 304 from the switch 306 (input d in the figure). When the target pixel is G, the output of the target pixel selection unit 301 is output from the switch 306 (output a in the figure), and when the target pixel is B, the second processing block 303 calculates The output intensity estimation value is output from the switch 306 (output c in the figure is output). When the target pixel is E, the intensity estimation value calculated in the first processing block 302 is output from the switch 306 ( The input b in the figure is output). Further, as shown in FIG. 9, when the target pixel is R, the switch 306 of the E intensity estimating unit 292 outputs the intensity estimated value calculated in the second processing block 303 from the switch 306 (in the drawing). Output c), and when the target pixel is G, the estimated intensity calculated in the first processing block 302 is output from the switch 306 (output b in the figure is output), and the target pixel is B At this time, the intensity estimation value calculated in the third processing block 304 is output from the switch 306 (output d in the figure), and when the target pixel is E, the output of the target pixel selection unit 301 is output from the switch 306. Output (output the input a in the figure).

  FIG. 10 is a block diagram illustrating a configuration of the first intensity estimating unit 311, the second intensity estimating unit 321, the third intensity estimating unit 322, or the fourth intensity estimating unit 323 of FIG. 8. When the first intensity estimation unit 311, the second intensity estimation unit 321, the third intensity estimation unit 322, and the fourth intensity estimation unit 323 in FIG. 8 perform coarse interpolation in the coarse interpolation processing unit 341. The second color acquisition method basically has the same configuration except that the second color selection method is different.

  The coarse interpolation processing unit 341 uses a predetermined pixel position and the surrounding eight pixels so that a set of intensities of the first color Cr and the second color Ce can be created at a plurality of pixel positions in the local region. The pixel intensity estimation values (hereinafter referred to as rough estimation values) of the first color and the second color are calculated by a simple calculation using an interpolation method described later. In the present invention, in order to estimate the color distribution shape, it is necessary to calculate a second-order statistic or a value corresponding thereto, and for this reason, the intensity of each color at the same pixel position can be taken as a set. Need to be. In order to generate this intensity set, the coarse interpolation processing unit 341 has a plurality of pixels in the local region (here, the pixel of interest is centered on the local region of n × n pixels (n−2)). A rough estimated value of the first color and the second color in (× (n−2) pixels) is calculated.

  The statistic calculation unit 342 calculates a statistic between two colors by using a pair of intensities of the first color and the second color in each pixel in the local area calculated by the coarse interpolation processing unit 341. . Specifically, the statistic calculation unit 342 calculates the average value of the first color, the average value of the second color, the dispersion value of the first color, and the covariance of the first color and the second color. Calculate and output the value.

  The regression calculation processing unit 343 performs linear regression calculation of the color distribution shape based on the pixel intensity of the pixel of interest and the statistic calculated by the statistic calculator 342, and calculates an intensity estimation value at the pixel position of interest.

  As described above, the first intensity estimator 311, the second intensity estimator 321, the third intensity estimator 322, and the fourth intensity estimator 323 in FIG. Based on the relationship, the intensity estimate value of the second color is calculated from the intensity estimate value of the first color. The difference between the four types of intensity estimation units is the difference in how to select these two colors. The coarse interpolation processing unit 341 selects two colors as the first color Cr and the second color Ce. The processing of the statistic calculation unit 342 and the regression calculation processing unit 343 is common. The processing of the coarse interpolation processing unit 341 of each of the first intensity estimating unit 311, the second intensity estimating unit 321, the third intensity estimating unit 322, and the fourth intensity estimating unit 323 is described with reference to FIGS. Will be described.

  The coarse interpolation processing unit 341 of the first intensity estimating unit 311 selects a color at the target pixel position and a color adjacent thereto obliquely as two colors. FIG. 11 shows a local region when the pixel of interest is E. When the target pixel is E, since G is present obliquely, the coarse interpolation processing unit 341 of the first intensity estimating unit 311 obtains a pair of G and E interpolation values.

  The coarse interpolation processing unit 341 of the second intensity estimation unit 321 selects a color at the target pixel position and two colors adjacent to the left and right as two colors. FIG. 12 shows a local region when the target pixel is B. When the target pixel is B, since G is present on the left and right, the coarse interpolation processing unit 341 of the second intensity estimating unit 321 obtains a pair of G and B interpolation values.

  The coarse interpolation processing unit 341 of the third intensity estimation unit 322 selects a color at the target pixel position and two adjacent colors above and below it as two colors. FIG. 13 shows a local region when the target pixel is R. When the target pixel is R, since G is present above and below, the coarse interpolation processing unit 341 of the third intensity estimation unit 322 obtains a pair of G and R interpolation values.

  The coarse interpolation processing unit 341 of the fourth intensity estimation unit 323 selects two colors, the color adjacent to the left and right of the target pixel position and the color adjacent to the top and bottom as the two colors. FIG. 14 shows a local region when the target pixel is R. When the pixel of interest is R, E and G are present on the left and right and top and bottom, so the coarse interpolation processing unit 341 of the fourth intensity estimation unit 323 obtains a pair of G and E interpolation values.

  FIG. 15 is a block diagram illustrating a configuration of the statistic calculation unit 342 of FIG.

  The statistic calculation unit 342 includes an average value calculation unit 361, an average value calculation unit 363, a variance calculation unit 362, and a covariance calculation unit 364. The average value calculation unit 361 calculates the average value of the values of the first color among the rough two-color interpolation values in the local region calculated by the rough interpolation processing unit 341. Similarly, the average value calculation unit 363 calculates the second average value of the rough two-color interpolation values in the local region calculated by the rough interpolation processing unit 284. The variance calculation unit 362 calculates the variance value of the first color from the average value of the first color calculated by the average value calculation unit 361 and the rough interpolation value of the first color in the local region. The covariance calculation unit 364 calculates the average value of the first color calculated by the average value calculation unit 361, the rough interpolation value of the first color, and the average value of the second color calculated by the average value calculation unit 363. And a rough interpolated value of the second color, a covariance value of the first color and the second color is calculated.

  The average value calculation unit 361 and the average value calculation unit 363 calculate the average value Mc using, for example, the following equation (1).

・・・（１）

... (1)

In Expression (1), C _roughi is i (i = 1 to N) th data of the rough intensity interpolation value of the supplied color C (R, G, B, or E). The color C is G in the average value calculation unit 361 and R in the average value calculation unit 363. Wi is a weight value (weighting coefficient) for the i-th data.

  The weight value wi is a value set in advance using the distance from the position of the i-th data to the target pixel position as an index. It is preferable that the weight value wi is set so that the weight value becomes larger as the pixel position is closer.

The variance calculating unit 362 calculates the variance value V _C1C1 of the first color intensity using, for example, the calculation formula shown in the following formula (2).

・・・（２）

... (2)

In this equation, C _1roughi is the i-th data of the rough intensity interpolation value of the color C1, M _C1 is the average value of the color C1, and wi is a weight value for the i-th data. The weight value wi may be the same as that used by the average value calculation unit 361 and the average value calculation unit 363.

The covariance calculation unit 364 calculates the covariance value V _C1C2 of the first signal (here, G) and the second signal (here, R), for example, by the calculation formula shown in the following formula (3). .

In this equation, C _2roughi is the i-th data of the rough intensity interpolation value of the color C2 (here, R), and M _C2 is the average value of the color C2.

・・・（３）

... (3)

  When the calculation of the variance value described using the equation (2) and the calculation of the covariance described using the equation (3) are executed, accurate statistics as defined can be obtained. However, when the calculations of Formula (2) and Formula (3) are executed, a large amount of integration appears, so that the calculation time becomes long, and the circuit scale when this calculation is realized by hardware increases. Therefore, the variance and covariance may be calculated by simpler calculation.

  For example, the following equation (4) may be used for the covariance calculation.

・・・（４）

... (4)

  Here, the function sgn (a, b) is for checking the coincidence of the signs of the two variables a and b. Since the output of sgn (a, b) is one of {1, 0, -1}, the operation of sgn {a, b} does not actually require integration. The calculation of equation (4) replaces the i-th deviation of G and R in the calculation of equation (3) with the sum of absolute deviations of G and R. Even if the integration is replaced with summation using equation (4), it is possible to calculate an approximate value having sufficient accuracy for estimating the color distribution shape of the local region targeted by the present invention.

  Further, for example, the following equation (5) may be used for the covariance calculation.

・・・（５）

... (5)

  Here, the calculation of Expression (5) is obtained by replacing the integration of deviations of each i-th G and R in the calculation of Expression (3) with the process of selecting the minimum value of G and R. Note that the calculation of equation (5) has better approximation accuracy to the covariance calculation in equation (3) than the calculation of equation (4).

The expression (4) or the expression (5) is used to explain that the approximation operation for the covariance operation shown in the expression (3) is possible, but the dispersion operation described using the expression (2) is Since it is equivalent to the covariance when the two inputs are equal, the variance calculation described using equation (2) can be similarly approximated using equation (4) or equation (5). Needless to say. Specifically, when the variance calculation is approximated, (C _1roughi −M _C1 ) ² is approximated to (C _1roughi −M _C1 ) _{regardless of which} of the equations (4) and (5) is used.

  Next, FIG. 16 is a block diagram illustrating a configuration of the regression calculation processing unit 343. The regression calculation processing unit 286 estimates a predetermined pixel intensity by obtaining a regression line represented by the following equation (6).

・・・（６）

... (6)

  The regression calculation processing unit 343 includes an inclination calculation unit 381 and a pixel intensity estimation unit 382.

The inclination calculation unit 381 calculates the inclination Km of the GR color distribution based on the average value MR of _R and the average value MG of _G. Specifically, the regression calculation processing unit 286 calculates the ratio of the two average values as shown in the equation (7). Here, C1 is G and C2 is R.

・・・（７）

... (7)

M _threshold is a constant to avoid divergence by dividing by zero,
A sufficiently small positive value is preset.

Alternatively, the slope calculation unit 381 calculates the slope Ks of the C1-C2 color distribution based on the C1 and C2 covariance values V _{C1, C2} and the C1 variance values V _{C1, C1} . Here, C1 is G and C2 is R. Specifically, the slope calculation unit 381 calculates the ratio between the variance and the covariance value as shown in Expression (8).

・・・（８）

... (8)

V _threshold is a constant for avoiding the value to diverge by dividing by zero, and a sufficiently small positive value is set in advance. Using V _threshold in Equation (8) to clip the variance of C1 in the denominator, division by zero can be avoided, but clipping with V _threshold further reduces noise in the flat part of the image. Can be used to reduce.

  That is, the variance of C1 as the denominator is a value reflecting the variation in luminance of the local region, and a small value is synonymous with the fact that the local region is flat. Since the noise of a solid-state image sensor stands out better as the image becomes flat, if noise reduction processing is performed only on the flat part of the image, the noticeable noise is effectively reduced without degrading the overall image quality. This can be reduced and is preferable. Also, by clipping the denominator of the gradient Ks of the color distribution so that the denominator does not become any smaller, the absolute value of Ks can be suppressed to be smaller than the original value. By suppressing the absolute value of the slope Ks to be small, the rate of change of C2 with respect to C1 is reduced, and as a result, the effect that the amplitude of C2 in the local region can be suppressed occurs.

  As described above, the clipping process for the denominator of the slope Ks has the effect of reducing noise, as in the case where both the process of determining whether the image is flat and the process of suppressing the output amplitude are executed. Thus, by using the color estimation method of the present invention, it is possible to suppress noticeable noise in a flat portion of an image without adding a separate noise reduction process.

  That is, the inclination calculation unit 381 supplies either Ks or Km to the pixel intensity estimation unit 382 as the inclination K of the regression line. Here, Km is easier to calculate than Ks, but if a strong correlation between the first color and the second color cannot be expected so much, a more accurate estimated value can be obtained by using Ks. Conversely, when a strong correlation between the first color and the second color can be expected, a sufficient estimated value can be obtained even if Km is used. That is, as described with reference to FIG. 4, when there is a strong correlation between the spectral characteristics of G and E, a strong correlation may be expected for the obtained G and E image signals. In the first intensity estimating unit 311 and the fourth intensity estimating unit 323 using the first and second colors as the first color and the second color, it is possible to perform the intensity estimation by simpler calculation using Km as the slope K. . Conversely, in the second intensity estimation unit 321 and the third intensity estimation unit 322 that always use either R or B, it is better to use Ks for the slope K.

  FIG. 17 is a block diagram illustrating a configuration of the R intensity estimating unit 283-1 to the E intensity estimating unit 283-4 of FIG. In FIG. 17, each of the reliability calculation unit 471, the rough interpolation processing unit 472, and the synthesis processing unit 473 is illustrated as one RGBE intensity estimating unit 283, but the R intensity estimating unit 283-1 is illustrated. It goes without saying that the reliability calculation unit 471, the rough interpolation processing unit 472, and the synthesis processing unit 473 may be provided in each of the E intensity estimation units 283-4.

  The reliability calculation unit 471 includes n × n (for example, 9 × 9) pixels in the rectangular local region extracted by the local region extraction unit 281 and the G + E intensity of the target pixel position calculated by the G + E intensity estimation processing unit 282. Are used to calculate the reliability of color distribution shape estimation at the target pixel position. In the color distribution shape estimation in the demosaic process to which the present invention is applied, it is assumed that there are two different color regions in the extracted local region. Since this assumption is often applied to a normal object contour region, sufficiently accurate contour color reproduction is possible with this assumption. However, the assumption is not true in the image, and there may be cases where three different colors exist in the local region. Typically, this is a case of a thin striped texture. In such a texture, many colors are likely to be included in the local region. Therefore, the reliability calculation unit 471 detects such a thin striped texture, and the reliability is a value that predicts the accuracy of estimation of the color distribution shape in this local region based on the intensity of the texture. Output the value. How to calculate the reliability will be described later.

  The coarse interpolation processing unit 472 estimates the R, G, B, and E pixel intensities by a simple calculation so that a set of intensities of R, G, B, and E colors can be created at a plurality of pixel positions in the local region. (Hereinafter, referred to as a rough estimated value) is calculated using an interpolation method described later. In the present invention, in order to estimate the color distribution shape, it is necessary to calculate a second-order statistic or a value corresponding thereto, and for this reason, the intensity of each color at the same pixel position can be taken as a set. Need to be. In order to generate this set of intensities, the coarse interpolation processing unit 742 has a plurality of pixels in the local region (here, the pixel of interest is centered on the n × n pixel local region (n−2)). A rough estimated value of R, G, B, E in (× (n−2) pixels) is calculated.

  The synthesis processing unit 473 combines the rough estimation values of G and E among the rough estimation values of R, G, B, and E calculated by the rough interpolation processing unit 472, and generates a rough estimation by combining the GEs. Calculate the value.

  The statistic calculators 342-1 to 342-4 have basically the same configuration as the statistic calculators 342 described with reference to FIG. The statistic calculation unit 342-1 calculates the statistic based on the rough estimated value synthesized by the GE supplied from the synthesis processing unit 473 and the rough estimated value of R calculated by the rough interpolation processing unit 472. It calculates and supplies to the regression calculation process part 474-1. The statistic calculation unit 342-2 calculates a statistic based on the rough estimated value obtained by combining the GE supplied from the combining processing unit 473 and the rough estimated value of G calculated by the rough interpolation processing unit 472. It calculates and supplies to the regression calculation process part 474-2. The statistic calculation unit 342-3 calculates the statistic based on the rough estimated value obtained by combining the GE supplied from the combining processing unit 473 and the rough estimated value of B calculated by the rough interpolation processing unit 472. It calculates and supplies to the regression calculation process part 474-3. The statistic calculation unit 342-4 calculates the statistic based on the rough estimated value obtained by combining the GE supplied from the combining processing unit 473 and the rough estimated value of E calculated by the rough interpolation processing unit 472. It calculates and supplies to the regression calculation process part 474-4.

  The regression calculation processing units 474-1 to 474-4 include the G + E intensity of the target pixel position calculated by the G + E intensity estimation processing unit 282, the reliability information calculated by the reliability calculation unit 471, and the statistic calculation unit 342-1. Thru | or the statistic calculation part 342-4 based on the statistic which the corresponding one calculated, performs a regression calculation, and the regression calculation process part 474-1 uses the estimated value of R as a regression calculation process part 474-. 2 calculates an estimated value of G, a regression calculation processing unit 474-3 calculates an estimated value of B, and a regression calculation processing unit 474-4 calculates and outputs an estimated value of E.

  FIG. 18 is a block diagram illustrating a configuration of the reliability calculation unit 471. Specifically, the reliability calculation unit 471 calculates a low-frequency filter for each direction with respect to the (G + E) intensity around the target pixel position in order to detect a texture with a thin striped pattern. G + E) By performing a difference calculation process with intensity, high-frequency extraction is performed, and when there is a luminance change such as “bright-dark-bright” or “dark-bright-dark” around the target pixel position, this is indicated as a direction. The reliability is calculated by separately detecting and integrating the detection results for each direction. The reliability calculation unit 471 includes high frequency extraction units 481-1 to 481-6, an addition processing unit 482, and a clip processing unit 483. The six high-frequency extraction units 481-1 to 481-6 perform high-frequency extraction in six different directions around the target pixel and output the intensity.

  A specific example of direction-specific high-frequency extraction by the frequency extraction units 481-1 to 481-6 of the reliability calculation unit 471 will be described with reference to FIG. FIG. 19 shows a local area of 9 × 9 pixels extracted by the local area extraction unit 281. For purposes of illustration, numbers 1-9 are assigned to pixel rows and columns in the figure. 19A shows the case where the pixel of interest (the pixel at the position of the fifth row and the fifth column in the figure) is E, and FIG. 19B shows the case where the pixel of interest is R. Since only the G and E pixels are used for the calculation of the reliability, when the target pixel is G, the same processing as that for E is executed, and when the target pixel is B, it is R. The same processing as in the case is executed.

  First, 12 positions from positions close to the target pixel (positions indicated by circles in the figure) that are shifted from the pixel by exactly ½ in the horizontal and vertical directions (that is, surrounded by four pixels). ) (G + E) intensity is calculated. Specifically, since the G pixel and the E pixel are diagonally adjacent to each other at the 12 positions shown in FIG. 19, the average value of the intensity of the two pixels is used to determine the 12 positions. Each (G + E) intensity is used. Here, when the target pixel is G or E, and when the target pixel is R or B, the directions of adjacent G are different, but in either case, G and E exist at each position. The direction to be determined can be determined based on whether the target pixel is G or E, or R or B.

  Thus, after the (G + E) intensity is calculated at 12 locations, a low frequency filter in 6 directions is calculated using them. The direction of the filter will be described with reference to FIG. In FIG. 20, only 5 × 5 pixels in the vicinity of the target pixel are shown in the local region.

  The 12 (G + E) intensity calculation positions are arranged symmetrically with respect to the pixel position of interest (the third row and the third column in the drawing in FIG. 20), and the (G + E) intensities are in mutually symmetrical positions. In the figure, a pair of (G + E) intensity indicated by A, a group of (G + E) intensity indicated by D, a group of (G + E) intensity indicated by VA, and a group of (G + E) intensity indicated by VD , (G + E) intensity set indicated by HA and (G + E) intensity set indicated by HD. In these six (G + E) intensity pairs, if a filter that calculates an average value is used, a low-frequency filter in six directions is calculated. Further, if the difference between the output of the low-frequency filter and the (G + E) intensity at the target pixel position is taken, high-frequency extraction for each direction can be performed.

  Then, the addition processing unit 482 outputs a value obtained by adding the six high-frequency extraction results with an appropriate balance. The clip processing unit 483 performs clipping processing on the addition processing result of the addition processing unit 482, and outputs the added value of the clipped high frequency extraction as a reliability value.

  The calculated reliability value increases as the strength of the thin striped texture increases, so the smaller the reliability value is, the more likely it is that the estimation of the color distribution shape by statistics is more certain. It will be.

  Next, the coarse interpolation processing unit 284 uses a method that is not a complicated calculation method so that a set of intensities of R, G, B, and E colors can be created at a plurality of positions in the local region. An estimated value (rough estimated value) is calculated by interpolation. The coarse interpolation process by the coarse interpolation processing unit 284 will be described with reference to FIG.

  FIG. 21 shows the pixels in the local area of 9 × 9 pixels extracted by the local area extraction unit 281. The rough interpolation processing unit 284 uses the rough interpolation to perform R, R on the 7 × 7 pixel positions in the second to eighth rows and the second to eighth columns in the 9 × 9 pixel region. Make the G, B, and E intensities uniform. For example, when attention is paid to E, as shown in the pixel in the second row and the second column in the figure, the interpolation value of G can be calculated from the pixels in the surrounding oblique four directions, and is shown in the pixel in the second row and the fourth column. Thus, the interpolation value of R can be calculated from the upper and lower two pixels, and the interpolation value of B can be calculated from the two left and right pixels as shown by the pixel in the second row and the sixth column. If the color arrangement of the color filter is a Bayer arrangement, interpolation from four diagonal pixels, two upper and lower pixels, and two left and right pixels can be performed at any pixel position. Interpolated value of any of G, B, E. That is, if the color of the signal at each pixel position (color before correction) is included, each pixel always has one R interpolation value, one G interpolation value, one B interpolation value, and E interpolation. One value is obtained. In this way, the rough interpolation processing unit 284 can calculate rough estimated values of R, G, B, and E at each pixel position.

  Here, the estimated values of R, G, B, and E obtained by the coarse interpolation processing unit 284 only need to obtain a set of estimated values of R, G, B, and E at the same position. It does not have to be on the pixel position of the sensor.

  Next, FIG. 22 is a block diagram illustrating a configuration of the regression calculation processing unit 474. In FIG. 22, the regression calculation processing unit 474 (that is, the regression calculation processing unit 474-1) relating to R is shown, but the operation relating to G, B, and E only needs to replace R with G, B, or E. Basically, the same processing is executed.

  The regression calculation processing unit 474 includes an inclination calculation unit 501, an inclination calculation unit 502, an inclination synthesis unit 503, and a pixel intensity estimation unit 504.

The inclination calculation unit 501 calculates an inclination Km of the GR color distribution based on the average value MR of _R and the average value MG of _G. Specifically, the regression calculation processing unit 474 calculates the ratio of the two average values using the above-described equation (7). Here, C1 is G and C2 is R.

M _threshold is a constant for avoiding the value to diverge by dividing by zero, and a sufficiently small positive value is set in advance.

The slope calculation unit 502 calculates the slope Ks of the color distribution of C1-C2 based on the covariance values V _{C1, C2 of C1 and C2} and the variance values V _{C1, C1} of C1. Here, C1 is G and C2 is R. Specifically, the slope calculation unit 502 calculates the ratio of variance and covariance values using the above-described equation (8).

V _threshold is a constant for avoiding the value to diverge by dividing by zero, and a sufficiently small positive value is set in advance. Using V _threshold in Equation (8) to clip the variance of C1 in the denominator, division by zero can be avoided, but clipping with V _threshold further reduces noise in the flat part of the image. Can be used to reduce.

  That is, the variance of C1 as the denominator is a value reflecting the variation in luminance of the local region, and a small value is synonymous with the fact that the local region is flat. Since the noise of a solid-state image sensor stands out better as the image becomes flat, if noise reduction processing is performed only on the flat part of the image, the noticeable noise is effectively reduced without degrading the overall image quality. This can be reduced and is preferable. Also, by clipping the denominator of the gradient Ks of the color distribution so that the denominator does not become any smaller, the absolute value of Ks can be suppressed to be smaller than the original value. By suppressing the absolute value of the slope Ks to be small, the rate of change of C2 with respect to C1 is reduced, and as a result, the effect that the amplitude of C2 in the local region can be suppressed occurs.

  As described above, the clipping process for the denominator of the slope Ks has the effect of reducing noise, as in the case where both the process of determining whether the image is flat and the process of suppressing the output amplitude are executed. Thus, by using the color estimation method of the present invention, it is possible to suppress noticeable noise in a flat portion of an image without adding a separate noise reduction process.

  The gradient combining unit 503 combines the two estimated gradient values Km and Ks based on the reliability h, and calculates the estimated gradient value K. As described above, the inclination estimation value Ks based on the variance and covariance cannot always be correctly estimated in a region having a thin striped texture. Therefore, the inclination synthesis unit 503 uses the reliability h that reflects the strength of the thin striped texture to synthesize the inclination estimation value Km based on the average value by using, for example, Equation (9) to estimate the inclination. The value K is calculated.

・・・（９）

... (9)

Then, the pixel intensity estimation unit 354 applies the obtained gradient estimation value K, the two average values M _G and M _R , and the G intensity of the target pixel position to the linear regression equation of the following equation (10). Thus, an estimated value of the R intensity at the target pixel position is calculated.

・・・（１０）

... (10)

Here, C _1center and C _2center are respectively the intensity of the color C1 (that is, G) corresponding to the first signal at the target pixel position and the color C2 (corresponding to the second signal at the target pixel position). That is, the estimated intensity value of R).

  Further, the pixel intensity estimation unit 504 may calculate the intensity estimation value using a regression equation different from the equation (10). For example, as shown in the equation (11), the linear regression calculation may be performed by multiplying the slope estimation value K by an appropriate constant u.

・・・（１１）

(11)

  In the equation (11), the first term can be considered as a high frequency component of the color C2 (ie, R) intensity corresponding to the second signal, and the second term can be considered as a low frequency component. In the equation (11), by slightly enhancing the high frequency component with an appropriate constant u, it is possible to obtain the same effect as when appropriate high-frequency correction for R is executed. By doing so, it is possible to obtain an image in which the high frequency is corrected without adding a separate high frequency correction process.

  In the above description, the case where the regression line slope K is calculated in the regression calculation processing unit 474 based on the average value, the variance value, and the covariance value calculated in the statistic calculation unit 342 has been described. The slope of the regression line may be calculated using a method other than this.

  For example, a definition formula for calculating the slope of the regression line using the standard deviation and the correlation coefficient is represented by the following formula (12).

・・・（１２）

(12)

  The standard deviations Sx and Sy are statistics indicating how much the data values are distributed around the average, and are values close to dx and dy representing the variation in the xy direction of two variables. Conceivable. You may make it obtain | require a regression line using this Formula (12).

  In particular, when data is distributed on a straight line having a positive slope and Rxy is 1, Sx and Sy are equivalent to dx and dy. That is, instead of using a complicated standard deviation of calculation, if there is another statistic that represents the data distribution width and is easier to calculate, Sx and Sy may be replaced with other statistic that represents the data distribution width. The slope of the regression line can be expected to show close behavior.

  Therefore, as another statistic representing the data distribution width, an average deviation used to represent the data distribution width along with the standard deviation is used instead. The definition of the average deviation Ax of x is shown by the following formula (13).

・・・（１３）

... (13)

  Similarly, the average deviation Ay of y is also expressed by the following equation (14).

・・・（１４）

(14)

  When Rxy is rewritten using the average deviations Ax and Ay, the following equation (15) is obtained.

・・・（１５）

... (15)

  The average deviation can be calculated with a small amount of calculation compared to the fact that a square root or multiplication is necessary for the calculation using the standard deviation. Further, approximation of Rxy can be calculated at high speed by performing approximate calculation of multiplication used to calculate Vxy and multiplication of Ax and Ay.

  As described above, instead of the process of calculating the slope K of the regression line in the regression calculation processing unit 474 based on the average value, the variance value, and the covariance value calculated in the statistic calculation unit 342, the standard deviation And using the correlation function to calculate the slope of the regression line in the two-dimensional distribution of the two systems of data, or after approximating the deviation and correlation, respectively, the slope of the regression line in the two-dimensional distribution of the two systems of data May be calculated.

  Next, processing of the DSP block 216 in FIG. 5 will be described with reference to the flowchart in FIG.

  In step S1, the image RAM 241 is composed of periodic pattern intensity signals in an array (for example, the four-color array described with reference to FIG. 3) defined by the color filter used in the CCD image sensor 213. Obtain a mosaic image and save it temporarily.

  In step S2, the white balance adjustment unit 251 of the signal processing processor 242 applies an appropriate color according to the color of each pixel intensity so that the color balance of the achromatic subject region is achromatic with respect to the mosaic image. A white balance adjustment process, which is a process for applying a coefficient, is performed.

  In step S3, the gamma correction unit 252 performs the first operation so that the brightness and color saturation of the image displayed on the display unit 220 are correctly displayed for each pixel intensity of the mosaic image with the white balance. Perform gamma correction.

  In step S4, the demosaic processing unit 253 executes a demosaic process described later with reference to FIG.

In step S5, the gamma reverse correction unit 254 applies the reverse of the gamma characteristics applied to the four-channel images of R ^γ , G ^γ , B ^γ , and E ^γ that are outputs from the demosaic processing unit 253 to perform the gamma reverse correction processing. The gamma characteristic is converted to a linear characteristic.

  In step S <b> 6, the matrix processing unit 255 applies a 3 × 4 linear matrix with coefficients set in advance to the R, G, B, and E four-channel images supplied from the gamma inverse correction unit 254. The intensity values are converted into primary color intensity values [Rm, Gm, Bm]. Since this matrix processing needs to be applied to the image intensity of linear characteristics, it is necessary to perform gamma reverse correction by the gamma reverse correction unit 254 in the previous stage of the processing. The coefficient of the linear matrix is an important design item for achieving optimum color reproduction, and the matrix processing is applied after the demosaic processing. Therefore, the linear matrix coefficient used by the matrix processing unit 255 is an appropriately designed value. It may be. The output of the matrix processing unit 255 is three images corresponding to the three colors Rm, Gm, and Bm that have been color corrected.

  In step S7, the gamma correction unit 256 performs second gamma correction on the color-corrected three-channel image.

  In step S8, the YC conversion unit 257 performs YC conversion on the R, G, and B three-channel images supplied from the gamma correction unit 256 by performing YC matrix processing and band limitation on chroma components in a row. An image and a C image are generated, and in step S9, the generated Y image and C image are output, and the process ends.

  Through such processing, the DSP block 216 performs various processing on the supplied mosaic image signal to generate a Y image and a C image, and displays the image data on the display unit 220 under the control of the CPU 223. Is displayed in the D / A converter 218 and is stored in the memory 222, and is supplied to the codec processing unit 221.

  Next, the demosaic process 1 executed in step S4 of FIG. 23 will be described with reference to the flowchart of FIG.

  In step S21, the local region extraction unit 281 extracts one of the unprocessed pixels as a target pixel, and in step S22 extracts a predetermined number (n × n) of pixels around the target pixel position as a local region. To the G + E intensity estimation processing unit 282 and the RGBE intensity estimation unit 283.

  In step S23, a G + E intensity estimation process, which will be described later, is executed using the flowchart of FIG.

  In step S24, color intensity estimation processing 1 described later using the flowchart of FIG. 50 is executed in parallel in each of the RGBE intensity estimation units 283 of FIG.

  In step S25, the local region extraction unit 281 determines whether or not processing has been completed for all pixels. If it is determined in step S25 that the process has not been completed for all pixels, the process returns to step S21, and the subsequent processes are repeated. If it is determined in step S27 that the process has been completed for all pixels, the process proceeds to step S5 in FIG.

  In other words, each unit constituting the demosaic processing unit 253 performs respective processing at the target pixel position when a certain target pixel position is determined, and the processing of step S21 to step S26 is completed for all pixels. If so, the process is terminated.

  By such processing, the mosaic image obtained based on the color filter array of the CCD image sensor 213 is demosaiced (color interpolation or synchronization), and each color constituting the color filter is interpolated in each pixel. Image data can be obtained.

  Next, the G + E intensity estimation process executed in step S23 of FIG. 24 will be described with reference to the flowchart of FIG.

  In step S41, the G intensity estimating unit 291 executes a G intensity estimated value calculation process for the target pixel position, which will be described later with reference to FIG.

  In step S42, the E intensity estimation unit 292 executes an E intensity estimated value calculation process for the target pixel position, which will be described later with reference to FIG.

  In step S43, the addition processing unit 293 calculates the G + E intensity estimated value by adding the G intensity estimated value calculated in step S41 and the E intensity estimated value calculated in step S42. In step S44, The G + E intensity estimated value is output, and the process returns to step S24 in FIG.

  By such processing, a G + E intensity estimation value, which is a new intensity estimation value composed of G and E arranged in a checkered pattern, can be calculated. Therefore, it is used for an RGBE color intensity estimation calculation described later. Is possible.

  In addition, the process of step S41 and the process of step S42 may be performed in parallel, or the order of the processes may be reversed.

  Next, referring to the flowchart of FIG. 26, the G intensity estimated value calculation process for the target pixel position and the E intensity estimated value calculation process for the target pixel position executed in steps S41 and S42 of the G + E intensity estimation process of FIG. Will be described.

  That is, the G intensity estimated value calculation process for the target pixel position performed by the G intensity estimation unit 291 and the E intensity estimated value calculation process for the target pixel position performed by the E intensity estimation unit 292 are G to be interpolated. Basically, the same processing is executed only by whether or not there is E.

  Further, in the description of the block diagram of the G intensity estimating unit 291 or the E intensity estimating unit 292 in FIG. 8, estimated values are calculated in advance for all four types of operations based on the color of the target pixel position, Finally, the switch 306 selects which one to output. However, in the case where the same processing is realized by software, it is more efficient to first determine the color of the pixel of interest and select an operation to be executed according to the color.

  In step S61, the G intensity estimator 291 or the E intensity estimator 292 uses the color of the pixel of interest to be interpolated (G in the process of the G intensity estimator 291 and E in the process of the E intensity estimator 292). Judge whether there is.

  If it is determined in step S61 that the color of the target pixel is a color to be interpolated, in step S62, the G intensity estimation unit 291 or the E intensity estimation unit 292 outputs the pixel intensity at the target pixel position and performs processing. Advances to step S42 or step S43 of FIG.

  When it is determined in step S61 that the color of the target pixel is not the color to be interpolated, in step S63, the G intensity estimation unit 291 or the E intensity estimation unit 292 determines that the color of the target pixel is not the color to be interpolated. It is determined whether or not any one of E (E in the process of the G intensity estimating unit 291 and G in the process of the E intensity estimating unit 292).

  If it is determined in step S63 that the color of the target pixel is either G or E that is not a color to be interpolated, in step S64, the G intensity estimation unit 291 or the E intensity estimation unit 292 performs the first process. The intensity estimation value to be interpolated is calculated and output by the block 302, and the process proceeds to step S42 or step S43 in FIG.

  If it is determined in step S63 that the color of the target pixel is not G or E that is not the color to be interpolated, in step S65, the G intensity estimation unit 291 or the E intensity estimation unit 292 determines the color of the target pixel. Are arranged on the left and right sides of the color to be interpolated (for example, when the four-color array in FIG. 3 is used as a color filter, in the process of the G intensity estimation unit 291, Is R).

  When it is determined in step S65 that the color of the target pixel is a color arranged on the left and right of the color to be interpolated, in step S66, the G intensity estimating unit 291 or the E intensity estimating unit 292 performs the second processing block. In step 303, the estimated intensity value to be interpolated is calculated and output, and the process proceeds to step S42 or step S43 in FIG.

  When it is determined in step S65 that the color of the target pixel is a color arranged on the left and right of the color to be interpolated, in step S67, the G intensity estimation unit 291 or the E intensity estimation unit 292 performs the third processing block. The estimated intensity value to be interpolated is calculated and output by 304, and the process proceeds to step S42 or step S43 in FIG.

  Through such processing, the G intensity estimation unit 291 or the E intensity estimation unit 292 can estimate the G or E intensity at the target pixel position.

  In the processing in step S64, step S66, or step S67 in FIG. 26, that is, the processing executed by each of the first processing block 302 to the third processing block 304, the first intensity described with reference to FIG. In the estimation unit 311, the second intensity estimation unit 321, the third intensity estimation unit 322, or the fourth intensity estimation unit 323, the coarse interpolation processing 1 by the coarse interpolation processing unit 341 and the two colors by the statistic calculation unit 342 The distribution shape statistic calculation process and the interpolation pixel estimation process 1 by the regression calculation processing unit 343 are executed.

  27 to 34, the rough interpolation processing 1 by the rough interpolation processing unit 341, the statistic calculation processing of the two-color distribution shape by the statistic calculation unit 342, and the interpolation pixel estimation processing 1 by the regression calculation processing unit 343. explain.

  First, the rough interpolation process 1 will be described with reference to the flowchart of FIG.

  Here, it is assumed that the coarse interpolation processing unit 341 of the first intensity estimation unit 311 selects the color at the target pixel position as the first color and the color adjacent to the diagonal as the second color. In addition, the rough interpolation processing unit 341 of the second intensity estimation unit 321 selects the color at the target pixel position as the first color and the colors adjacent to the left and right as the second color. The rough interpolation processing unit 341 of the third intensity estimation unit 322 selects the color at the target pixel position as the first color and selects the adjacent colors above and below it as the second color. The coarse interpolation processing unit 341 of the fourth intensity estimation unit 323 selects two colors, a color adjacent to the left and right of the target pixel position and a color adjacent to the top and bottom, as the first color and the second color.

  In step S71, the coarse interpolation processing unit 341 initializes the value s of the first register indicating the pixel position where the processing is performed among the pixels in the supplied local region, and sets s = 2. Among the supplied local area pixels, the value t of the second register indicating the pixel position where the process is executed is initialized to t = 2.

  In step S73, the coarse interpolation processing unit 341 extracts the pixel value of the first color from the pixel (s, t) and the surrounding eight pixels, and calculates the average value as the interpolation value of the first pixel. .

  In step S74, the coarse interpolation processing unit 341 extracts the pixel value of the second color from the pixel (s, t) and the surrounding eight pixels, and calculates the average value as the interpolation value of the second pixel. .

  In step S75, the rough interpolation processing unit 341 refers to the value t of the second register and determines whether t = n−1.

  If it is determined in step S75 that t = n−1 is not true, the coarse interpolation processing unit 341 sets the value t of the second register to t = t + 1 in step S76, and the process returns to step S73. The subsequent processing is repeated.

  If it is determined in step S75 that t = n−1, in step S77, the coarse interpolation processing unit 341 refers to the value s of the first register and determines whether s = n−1. to decide.

  If it is determined in step S77 that s = n−1 is not satisfied, in step S78, the coarse interpolation processing unit 341 sets the value s of the first register to s = s + 1, and the process returns to step S72. The subsequent processing is repeated.

  If it is determined in step S77 that s = n−1, in step S79, the coarse interpolation processing unit 341 determines the first color of (n−2) × (n−2) pixels near the target pixel and A set of interpolation values for the second color is output, and the process is terminated.

  By such processing, the rough interpolation processing unit 341 can calculate the rough interpolation value of the first color and the rough interpolation value of the second color from the pixels near the target pixel.

  Next, the two-color distribution shape statistic calculation process executed by the statistic calculation unit 342 will be described with reference to the flowchart of FIG.

  In step S <b> 91, the average value calculation unit 361 of the statistic calculation unit 342 acquires a first signal among the coarse interpolation values supplied from the coarse interpolation value processing unit 284.

  In step S92, the average value calculation unit 361 executes an average value calculation process described later using the flowchart of FIG. 29 or FIG.

  In step S <b> 93, the average value calculation unit 363 acquires a second signal among the rough interpolation values supplied from the rough interpolation value processing unit 284.

  In step S94, the average value calculation unit 363 executes an average value calculation process to be described later using the flowchart of FIG. 29 or FIG.

  In step S95, the variance calculation unit 362 executes a variance calculation process, which will be described later using the flowchart of FIG.

  In step S96, the covariance calculation unit 364 executes a covariance approximation calculation process, which will be described later using the flowchart of FIG.

  Next, the average value calculation process 1 executed in step S92 or step S94 in FIG. 28 will be described with reference to the flowchart in FIG.

  In step S111, the average value calculation unit 361 or the average value calculation unit 363 calculates the average value Mx (the subscript x of the average value Mx is the Cr if the average value of the first color Cr is calculated). When the average value of the second color Ce is calculated, it is replaced with Ce respectively).

  In step S112, the average value calculation unit 361 or the average value calculation unit 363 sums the acquired interpolation values (here, (n−2) square interpolation values).

  In step S113, the average value calculation unit 361 or the average value calculation unit 363 divides the total value calculated in step S112 by the number of acquired values (here, the square of (n−2)).

  In step S114, the average value calculation unit 361 or the average value calculation unit 363 outputs the division result calculated by the process of step S113.

  Further, when obtaining the average value, the first and second colors subjected to coarse interpolation are weighted according to the distance from the target pixel, and based on the weighted RGB intensity, An average value may be obtained.

  The average value calculation process 2 executed in step S92 or step S94 in FIG. 28 will be described with reference to the flowchart in FIG.

  In step S121, the average value calculation unit 361 or the average value calculation unit 363 calculates the average value Mx (the subscript x of the average value Mx is the calculation result when the average value of the first color Cr is calculated). When the average value of the second color Ce is calculated, it is replaced with Ce respectively).

  In step S122, the average value calculation unit 361 or the average value calculation unit 363 uses the acquired interpolation values (here, (n−2) square interpolation values) as shown in FIGS. 31 to 34, for example. Such weighting is performed, and the weighted values are summed.

The weight value w _i is set in advance using the distance from the position of the i-th data to the target pixel position as an index, and the first intensity estimation unit 311, the second intensity estimation unit 321, and the third intensity estimation. Different weight values may be set for each of the unit 322 and the fourth intensity estimating unit 323. FIG. 31 to FIG. 34 show examples of weight values used in the first intensity estimating unit 311, the second intensity estimating unit 321, the third intensity estimating unit 322, and the fourth intensity estimating unit 323, respectively. Show. The rectangle in the figure indicates the pixel position in the 9 × 9 local area, and the number in the rectangle indicates the weight value of the data at the corresponding position.

  In step S123, the average value calculation unit 361 or the average value calculation unit 363 calculates the sum of weights.

  In step S124, the average value calculation unit 361 or the average value calculation unit 363 calculates the above-described equation (1). That is, in step S124, the average value calculation unit 361 or the average value calculation unit 363 calculates the total value calculated in step S122 as the number of acquired values (here, the square of (n−2)), Divide by the product of the weighted sum calculated in step S123.

  In step S125, the average value calculation unit 361 or the average value calculation unit 363 outputs the division result calculated by the process of step S124.

  By such processing, the calculation of the weighted average described using Expression (1) is executed.

  Next, the distributed calculation process 1 executed in step S95 in FIG. 28 will be described with reference to the flowchart in FIG.

In step S141, the variance calculation unit 362 of the statistic calculation unit 342 outputs the variance value Vxx output as the calculation result (here, it is replaced with V _CrCr because it is the variance value of the first color Cr. Initialize).

In step S142, the variance calculation unit 362 calculates the average value M _x of the first signal calculated by the average value calculation unit 361 (here, the subscript _x of M _x is replaced with Cr. Hereinafter, description is omitted). To get.

  In step S143, the variance calculation unit 362 receives a rough interpolation value of a part of the local region (range of (n−2) × (n−2) centered on the target pixel), Among them, the value s ′ of the first register indicating the pixel position to be processed is initialized to s ′ = 2, and in step S144, a part of the local region (centered on the target pixel (n −2) × (n−2) range) of pixels, the second register value t ′ indicating the pixel position to be processed is initialized to t ′ = 2.

In step S145, the variance calculation unit 362 subtracts the average value M _Cr acquired in step S142 from the intensity X (that is, Crroughi, hereinafter omitted) of the pixel (s ′, t ′), and obtains the subtraction result. The value added to the current variance value V _CrCr is _squared and updated as the variance value V _CrCr . In other words, the variance calculating unit 362 calculates V _CrCr = V _CrCr + (| Intensity X of pixel (s ′, t ′) − Average value M _Cr |) ² . In this calculation, the variance calculation unit 362 may weight the intensity X of the pixel (s ′, t ′) as described with reference to FIG. 11 or FIG.

  In step S146, the variance calculation unit 362 refers to the value t ′ of the second register indicating the pixel position where the process is executed, and determines whether t ′ = n−1.

  If it is determined in step S146 that t ′ = n−1 is not satisfied, in step S147, the variance calculating unit 362 updates the value t ′ of the second register to t ′ = t ′ + 1 to perform processing. Returns to step S145, and the subsequent processing is repeated.

  When it is determined in step S146 that t ′ = n−1, in step S148, the variance calculation unit 362 refers to the value s ′ of the first register indicating the pixel position where the process is performed, and s ′ It is determined whether or not n = 1.

  If it is determined in step S148 that s ′ = n−1 is not satisfied, in step S149, the variance calculating unit 362 updates the value s ′ of the first register to s ′ = s ′ + 1 to perform processing. Returns to step S144, and the subsequent processing is repeated.

If it is determined in step S148 that s ′ = n−1, in step S150, the variance calculating unit 362 selects a part of the local area ((n−2) × ( The variance value V _CrCr of the coarse interpolation value in the range n-2) is output, and the process returns to step S96 in FIG.

For the calculation of the dispersion value, by definition, it is necessary to divide (| the intensity X of the pixel (s ′, t ′) − the average value M _Cr |) ² by the number of pixels used for the calculation of the dispersion value. Since the calculation result of the variance value is used to divide the calculation result of the covariance value by the process described later, when the covariance value is also divided by the same number, the division process is performed with the calculation of the variance value and the covariance value. It can be omitted in both of the calculation of the variance value.

In addition, in the calculation of the variance value, when weighting is performed, as shown in the definition of the variance value described using Expression (2), (| intensity X−average value of pixels (s ′, t ′)] The dispersion value can be calculated by dividing M _Cr |) ² by the sum of the weighting factors. However, since the calculation result of the variance value is used to divide the calculation result of the covariance value by the processing described later, when weighting is also applied in the calculation of the covariance value, the division by the sum of the weighting factors is performed. The processing can be omitted in both the variance value calculation and the covariance value calculation.

  In the distributed calculation process 1 described with reference to FIG. 35, since there is a square calculation, there is a problem that the calculation process takes time and the scale of hardware increases. Therefore, the square calculation may be omitted by approximate calculation.

  Next, the variance calculation process 2 executed in step S95 in FIG. 28 when the square calculation is omitted by the approximate calculation will be described with reference to the flowchart in FIG.

  In steps S171 to S174, the same processes as those in steps S141 to S144 described with reference to FIG. 35 are executed.

In step S175, the variance calculation unit 362 subtracts the average value M _Cr acquired in step S142 from the intensity X of the pixel (s ′, t ′), squares the subtraction result, and obtains the current variance value V _The value added to _CrCr is updated as the dispersion value V _CrCr . In other words, the variance calculating unit 362 calculates V _CrCr = V _CrCr + (| Intensity X of pixel (s ′, t ′) − Average value M _Cr |). In this calculation, the variance calculation unit 362 applies the same weighting to the intensity X of the pixel (s ′, t ′) as in the calculation of the average value as shown in FIGS. 31 to 34, for example. May be.

When the value | p | is normalized and 0 ≦ | P | <1, p ² can be approximated by | p |. This is similar to the fact that the approximation calculation for the covariance calculation shown in Expression (3) is possible using Expression (4) or Expression (5).

  In steps S176 to S180, processing similar to that in steps S146 to S150 described with reference to FIG. 35 is executed, and the processing returns to step S96 in FIG.

  In this way, by using the approximate calculation processing similar to the case of performing the approximate calculation for the covariance calculation shown in Expression (3) using Expression (4) or Expression (5), 2 necessary for calculating the variance value is used. It is possible to obtain an approximate value of the variance value by an approximation operation that omits the calculation of the power.

  Next, the covariance calculation process executed in step S96 of FIG. 28 will be described with reference to the flowchart of FIG.

In step S201, the covariance calculation unit 364 replaces the output value with a covariance value V _xy (here, since it is a covariance value between the first color Cr and the second color Cr, V _CrCe). (Hereinafter omitted) is initialized.

In step S202, the covariance calculation unit 364 performs the average value M _Cr of the first signal calculated by the process of step S92 of FIG. 28 by the average value calculation unit 361 and the step of FIG. 28 by the average value calculation unit 363. The second average value M _Ce calculated by the process of S94 is acquired.

  In step S <b> 203, the covariance calculation unit 364 receives a rough interpolation value of pixels in a part of the local region (range of (n−2) × (n−2) centered on the target pixel). Among them, the value s ′ of the first register indicating the pixel position where the process is executed is initialized to s ′ = 2.

  In step S204, the covariance calculation unit 364 performs processing among the coarse interpolation values of pixels in a part of the local region (range of (n−2) × (n−2) centered on the target pixel). Initialize the value t ′ of the second register indicating the pixel position to execute t ′ = 2.

  In step S205, integration processing described later with reference to FIG. 39 or FIG. 40 is executed.

  In step S206, the covariance calculation unit 364 refers to the value t ′ of the second register indicating the pixel position where the process is executed, and determines whether t ′ = n−1.

  If it is determined in step S206 that t ′ = n−1 is not satisfied, the covariance calculation unit 364 updates the value t ′ of the second register to t ′ = t ′ + 1 in step S207, The processing returns to step S205, and the subsequent processing is repeated.

  If it is determined in step S206 that t ′ = n−1, in step S208, the covariance calculation unit 364 refers to the value s ′ of the first register indicating the pixel position where the process is performed, and s It is determined whether or not '= n-1.

  If it is determined in step S208 that s ′ = n−1 is not satisfied, in step S209, the covariance calculation unit 364 updates the value s ′ of the first register to s ′ = s ′ + 1, The process returns to step S204, and the subsequent processes are repeated.

If it is determined in step S208 that s ′ = n−1, in step S209, the covariance calculation unit 364 outputs the covariance value V _CrCe , and the process ends.

The calculation of the covariance value is, by definition, (| intensity X of pixel (s ′, t ′) − average value M _Cr |) (| intensity y of pixel (s ′, t ′) − average value M _Ce | ) Must be divided by the number of pixels used in the calculation of the covariance value, but the calculation result of the covariance value is divided by the calculation result of the dispersion value by the processing described later. In the case of dividing by the same number, the division process can be omitted in both the variance value calculation and the covariance value calculation.

In addition, in the calculation of the covariance value, when weighting is performed, as shown in the definition of the dispersion value described using Expression (3), the intensity X−average of (| pixel (s ′, t ′) The covariance value can be calculated by dividing the value M _Cr |) (| intensity y of pixel (s ′, t ′) − average value M _Ce |) by the sum of the weighting factors. However, since the calculation result of the covariance value is divided by the calculation result of the dispersion value by a process described later, when weighting is applied also in the calculation of the dispersion value, the division process by the sum of the weighting coefficients is performed. It is possible to omit both the calculation of the variance value and the calculation of the covariance value.

  The integration process executed in step S205 of the above-described covariance calculation process of FIG. 37 includes the integration process 1 executed when the covariance value is calculated as defined using the above-described equation (3). There are two types of processing: integration processing 2 executed when the covariance value is calculated using approximate calculation.

  Next, the integration process 1 executed in step S205 of FIG. 37 will be described with reference to the flowchart of FIG. The integration process 1 is a process executed when a covariance value is calculated as defined using the above-described equation (3).

In step S221, the covariance calculation unit 364 (the intensity X of the first signal pixel (s ′, t ′) − the average value M _Cr of the first signal) and the (second signal pixel (s The product of the intensity Y of ', t') (that is, Ceroughi, hereinafter omitted) -average value M _Ce of the second signal) is calculated.

In step S222, the covariance calculation unit 364 sets V _CrCe = V _CrCe + wi (integration result), and the process proceeds to step S206 in FIG. Here, wi is a weighting value in the pixel (s ′, t ′).

  By executing such integration processing, a covariance value is calculated as defined using the above-described equation (3). When integration process 1 is executed, the calculation is as defined, so the calculation result is very accurate. Instead, the calculation takes time or the number of gates in the hardware implementation increases. Will be invited.

  Next, the integration process 2 executed in step S205 of FIG. 37 will be described with reference to the flowchart of FIG. The integration process 2 is a process executed when an approximate covariance value is calculated using the above-described equation (4) or equation (5).

In step S241, the covariance calculation unit 364 sets {the intensity X of the pixel (s ′, t ′) of the first signal−the average value M _Cr of the first signal} to p.

In step S242, the covariance calculating unit 364 sets {the intensity Y of the pixel (s ′, t ′) of the second signal−the average value M _Ce of the second signal} to q.

  In step S243, integration approximation processing described later with reference to FIG. 40 or FIG. 41 is executed.

In step S244, the covariance calculation unit 364 sets V _CrCe = V _CrCe + (approximate value of pq), and the process proceeds to step S206 in FIG.

  Next, with reference to the flowchart of FIG. 40, the integration approximation process 1 executed in step S243 of FIG. 39 will be described. The integrated approximation process 1 is a process executed when an approximate covariance value is calculated using the above-described equation (4).

  In step S261, the covariance calculation unit 364 determines whether or not | p | ≧ | q | using the values p and q replaced in steps S241 and S242 of FIG.

  If it is determined in step S261 that | p | ≧ | q |, in step S262, the covariance calculation unit 364 determines whether or not p ≧ 0.

  If it is determined in step S262 that p ≧ 0, in step S263, the covariance calculation unit 364 sets pq approximate value = + q, and the process returns to step S244 in FIG.

  If it is determined in step S262 that p ≧ 0 is not satisfied, in step S264, the covariance calculation unit 364 sets an approximate value of pq = −q, and the process returns to step S244 in FIG.

  If it is determined in step S261 that | p | ≧ | q | is not satisfied, the covariance calculation unit 364 determines whether q ≧ 0 in step S265.

  When it is determined in step S265 that q ≧ 0, in step S266, the covariance calculation unit 364 sets pq approximate value = + p, and the process returns to step S244 in FIG.

  If it is determined in step S265 that q ≧ 0 is not satisfied, in step S267, the covariance calculation unit 364 sets pq approximate value = −p, and the process returns to step S244 in FIG.

  In this process, when q or p is 0, the approximate value of pq is always 0. Specifically, when q is 0, | p | ≧ | q | always holds, so that the approximate value of pq is 0 regardless of the value of p. Also, when p is 0, | p | ≧ | q | does not always hold, so the approximate value of pq is 0 regardless of the value of q.

  By the processing described with reference to FIG. 40, the covariance value can be approximated using the above-described equation (4).

  Next, the integration approximation process 2 executed in step S243 of FIG. 39 will be described with reference to the flowchart of FIG. The integrated approximation process 2 is a process executed when an approximate covariance value is calculated using the above-described equation (5).

  In step S281, the covariance calculation unit 364 determines whether p or q is 0 using the values p and q replaced in steps S241 and S242 of FIG.

  If it is determined in step S281 that p or q is 0, in step S281, the covariance calculation unit 364 sets the approximate value of pq to 0, and the process returns to step S244 in FIG.

  If it is determined in step S281 that p and q are not 0, in step S283, the covariance calculation unit 364 determines that the relationship between p and q is p> 0 and q> 0, or p <0. In addition, it is determined whether any one of q <0 is satisfied.

  If it is determined in step S283 that the relationship between p and q is either p> 0 and q> 0, or p <0 and q <0, in step S284, the covariance calculation unit In step 364, the approximate value of pq is set to (| p | + | q |) / 2, and the process returns to step S244 in FIG.

  If it is determined in step S283 that the relationship between p and q is neither p> 0 and q> 0, or p <0 and q <0, the covariance calculation unit 364 determines in step S285. , Pq is set to (− | p | − | q |) / 2, and the process returns to step S244 in FIG.

  Note that the value p or q replaced in step S241 and step S242 in FIG. 39 may be 0 depending on the type of signal. Therefore, the processing in step S281 and step S282 may be omitted. By doing so, it is possible to increase the calculation processing speed and reduce the hardware implementation scale.

  With the processing described with reference to FIG. 41, it is possible to approximate the covariance value using the above-described equation (5).

  By executing the integration process described with reference to FIGS. 39 to 41, it is possible to approximate the covariance value using the above-described equation (4) or equation (5). In such a case, compared with the case where the calculation of the covariance value as defined in Expression (3) is executed, the calculation speed is increased or the hardware gate implemented for the calculation is used. Advantages such as being able to reduce the number occur.

  Next, the interpolation pixel estimation process 1 executed in the regression calculation processing unit 343 in FIG. 16 will be described with reference to the flowchart in FIG.

  In step S291, the inclination calculation unit 381 calculates the inclination K based on the variance value and the covariance value using the above-described equation (7) or equation (8).

  Here, the slope calculation unit 381 performs clipping processing on the denominator of Ks when the slope K is obtained by Ks, as shown in the above-described equation (8). This clipping process has the effect of reducing noise, as in the case where both the determination of whether an image is flat and the process of suppressing the output amplitude are executed. By executing this process, it is possible to suppress noise that is noticeable in a flat portion of an image without adding a separate noise reduction process.

In step S292, the pixel intensity estimating unit 382, the slope K, 2 single average value calculated in step S291 M _Cr and M _Ce, and, based on the pixel intensity of the pixel of interest, linear above equation (6) Using the regression equation, an intensity estimate value of the target pixel position is calculated, and the process is terminated.

  Through such processing, the estimated intensity of each color at the target pixel position can be calculated. In particular, by multiplying the slope K by an appropriate constant u greater than 1 in the linear regression equation for calculating the estimated intensity, the high-frequency component is slightly enhanced in the same manner as the equation (11) described above. The same effect as that obtained when appropriate high-frequency correction is executed can be obtained. By doing so, it is possible to obtain an image in which the high frequency is corrected without adding a separate high frequency correction process.

  Next, the texture direction determination process executed by the texture direction determination unit 305 in FIG. 8 will be described with reference to the flowchart in FIG.

  In step S301, the texture direction determination unit 305 calculates the horizontal gradient value dH at the target pixel (j, k) using the following equation (16), and uses the equation (17) to calculate the vertical gradient. The value dV is calculated. That is, the texture intensity calculation processing unit 334 calculates the absolute value of the difference between the mosaic image signal M at the pixel position two pixels left and right (horizontal direction) from the target pixel (j ′, k ′) and the mosaic image signal M of the target pixel. Is calculated as dH by adding the absolute value of the difference between the mosaic image signals M at pixel positions separated by one to the left and right (horizontal direction) of the target pixel, and above and below (vertical direction) the target pixel. The average of the absolute values of the differences between the mosaic image signal M at the pixel position two apart and the mosaic image signal M at the pixel of interest is the average of the mosaic image signals M at the pixel positions one distance above and below (vertical direction) the pixel of interest. The sum of the absolute values of the differences is calculated as dV.

dH = {(| M (j′−2, k ′) − M (j ′, k ′) | + | M (j ′ + 2, k ′) − M (j ′, k ′) |) / 2 + | M (j′−1, k ′) − M (j ′ + 1, k ′) |} / 2
... (16)
dV = {(| M (j ′, k′−2) −M (j ′, k ′) | + | M (j ′, k ′ + 2) −M (j ′, k ′) |) / 2 + | M (j ′, k′−1) −M (j ′, k ′ + 1) |} / 2
... (17)

In step S302, the texture direction determination unit 305 determines whether or not texture strength dH> texture strength dV. If it is determined that dH> dV, the texture strength calculation processing unit 334 checks in step S203 The texture intensity T _H of the pixel (j ′, k ′) is set to 0, and the texture intensity T _V of the pixel of interest (j ′, k ′) is set to 1.

In step S302, if it is determined not dH> dV, at step S204, the texture direction determination unit 305, the texture-intensity T _H of the pixel of interest C and 1, texture intensity in the target pixel (j ', k') T _{Let V} be 0.

After step S303 or step S304, in step S305, the texture direction determination unit 305 determines and outputs the texture direction from the set of the texture strengths T _H and T _V of the pixel of interest, and the process ends.

  By such processing, a determination result is obtained as to whether the texture at the target pixel position is close to horizontal stripes or vertical stripes, and is supplied to the switch 324.

  Next, the processing of the second processing block 303 in FIG. 8 will be described with reference to the flowchart in FIG.

  In step S <b> 311, the second intensity estimation unit 321 of the second processing block 303 performs intensity estimation of the target pixel using the target pixel and the pixels on the left and right sides thereof, and calculates a first estimated value.

  In step S312, the third intensity estimation unit 322 of the second processing block 303 estimates the intensity of the target pixel using the target pixel and pixels above and below it.

  In step S313, the fourth intensity estimation unit 323 of the second processing block 303 calculates the intensity estimation result of the target pixel using the target pixel calculated in step S312 and the pixels above and below the target pixel, and the upper, lower, left, and right of the target pixel. Intensity estimation is performed using these pixels, and a second estimated value is calculated.

  In step S314, the switch 324 of the second processing block 303 acquires the texture direction information supplied from the texture direction reversing unit 305.

  In step S315, the switch 324 of the second processing block 303 determines whether or not the texture direction is close to the horizontal direction.

  When it is determined in step S315 that the texture direction is close to the horizontal direction, in step S316, the switch 324 of the second processing block 303 determines the intensity estimation result of the target pixel using the target pixel and the pixels on the left and right sides thereof. Is output, and the process is terminated.

  When it is determined in step S315 that the texture direction is not close to the horizontal direction, that is, it is determined that the texture direction is close to the vertical direction, in step S316, the switch 324 of the second processing block 303 selects the upper, lower, left, and right pixels of the target pixel. The second estimated value, which is the intensity estimation result of the used pixel of interest, is output, and the process ends.

  By such processing, an estimated value with high reliability corresponding to the texture direction is selected and supplied to the switch 306.

  Next, the processing of the third processing block 304 of FIG. 8 will be described with reference to the flowchart of FIG.

  In step S331, the third intensity estimation unit 322 of the third processing block 303 performs intensity estimation of the target pixel using the target pixel and pixels above and below it, and calculates a first estimated value.

  In step S332, the second intensity estimation unit 321 of the third processing block 303 estimates the intensity of the target pixel using the target pixel and the pixels on the left and right sides thereof.

  In step S333, the fourth intensity estimation unit 323 of the third processing block 303 calculates the intensity estimation result of the target pixel using the target pixel calculated in step S331 and the pixels on the left and right, and the upper, lower, left, and right of the target pixel. Intensity estimation is performed using these pixels, and a second estimated value is calculated.

  In step S334, the switch 324 of the third processing block 303 acquires the texture direction information supplied from the texture direction reversing unit 305.

  In step S335, the switch 324 of the third processing block 303 determines whether or not the texture direction is close to the vertical direction.

  When it is determined in step S335 that the texture direction is close to the vertical direction, in step S336, the switch 324 of the third processing block 303 estimates the intensity of the target pixel using the target pixel and pixels above and below it. The first estimated value that is the result is output, and the process is terminated.

  If it is determined in step S335 that the texture direction is not close to the vertical direction, that is, close to the horizontal direction, in step S336, the switch 324 of the third processing block 303 sets the upper, lower, left, and right pixels of the target pixel. The second estimated value, which is the intensity estimation result of the used pixel of interest, is output, and the process ends.

  By such processing, an estimated value with high reliability corresponding to the texture direction is selected and supplied to the switch 306.

  The configuration of the G + E intensity estimating unit 282 described with reference to FIG. 7 is such that the G intensity estimating unit 291 and the E intensity estimating unit 292 respectively set the G intensity and E intensity of the target pixel position as described with reference to FIG. It was an estimate. Further, as described with reference to FIG. 8, the G intensity estimation unit 291 and the E intensity estimation unit 292 have the same configuration. Here, based on the operation of the switch 306 shown in FIG. 9 and the configurations of the G intensity estimating unit 291 and the E intensity estimating unit 292 described with reference to FIG. 8, the G intensity estimating unit 291 and the E intensity estimating unit 292 are used. There is a part that can share the configuration. That is, when the color of the target pixel position is G or E, the output of the switch 306 of the G intensity estimation 291 and the E intensity estimation unit 292 is a or b, and when it is R or B, the output of the switch 306 The output is c or d. Since the outputs of the G intensity estimation unit 401 and the E intensity estimation unit 402 are added by an adder, (a + b) is selected when the color of the target pixel position is G or E, and R or B is selected. If (c + d) is output, the common parts of the G intensity estimator 291 and the E intensity estimator 292 can be combined into a simple configuration.

  FIG. 46 shows a block diagram showing the configuration of the G + E intensity estimation unit 551 whose configuration has been changed as described above. The configurations of the pixel-of-interest selection unit 301, the first intensity estimation unit 311, the second intensity estimation unit 321, the third intensity estimation unit 322, the fourth intensity estimation unit 323, and the switch 324 are described with reference to FIG. The configurations of the G intensity estimation unit 291 and the E intensity estimation unit 292 described are the same.

  Then, the pixel of interest selected by the pixel-of-interest selection unit 301 and the intensity estimation value estimated by the first intensity estimation unit 311 are added by the addition processing unit 561-1 and input to the switch 562 as an input (a + b). Supplied. Then, by the processing of the second intensity estimation unit 321 and the fourth intensity estimation unit 323, two estimation values with high reliability are added by the addition processing unit 561-2 when the texture direction is close to the horizontal direction, Then, by the processes of the third intensity estimator 321 and the fourth intensity estimator 323, two estimated values with high reliability are added by the adder 561-3 when the texture direction is close to the vertical direction. Based on the texture direction determination result by the texture direction determination unit 305, the sum of the estimated values selected by the switch 324 is supplied to the switch 562 as an input (c + d).

  The operation of the switch 562 at this time is as shown in FIG. That is, when the color of the target pixel is R, the output of the switch 562 is (c + d), and when the color of the target pixel is G, the output of the switch 562 is (a + b), and the color of the target pixel is B , The output of the switch 562 is (c + d), and when the color of the pixel of interest is E, the output of the switch 562 is (a + b).

  In addition, the first intensity estimating unit 311, the second intensity estimating unit 321, the third intensity estimating unit 322, and the fourth intensity estimating unit 323 are all described with reference to FIG. A coarse interpolation processing unit 341, a statistic calculation unit 342, and a regression calculation processing unit 242 are included. Therefore, the G + E intensity estimator 551 described with reference to FIG. 46 is combined with the common processes of the coarse interpolation processor 341, the statistic calculator 342, and the regression calculation processor 242 to obtain FIG. The G + E intensity estimation unit 571 shown in FIG. The G + E intensity estimation unit 571 can, of course, execute the same processing as the G + E intensity estimation unit 551.

  That is, since the G + E intensity estimator 551 has the same statistical calculation for the first intensity estimator 311 and the fourth intensity estimator 323, the G + E intensity estimator 571 includes the statistic calculator 342. −1 is configured to be executed by one statistic calculation unit. In addition, since the outputs of the second intensity estimator 321 and the third intensity estimator 322 in the G + E intensity estimator 551 are used when the color of the pixel of interest is R or B, the G + E intensity estimator 571 , The switch 581 selects the interpolation value of the color that matches the target pixel from the rough interpolation value of R and the rough interpolation value of B supplied from the rough interpolation processing unit 427, and calculates the statistical amount calculation unit. 342-2 and the statistic calculator 342-3 are supplied.

  The coarse interpolation processing unit 427 calculates rough interpolation values of the four RGBE colors by processing described later using the flowchart of FIG. The operation of the switch 562 is the same as that described with reference to FIG.

  Next, referring to the flowchart of FIG. 49, the rough interpolation processing executed in the rough interpolation processing unit 472 of FIG. 48 (this processing is also described later in the rough interpolation processing unit 472 described with reference to FIG. 17). 50 executed in step S371).

  In step S351, the coarse interpolation processing unit 472 initializes the value s of the first register indicating the pixel position to be processed among the supplied pixels in the local region, and sets s = 2, and in step S352, Among the supplied local area pixels, the value t of the second register indicating the pixel position where the process is executed is initialized to t = 2.

  In step S353, the coarse interpolation processing unit 472 sets the pixel intensity of the pixel (s, t) as the interpolation value α1.

  In step S354, the rough interpolation processing unit 472 outputs the pixel (s-1, t), which is a pixel adjacent to the pixel (s, t), and the pixel (s-1, t) with respect to the pixel (s, t). And the average intensity of the pixel (s + 1, t) which is the adjacent pixel in the opposite direction is defined as an interpolation value α2.

  In step S355, the coarse interpolation processing unit 472 is a pixel (s, t) and a pixel (s−1, t) and a pixel that is a pixel adjacent to the pixel (s + 1, t) in a direction perpendicular to the direction of the pixel (s + 1, t). The average intensity of s, t−1) and the pixel (s, t + 1) that is the pixel adjacent to the pixel (s, t−1) in the opposite direction to the pixel (s, t) is defined as an interpolation value α3. .

  For example, when the pixel (s, t) is the pixel (2, 2) in FIG. 8, the pixel indicates the G intensity, and the pixel (s−1, t) and the pixel (s + 1, t) have the R intensity. The pixel (s, t−1) and the pixel (s, t + 1) indicate B intensity.

  In step S356, the rough interpolation processing unit 472 performs the pixel (s-1, t-1), pixel (s) that are in the upper right direction, the lower right direction, the upper left direction, and the lower left direction with respect to the target pixel. The average intensity of s−1, t + 1), pixel (s + 1, t−1), and pixel (s + 1, t + 1) is set as an interpolation value α4.

  For example, when the pixel (s, t) is the pixel (2, 2) in FIG. 8, the pixel indicates G intensity, and the pixel (s−1, t−1), pixel (s−1, t + 1) , Pixel (s + 1, t−1), and pixel (s + 1, t + 1) also each exhibit G intensity. In addition, when the pixel (s, t) is the pixel (2, 3) in FIG. 8, the pixel indicates B intensity, and the pixel (s−1, t−1), pixel (s−1, t + 1) , Pixel (s + 1, t−1) and pixel (s + 1, t + 1) each show R intensity. When the pixel (s, t) is the pixel (3, 2) in FIG. 8, the pixel indicates the R intensity, and the pixel (s−1, t−1), the pixel (s−1, t + 1) , Pixel (s + 1, t−1) and pixel (s + 1, t + 1) each show B intensity.

  In step S357, the coarse interpolation processing unit 472 determines which of the RGBE values the interpolation values α1 to α4 calculated by the processing of steps S353 to S356 correspond to.

  When the arrangement of the color filters is as described with reference to FIG. 3, when the pixel (s, t) is a G pixel, the interpolation value α1 is an interpolation value for G, and the interpolation value α2 is An interpolation value for B, an interpolation value α3 is an interpolation value for R, and an interpolation value α4 is an interpolation value for E. When the pixel (s, t) is an R pixel, the interpolation value α1 is an interpolation value for R, the interpolation value α2 is an interpolation value for E, and the interpolation value α3 is an interpolation value for G. The value α4 is an interpolation value for B. When (s, t) is a pixel of B, the interpolation value α1 is an interpolation value for B, the interpolation value α2 is an interpolation value for G, the interpolation value α3 is an interpolation value for E, and the interpolation value α4 is an interpolation value for R. When (s, t) is an E pixel, the interpolation value α1 is an interpolation value for E, the interpolation value α2 is an interpolation value for R, the interpolation value α3 is an interpolation value for B, and the interpolation value α4 is an interpolation value for G.

  In step S358, the coarse interpolation processing unit 472 sets the average value of the two interpolation values corresponding to G as the G interpolation value of the pixel (s, t).

  In step S359, the coarse interpolation processing unit 472 sets the interpolation value corresponding to R as the R interpolation value of the pixel (s, t).

  In step S360, the coarse interpolation processing unit 472 sets the interpolation value corresponding to B as the B interpolation value of the pixel (s, t).

  In step S361, the coarse interpolation processing unit 472 sets the interpolation value corresponding to E as the E interpolation value of the pixel (s, t).

  In step S362, the rough interpolation processing unit 472 refers to the value t of the second register indicating the pixel position to be processed, and determines whether t = n−1.

  If it is determined in step S362 that t = n−1 is not satisfied, in step S363, the coarse interpolation processing unit 472 sets the value t of the second register indicating the pixel position to be processed to t = t + 1, The processing returns to step S353, and the subsequent processing is repeated.

  In step S364, the coarse interpolation processing unit 472 refers to the value s of the first register indicating the pixel position where the process is executed, and determines whether s = n−1.

  If it is determined in step S364 that s = n−1 is not satisfied, in step S365, the coarse interpolation processing unit 472 sets the value s of the first register indicating the pixel position to be processed as s = s + 1, The processing returns to step S352, and the subsequent processing is repeated.

  When it is determined in step S364 that s = n−1, in step S366, the coarse interpolation processing unit 472 sets a set of RGB interpolation values of (n−2) square pixels near the target pixel. Is output and the process is terminated.

  With such a process, the RGBE interpolation values at the (n−2) × (n−2) pixel positions centered on the target pixel with respect to the n × n pixels extracted as the local region are obtained. Calculated.

  However, in the coarse interpolation process described with reference to FIG. 49, the order of steps S353 to S356 may be interchanged, and the order of steps S358 to S361 may also be interchanged. Absent.

  Even when the G + E intensity estimator 551 and the G + E intensity estimator 571 described with reference to FIG. 46 are replaced with the G + E intensity estimator 282 in FIG. 6, the R (GB or E) intensity estimator 283 at the subsequent stage is used. The processing executed by (or RGBE intensity estimation unit 283) is the same.

  Next, the color intensity estimation process 1 executed by the RGBE intensity estimation unit 283 described with reference to FIG. 17 in step S24 of FIG. 24 will be described with reference to the flowchart of FIG.

  In step S371, the coarse interpolation processing unit 472 executes the coarse interpolation process 2 described with reference to FIG.

  In step S372, the reliability calculation unit 471 acquires the G + E intensity calculated by the G + E intensity estimation processing unit 282.

  In step S373, the reliability calculation unit 471 executes a reliability calculation process to be described later using the flowchart of FIG.

  In step S374, the statistical amount calculation units 432-1 to 432-4 execute the two-color distribution shape statistical amount calculation processing described with reference to FIG.

  In step S375, the regression calculation processing units 474-1 to 474-4 execute interpolation pixel estimation processing 2 described with reference to FIG. 52, and the processing proceeds to step S25 in FIG.

  Next, the reliability calculation process executed in step S373 in FIG. 50 will be described with reference to the flowchart in FIG.

  In step S401, the high-frequency extraction units 481-1 to 481-6 of the reliability calculation unit 471 extract themselves from the G interpolation values at the predetermined 12 positions near the target pixel described with reference to FIG. The G + E interpolation value of the pixel position necessary for obtaining the high frequency to be calculated, that is, the average value of the pixel values of G and E that are in contact with each other obliquely is calculated.

  In step S402, the high frequency extraction units 481-1 to 481-6 extract and add high frequency components in six directions centered on the pixel of interest based on the G interpolation value calculated in the processing of step S401. This is supplied to the processing unit 482.

  In step S403, the addition processing unit 482 adds the six high frequency components supplied from the high frequency extraction units 481-1 to 481-6 and outputs the result to the clipping processing unit 483.

  In step S404, the clipping processing unit 483 clips the addition result supplied from the addition processing unit 482, and outputs the value as a reliability value to the regression calculation processing units 286-1 to 286-3. Proceed to step S374 in FIG.

  Through such processing, a reliability value indicating whether or not a high-frequency component exists in the vicinity of the target pixel is calculated and output to the regression calculation processing units 474-1 to 474-4. The calculated reliability value increases as the strength of the thin striped texture increases, so the smaller the reliability value, the higher the probability that the color distribution shape is estimated by the statistics.

  Next, the interpolation pixel estimation processing 2 executed in step S375 in FIG. 50 will be described with reference to the flowchart in FIG.

  In step S <b> 421, the inclination calculation unit 501 calculates the estimated inclination value Km based on the average value using the above-described equation (7), and outputs it to the inclination synthesis unit 503.

  In step S422, the inclination calculation unit 502 calculates the estimated inclination value Ks based on the variance value and the covariance value using the above-described equation (8), and outputs it to the inclination synthesis unit 503.

  Here, the inclination calculation unit 502 performs clipping processing on the denominator of the inclination Ks as shown in the above-described equation (8). This clipping process has the effect of reducing noise, as in the case where both the determination of whether an image is flat and the process of suppressing the output amplitude are executed. By executing this process, it is possible to suppress noise that is noticeable in a flat portion of an image without adding a separate noise reduction process.

  In step S423, the slope composition unit 503 uses the slope estimate value Km calculated in step S421 and the slope estimate value Ks calculated in step S422 based on the reliability h using the above-described equation (9). Are combined to calculate an estimated slope value K and output it to the pixel intensity estimating unit 504.

In step S424, the elementary intensity estimation unit 504, based on the estimated slope value K calculated in step S423, the two average values M _G and M _R (or M _B ), and the G intensity at the target pixel position, The estimated value of the R intensity (or B intensity) of the target pixel position is calculated using the linear regression equation of the above-described equation (10) or equation (11), and the process is terminated.

  Through such processing, the estimated intensity of each color at the target pixel position can be calculated. In particular, the high frequency component is slightly enhanced by multiplying the slope K by an appropriate constant u greater than 1 using the above-described equation (11) in the linear regression equation when calculating the estimated intensity, An effect similar to that obtained when appropriate high-frequency correction for R is executed can be obtained. By doing so, it is possible to obtain an image in which the high frequency is corrected without adding a separate high frequency correction process.

  The above-described processing can also be applied when the mosaic pattern used for the color filter is a six-color name array as shown in FIG. The six-color array shown in FIG. 53 is an array in which half of the R and B pixels in the four-color array in FIG. 3 are each replaced with one having a slightly different spectral characteristic from R and B (in other words, For example, the spectral characteristics of R1 and R2, B1 and B2 are highly correlated). That is, the color filter shown in FIG. 53 is arranged in a checkered pattern so that G and E used for luminance value estimation are not the same line in the horizontal direction and the vertical direction, respectively. Different colors R1 and R2 are located on the same line as E (ie, a line different from G), and colors B1 and B2 different from G, E, R1, and R2 are located on the same line as G (ie, E and Are color filters that are located on different lines.

  In this six-color array, the G and E arrays are the same as the four-color array in FIG. 3, so it is clear that the luminance component calculation method using the estimated value of G + E intensity can be applied as it is. . In addition, although the six-color array shown in FIG. 53 divides both R and B into two colors having similar spectral characteristics, the present invention can be changed even if only one of the five-color arrays is configured. Needless to say, it is applicable. By using a color filter having such a six-color arrangement, it is possible to obtain an image signal that is more advantageous for color reproduction by multispectral imaging. Furthermore, since the demosaic process interpolates G + E for all pixels using G and E having high correlation, it is possible to accurately calculate even the high frequency component of the luminance.

  The processing of the demosaic processing unit 253 described with reference to FIGS. 6 to 53 was obtained using the four color filters described with reference to FIG. 3 or the six color filters illustrated in FIG. G + E intensity estimation means 282, G + E intensity estimation means 401, or G + E intensity estimation means 431 estimates the G + E intensity for the mosaic image, and an RGBE four-color image is obtained based on this. It was. On the other hand, when a color filter in which a certain color is arranged in a checkered pattern is used, the luminance value of the pixel of interest is estimated using information on the color arranged in a checkered pattern instead of the G + E intensity. And based on this, you may make it obtain the image of four colors of RGBE.

  When estimating the luminance value of the pixel of interest instead of the G + E intensity, the demosaic processing unit 601 shown in FIG. 54 is used instead of the demosaic processing unit 253 of the DSP block 216 described with reference to FIG.

  54 includes a luminance estimation processing unit 611 instead of the G + E intensity estimation unit 282, and an R intensity estimation unit instead of the R intensity estimation units 283-1 to 283-4. It has the same configuration as the demosaic processing unit 253 described with reference to FIG. 6 except that 612-1 to E intensity estimation unit 612-4 are provided.

  The demosaic processing unit 601 in FIG. 54 replaces not only the Bayer array shown in FIG. 1 but also, for example, as shown in FIG. 55, half of the Bayer array B shown in FIG. In the case where four color filters arranged in every other row are used in both the vertical and horizontal directions, a suitable demosaic process can be performed.

  The luminance estimation processing unit 611 calculates the luminance estimation value of the target pixel position using the pixels in the rectangular region of the n × n (for example, 9 × 9) local region pixels extracted by the local region extraction unit 281. Process.

  The R intensity estimation units 612-1 through E intensity estimation units 612-4 use the estimated luminance values calculated by the luminance estimation processing unit 611 and the pixels of the rectangular area extracted by the local area extraction unit 281 respectively. Calculate the estimated value of the intensity. The R intensity estimating unit 612-1, the G intensity estimating unit 612-2, the B intensity estimating unit 612-3, and the E intensity estimating unit 612-4 basically have the same configuration. When it is not necessary to individually distinguish the R intensity estimating unit 612-1, the G intensity estimating unit 612-2, the B intensity estimating unit 612-3, and the E intensity estimating unit 612-4, R (G, B, or E) Intensity estimation unit 612, or R intensity estimation unit 612-1, G intensity estimation unit 612-2, B intensity estimation unit 612-3, and E intensity estimation unit 612-4 are all RGBE It is assumed to be called an intensity estimation unit 612.

  FIG. 56 is a block diagram showing a configuration of the luminance estimation processing unit 611.

  The luminance estimation processing unit 611 includes a target pixel selection unit 301, a texture direction determination unit 305, a first intensity estimation unit 621, a second intensity estimation unit 622, a switch 623, and a switch 624. The pixel-of-interest selection unit 301 and the texture direction determination unit 305 have the same functions as described with reference to FIG. The first intensity estimator 621 and the second intensity estimator 622 calculate the intensity of the color to be estimated based on the intensity estimation value of the color at the local region pixel and the target pixel position. The color to be estimated here is G arranged in a checkered pattern in the color filter described with reference to FIG. The luminance estimation processing unit 611 uses the fact that G exists in a checkered pattern, and calculates the G intensity as a luminance component. In other words, the luminance estimation process executed by the luminance estimation processing unit 611 estimates the G intensity, but the G intensity as the output result of the demosaic means is calculated separately from this, so that it is distinguished from it. The G intensity estimated by the luminance estimation processing unit 611 is referred to as luminance. The first intensity estimating unit 621 calculates the luminance using the pixel values of the left and right pixels of the target pixel, and the second intensity estimating unit 622 calculates the luminance using the pixel values of the pixels above and below the target pixel. To do.

  The switch 623 selects which of the two input intensity estimation values should be used based on the texture direction determination result output by the texture direction determination unit 305. If it is determined that the texture direction is closer to horizontal stripes, the intensity estimation value calculated by the first intensity estimation unit 621 is output from the switch 623, and if it is determined that the texture direction is closer to vertical stripes, the second intensity is calculated. The intensity estimated value calculated by the estimation unit 622 is output from the switch 623. The switch 624 selects which of the two input intensity estimation values should be used based on the color of the target pixel position in the original mosaic image. If the target pixel position of the mosaic image is G itself, the intensity of the target pixel of the mosaic image may be output as it is without the need for special estimation processing, so the output of the target pixel selection unit 301 becomes the output of the switch 624. As described above, when the target pixel position of the mosaic image is not G, the output signal from the switch 624 is selected so that the output of the switch 623 becomes the output of the switch 624.

  FIG. 57 is a block diagram showing a configuration of the first intensity estimating unit 621 or the second intensity estimating unit 622 of FIG. The rough interpolation processing unit 631 executes processing for obtaining rough interpolation values between the first color and the second color based on the mosaic image. Detailed processing of the coarse interpolation processing unit 631 will be described later with reference to the flowcharts of FIGS. 64 and 65. The statistic calculation unit 342 and the regression calculation processing unit 343 have the same functions as those described with reference to FIG.

  The difference between the first intensity estimator 621 and the second intensity estimator 622 is a method for selecting the first color and the second color in the coarse interpolation process executed by the coarse interpolation processor 631. These differences will be described with reference to FIGS. 58 and 59, the case where the target pixel is R will be described, but the same applies to the case where the target pixel is B or E.

  As shown in FIG. 58, the first intensity estimating unit 621 uses a color at the target pixel position as the first color Cr (R in FIG. 58), and a color adjacent to the left and right of Cr as the second color Ce. That is, G is selected. In the drawing, hatched portions of R and G pixels are pixels used in the coarse interpolation processing unit 631 of the first intensity estimation unit 621. On the other hand, as shown in FIG. 59, the second intensity estimation unit 622 is adjacent to the color at the target pixel position as the first color Cr (R in FIG. 58) and above and below Cr as the second color Ce. Color to be used, that is, G is selected. In the drawing, hatched portions of R and G pixels are pixels used in the coarse interpolation processing unit 631 of the second intensity estimating unit 622.

  The calculation of the average value, variance value, and covariance value executed in the statistic calculation unit 342 is the same as that described above. For example, as the weighting coefficient, as shown in FIG. A position where the weighting coefficient is the highest at the position and the weighting coefficient becomes smaller as the distance from the target pixel position is increased is used.

  61 is a block diagram showing a configuration of the R intensity estimating unit 612-1 to E intensity estimating unit 612-4 of FIG. In FIG. 60, each of the reliability calculation unit 471 and the rough interpolation processing unit 641 is illustrated as an RGBE intensity estimation unit 612, but the R intensity estimation unit 612-1 through the E intensity estimation unit 612 are illustrated. Needless to say, the reliability calculation unit 471 and the coarse interpolation processing unit 641 may be provided in each of the four. In addition, the same code | symbol is attached | subjected to the part corresponding to the RGBE intensity estimation part 283 demonstrated using FIG. That is, the RGBE intensity estimation unit 612 in FIG. 60 is the same as the RGBE intensity estimation unit 283 in FIG. 17 except that the synthesis processing unit 473 is omitted and a coarse interpolation processing unit 641 is provided instead of the coarse interpolation processing unit 472. It has basically the same configuration.

  The rough interpolation processing unit 641 uses pixels in an n × n pixel local region and performs rough interpolation of two different colors (first color is Cr and second color is Ce) at the same position. Generate multiple value pairs. Details of the processing executed by the coarse interpolation processing unit 641 will be described later with reference to the flowchart of FIG.

  Next, the demosaic process 2 executed by the demosaic processing unit 601 in FIG. 54 in step S4 in FIG. 23 will be described with reference to the flowchart in FIG.

  In step S451, the local region extraction unit 281 selects one of the unprocessed pixels as a target pixel, and in step S452, extracts a predetermined number (n × n) of pixels around the target pixel position as a local region. And supplied to the luminance estimation processing unit 611 and the R (G, B, or E) intensity estimation unit 612.

  In step S453, luminance estimation processing described later is executed using the flowchart of FIG.

  In step S454, a color intensity estimation process, which will be described later using the flowchart of FIG. 63, is executed in parallel in each of the R (G, B, or E) intensity estimation units 612 of FIG.

  In step S455, the local region extraction unit 281 determines whether the processing has been completed for all pixels. If it is determined in step S457 that the processing has not been completed for all pixels, the processing returns to step S451, and the subsequent processing is repeated. If it is determined in step S455 that the process has been completed for all pixels, the process proceeds to step S5 in FIG.

  In other words, each unit constituting the demosaic processing unit 253 executes each process at the target pixel position when a certain target pixel position is determined, and the processing of steps S451 to S456 is completed for all the pixels. If so, the process is terminated.

  By such processing, the mosaic image obtained based on the color filter array of the CCD image sensor 213 is demosaiced (color interpolation or synchronization), and each color constituting the color filter is interpolated in each pixel. Image data can be obtained.

  Next, the luminance estimation process executed in step S453 in FIG. 61 will be described with reference to the flowchart in FIG.

  In the processing described with reference to FIG. 56, three estimated values are obtained in advance based on whether the color of the pixel of interest is G and the texture direction, and the pixel of interest is obtained by the switch 623 and the switch 624. However, if the same processing is executed by software, whether or not the color of the pixel of interest is G, and the texture It is preferable to use a suitable intensity estimation value calculation method after detecting the direction.

  In step S471, the switch 624 determines whether or not the color of the target pixel is G.

  When it is determined in step S471 that the color of the target pixel is G, in step S472, the switch 624 outputs the pixel intensity at the target pixel position selected by the crowd pixel selection unit 301. The process advances to step S454 of 62.

  If it is determined in step S471 that the color of the target pixel is not G, in step S473, the switch 623 determines whether the texture direction of the local region is close to a horizontal stripe.

  When it is determined in step S473 that the texture direction of the local region is close to a horizontal stripe, in step S474, the switch 623 calculates and outputs an intensity estimation value to be interpolated by the first intensity estimation unit 621. The processing proceeds to step S454 in FIG.

  If it is determined in step S473 that the texture direction of the local region is not close to horizontal stripes, that is, close to vertical stripes, in step S475, the switch 623 determines the intensity estimation value to be interpolated by the second intensity estimation unit 622. After calculating and outputting, the process proceeds to step S454 in FIG.

  Through such processing, an estimated value of the luminance value corresponding to whether or not the color of the pixel of interest is G and the texture direction is calculated and output.

  Next, the rough interpolation process 3 executed by the rough interpolation processing unit 631 of the first intensity estimating unit 621 of FIG. 57 will be described with reference to the flowchart of FIG.

  In step S481, the coarse interpolation processing unit 631 initializes the value s of the first register indicating the pixel position where the processing is performed, out of the pixels in the supplied local region, and sets s = 2, and in step S482, Among the supplied local area pixels, the value t of the second register indicating the pixel position where the process is executed is initialized to t = 2.

  In step S483, the rough interpolation processing unit 631 determines whether or not the pixel (s, t) has the same color as the target pixel. If it is determined in step S483 that the pixel (s, t) is not the same color as the target pixel, the process proceeds to step S486.

  If it is determined in step S483 that the pixel (s, t) has the same color as the target pixel, in step S484, the coarse interpolation processing unit 631 determines that the pixel (s, t) is the first color. , A set of pixel intensities of the second color located on the left side of the pixel (s, t) is created.

  In step S485, the rough interpolation processing unit 631 creates a set of intensity of the pixel (s, t) that is the first color and the pixel that is the second color that is located to the right of the pixel (s, t). .

  In step S486, the rough interpolation processing unit 631 refers to the value t of the second register and determines whether t = n−1.

  If it is determined in step S486 that t = n−1 is not satisfied, in step S487, the coarse interpolation processing unit 631 sets the value t of the second register to t = t + 1, and the process returns to step S483. The subsequent processing is repeated.

  If it is determined in step S486 that t = n−1, in step S488, the coarse interpolation processing unit 631 refers to the value s of the first register and determines whether s = n−1. to decide.

  If it is determined in step S488 that s = n−1 is not satisfied, in step S489, the coarse interpolation processing unit 631 sets the value s of the first register to s = s + 1, and the process returns to step S482, The subsequent processing is repeated.

  When it is determined in step S488 that s = n−1, in step S490, the coarse interpolation processing unit 631 outputs the first color and the second color set, and the process is ended.

  By such processing, a luminance estimation value with high reliability is calculated for the texture in the horizontal direction.

  Next, the rough interpolation process 4 executed by the coarse interpolation processing unit 631 of the second intensity estimating unit 622 in FIG. 57 will be described with reference to the flowchart in FIG.

  In step S491, the coarse interpolation processing unit 631 initializes the value s of the first register indicating the pixel position where the processing is performed, out of the pixels in the supplied local region, and sets s = 2, and in step S492, Among the supplied local area pixels, the value t of the second register indicating the pixel position where the process is executed is initialized to t = 2.

  In step S493, the coarse interpolation processing unit 631 determines whether or not the pixel (s, t) has the same color as the target pixel. If it is determined in step S493 that the pixel (s, t) is not the same color as the target pixel, the process proceeds to step S496.

  If it is determined in step S493 that the pixel (s, t) has the same color as the target pixel, in step S494, the coarse interpolation processing unit 631 determines that the pixel (s, t) is the first color. , A set of pixel intensities that are the second color located above the pixel (s, t) is created.

  In step S495, the coarse interpolation processing unit 631 creates a set of intensity of the pixel (s, t) as the first color and the pixel as the second color located below the pixel (s, t). To do.

  In step S496, the rough interpolation processing unit 631 refers to the value t of the second register and determines whether t = n−1.

  If it is determined in step S496 that t = n−1 is not true, the coarse interpolation processing unit 631 sets the value t of the second register to t = t + 1 in step S497, and the process returns to step S493, The subsequent processing is repeated.

  If it is determined in step S496 that t = n−1, in step S498, the coarse interpolation processing unit 631 refers to the value s of the first register and determines whether s = n−1. to decide.

  When it is determined in step S498 that s = n−1 is not satisfied, in step S499, the coarse interpolation processing unit 631 sets the value s of the first register to s = s + 1, and the process returns to step S492. The subsequent processing is repeated.

  If it is determined in step S498 that s = n−1, in step S490, the coarse interpolation processing unit 631 outputs the first color and the second color set, and the process is terminated.

  With such processing, a highly reliable luminance estimated value is calculated for the texture in the vertical direction.

  Next, the color intensity estimation process 2 executed by the RGBE intensity estimation unit 283 described with reference to FIG. 60 in step S454 of FIG. 61 will be described with reference to the flowchart of FIG.

  In step S501, the coarse interpolation processing unit 472 executes coarse interpolation processing 5 described later with reference to FIG.

  In step S <b> 502, the reliability calculation unit 471 acquires the estimated luminance value calculated by the luminance estimation processing unit 611.

  In step S503, the reliability calculation unit 471 executes the reliability calculation process described with reference to the flowchart of FIG.

  In step S504, the statistic calculation units 432-1 to 432-4 execute the two-color distribution shape statistic calculation process described with reference to FIG.

  In step S505, the regression calculation processing units 474-1 to 474-4 execute the interpolation pixel estimation processing 2 described with reference to FIG. 52, and the processing proceeds to step S455 in FIG.

  Next, with reference to the flowchart of FIG. 67, the rough interpolation process 5 which the rough interpolation process part 641 of FIG. 61 performs in step S511 of FIG. 66 is demonstrated.

  In step S531, the rough interpolation processing unit 641 determines that the color of the pixel at the predetermined pixel position in the (n−2) × (n−2) area centered on the target pixel in the supplied local area. Whether or not it is G is determined.

  When it is determined in step S531 that the color of the pixel at the predetermined pixel position is G, in step S532, the coarse interpolation processing unit 641 sets the set of the intensity Ce of the target pixel and the intensity Ce of the target pixel (Ce, Make Ce).

  In step S533, the coarse interpolation processing unit 641 adds (Ce, Ce) as a set of Lg and G, and the process proceeds to step S455 of FIG.

  If it is determined in step S531 that the color of the pixel at the predetermined pixel position is not G, in step S534, the coarse interpolation processing unit 641 determines the vicinity of the predetermined pixel, that is, the upper, lower, left, and right pixels of the pixel. Two G intensity values are extracted.

In step S535, the coarse interpolation processing unit 641 converts the extracted G intensity to L1 and L2.
And sets (L1, Ce) and (L2, Ce) with the intensity Ce of a predetermined pixel.

In step S536, the coarse interpolation processing unit 641 determines whether or not the color of the predetermined pixel is R.

  If it is determined in step S536 that the color of the predetermined pixel is R, in step S537, the coarse interpolation processing unit 641 adds (L1, Ce) and (L2, Ce) as a pair of Lr and R. Then, the process proceeds to step S455 in FIG.

  If it is determined in step S536 that the color of the predetermined pixel is not R, the coarse interpolation processing unit 641 determines whether or not the color of the predetermined pixel is B in step S538.

  If it is determined in step S538 that the color of the predetermined pixel is B, in step S539, the coarse interpolation processing unit 641 adds (L1, Ce) and (L2, Ce) as a pair of Lb and B. Then, the process proceeds to step S455 in FIG.

  If it is determined in step S538 that the color of the predetermined pixel is not B, that is, E, in step S540, the coarse interpolation processing unit 641 changes (L1, Ce) and (L2, Ce) to Le. And the process proceeds to step S455 in FIG.

  The coarse interpolation processing unit 641 calculates a rough set of interpolation values in the (n−2) × (n−2) region centered on the target pixel in the local region by such processing, A set of all rough interpolation values calculated in the (n-2) × (n-2) region is output to the statistic calculation units 342-1 to 341-4.

  In the process of FIG. 67, in step S534, the coarse interpolation processing unit 641 extracts two G intensity values from four neighborhoods of a predetermined pixel, that is, from the upper, lower, left, and right pixels of the pixel. Different methods may be used as the method for selecting the two G intensities. For example, specifically, the difference value is calculated for each of the upper and lower G intensities and the left and right G intensities. The one with the smaller difference value is selected from the left and right. In this way, when the pixel position of the predetermined pixel is at a place where the color change is large, such as on the object outline, it is possible to select G in the direction where the change is as small as possible. Similarly, as a second method for selecting a direction in which the color change is as small as possible, there is also a method of selecting two of the upper, lower, left, and right G intensities excluding the maximum value and the minimum value. As a third method, the texture direction determination result by the texture direction determination unit 305 in the luminance estimation processing unit 51 is diverted to select the left and right G intensity for horizontal stripes and the upper and lower G intensity for vertical stripes. Also good. In addition, when n × n pixels in the entire local area are selected as the predetermined pixel position, when the pixel position is a corner of the local area, there are only two adjacent G, so that it is selected. You can do it.

  Also, the demosaic processing unit 601 in FIG. 54 is not limited to the Bayer arrangement shown in FIG. 1 or the arrangement of the four primary color filters shown in FIG. Even when a filter is used, suitable demosaic processing can be performed. This five-color array is an array in which half of the R and B pixel positions of the three-color Bayer array are replaced with ones having slightly different spectral characteristics (in other words, the spectrum of R1 and R2, B1 and B2). The characteristics are highly correlated). In this five-color array, the G array is the same as the four-color array in FIG. 55, and thus the luminance component calculation method using G by the demosaic processing unit 601 in FIG. 54 can be applied as it is.

  68, for example, a four-color array in which half of the R pixel positions of the three-color Bayer array are replaced with ones having slightly different spectral characteristics, or a three-color Bayer array B Needless to say, the present invention can also be applied to a four-color array in which half of the pixel positions are replaced with those having slightly different spectral characteristics.

  As described above, the present invention is also applied to a color arrangement of four or more colors, and similarly, it is possible to reduce jegginess and improve the sense of resolution.

  The image processing in the digital still camera 201 has been described above, but the present invention can also be applied to a digital video camera that can capture a moving image. When the present invention is applied to a digital video camera, the codec processing unit 221 performs processing using a compression or expansion algorithm of digital moving image data such as MPEG (Moving Picture Coding Experts Group / Moving Picture Experts Group) or MotionJPEG. Execute.

  The series of processes described above can be executed by hardware, but can also be executed by software.

  In this case, the above-described functions are realized by executing the software by the DSP block 216. Further, for example, part of the processing of the digital still camera 201 can be executed by a personal computer 701 as shown in FIG.

  69, a CPU (Central Processing Unit) 711 performs various processes according to a program stored in a ROM (Read Only Memory) 712 or a program loaded from a storage unit 718 to a RAM (Random Access Memory) 713. Execute. The RAM 713 also appropriately stores data necessary for the CPU 711 to execute various processes.

  The CPU 711, the ROM 712, and the RAM 713 are connected to each other via the bus 714. An input / output interface 715 is also connected to the bus 714.

  The input / output interface 715 includes an input unit 716 including a keyboard and a mouse, an output unit 717 including a display and a speaker, a storage unit 718 including a hard disk, and a communication unit 719 including a modem and a terminal adapter. It is connected. The communication unit 719 performs communication processing via a network including the Internet.

  A drive 720 is also connected to the input / output interface 715 as necessary, and a magnetic disk 731, an optical disk 732, a magneto-optical disk 733, a semiconductor memory 734, or the like is appropriately mounted, and a computer program read from these is loaded. It is installed in the storage unit 718 as necessary.

  When a series of processing is executed by software, a program constituting the software is dedicated hardware (for example, the DSP block 216 or the demosaic processing unit 253 or the demosaic processing unit 601 included therein). For example, a general-purpose personal computer or the like capable of executing various functions by installing various programs installed therein is installed from a network or a recording medium.

  As shown in FIG. 69, this recording medium includes a magnetic disk 731 (including a floppy disk) and an optical disk 732 (including a floppy disk) that are distributed to supply a program to the user separately from the apparatus main body. Package media including CD-ROM (compact disk-read only memory), DVD (including digital versatile disk), magneto-optical disk 733 (including MD (mini-disk) (trademark)), or semiconductor memory 734 In addition to being configured, it is configured by a ROM 712 storing a program and a hard disk included in the storage unit 718 supplied to the user in a state of being incorporated in the apparatus main body in advance.

  In the present specification, the step of describing the program stored in the recording medium is not limited to the processing performed in chronological order in the order in which they are included, but is not necessarily processed in chronological order, either in parallel or individually. The process to be executed is also included.

It is a figure for demonstrating a Bayer arrangement | sequence. It is a block diagram which shows the structure of the digital still camera to which this invention is applied. It is a figure for demonstrating the color filter used for the CCD image sensor of FIG. It is a figure which shows the example of the spectral characteristic of the color filter of FIG. It is a block diagram which shows the structure of the DSP block of FIG. It is a block diagram which shows the structure of the demosaic process part of FIG. It is a block diagram which shows the structure of the G + E intensity | strength estimation process part of FIG. It is a block diagram which shows the structure of the G intensity estimation part of FIG. 7, and an E intensity estimation part. It is a figure for demonstrating the process of the switch of FIG. It is a block diagram which shows the structure of the 1st thru | or 4th intensity | strength estimation part of FIG. It is a figure for demonstrating the 1st color and 2nd color which the rough interpolation process part of FIG. 10 selects. It is a figure for demonstrating the 1st color and 2nd color which the rough interpolation process part of FIG. 10 selects. It is a figure for demonstrating the 1st color and 2nd color which the rough interpolation process part of FIG. 10 selects. It is a figure for demonstrating the 1st color and 2nd color which the rough interpolation process part of FIG. 10 selects. It is a block diagram which shows the structure of the statistic calculation part of FIG. It is a block diagram which shows the structure of the regression calculation process part of FIG. It is a block diagram which shows the structure of the RGBE intensity estimation part of FIG. It is a block diagram which shows the structure of the reliability calculation part of FIG. It is a figure for demonstrating the extraction position of the pixel intensity | strength in a reliability calculation part. It is a figure for demonstrating the group of the pixel intensity | strength for extracting a high frequency component in a reliability calculation part. It is a figure for demonstrating the process of the rough interpolation process part of FIG. It is a block diagram which shows the structure of the regression calculation process part of FIG. 6 is a flowchart for explaining image processing executed by the DSP block of FIG. 5. 10 is a flowchart for explaining demosaic processing 1; It is a flowchart for demonstrating a G + E intensity estimation process. It is a flowchart for demonstrating the G (E) intensity | strength estimated value calculation process of an attention pixel position. 6 is a flowchart for explaining rough interpolation processing 1; It is a flowchart for demonstrating the statistic amount calculation process of 2 color distribution shape. 6 is a flowchart for explaining an average value calculation process 1; 10 is a flowchart for explaining average value calculation processing 2; It is a figure for demonstrating a weighting value. It is a figure for demonstrating a weighting value. It is a figure for demonstrating a weighting value. It is a figure for demonstrating a weighting value. 10 is a flowchart for explaining distributed calculation processing 1; 10 is a flowchart for explaining distributed calculation processing 2; It is a flowchart for demonstrating a covariance calculation process. 5 is a flowchart for explaining an integration process 1; 5 is a flowchart for explaining an integration process 2; 5 is a flowchart for explaining integration approximation processing 1; 10 is a flowchart for explaining integration approximation processing 2; 6 is a flowchart for explaining interpolation pixel estimation processing 1; It is a flowchart for demonstrating a texture direction determination process. It is a flowchart for demonstrating the process of a 2nd process block. It is a flowchart for demonstrating the process of a 3rd process block. It is a block diagram which shows a different structure of a G + E intensity estimation process part. FIG. 47 is a diagram for explaining processing of the switch of FIG. 46. It is a block diagram which shows a different structure of a G + E intensity estimation process part. 10 is a flowchart for explaining rough interpolation processing 2; 5 is a flowchart for explaining color intensity estimation processing 1; It is a flowchart for demonstrating reliability calculation processing. It is a flowchart for demonstrating the interpolation pixel estimation process 2. FIG. It is a figure for demonstrating the example from which the color filter used for the CCD image sensor of FIG. 2 differs. It is a block diagram which shows the example of a different structure of the demosaic process part of FIG. FIG. 55 is a diagram for describing an example of a color filter used in the CCD image sensor in FIG. 2 when the demosaic processing unit in FIG. 54 is used. It is a block diagram which shows the structure of the brightness | luminance estimation process part of FIG. FIG. 57 is a block diagram showing a configuration of first and second intensity estimation units in FIG. 54. FIG. 58 is a diagram for describing a first color and a second color selected by the rough interpolation processing unit in FIG. 57. FIG. 58 is a diagram for describing a first color and a second color selected by the rough interpolation processing unit in FIG. 57. It is a figure for demonstrating a weighting value. It is a block diagram which shows the structure of the RGBE intensity | strength estimation part of FIG. 10 is a flowchart for explaining demosaic processing 2; It is a flowchart for demonstrating a brightness | luminance estimation process. 10 is a flowchart for explaining rough interpolation processing 3; 10 is a flowchart for explaining coarse interpolation processing 4; 10 is a flowchart for explaining color intensity estimation processing 2; 10 is a flowchart for explaining rough interpolation processing 5; FIG. 55 is a diagram for describing a different example of a color filter used in the CCD image sensor in FIG. 2 when the demosaic processing unit in FIG. 54 is used. It is a block diagram which shows the structure of a personal computer.

Explanation of symbols

  201 digital still camera, 216 DSP block, 242 signal processing processor, 253 demosaic processing unit, 281 local region extraction unit, 282 G + E intensity estimation processing unit, 283 RGBE intensity estimation unit, 291 G intensity estimation unit, 292 E intensity estimation unit , 293 addition processing unit, 301 target pixel selection unit, 305 texture intensity determination unit 306 switch, 311 first intensity estimation unit, 321 second intensity estimation unit, 322 third intensity estimation unit, 323 fourth intensity Estimator, 324 switch, 341 coarse interpolation processor, 342 statistic calculator, 343 regression calculation processor, 361 average value calculator, 362 variance value calculator, 363 average value calculator, 364 covariance calculator, 381 slope A calculation unit, a 382 pixel intensity estimation unit, 471 reliability calculation unit, 472 rough interpolation processing unit, 473 regression calculation processing unit, 481 high frequency extraction unit, 482 addition processing unit, 483 clip processing unit, 501 502 slope calculation unit, 503 tilt synthesis unit, 504 pixel intensity estimation unit , 551 G + E intensity estimation processing unit, 562 switch, 571 G + E intensity estimation processing unit, 581 switch, 601 demosaic processing unit, 611 luminance estimation processing unit, 612 RGBE intensity estimation unit, 621 first intensity estimation unit, 622 second Strength estimation unit, 623 switch, 624 switch, 631 coarse interpolation processing unit, 641 coarse interpolation processing unit

Claims (32)

In an image processing apparatus that captures and processes an image,
Image acquisition means for acquiring a mosaic image by an image sensor having at least four types of filters having different spectral sensitivities, and any of the plurality of types of filters being used for each pixel;
Image processing means for generating, from the mosaic image acquired by the image acquisition means, a color image in which intensity information for each pixel position corresponding to a plurality of colors determined by the spectral sensitivity of the filter is aligned in all pixels; Prepared,
The color filter array of the image sensor of the image acquisition means has a spectral characteristic in which a filter of pixels arranged in a checkered pattern in half of all pixels has a strong correlation with a spectral characteristic of luminance,
The image processing means includes
An image processing apparatus, wherein an estimated intensity value of a color of pixels arranged in a checkered pattern is calculated for each pixel, and a luminance value of each pixel is calculated based on the estimated intensity value. The color filter array of the image sensor of the image acquisition means is
A first filter corresponding to a first color having a spectral characteristic having a strong correlation with a spectral characteristic of luminance among the plurality of types of filters is disposed every other line in the horizontal and vertical directions,
The second filter corresponding to the second color different from the first color and having a spectral characteristic having a strong correlation with the spectral characteristic of luminance is arranged every other line in the horizontal and vertical directions, and the first filter Placed on a different line from
The image processing means includes
First calculating means for calculating a first intensity estimated value that is an intensity estimated value of the first color at a target pixel position in the mosaic image;
Second calculating means for calculating a second intensity estimated value that is an intensity estimated value of the second color at the target pixel position;
And combining means for combining the first intensity estimated value calculated by the first calculating means and the second intensity estimated value calculated by the second calculating means. The image processing apparatus according to claim 1. The first calculation means includes
An intensity value corresponding to a known color already obtained at the target pixel position;
Third calculation means for calculating an intensity estimation value corresponding to the target color at the target pixel position based on the correlation between the known color calculated from pixels near the target pixel and the target color to be estimated Including
If the pixel of interest is not the first color,
3. The first intensity estimation value is calculated by the third calculation unit using the filter color at the target pixel position as the known color and the first color as the target color. An image processing apparatus according to 1. The third calculation means includes:
Generating means for generating a plurality of sets of pixel intensities of pixels corresponding to the known color and pixel intensities of pixels corresponding to the target color in the vicinity of the target pixel;
From the plurality of sets of the pixel intensity of the pixel corresponding to the known color and the pixel intensity of the pixel corresponding to the target color generated by the generation unit, the centroid and inclination of the color distribution between the known color and the target color are calculated. A fourth calculating means for calculating;
Intensity estimation corresponding to the target color at the target pixel position based on the center of gravity and inclination of the color distribution calculated by the fourth calculation means and the intensity value corresponding to the known color at the target pixel position The image processing apparatus according to claim 3, further comprising: a fifth calculation unit that calculates a value. The image processing apparatus according to claim 4, wherein the fifth calculation unit calculates an intensity estimation value corresponding to the target color at the target pixel position using regression calculation. The second calculation means includes:
An intensity value corresponding to a known color already obtained at the target pixel position;
Third calculation means for calculating an intensity estimation value corresponding to the target color at the target pixel position based on the correlation between the known color calculated from pixels near the target pixel and the target color to be estimated Including
If the pixel of interest is not the second color,
3. The second intensity estimation value is calculated by the third calculation means, with the filter color at the target pixel position as the known color and the second color as the target color. An image processing apparatus according to 1. The third calculation means includes:
Generating means for generating a plurality of sets of pixel intensities of pixels corresponding to the known color and pixel intensities of pixels corresponding to the target color in the vicinity of the target pixel;
From the plurality of sets of the pixel intensity of the pixel corresponding to the known color and the pixel intensity of the pixel corresponding to the target color generated by the generation unit, the centroid and inclination of the color distribution between the known color and the target color A fourth calculating means for calculating;
Intensity estimation corresponding to the target color at the target pixel position based on the center of gravity and inclination of the color distribution calculated by the fourth calculation means and the intensity value corresponding to the known color at the target pixel position The image processing apparatus according to claim 6, further comprising: a fifth calculation unit that calculates a value. The image processing apparatus according to claim 7, wherein the fifth calculation unit calculates an intensity estimation value corresponding to the target color at the target pixel position using regression calculation. The first calculation means includes
An intensity value corresponding to a known color already obtained at the target pixel position;
Third calculation means for calculating an intensity estimation value corresponding to the target color at the target pixel position based on the correlation between the known color calculated from pixels near the target pixel and the target color to be estimated 2 or more
If the pixel of interest is not the first color,
The first third calculating means includes:
An intensity value corresponding to the second color at the target pixel position is calculated using the filter color at the target pixel position as the known color and the second color as the target color,
The second third calculating means includes:
The image processing apparatus according to claim 2, wherein the first intensity estimation value is calculated using the second color as the known color and the first color as the target color. The third calculation means includes:
Generating means for generating a plurality of sets of pixel intensities of pixels corresponding to the known color and pixel intensities of pixels corresponding to the target color;
From the plurality of sets of the pixel intensity of the pixel corresponding to the known color and the pixel intensity of the pixel corresponding to the target color generated by the generation unit, the centroid and inclination of the color distribution between the known color and the target color A fourth calculating means for calculating;
Intensity estimation corresponding to the target color at the target pixel position based on the center of gravity and inclination of the color distribution calculated by the fourth calculation means and the intensity value corresponding to the known color at the target pixel position The image processing apparatus according to claim 9, further comprising: a fifth calculation unit that calculates a value. The image processing apparatus according to claim 10, wherein the fifth calculation unit calculates an intensity estimation value corresponding to the target color at the target pixel position using regression calculation. The second calculation means includes:
An intensity value corresponding to a known color already obtained at the target pixel position;
Third calculation means for calculating an intensity estimation value corresponding to the target color at the target pixel position based on the correlation between the known color calculated from pixels near the target pixel and the target color to be estimated 2 or more
If the pixel of interest is not the second color,
The first third calculating means includes:
An intensity value corresponding to the first color at the target pixel position is calculated using the filter color at the target pixel position as the known color and the first color as the target color,
The second third calculating means includes:
The image processing apparatus according to claim 2, wherein the second intensity estimation value is calculated using the first color as the known color and the second color as the target color. The third calculation means includes:
Generating means for generating a plurality of sets of pixel intensities of pixels corresponding to the known color and pixel intensities of pixels corresponding to the target color;
From the plurality of sets of the pixel intensity of the pixel corresponding to the known color and the pixel intensity of the pixel corresponding to the target color generated by the generation unit, the centroid and inclination of the color distribution between the known color and the target color are calculated. A fourth calculating means for calculating;
Intensity estimation corresponding to the target color at the target pixel position based on the center of gravity and inclination of the color distribution calculated by the fourth calculation means and the intensity value corresponding to the known color at the target pixel position The image processing apparatus according to claim 12, further comprising: a fifth calculation unit that calculates a value. The image processing apparatus according to claim 13, wherein the fifth calculation unit calculates an intensity estimation value corresponding to the target color at the target pixel position using regression calculation. The first calculation means includes
First intensity estimating means for calculating a first estimated value of the first intensity estimated value;
Second intensity estimating means for calculating a second estimated value of the first intensity estimated value;
Discrimination means for discriminating a texture direction in the vicinity of the target pixel;
Of the first estimated value estimated by the first intensity estimating means and the second estimated value estimated by the second intensity estimating means based on the discrimination result by the discriminating means, suitable Selecting means for selecting the estimated value,
The first intensity estimating means includes
An intensity value corresponding to a known color already obtained at the target pixel position;
Third calculation means for calculating an intensity estimation value corresponding to the target color at the target pixel position based on the correlation between the known color calculated from pixels near the target pixel and the target color to be estimated Including
If the pixel of interest is not the first color,
The first estimated value of the first intensity estimated value is calculated by the third calculating means using the color of the filter at the target pixel position as the known color and the first color as the target color,
The second intensity estimating means includes
Including two or more third calculation means,
If the pixel of interest is not the first color,
The first third calculating means uses the filter color at the target pixel position as the known color, the second color as the target color, and the intensity corresponding to the second color at the target pixel position. Calculate the value,
The second third calculating means calculates the second estimated value of the first intensity estimated value using the second color as the known color and the first color as the target color. ,
The selection unit is based on the determination result by the determination unit and whether the filter corresponding to the first color is positioned in the horizontal direction or the vertical direction with respect to the target pixel position. Selecting a suitable estimated value from among the first estimated value estimated by the first intensity estimating means and the second estimated value estimated by the second intensity estimating means. The image processing apparatus according to claim 2. The second calculation means includes:
First intensity estimating means for calculating a first estimated value of the second intensity estimated value;
Second intensity estimating means for calculating a second estimated value of the second intensity estimated value;
Discrimination means for discriminating a texture direction in the vicinity of the target pixel;
Of the first estimated value estimated by the first intensity estimating means and the second estimated value estimated by the second intensity estimating means based on the discrimination result by the discriminating means, suitable Selecting means for selecting the estimated value,
The first intensity estimating means includes
An intensity value corresponding to a known color already obtained at the target pixel position;
Third calculation means for calculating an intensity estimation value corresponding to the target color at the target pixel position based on the correlation between the known color calculated from pixels near the target pixel and the target color to be estimated Including
If the pixel of interest is not the second color,
The first estimated value of the second intensity estimated value is calculated by the third calculating means with the filter color at the target pixel position as the known color and the second color as the target color,
The second intensity estimating means includes
Including two or more third calculation means,
If the pixel of interest is not the second color,
The first third calculating means uses the filter color at the target pixel position as the known color, the first color as the target color, and the intensity corresponding to the first color at the target pixel position. Calculate the value,
The second third calculating means calculates the second estimated value of the second intensity estimated value using the first color as the known color and the second color as the target color. ,
The selection unit is based on the determination result by the determination unit and whether the filter corresponding to the second color is positioned in the horizontal direction or the vertical direction with respect to the target pixel position. Selecting a suitable estimated value from among the first estimated value estimated by the first intensity estimating means and the second estimated value estimated by the second intensity estimating means. The image processing apparatus according to claim 2. The filter arrangement of the plurality of types of the filters of the image sensor of the image acquisition means is:
A third color filter different from any of the first color and the second color of the filters is horizontally and vertically spaced every other line, and is different from the first filter. Placed in line,
A filter of a fourth color different from any of the first color, the second color, and the third color of the filters is arranged every other line in the horizontal and vertical directions, and The image processing apparatus according to claim 2, wherein the image processing apparatus is arranged on a horizontal line different from the second filter. The filter arrangement of the plurality of types of the filters of the image sensor of the image acquisition means is:
A third color and a fourth color filter different from any of the first color and the second color of the filters are arranged every other line in the horizontal and vertical directions, and the first color Alternatingly arranged on a horizontal line different from the filter,
The filters of the fifth color and the sixth color different from any of the first color, the second color, the third color, and the fourth color of the filters are horizontal. The image processing apparatus according to claim 2, wherein the image processing apparatus is alternately arranged every other line vertically and on a horizontal line different from the second filter. The third color and the fourth color have spectral characteristics that are highly correlated with each other,
The image processing apparatus according to claim 18, wherein the fifth color and the sixth color have spectral characteristics having a high correlation with each other. The image processing apparatus according to claim 18, wherein the third color and the fourth color are the same color. The image processing apparatus according to claim 18, wherein the fifth color and the sixth color are the same color. The color filter array of the image sensor of the image acquisition means is
A first filter corresponding to a first color having spectral characteristics having a strong correlation with spectral characteristics of luminance among the plurality of types of filters is arranged in a checkered pattern,
The image processing means includes
An intensity value corresponding to a known color already obtained at the target pixel position;
First calculation means for calculating an intensity estimation value corresponding to the target color at the target pixel position based on the correlation between the known color calculated from pixels near the target pixel and the target color to be estimated Including
2. The first intensity estimation value is calculated by the first calculation unit with the filter color at the target pixel position as the known color and the first color as the target color. An image processing apparatus according to 1. The first calculation means includes
Generating means for generating a plurality of sets of pixel intensities of pixels corresponding to the known color and pixel intensities of pixels corresponding to the target color;
From the plurality of sets of the pixel intensity of the pixel corresponding to the known color and the pixel intensity of the pixel corresponding to the target color generated by the generation unit, the centroid and inclination of the color distribution between the known color and the target color are calculated. A second calculating means for calculating;
Intensity estimation corresponding to the target color at the target pixel position based on the center of gravity and inclination of the color distribution calculated by the second calculation means and the intensity value corresponding to the known color at the target pixel position The image processing apparatus according to claim 24, further comprising: a third calculation unit that calculates a value. The image processing apparatus according to claim 23, wherein the second calculation unit calculates an intensity estimation value corresponding to the target color at the target pixel position using regression calculation. The filter arrangement of the plurality of types of the filters of the image sensor of the image acquisition means is:
A second color filter different from the first color among the filters is arranged every other line in the horizontal and vertical directions;
At the filter position where the filters of the first color and the second color of the filters are not arranged, it is different from any of the first color and the second color of the filters. The image processing apparatus according to claim 22, wherein the filters of the third color and the fourth color are arranged so as to form an oblique lattice pattern every other pixel in the oblique direction. The second color is a filter color having a spectral characteristic that has sensitivity on a longer wavelength side than the first color,
26. At least one of the third color and the second color is a filter color having a spectral characteristic that has sensitivity on a shorter wavelength side than the first color. Image processing apparatus. The filter arrangement of the plurality of types of the filters of the image sensor of the image acquisition means is:
Of the filters, the second color, third color, fourth color, and fifth color filters different from the first color are arranged every two lines horizontally and vertically and in an oblique direction. The second color and the third color, and the fourth color and the fifth color are arranged so as to be positioned on different horizontal and vertical lines. The image processing apparatus according to claim 22. The second color and the third color have spectral characteristics that are highly correlated with each other,
The image processing apparatus according to claim 27, wherein the fourth color and the fifth color have spectral characteristics having high correlation with each other. The image processing apparatus according to claim 27, wherein the second color and the third color are the same color. The image processing apparatus according to claim 27, wherein the fourth color and the fifth color are the same color. In an image processing method of an image processing apparatus that captures and processes an image,
An acquisition step of acquiring a mosaic image by an image sensor having at least four types of filters having different spectral sensitivities, and any one of the plurality of types of filters is used for each pixel;
An image processing step of generating, from the mosaic image acquired by the processing of the acquisition step, a color image in which intensity information for each pixel position corresponding to a plurality of colors determined by the spectral sensitivity of the filter is aligned in all pixels; Including
In the process of the acquisition step,
A mosaic image having a color arrangement such that the filter of the pixels arranged in a checkered pattern in half of all the pixels has a spectral characteristic having a strong correlation with the spectral characteristic of luminance is acquired,
The image processing step includes
Image processing comprising: calculating a color intensity estimation value of pixels arranged in the checkered pattern at each pixel and calculating a luminance value of each pixel based on the intensity estimation value Method. A program for causing a computer to execute processing for capturing an image and executing processing of the captured image,
An acquisition step of acquiring a mosaic image by an image sensor having at least four types of filters having different spectral sensitivities, and any one of the plurality of types of filters is used for each pixel;
An image processing step of generating, from the mosaic image acquired by the processing of the acquisition step, a color image in which intensity information for each pixel position corresponding to a plurality of colors determined by the spectral sensitivity of the filter is aligned in all pixels; Including
In the process of the acquisition step,
A mosaic image having a color arrangement such that the filter of the pixels arranged in a checkered pattern in half of all the pixels has a spectral characteristic having a strong correlation with the spectral characteristic of luminance is acquired,
The image processing step includes
Processing including calculating a color intensity estimation value of pixels arranged in the checkered pattern in each pixel and calculating a luminance value of each pixel based on the intensity estimation value. A program to be executed by a computer.

Images