JP4292950B2 - Image processing apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program, and more particularly, to an image processing apparatus and an image processing method capable of performing color component interpolation processing on all pixels while suppressing generation of false colors and reduction in resolution. , As well as programs.

  When photographing with a photographing device using a single plate type fixed imaging element, only a single spectral sensitivity can be obtained. Therefore, in general, a method is used in which different color filters are attached to a large number of pixels and arranged in a specific pattern to perform color photographing. Since an image captured by this method can obtain only one color for each pixel, an image having a mosaic shape with respect to the color is generated. Therefore, by performing interpolation using color information obtained from surrounding pixels, it is possible to generate an image in which all colors (for example, RGB) are aligned in all pixels.

  In recent years, in such an imaging device, as the number of pixels of the solid-state imaging device has increased dramatically, the processing time for charge transfer and pixel interpolation has increased, making it difficult to obtain moving images by continuously shooting at high speed. It has become to.

Therefore, in order to solve this problem, methods for thinning out pixels, that is, imaging with a low resolution have been studied. The principle of thinning is that all the colors of the filter are aligned in the local area even after thinning. If the image sensor is a CCD (Charge Coupled Device), thinning is performed for each horizontal line due to the convenience of the circuit. It is mostly done. For example, Patent Document 1 shows an example of a thinning method in a color array called a Bayer array as shown in FIG. 1, and Patent Document 2 shows a G stripe as shown in FIG. An example of an RB complete checkered filter array and a soot RGB stripe color filter array as shown in FIG. 3 is shown. Further, for example, Patent Document 3 discloses a method of performing thinning with two adjacent lines as a set.
JP 2001-86520 A Japanese Patent Laid-Open No. 7-312714 JP 2001-86520 A

  However, in the method of thinning out adjacent lines as a set as described above, since all color information is obtained from the adjacent lines, false colors are not easily generated, but spatial information is not obtained much. Referring to FIG. 4, when sampling the striped pattern drawn on the right side by thinning out the filter array on the left side to 1/4, if a G signal is obtained in the adjacent first and second rows, Although only fringes can be detected, if the first and fourth rows that are not adjacent to each other are used, the fringes a and b can be detected. In the case where false colors are generated by using non-adjacent lines, in the conventional technology, processing for limiting the band of color components and reducing false colors has been performed. However, it is difficult to specify the generation position again after the false color is generated. However, if all pixels are processed, there is a problem that the saturation of the entire interpolated image is lowered.

  Further, the filter of the single-plate solid-state image sensor is not limited to three colors, and there are cases where four colors are used. A well-known one is a so-called complementary color filter array, and the primary color filter array further improves the three colors of RGB as disclosed in Patent Document 4 with the aim of improving color reproducibility. By adding another color (hereinafter referred to as E), an array of four colors can be obtained. When thinning is performed in such a four-color arrangement, the same problem as in the Bayer arrangement occurs. Even in the Bayer array, the difference in spectral characteristics caused by the difference in the arrangement of R and B adjacent to G and the positional deviation of the filter in the manufacturing process as mentioned in Patent Document 5 and Patent Document 6 Or two gains in a 2 × 2 filter array due to a gain error between lines generated in a solid-state imaging device having different signal output paths in even lines and odd lines as disclosed in Patent Document 7. G does not necessarily have the same characteristics, and may actually be an array of four colors rather than an array of three colors. For any reason, if the number of colors of the filter increases and the number of samplings for one color decreases, the problem of false color generation due to thinning becomes more serious.

The present invention has been made in view of such a situation, and using a mosaic signal of a color imaged by thinning out in a four-color filter array, while suppressing occurrence of false colors and a decrease in resolution, all pixels In addition, the color component interpolation process can be performed.
JP 2002-271804 A JP 7-135302 A JP-A-8-191141 Japanese Patent Laid-Open No. 4-154281

  An image processing apparatus according to the present invention is an image processing apparatus that generates a color image in which each pixel has a plurality of color components based on a mosaic image captured using a single-plate color image sensor, and the first color The component is arranged on the first horizontal line, the second color component is arranged on the second horizontal line, and any one of the first color component and the second color component is included in all the horizontal lines. A first interpolation value that is an interpolation value of the first color component in a pixel that does not have the first color component is calculated using only pixels in the first horizontal line that are arranged and in the first horizontal line. In the second horizontal line, only the pixels in the second horizontal line are used to calculate a second interpolation value that is an interpolation value of the second color component in a pixel having no second color component. Interpolation value calculation means and pixels near the pixel of interest in the mosaic image. Based on the first interpolation value and the first color component, and the second interpolation value and the second color component in the pixel in the vicinity of the target pixel, the interpolation values of the first all colors in the target pixel are calculated. A second interpolation value calculating means, a generating means for generating a third color component composed of a first color component and a second color component in each pixel based on the pixel value of the mosaic image, A third interpolation value calculating means for calculating an interpolation value for all the second colors in the target pixel based on the third color component in the neighboring pixel and the pixel value of the pixel in the vicinity of the target pixel; And a fourth interpolation value calculating means for calculating the interpolation values of the third all colors by synthesizing the interpolation values of all the colors and the interpolation values of the second all colors.

  The fourth interpolation value calculation means is based on the saturation calculation means for calculating the saturation based on the interpolation values of the first all colors, the interpolation values of the first all colors and the interpolation values of the second all colors. A chromaticity difference calculating means for calculating a chromaticity difference, and a blend ratio calculation for calculating a blend ratio based on the saturation calculated by the saturation calculating means and the chromaticity difference calculated by the chromaticity difference calculating means. And means for synthesizing the interpolated values of the first all colors and the interpolated values of the second all colors using the blend ratio.

  The first color component and the second color component can have spectral sensitivity highly correlated with each other.

  Before calculating the first interpolation value and the second interpolation value, it is possible to further include adjustment means for adjusting the white balance for the pixels of the mosaic image.

  The image processing method of the present invention is an image processing method of an image processing apparatus that generates a color image in which each pixel has a plurality of color components based on a mosaic image captured using a single-plate color image sensor, The first color component is arranged on the first horizontal line, the second color component is arranged on the second horizontal line, and all the horizontal lines have the first color component and the second color component. The first interpolation which is an interpolation value of the first color component in the pixel having no first color component using only the pixels in the first horizontal line in the first horizontal line. A first interpolation value calculation step for calculating a value, and an interpolation value of the second color component in a pixel having no second color component by using only pixels in the second horizontal line in the second horizontal line The second interpolation value calculation step for calculating the second interpolation value is And the first interpolation value and the first color component in the pixel in the vicinity of the target pixel of the mosaic image, and the second interpolation value and the second color component in the pixel in the vicinity of the target pixel. A third interpolation value calculating step for calculating the interpolation values of all the first colors in the pixel, and a third interpolation unit including a first color component and a second color component in each pixel based on the pixel value of the mosaic image. Based on the generation step for generating the color component, the third color component in the pixel in the vicinity of the target pixel, and the pixel value of the pixel in the vicinity of the target pixel, the interpolation values of the second all colors in the target pixel are calculated. A fourth interpolation value calculating step; a fifth interpolation value calculating step for calculating an interpolation value for the third all color by synthesizing the interpolation value for the first all color and the interpolation value for the second all color; It is characterized by including.

  The program of the present invention is a program for generating a color image in which each pixel has a plurality of color components based on a mosaic image captured using a single-plate color imaging device, and the first color component is the first color component. And the second color component is arranged on the second horizontal line, and any one of the first color component and the second color component is arranged on all the horizontal lines. 1st interpolation which calculates the 1st interpolation value which is the interpolation value of the 1st color component in the pixel which does not have the 1st color component, using only the pixel in the 1st horizontal line in 1 horizontal line In the value calculating step, in the second horizontal line, only the pixels in the second horizontal line are used, and the second interpolation value that is the interpolation value of the second color component in the pixel having no second color component is obtained. A second interpolation value calculating step to calculate, and a mosaic image Based on the first interpolation value and the first color component in the pixel in the vicinity of the target pixel, and the second interpolation value and the second color component in the pixel in the vicinity of the target pixel. Based on the third interpolation value calculation step for calculating the interpolation values of all colors and the pixel value of the mosaic image, a third color component composed of the first color component and the second color component in each pixel is generated. Based on the generation step, the third color component in the pixel in the vicinity of the target pixel, and the pixel value in the pixel in the vicinity of the target pixel, the fourth interpolation value for calculating the interpolation values of the second all colors in the target pixel A computer including a calculation step and a fifth interpolation value calculation step of combining the interpolation values of the first all colors and the interpolation values of the second all colors to calculate the interpolation values of the third all colors. It is made to perform.

  In the present invention, the first color component is arranged on the first horizontal line, the second color component is arranged on the second horizontal line, and all the horizontal lines have the first color component and Any one of the second color components is arranged, and in the first horizontal line, only the pixels in the first horizontal line are used, and the interpolation value of the first color component in the pixel having no first color component is used. A first interpolation value is calculated, and in the second horizontal line, only the pixels in the second horizontal line are used, and the interpolation value of the second color component in the pixel having no second color component is used. 2 interpolation values are calculated, the first interpolation value and the first color component in the pixel near the target pixel of the mosaic image, and the second interpolation value and the second color component in the pixel near the target pixel Based on the above, the interpolation values of the first all colors in the target pixel are calculated. Based on the pixel value of the mosaic image, a third color component including the first color component and the second color component in each pixel is generated, and the third color component in the pixel near the target pixel and the target pixel are generated. Based on the pixel values of the pixels in the vicinity of the second pixel, the interpolation values of the second all colors in the pixel of interest are calculated, and the first all color interpolation values and the second all color interpolation values are synthesized to obtain the third Interpolated values of all the colors are calculated.

  According to the present invention, it is possible to interpolate color components for all pixels while suppressing generation of false colors and reduction in resolution. In particular, according to the present invention, it is possible to accurately determine the occurrence position of a false color. In addition, high-quality color components can be interpolated. Furthermore, a more natural interpolation image can be obtained.

  Embodiments of the present invention will be described below. The correspondence relationship between the invention described in this specification and the embodiments of the invention is exemplified as follows. This description is intended to confirm that the embodiments supporting the invention described in this specification are described in this specification. Therefore, even if there is an embodiment that is described in the embodiment of the invention but is not described here as corresponding to the invention, the fact that the embodiment is not It does not mean that it does not correspond to the invention. Conversely, even if an embodiment is described herein as corresponding to an invention, that means that the embodiment does not correspond to an invention other than the invention. Absent.

  Further, this description does not mean all the inventions described in this specification. In other words, this description is for the invention described in the present specification, which is not claimed in this application, that is, for the invention that will be applied for in the future or that will appear and be added by amendment. It does not deny existence.

  According to the present invention, an image processing apparatus is provided. In this image processing apparatus (for example, the digital camera 201 in FIG. 5), each pixel has a plurality of colors based on a mosaic image captured using a single-plate color image sensor (for example, the CCD image sensor 213 in FIG. 5). An image processing apparatus that generates a color image having components, wherein a first color component (for example, G) is arranged on a first horizontal line, and a second color component (for example, E) is second. The first color component and the second color component are arranged in all the horizontal lines, and only the pixels in the first horizontal line are used in the first horizontal line. Thus, the first interpolation value (for example, the processing of step S108 in FIG. 12) that is the interpolation value of the first color component (for example, G) in the pixel that has no first color component (for example, B) is calculated. L (j, k) (However, (j, k) Processing pixel position)), and in the second horizontal line, only the pixels in the second horizontal line are used, and the interpolation of the second color component in the pixel having no second color component (for example, R) is performed. First interpolation value calculation means (for example, G and E interpolation processes in FIG. 8) for calculating a second interpolation value (for example, L (j, k) calculated in the process of step S104 in FIG. 12). 262), the first interpolation value and the first color component in the pixel in the vicinity of the target pixel of the mosaic image, and the second interpolation value and the second color component in the pixel in the vicinity of the target pixel. Second interpolation value calculation means (all colors in FIG. 8) for calculating the interpolation values of the first all colors in the target pixel (for example, RF, GF, BF, EF calculated in step S154 in FIG. 14) Based on interpolation processing unit 264) and pixel values of the mosaic image Generating means for generating a third color component (for example, G + E) composed of a first color component and a second color component in each pixel, a third color component in a pixel in the vicinity of the target pixel, and a target Based on the pixel values of the pixels in the vicinity of the pixel, the third interpolation value for the second all colors in the target pixel (for example, RU, GU, BU, EU calculated in the process of step S507 in FIG. 23) is calculated. The interpolation value calculation means (all color interpolation processing unit 265 in FIG. 8) and the interpolation values of the first all colors and the interpolation values of the second all colors are synthesized to obtain the interpolation values of the third all colors (for example, , Fourth interpolation value calculation means (for example, the synthesis processing unit 266 in FIG. 8) for calculating C ″) output in step S559 in FIG.

  The fourth interpolation value calculating means calculates saturation (for example, saturation S) based on the interpolation values of the first all colors (for example, saturation calculation processing unit 302 in FIG. 9). And a chromaticity difference calculating means (for example, FIG. 9) that calculates a chromaticity difference (for example, chromaticity difference ΔCc) based on the interpolation values of the first all colors and the second all colors. A blend ratio (for example, r) is calculated based on the chromaticity difference calculation processing unit 304) and the saturation calculated by the saturation calculation unit and the chromaticity difference calculated by the chromaticity difference calculation unit. A blend ratio calculating unit (for example, blend ratio calculation processing unit 303 in FIG. 9), and a composition for combining the interpolation values of the first all colors and the interpolation values of the second all colors using the blend ratio. A means (for example, blend processing unit 305 in FIG. 9). .

  The first color component and the second color component can have spectral sensitivity highly correlated with each other.

  Before calculating the first interpolation value and the second interpolation value, adjustment means (for example, the white balance adjustment unit 251 in FIG. 7) for adjusting the white balance for the pixels of the mosaic image is further provided. be able to.

  According to the present invention, an image processing method is provided. This image processing method is an image processing apparatus (for example, the digital camera 201 in FIG. 5) that generates a color image in which each pixel has a plurality of color components based on a mosaic image captured using a single-plate color image sensor. The first color component (for example, G) is disposed on the first horizontal line, and the second color component (for example, E) is disposed on the second horizontal line. All the horizontal lines are provided with either the first color component or the second color component, and in the first horizontal line, using only the pixels in the first horizontal line, A first interpolation value (for example, L calculated in the process of step S108 in FIG. 12) that is an interpolation value of the first color component (for example, G) in a pixel that has no first color component (for example, B). (J, k)) to calculate the first interpolation value In the calculation step (for example, step S108 in FIG. 12), and in the second horizontal line, only the pixel in the second horizontal line is used and the pixel without the second color component (for example, R) is used. A second interpolation value calculating step (for example, calculating a second interpolation value (for example, L (j, k) calculated in the process of step S104 in FIG. 12)) that is the interpolation value of the second color component. Step S104 in FIG. 12, the first interpolation value and the first color component in the pixel in the vicinity of the target pixel of the mosaic image, the second interpolation value in the pixel in the vicinity of the target pixel and the Based on the second color component, a third interpolation that calculates the interpolation values (for example, RF, GF, BF, EF calculated in the process of step S154 in FIG. 14) of the first all colors in the pixel of interest. Value calculation Based on the step (for example, step S54 in FIG. 11) and the pixel value of the mosaic image, a third color component (for example, G + E) composed of the first color component and the second color component in each pixel. Based on the generation step (for example, step S55 in FIG. 11), the third color component (for example, G + E) in the pixel in the vicinity of the target pixel, and the pixel value of the pixel in the vicinity of the target pixel. A fourth interpolation value calculating step (for example, FIG. 11) for calculating the interpolation values (for example, RU, GU, BU, EU calculated in the process of step S507 in FIG. 23) of the second all colors in the target pixel. Step S56) and the interpolation values of the first all colors and the second all colors are synthesized and output in the third all colors (for example, step S559 in FIG. 24). C '') Fifth interpolation value calculation step of (e.g., step S57 of FIG. 11) and a.

  According to the present invention, a program is provided. This program is a program for generating a color image in which each pixel has a plurality of color components based on a mosaic image captured using a single-plate color image sensor, and the first color component (for example, G) Are arranged on the first horizontal line, and a second color component (for example, E) is arranged on the second horizontal line, and all the horizontal lines include the first color component and the second color component. Any one of the color components is arranged, and in the first horizontal line, only the pixels in the first horizontal line are used, and the first in the pixel (for example, B) without the first color component is used. A first interpolation value calculating step (for example, a first interpolation value (for example, L (j, k) calculated in the process of step S108 in FIG. 12)) that is an interpolation value of a color component (for example, G). , Step S108 in FIG. In the second horizontal line, only the pixels in the second horizontal line are used, and the second color component is an interpolation value of the second color component in a pixel having no second color component (for example, R). A second interpolation value calculation step (for example, step S104 in FIG. 12) for calculating an interpolation value (for example, L (j, k) calculated in the process of step S104 in FIG. 12), and the target pixel of the mosaic image Based on the first interpolation value and the first color component in the pixel in the vicinity of the pixel, and the second interpolation value and the second color component in the pixel in the vicinity of the pixel of interest. A third interpolation value calculation step (for example, step S54 in FIG. 11) for calculating interpolation values for all the colors of one (for example, RF, GF, BF, and EF calculated in the process of step S154 in FIG. 14); Mosai A generating step (for example, step S55 in FIG. 11) that generates a third color component (for example, G + E) composed of the first color component and the second color component in each pixel based on the pixel value of the image. Based on the third color component (for example, G + E) in the pixel in the vicinity of the target pixel and the pixel value of the pixel in the vicinity of the target pixel. For example, a fourth interpolation value calculation step (for example, step S56 in FIG. 11) for calculating RU, GU, BU, EU calculated in the process of step S507 in FIG. 23, and interpolation of the first all colors A fifth interpolation value calculation step of calculating the interpolation value of the third all colors (for example, C ″ output in step S559 of FIG. 24) by combining the values and the interpolation values of the second all colors (For example, step of FIG. 57) and causes the computer to execute.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 5 is a block diagram showing a configuration of a digital camera 201 to which the present invention is applied.

  As shown in FIG. 5, a digital camera 201 includes a lens 211, a diaphragm 212, a CCD (Charge Coupled Devices) image sensor 213, a correlated double sampling (CDS) circuit 214, an A / D converter 215, and a DSP (DSP). Digital Signal Processor) block 216, timing generator 217, D / A converter 218, video encoder 219, display unit 220, CODEC (CODEC: COmpression / DECompression) processing unit 221, memory 222, CPU (Central Processing Unit) 223, and The operation input unit 224 is configured.

  The CCD is a semiconductor element that converts light information into an electrical signal (photoelectric conversion). The CCD image sensor 213 has a plurality of light receiving elements (pixels) that convert light into electricity, and changes in light for each pixel. It converts to an electric signal independently. The correlated double sampling circuit 214 samples the reset noise, which is the main component of the noise included in the output signal of the CCD image sensor 213, from the sampled video signal period of the output pixel signals, and the reference period. Is a circuit that removes the signal by subtracting the sampled signal. The A / D converter 215 converts the supplied analog signal after noise removal into a digital signal.

  The DSP block 216 is a block having a signal processing processor and an image RAM (Random Access Memory), and the signal processing processor performs pre-programmed image processing on image data stored in the image RAM, or It performs image processing configured as arithmetic processing by hardware. The timing generator 217 is a logic circuit that generates various horizontal and vertical driving pulses necessary for driving the CCD and pulses used in the analog front processing in synchronization with the reference clock. The timing clock generated by the timing generator 217 is also supplied to the codec processing unit 221, the memory 222, and the CPU 223 via the bus 225.

  The D / A converter 218 converts the supplied digital signal into an analog signal and outputs the analog signal. The video encoder 219 encodes the supplied analog signal into video data in a format that can be displayed on the display unit 220. The display unit 220 is configured by, for example, an LCD (Liquid Crystal Display) and displays the video signal supplied from the video encoder 219.

  The codec processing unit 221 performs processing by digital video data compression or expansion algorithm such as MPEG (Moving Picture Coding Experts Group / Moving Picture Experts Group), for example. The memory 222 is configured by, for example, a semiconductor memory, a magnetic disk, a magneto-optical disk, or an optical disk, and stores supplied data or outputs stored data based on control of the CPU 223. . Note that the memory 222 may be detachable from the digital camera 201.

  The CPU 223 controls each unit of the digital camera 201 based on a user operation input supplied from the operation input unit 224 via the bus 225. The operation input unit 224 includes, for example, a jog dial, a key, a lever, a button, or a touch panel as well as a button for instructing recording, and receives an operation input by a user.

  Light input through the lens 211 and the diaphragm 212 is incident on the CCD image sensor 213, converted into an electrical signal by photoelectric conversion at the light receiving element, and supplied to the correlated double sampling circuit 214. The correlated double sampling circuit 214 removes noise by subtracting the sampled video signal period and the sampled reference period from the pixel signals output from the CCD image sensor 213, and A / D This is supplied to the converter 215. The A / D converter 215 converts the supplied analog signal after noise removal into a digital signal, and temporarily stores it in the image RAM of the DSP block 216.

  The timing generator 217 controls the CCD image sensor 213, the correlated double sampling circuit 214, the A / D converter 215, and the DSP block 216 so as to maintain image capture at a constant frame rate during imaging. .

  The DSP block 216 receives supply of pixel stream data at a constant rate, temporarily stores it in the image RAM, and executes image processing (to be described later) on the temporarily stored image data in the signal processing processor. When the image block is displayed on the display unit 220 after the image processing is completed, the DSP block 216 displays the code data in the D / A converter 218 and the codec processing unit 221 in the case where the image data is stored in the memory 222. Supply image data.

  The D / A converter 218 converts the digital image data supplied from the DSP block 216 into an analog signal and supplies the analog signal to the video encoder 219. The video encoder 219 converts the supplied analog image signal into a video signal, and outputs the video signal to the display unit 220 for display. That is, the display unit 220 plays the role of a camera finder in the digital camera 201. The codec processing unit 221 performs a predetermined encoding on the image data supplied from the DSP block 216 and supplies the encoded image data to the memory 222 for storage.

  Further, the codec processing unit 221 reads data designated by the user from among the data stored in the memory 222 based on the control of the CPU 223 that has received the user's operation input from the operation input unit 224, and performs predetermined decoding. The decoded signal is output to the DSP block 216. Thus, the decoded signal is supplied to the D / A converter 218 via the DSP block 216, converted into an analog signal, encoded by the video encoder 219, and displayed on the display unit 220.

  Incidentally, the on-chip color filter of the CCD image sensor 213 in FIG. 5 normally uses three or four types of colors, and these on-chip color filters have different colors alternately for each light receiving element. They are arranged in a mosaic. In the present invention, four colors obtained by adding a color (hereinafter referred to as E (Emerald)) that has a spectral characteristic close to G to three colors of RGB (Red, Green, Blue). An on-chip color filter with an array of is used. An example of this four-color mosaic arrangement is shown in FIG. In the present specification, the fourth color having spectral characteristics close to G is E, but this color is not limited to Emerald, and any color may be used as long as the spectral characteristics are close to G.

  As shown in FIG. 6, the four-color arrangement has one R filter that transmits only red (R) light, one G filter that transmits only green (G) light, and blue (B). One filter for transmitting only B light and one E filter for transmitting only light of emerald (E), a total of four is configured as a minimum unit. That is, the R, G, B, and E filters exist at the same density.

  As described above, when the four-color mosaic as shown in FIG. 6 is used for the on-chip color filter of the CCD image sensor 213, the image temporarily stored in the image RAM of the DSP block 216 is stored in each pixel. Both have only one color of R, G, B, and E. Therefore, the signal processing processor of the DSP block 216 processes this image by using an image processing program or hardware incorporated in advance, and generates image data having data of all colors in all pixels. Specifically, the signal processing processor of the DSP block 216 first generates a new color (hereinafter referred to as G + E) by combining G and E for all pixels, and G is set to all pixels in the three-color Bayer array. The interpolation method based on the premise of being aligned is also applicable to the mosaic arrangement of four colors.

  FIG. 7 is a block diagram showing a more detailed configuration of the DSP block 216 of FIG.

  As described above, the DSP block 216 includes the image RAM 241 and the signal processing processor 242, and the signal processing processor 242 includes the white balance adjustment unit 251, the gamma correction unit 252, and the demosaic processing unit 253. .

  The mosaic image converted into a digital signal by the A / D converter 215 is temporarily stored in the image RAM 241. The mosaic image is an intensity signal corresponding to any color of R, G, B, or E for each pixel, that is, an array determined by a color filter used in the CCD image sensor 213 (for example, using FIG. 6). This is composed of periodic pattern intensity signals of the four-color mosaic arrangement described.

  The white balance adjustment unit 251 performs processing (white balance adjustment processing) for applying an appropriate coefficient to the mosaic image according to the color of each pixel intensity so that the color balance of the achromatic subject region becomes an achromatic color. Or level balance adjustment processing). The gamma correction unit 252 performs gamma correction on each pixel intensity of the mosaic image whose white balance (level balance) has been adjusted. A numerical value “gamma (γ)” is used to represent the response characteristics of the gradation of the image. The gamma correction is a correction process for correctly displaying the brightness and color saturation of the image displayed on the display unit 220. The signal output to the display unit 220 reproduces the brightness and color of the image by applying a specific voltage for each pixel. However, due to the characteristics (gamma value) of the display unit 220, the brightness and color of the actually displayed image does not double the brightness of the CRT even if the input voltage is doubled (has non-linearity). For this reason, the gamma correction unit 252 performs a correction process so that the brightness and color saturation of the image displayed on the display unit 220 are displayed correctly.

The demosaic processing unit 253 performs a demosaic process for aligning the intensities of the four colors R, G, B, and E at each pixel position of the mosaic signal subjected to the gamma correction, and performs a band limitation on the matrix process and the chroma component. it is, to generate an image (Y image and the C image) corresponding to the color of the YC _b C _r. Thus, the output signal from the demosaic processing unit 253, image (the Y and C images) corresponding to the color of the YC _b C _r becomes a signal.

  In this example, an example using the four-color filter array shown in FIG. 6 will be described. The thinning method uses 8 lines as one set, and uses the 4th and 7th lines. A new color component G + E is generated by synthesizing G and E with interpolation values of all colors at the target pixel position obtained based on luminance in which G and E are generated only by interpolation within the horizontal line, and G + E An image processing method capable of obtaining interpolated values of all colors with suppressed false colors by combining interpolated values of all colors at the target pixel position obtained based on the above will be described.

  8 executes a demosaic process, which is a process of sequentially interpolating or estimating the intensity of a color that is not present at each pixel position so that there are four RGBE colors at all pixel positions. 4 is a block diagram showing a more detailed configuration of a demosaic processing unit 253. FIG.

  The demosaic processing unit 253 includes a local region extraction unit 261, a G / E interpolation processing unit 262, a G + E generation processing unit 263, an all color interpolation processing unit 264, an all color interpolation processing unit 265, and a synthesis processing unit 266. . The local area extraction unit 261 cuts out pixels in a local area having a predetermined size around the target pixel position from the gamma-corrected mosaic image. Here, the local region to be cut out is a 5 × 5 pixel rectangular region centered on the target pixel position. The G / E interpolation processing unit 262 interpolates G using pixels adjacent to the left and right at pixel positions where G does not exist in the line where G exists, using pixels existing in the local region. , E is present in a line where E is not present in the line, and the adjacent E is interpolated so that either G or E exists at any pixel position. E is regarded as a luminance signal L. The all-color interpolation processing unit 264 uses the luminance signal L and the mosaic signal M to calculate R, G, B, and E interpolation values RF, GF, BF, and EF (hereinafter referred to as a signal C) at the target pixel position. To do. The G + E generation processing unit 263 generates a new color component G + E based on the mosaic signal M. Based on G + E and the mosaic signal M, the all-color interpolation processing unit 265 calculates R, G, B, and E interpolation values RU, GU, BU, and EU (hereinafter referred to as a signal C ′) at the target pixel position. . The synthesis processing unit 266 synthesizes the interpolation values RF, GF, BF, EF and RU, GU, BU, EU to calculate a final all-color interpolation signal.

  FIG. 9 is a block diagram illustrating a detailed configuration example of the composition processing unit 266 of FIG. The composition processing unit 266 includes a color space conversion processing unit 301-1, a color space conversion processing unit 301-2, a saturation calculation processing unit 302, a blend rate calculation processing unit 303, a chromaticity difference calculation processing unit 304, and a blend processing unit 305. Consists of.

The color space conversion processing unit 301-1 converts the signal C from the RGBE color space to the RGB color space, and further converts the signal C into a color space represented by luminance and chromaticity. The color space conversion processing unit 301-2 converts the signal C ′ from the RGBE color space to the RGB color space, and further converts the signal C ′ to a color space represented by luminance and chromaticity. Among such color space, in this embodiment, a YC _b C _r color space defined by ITU-R BT.601 as a representative example.

The saturation calculation processing unit 302 calculates the saturation S based on the chromaticity signal C _c obtained from the signal C. The chromaticity difference calculation processing unit 304 calculates the chromaticity difference ΔC _c based on the chromaticity signal C ′ _c obtained from the chromaticity signal C _c and the signal C ′. The blend rate calculation processing unit 303 calculates a blend rate r that is a ratio when the chromaticity signal C _c and the chromaticity signal C ′ _c are combined based on the saturation S and the chromaticity difference ΔC _c . Blending processing unit 305, C _c and C 'on the basis of _c, new chroma signal C' to calculate the _'c,' brightness signal obtained from the C 'signal C _l and the chromaticity signal C'_'c Are combined to calculate a new YC _{b Cr} signal C ″.

  Next, processing of the DSP block 216 in FIG. 7 will be described with reference to the flowchart in FIG.

  In step S1, the image RAM 241 is composed of periodic pattern intensity signals in an array (for example, the four-color array described with reference to FIG. 6) defined by the color filter used in the CCD image sensor 213. Obtain a mosaic image and save it temporarily.

  In step S2, the white balance adjustment unit 251 of the signal processing processor 242 applies an appropriate color according to the color of each pixel intensity so that the color balance of the achromatic subject region is achromatic with respect to the mosaic image. White balance adjustment processing (level balance adjustment processing), which is processing for applying a coefficient, is performed.

  In step S <b> 3, the gamma correction unit 252 performs gamma correction so that the brightness and color saturation of the image displayed on the display unit 220 are correctly displayed for each pixel intensity of the mosaic image with white balance. Do.

  In step S4, the demosaic processing unit 253 performs a demosaic process described later with reference to FIG. 11, outputs the generated Y image and C image, and ends the process.

  Through such processing, the DSP block 216 performs various processing on the supplied mosaic image signal to generate a Y image and a C image, and displays the image data on the display unit 220 based on the control of the CPU 223. Is displayed in the D / A converter 218 and is stored in the memory 222, and is supplied to the codec processing unit 221.

  Next, the demosaic process executed in step S4 of FIG. 10 will be described with reference to the flowchart of FIG.

  In step S51, the local area extraction unit 261 sets one of unprocessed pixels as a target pixel, and in step S52, determines a predetermined number (2n + 1) × (2n + 1) of pixels around the target pixel position. Are extracted and supplied to the G, E interpolation processing unit 262, the G + E generation processing unit 263, the all color interpolation processing unit 264, and the all color interpolation processing unit 265. Note that n is a positive integer.

  In step S53, the G / E interpolation processing unit 262 executes G / E interpolation processing, which will be described later with reference to the flowcharts of FIGS. 12 and 13, and supplies the calculated value L to the all-color interpolation processing unit 264. .

  In step S <b> 54, the all-color interpolation processing unit 264 performs a first all-color interpolation process, which will be described later with reference to FIG. 14, and supplies the calculated signal C to the synthesis processing unit 266.

  In step S55, the G + E generation processing unit 263 performs a G + E generation process, which will be described later with reference to FIG. 20, and supplies the generated G + E to the all-color interpolation processing unit 265.

  In step S <b> 56, the all-color interpolation processing unit 265 performs a second all-color interpolation process, which will be described later with reference to FIG. 23, and supplies the calculated C ′ to the synthesis processing unit 266.

In step S57, the synthesis processing unit 266 performs a combination process described below with reference to FIG. 24, and outputs the color signal of the YC _b C _r.

  In step S58, the local region extraction unit 261 determines whether or not processing has been completed for all pixels. If it is determined in step S58 that processing has not been completed for all pixels, the processing returns to step S51, and the subsequent processing is repeated. If it is determined in step S58 that the processing has been completed for all pixels, the processing is terminated.

  In other words, each unit constituting the demosaic processing unit 253 executes the respective processes at the target pixel position when a certain target pixel position is determined, and the processes of steps S51 to S57 are completed for all the pixels. If so, the process is terminated.

  Note that the order of the processes of step S53 and step S54 and the processes of step S55 and step S56 may be reversed or may be executed in parallel.

  Next, the G and E interpolation processing executed in step S53 in FIG. 11 will be described with reference to the flowchart in FIG. This process is a process executed by the G and E interpolation processing unit 262 in FIG. Note that n is a positive integer. Further, (j, k) indicates a pixel position (processing pixel position) for executing processing when the center (target pixel position) in the local region is (0, 0).

  In step S101, the G and E interpolation processing unit 262 initializes the value j of the first register indicating the pixel position to be processed in the local region to j = −n + 1, and in step S102, the local region Among them, the value k of the second register indicating the pixel position to be processed is initialized to k = −n. Here, processing is performed on a pixel group of (2n + 1) × (2n + 1) centered on the pixel (0, 0). Note that n is a positive integer. Here, the initial values of j and k are j = −n + 1 and k = −n based on the left and right E or G when the processing pixel position is R or B in the processing of FIG. This is because G and E interpolation is performed. Note that the value of n can be considered as an arbitrary constant required for processing. Therefore, when this embodiment is implemented, such detailed setting of register values can be omitted. In particular, if n is a sufficiently large value, it is considered that substantially the same result can be obtained. The same applies to n ′, n ″, and n ″ ″ described later.

  In step S103, the G / E interpolation processing unit 262 determines whether or not the color of the pixel position (j, k) of the mosaic signal M is R. Since the above-described four-color filter arrangement shown in FIG. 6 is known, in this process, the color can be determined using the pixel position.

  When it is determined in step S103 that the color of the pixel position (j, k) of the mosaic signal M is R, in step S104, the G and E interpolation processing unit 262 averages the left and right Es of the pixel position R. And the luminance L (j, k). That is, the average value of the pixel (j−1, k) and the pixel (j + 1, k) is the luminance L (j, k).

  If it is determined in step S103 that the color of the pixel position (j, k) of the mosaic signal M is not R, in step S105, the G and E interpolation processing unit 262 determines whether the pixel (j, k) is G. Determine whether or not.

  When it is determined in step S105 that the pixel position (j, k) of the mosaic signal M is G, in step S106, the G and E interpolation processing unit 262 sets the pixel position G (j, k) to L (j , K). That is, G at the processing pixel position is set to L.

  If it is determined in step S105 that the pixel position (j, k) of the mosaic signal M is not G, in step S107, the G and E interpolation processing unit 262 determines that the pixel position (j, k) of the mosaic signal is B. It is determined whether or not there is. If it is determined in step S107 that the pixel position (j, k) of the mosaic signal is B, in step S108, the G and E interpolation processing unit 262 averages the left and right G of the pixel B, and the luminance L (J, k). That is, the average value of the pixel (j−1, k) and the pixel (j + 1, k) is the luminance L (j, k).

  When it is determined in step S107 that the pixel position (j, k) of the mosaic signal M is not B, that is, E, in step S109, the G and E interpolation processing unit 262 converts E of the pixel position to L (j , K). That is, E at the processing pixel position is set to L.

  After step S104, step S106, step S108, or step S109, in step S110, the G and E interpolation processing unit 262 refers to the value k of the second register and determines whether or not k = n. judge. If it is determined in step S110 that k = n is not satisfied, in step S111, the G and E interpolation processing unit 262 updates the value k of the second register to k = k + 1, and the process returns to step S103. The subsequent processing is repeated.

  When it is determined in step S110 that k = n, in step S112, the G and E interpolation processing unit 262 refers to the value j of the first register indicating the pixel position where the process is performed, and j = n− It is determined whether or not 1. If it is determined in step S112 that j = n−1 is not satisfied, the G and E interpolation processing unit 262 updates the value j of the first register to j = j + 1 in step S113, and the process returns to step S102. The subsequent processing is repeated.

  When it is determined in step S112 that j = n−1, in step S114, the G and E interpolation processing unit 262 outputs a set of luminance signals L of (2n + 1) × (2n−1) pixels near the target pixel. Then, the process proceeds to step S54 in FIG.

  12 and FIG. 13, the G and E interpolation processing unit 262 uses the G adjacent to the left and right at the pixel position (B) where G does not exist in the line where G exists. In a line where E is present, E is interpolated using E adjacent to the left and right at the pixel position (R) where E does not exist in the line, and either G or E is used at any pixel position. It is assumed to exist and is regarded as the luminance signal L.

  Next, the first all-color interpolation process executed in step S54 in FIG. 11 will be described with reference to the flowchart in FIG. This process is a process executed by the all-color interpolation processing unit 264 in FIG.

In step S151, the all-color interpolation processing unit 264 performs an R, G, B, E estimated value R ′ _c , G ′ _c , B ′ _c , E ′ _c calculation process at the target pixel position. Details of this processing will be described later with reference to FIG.

In step S152, the all-color interpolation processing unit 264 performs an R, G, B, E estimated value R ′ _u , G ′ _u , B ′ _u , E ′ _u calculation process on one pixel at the target pixel position. . Details of this processing will be described later with reference to FIG.

In step S153, the all-color interpolation processing unit 264 performs an R, G, B, E estimated value R ′ _d , G ′ _d , B ′ _d , E ′ _d calculation process one pixel below the target pixel position. . Details of this processing will be described later with reference to FIG.

  In step S154, the all-color interpolation processing unit 264 adds and averages the results interpolated above and below the target pixel position, and further adds and averages the addition average and the result interpolated to the target pixel position, and estimates RF and GF. , BF, EF are calculated. Specifically, the calculations shown in the equations (1) to (4) are performed.

RF = (R ′ _c + (R ′ _u + R ′ _d ) / 2) / 2 (1)
GF = (G ′ _c + (G ′ _u + G ′ _d ) / 2) / 2 (2)
BF = (B ′ _c + (B ′ _u + B ′ _d ) / 2) / 2 (3)
EF = (E ′ _c + (E ′ _u + E ′ _d ) / 2) / 2 (4)

  After the process of step S154, the process proceeds to step S55 of FIG.

  Note that the order of the processing of step S151, step S152, and step S153 may be reversed or may be executed in parallel.

  Next, the estimated value calculation process executed in step S151 in FIG. 14 will be described with reference to the flowchart in FIG. FIG. 15 is a process for estimating R, G, B, and E by color balance interpolation. Since the same process is performed for all of R, G, B, and E, X is described in FIG. The portion is appropriately read as R, G, B, E. The color balance interpolation is to interpolate an unknown color component by multiplying the ratio of the DC component of the luminance signal and the DC component of the color signal at the target pixel position by the luminance signal.

  In step S201, the all-color interpolation processing unit 264 executes linear interpolation processing. Details of this processing will be described later with reference to FIGS. 16 and 17.

In step S202, the all-color interpolation processing unit 264 executes a weighted average M _xa calculation process. Details of this processing will be described later with reference to FIG.

In step S203, all the color interpolation section 264 performs a weighted average M _L calculation process. Details of this processing will be described later with reference to FIG.

In step S204, the all color interpolation processing unit 264 determines whether or not M _L <(first constant), and if it is determined that M _L <(first constant), in step S205. Set M _L '= (first constant). This first constant is a predetermined value determined empirically or experimentally, for example.

When it is determined in step S204 that M _L <(first constant) is not satisfied, that is, M _L ≧ (first constant), in step S206, the all-color interpolation processing unit 264 determines that M _L ′ = M _L And set.

After the process of step S205, or after the processing in step S206, in step S207, all the color interpolation section 264, and _{X '= M xa / M L} ' (L-M L) + M xa, process 14 The process proceeds to step S152.

  In this example, the processing in FIG. 15 has been described as the details of the processing in step S151 in FIG. 14, but the same processing is performed in steps S152 and S153 in FIG. The estimated value calculation process is simply performed for R, G, B, and E one pixel above and one pixel below the target pixel position.

  Note that the order of the processes in step S202 and step S203 may be reversed or executed in parallel.

  Next, the linear interpolation process executed in step S201 in FIG. 15 will be described with reference to the flowcharts in FIGS. This process is a flowchart showing a processing procedure for linearly interpolating R, G, B, and E near the target pixel. Since the same processing is executed for all of R, G, B, and E, the portion described as X in FIGS. 16 and 17 is appropriately read as R, G, B, and E. Note that n ′ is a positive integer, and (j ′, k ′) indicates a pixel position where the process is executed when the center (target pixel position) in the local region is (0, 0).

  In step S251, the all color interpolation processing unit 264 initializes the value j ′ of the first register indicating the pixel position to be processed in the local region to j ′ = − n ′ + 1, and in step S252, A value k ′ of a second register indicating a pixel position to be processed in the local area is initialized to k ′ = − n ′ + 1. Here, processing is performed on a pixel group of (2n ′ + 1) × (2n ′ + 1) centered on the pixel (0, 0).

  In step S253, the all color interpolation processing unit 264 determines whether the M (mosaic signal M) color of the pixel (j ′, k ′) of the signal M is X or not. Here, X is appropriately read as R, G, B, and E. Since the above-described four-color filter arrangement shown in FIG. 6 is known, in this process, the color can be determined using the pixel position.

  If it is determined in step S253 that the color of the pixel (j ′, k ′) of the signal M is X, in step S254, the all-color interpolation processing unit 264 determines the pixel (j ′, k ′) of the signal M. Is the pixel (j ′, k ′) of the signal M. That is, when it is determined that the color of the pixel position of the signal M is X, no interpolation is required and the signal is used as it is.

  When it is determined in step S253 that the pixel (j ′, k ′) of the signal M is not X, in step S255, the all-color interpolation processing unit 264 determines that the pixel (j′−1, k ′) is X. It is determined whether or not. That is, it is determined whether or not the color of the pixel adjacent to the processing pixel is X.

  When it is determined in step S255 that the color of the pixel (j′−1, k ′) of the signal M is X, in step S256, the all-color interpolation processing unit 264 determines the pixel (j ′, k ′). Interpolated value XA is an average value of pixels (j′−1, k ′) and (j ′ + 1, k ′). That is, when it is determined that the color of the horizontally adjacent pixel is X, interpolation is performed from the left and right.

  When it is determined in step S255 that the color of the pixel (j′−1, k ′) of the signal M is not X, in step S257, the all-color interpolation processing unit 264 determines the pixel (j ′, k′− of the signal M). It is determined whether the color of 1) is X, and if it is determined that the color of the pixel (j ′, k′−1) is X, the pixel (j ′, k) of the signal M is determined in step S258. The interpolation value XA of ') is the average of the pixel (j', k'-1) and the pixel (j ', k' + 1). That is, when it is determined that the color of the vertically adjacent pixel is X, vertical interpolation is performed.

  If it is determined in step S257 that the pixel (j ′, k′−1) is not X, in step S259, the all-color interpolation processing unit 264 determines the interpolation value XA of the pixel (j ′, k ′) of the signal M. Of pixel (j′−1, k′−1), pixel (j ′ + 1, k′−1), pixel (j′−1, k ′ + 1), pixel (j ′ + 1, k ′ + 1) Average. That is, the pixel (j ′, k ′) of the signal M, the pixel (j′−1, k ′) of the signal M adjacent to the side of the pixel, and the pixel (j ′, k ′) of the signal M adjacent to the pixel vertically. When it is determined that the color of -1) is not X, interpolation is performed from the upper left, upper right, lower left, and lower right.

  After step S254, step S256, step S258, or step S259, in step S260, the all color interpolation processing unit 264 refers to the value k ′ of the second register, and k ′ = n′−1. It is determined whether or not. If it is determined in step S260 that k ′ = n′−1 is not satisfied, in step S261, the all color interpolation processing unit 264 updates the value k ′ of the second register to k ′ = k ′ + 1, and performs processing. Returns to step S253, and the subsequent processing is repeated.

  If it is determined in step S260 that k ′ = n′−1, in step S262, the all-color interpolation processing unit 264 refers to the value j ′ of the first register indicating the pixel position to be processed, It is determined whether j ′ = n′−1. When it is determined in step S262 that j ′ = n′−1 is not satisfied, in step S263, the all-color interpolation processing unit 264 updates the value j ′ of the first register to j ′ = j ′ + 1 and performs processing. Returns to step S252, and the subsequent processing is repeated.

  If it is determined in step S262 that j ′ = n′−1, in step S264, the all-color interpolation processing unit 264 determines X of (2n′−1) × (2n′−1) pixels near the target pixel. The set of interpolation values XA is output, and the process proceeds to step S202 in FIG.

  The processing in FIG. 16 and FIG. 17 is summarized as follows. When it is determined that the color of the processing pixel position of the signal M is X, the all-color interpolation processing unit 264 uses the signal as it is because there is no need for interpolation, and the adjacent pixel beside the processing pixel of the signal M When it is determined that the pixel is X, interpolation is performed from the left and right. When it is determined that the pixel vertically adjacent to the processing pixel of the signal M is X, interpolation is performed from the top and bottom. Interpolate from the upper right, lower left, and lower right. Such processing is performed for all of R, G, B, and E, and RA, GA, BA, and EA are calculated.

Next, referring to the flowchart of FIG. 18, the weighted average value M _x (the subscript _x of the weighted average value M _x is executed in step S202 or step S203 of FIG. 15 corresponds to step S202 of FIG. The calculation process will be described (replaced by xa in the case of the process to be performed and L in the case of the process corresponding to step S203 in FIG. 15). 18 is executed for RA, GA, BA, EA, and L. Therefore, in the portion described as X in FIG. 18, RA, GA, BA, EA, and L are appropriately selected. To be read as The RA, GA, BA, and EA are values calculated by the processes shown in FIGS. 16 and 17 described above. Note that n ″ is a positive integer, and (j ″, k ″) is a pixel position for executing processing when the center (target pixel position) in the local region is (0, 0). Show.

In step S301, the all-color interpolation processing unit 264 initializes the weighted average value M _x to 0.

  In step S302, the all-color interpolation processing unit 264 initializes the value j ″ of the first register indicating the pixel position where the processing is to be performed among the supplied pixels in the local region, and j ″ = − n ′. ', And among the pixels in the local area supplied in step S303, the value k ″ of the second register indicating the pixel position to be processed is initialized to k ″ = − n ″.

In step S304, the all-color interpolation processing unit 264 sets M _x = M _x + ( _{X of} the pixel (j ″, k ″) to which the weight coefficient wi is applied). That, M pixels weighted wi is applied to _{x (j '', k '} ') obtained by adding the X in is the new M _x.

  The weighting factor wi is a value set in advance using the distance from the position of the i-th data to the target pixel as an index. For example, it is set as shown in FIG. In the figure, squares indicate the pixel positions of the 5 × 5 local area, and the numbers in the squares indicate the weighting factors of the data at the corresponding positions. Needless to say, these weighting factors are not limited to those shown in FIG. 19, but as shown in FIG. 19, it is preferable that the weighting factors become larger as they are closer to the target pixel. In FIG. 19, since the outer periphery is 0, the calculation is actually performed in the 3 × 3 range of the inner periphery. Therefore, the calculation may be performed only for the 3 × 3 range of the inner periphery from the beginning.

  In step S305, the all-color interpolation processing unit 264 refers to the value k ″ of the second register and determines whether k ″ = n ″. If it is determined in step S305 that k ″ = n ″ is not satisfied, in step S306, the all color interpolation processing unit 264 updates the value k ″ of the second register to k ″ = k ″ +1. Then, the process returns to step S304, and the subsequent processes are repeated.

  When it is determined in step S305 that k ″ = n ″, in step S307, the all color interpolation processing unit 264 refers to the value j ″ of the first register indicating the pixel position where the processing is performed. Then, it is determined whether j ″ = n ″. When it is determined in step S307 that j ″ = n ″ is not satisfied, in step S308, the all color interpolation processing unit 264 updates the value j ″ of the first register to j ″ = j ″ +1. Then, the process returns to step S303, and the subsequent processes are repeated.

If it is determined in step S307 that j ″ = n ″, in step S309, the all-color interpolation processing unit 264 sets M _x = M _x / W. This W is the total sum W of the weighting factors wi.

In step S310, the all-color interpolation processing unit 264 outputs the weighted average value M _x , and the process proceeds to step S203 or step S204 in FIG.

The weighted average values M _x (x is appropriately read as RA, GA, BA, EA, L) in the vicinity of the processing pixel are calculated by the processing in FIG.

  Next, the G + E generation process executed in step S55 of FIG. 11 will be described with reference to the flowchart of FIG. This process is a process executed by the G + E generation processing unit 263 in FIG. Note that n ″ ′ is a positive integer, and (j ″ ′, k ″ ′) executes processing when the center (target pixel position) in the local region is (0, 0). Indicates the pixel position.

  In step S <b> 351, the G + E generation processing unit 263 initializes a value j ′ ″ of the first register indicating a pixel position where the process is executed among the pixels of the supplied local region, and j ′ ″ = − n. '' ', And among the pixels in the local area supplied in step S352, the value k' '' of the second register indicating the pixel position for executing the process is initialized, and k '' '=-n' '' And

  In step S353, the G + E generation processing unit 263 determines whether the mosaic signal M at the pixel position (j ″ ′, k ″ ′) is G or E, and is determined to be G or E. In this case, in step S354, the G + E generation processing unit 263 executes a first G + E generation process. Details of this processing will be described later with reference to FIG.

  In step S353, when it is determined that the mosaic signal M at the processing pixel position is not G or E, that is, B or R, in step S355, the G + E generation processing unit 263 executes a second G + E generation process. Details of this processing will be described later with reference to FIG.

  After step S354 or step S355, in step S356, the G + E generation processing unit 263 refers to the value k ′ ″ of the second register and determines whether k ′ ″ = n ″ ′. judge. If it is determined in step S356 that k ′ ″ = n ″ ′ is not satisfied, in step S357, the G + E generation processing unit 263 sets the value k ′ ″ of the second register to k ′ ″ = k ″. After updating to '+1', the process returns to step S353, and the subsequent processes are repeated.

  If it is determined in step S356 that k ′ ″ = n ″ ′, in step S358, the G + E generation processing unit 263 sets the value j ′ ″ of the first register indicating the pixel position for executing the processing. To determine whether j ′ ″ = n ″ ′. When it is determined in step S358 that j ′ ″ = n ″ ′ is not satisfied, in step S359, the G + E generation processing unit 263 sets the value j ′ ″ of the first register to j ′ ″ = j ′ ″. After updating to +1, the processing returns to step S352, and the subsequent processing is repeated.

  When it is determined in step S358 that j ′ ″ = n ″ ′, in step S360, the G + E generation processing unit 263 determines (2n ″ ′ + 1) × (2n ″ ′ + 1) near the target pixel. ) Output G + E of the pixel, and the process proceeds to step S56 in FIG.

  Next, the first G + E generation process executed in step S354 of FIG. 20 will be described with reference to the flowchart of FIG. This process is a process executed by the G + E generation processing unit 263 in FIG. Further, since the same processing is performed at the pixel position where either G or E exists, at the pixel position where G exists, the portion indicated as X in FIG. The portion which is present is read as E, and in the pixel position where E is present, X is read as E and Y is read as G in FIG. In the following processing, X is treated as luminance and Y is treated as a color signal.

  In step S401, the G + E generation processing unit 263 performs linear interpolation processing. Since this process is the same as the process of FIG. 16 and FIG. 17 described above, its details are omitted.

In step S402, the G + E generation processing unit 263 executes a weighted average M _xa calculation process. Since this process is the same as the process of FIG. 18 described above, its details are omitted.

In step S403, the G + E generation processing unit 263 executes a weighted average _Mya calculation process. Since this process is the same as the process of FIG. 18 described above, its details are omitted.

In step S404, G + E generation processing unit 263 determines whether the weighted average M _xa <(second constant) was determined to be weighted average M _xa <(second constant) In this case, in step S405, M _xa '= (second constant) is set. This second constant is a predetermined value determined empirically or experimentally, for example. When it is determined in step S404 that M _xa <(second constant) is not satisfied, that is, M _xa ≧ (second constant), in step S406, the G + E generation processing unit 263 determines that M _xa ′ = M _xa Set.

After the processing in step S405 or after the processing in step S406, in step S407, the G + E generation processing unit 263 sets the target pixel (x, y) when the target pixel position on the mosaic image is (x, y). The value of G + E at is set to (( _Mya / _Mxa '(M (x, y) _-Mxa ) + _Mya ) + M (x, y)) / 2, and the process proceeds to step S356 in FIG.

  21, E is interpolated by color balance interpolation at the pixel position where G exists, and G is interpolated by color balance interpolation at the pixel position where E exists, and a new color component G + E is synthesized by combining G and E on the same pixel. Is generated.

  Note that the processing in steps S401 to S403 may be performed in reverse order or in parallel.

  Next, the second G + E generation process executed in step S355 of FIG. 20 will be described with reference to the flowchart of FIG. This process is a process executed by the G + E generation processing unit 263 in FIG. Since the same processing is performed at a pixel position where either R or B exists, the portion described as X in FIG. 22 is appropriately read as R or B. In the following processing, X is treated as luminance and G and E are treated as color signals.

  In step S451, the G + E generation processing unit 263 performs linear interpolation processing. Since this process is the same as the process of FIG. 16 and FIG. 17 described above, its details are omitted.

In step S452, the G + E generation processing unit 263 executes a weighted average M _xa calculation process. Since this process is the same as the process of FIG. 18 described above, its details are omitted.

In step S453, the G + E generation processing unit 263 executes a weighted average M _ga calculation process. Since this process is the same as the process of FIG. 18 described above, its details are omitted.

In step S454, the G + E generation processing unit 263 executes a weighted average _Mea calculation process. Since this process is the same as the process of FIG. 18 described above, its details are omitted.

In step S455, G + E generation processing unit 263 determines whether the weighted average M _xa <(third constant) was determined to be weighted average M _xa <(third constant) In this case, in step S456, M _xa ′ = (third constant) is set. This third constant is a predetermined value determined empirically or experimentally, for example. If it is determined in step S455 that M _xa <(third constant) is not satisfied, that is, M _xa ≧ (third constant), the G + E generation unit 263 sets M _xa ′ = M _xa in step S457. To do.

After the process of step S456 or the process of step S457, in step S458, the G + E generation processing unit 263 sets the value of G + E at the target pixel (x, y) to ((M _ga / M _xa '(M (x , Y) −M _xa ) + M _ga ) + (M _ea / M _xa ′ (M (x, y) −M _xa ) + M _ea )) / 2, and the process proceeds to step S356 in FIG.

  22, when the mosaic signal at the target pixel position is R or B, G and E are interpolated at the pixel position by color balance interpolation, and they are combined to generate a new color component G + E.

  Note that the processing in steps S451 to S454 may be performed in reverse order or in parallel.

  Next, the second all-color interpolation process executed in step S56 in FIG. 11 will be described with reference to the flowchart in FIG. This process is a process executed by the all-color interpolation processing unit 265 of FIG. Here, R, G, B, and E are estimated by color balance interpolation. In this processing, the same processing is executed for all of R, G, B, and E. Therefore, the portion described as X in the figure is appropriately read as R, G, B, and E. In this process, G + E is treated as luminance and X is treated as a color signal.

  In step S501, the all-color interpolation processing unit 265 acquires the result of the linear interpolation processing calculated by the G + E generation processing unit 263 in step S401 of FIG. 21 or step S451 of FIG.

In step S502, the all-color interpolation processing unit 265 executes a weighted average M _xa calculation process. Since this process is the same as that in FIG. 18 described above, the description thereof is omitted. In step S502, the weighted average M _xa may be obtained using a weighting factor different from the weighting factor wi used in the processing in step S402 or step S452 of FIG. In this case, a weighted average M _xa calculated in step S502, the value of the weighted average M _xa calculated in step S402 or step S452 of FIG. 21 will be different.

In step S503, the all-color interpolation processing unit 265 executes a weighted average M _{G + E} calculation process. Since this process is the same as that in FIG. 18 described above, the description thereof is omitted. Specifically, in FIG. 18, x is merely changed to G + E obtained by the above-described processing of FIG.

In step S504, the all-color interpolation processing unit 265 determines whether or not M _{G + E} <(fourth constant), and if it is determined that M _{G + E} <(fourth constant). In step S505, the all-color interpolation processing unit 265 sets M _{G + E} ′ = (fourth constant). The fourth constant is a predetermined value determined empirically or experimentally, for example. If it is determined in step S504 that M _{G + E} <(fourth constant) is not satisfied, that is, M _{G + E} ≧ (fourth constant), in step S506, the all-color interpolation processing unit 265 determines that M _{G + E} <(fourth constant). Set _{G + E} '= _{MG + E.}

After step S505 or after step S506, in step S507, the all-color interpolation processing unit 265 converts the estimated value XU of X to M _xa / M _{G + E} ′ ((G + E) (x, y ) −M _{G + E} ) + M _xa , and the process proceeds to step S58 in FIG.

  Note that the processing in steps S501 to S503 may be performed in reverse order or in parallel.

  Next, the synthesis process executed in step S57 in FIG. 11 will be described with reference to the flowchart in FIG. This process is a process executed by the synthesis processing unit 266 of FIG.

  In step S551, the composition processing unit 266 acquires the interpolated values RF, GF, BF, and EF of R, G, B, and E at the target pixel position as the signal C. This value is a value calculated by the all-color interpolation processing unit 264 (the process in step S54 in FIG. 11).

  In step S552, the composition processing unit 266 acquires R, G, B, and E interpolation values RU, GU, BU, and EU at the target pixel position as a signal C ′. This value is a value calculated by the all-color interpolation processing unit 265 (processing in step S56 in FIG. 11).

In step S553, the color space conversion processing unit 301-1 of the composition processing unit 266 converts the signal represented by the RGBE color space of the signal C acquired in the processing of step S551 into an RGB color space, and performs RGB It is converted to YC _b C _r. In step S554, the color space conversion processing unit 301-2 of the synthesis processing unit 266 converts the signal represented by the RGBE color space of the signal C ′ acquired in the processing of step S552 into the RGB color space. together, converts the RGB to YC _b C _r. Here, the color space conversion processing unit 301-1 and the color space conversion processing unit 301-2 execute similar processing. Hereinafter, the color space conversion processing unit 301-1 and the color space conversion processing unit 301-2 will be referred to as the color space conversion processing unit 301 when it is not necessary to distinguish them individually. A conversion matrix from the RGBE color space to the RGB color space (RGB color system) is expressed by Equation (5).

  In Expression (5), parameters Tij (i = 1 to 3, j = 1 to 4) in the matrix are portions relating to color reproducibility, and are arbitrarily given according to the spectral characteristics of the filter to be used. For example, Japanese Patent Application Laid-Open No. 2002-271804 discloses parameters for converting to an XYZ color space in the case of a filter set of four colors having 465 nm, 530 nm, 585 nm, and 615 nm as peak wavelengths of the spectral sensitivity curve. ing. The conversion from the XYZ color space to the RGB color space is shown in CCIR 601.

Also, the conversion matrix from the RGB color space to the YC _{b Cr} color space is expressed by Equation (6).

In Equation (6), Y is the luminance signal C ′ _l in the present embodiment, and C _b and C _r are the chromaticity signals C _c and C ′ _c . Hereinafter, two signals included in the chromaticity signal C _c are described as C _b and C _r .

In step S555, the saturation calculation processing unit 302 of the synthetic processing unit 266, based on the chromaticity signal C _c, to calculate the saturation S. Specifically, the calculation shown in Expression (7) is performed.

In step S556, the chromaticity difference calculation processing unit 304 of the composition processing unit 266 calculates the chromaticity difference ΔC _c based on the chromaticity signals C _c and C ′ _c . Specifically, the calculation shown in Expression (8) is performed.

In step S557, the blend rate calculation processing unit 303 of the synthesis processing unit 266 calculates the blend rate r based on the saturation S and the chromaticity difference ΔC _c . Specifically, the blend ratio r is obtained by the calculation shown in Expression (9).

  In Expression (9), the fifth constant and the sixth constant are arbitrary constants. If r obtained by Equation 6 is greater than 1, it is clipped to 1. The fifth constant and the sixth constant are predetermined values determined empirically or experimentally, for example.

In step S558, the blend processing unit 305 of the synthesis processing unit 266 blends the chromaticity signals C _c and C ′ _c based on the blend ratio r to generate the chromaticity signal C ″ _c . Specifically, the chromaticity signal C ″ _c is generated by the operations shown in the equations (10) and (11).

C ″ _b = C _b × r + C ′ _b × (1−r) (10)
C ″ _r = C _r × r + C ′ _r × (1−r) (11)

In step S559, the blend processing unit 305 of the composition processing unit 266 newly adds the chromaticity signal C ″ _c and the luminance signal C ′ _l calculated by the color space conversion processing unit 301-2 in the process of step S552. YC _{b Cr} signal C ″ is output.

  Note that the order of the processes in steps S553 and S554 may be reversed, or may be executed in parallel.

  In the above example, the color space conversion processing unit 301 converts the signal expressed in the RGBE color space into the color space expressed by the luminance and the color difference, and then performs the blending process. By adopting such a configuration, the signal C ″ that is finally obtained even when the signal C whose longitudinal resolution is lower than that of the signal C ′ is used for blending uses the luminance signal of the signal C ′. Therefore, the same resolution as the signal C ′ can be obtained. On the other hand, if resolution reduction due to blending does not become a problem, for example, a simpler method that does not perform color space conversion as shown in FIG. 25 may be used.

  FIG. 25 is a block diagram showing a detailed configuration of the composition processing unit 266 of FIG. 8, which is another embodiment to which the present invention is applied. That is, this is a modification (another embodiment) of the composition processing unit 266 of FIG. 25 includes a luminance calculation processing unit 401, a saturation calculation processing unit 402, a blend rate calculation processing unit 403, a chromaticity difference calculation processing unit 404, and a blend processing unit 405. The signal C is supplied from the all-color interpolation processing unit 264 of FIG. 8 described above, and the signal C ′ is supplied from the all-color interpolation processing unit 265 of FIG. 8 described above. Hereinafter, four color components included in the signal C are represented by R, G, B, and E, and four color components included in the signal C ′ are represented by R ′, G ′, B ′, and E ′.

  The luminance calculation processing unit 401 calculates the luminance from the signal C. In this embodiment (another embodiment), the luminance L is defined as shown in the following formula (12).

  L = (R + G + B + E) / 4 (12)

  The saturation calculation processing unit 402 calculates the saturation S based on the signal C and the luminance L obtained by the luminance calculation processing unit 401. In this embodiment (another embodiment), the saturation S is defined as shown in the following equation (13).

  The chromaticity difference calculation processing unit 404 calculates the chromaticity difference based on the signals C and C ′. In this embodiment (another embodiment), the chromaticity difference ΔC is defined as shown in the following equation (14).

  Based on the saturation S obtained again by the calculation processing unit 402 and the chromaticity difference ΔC obtained by the chromaticity difference calculation processing unit 404, the blend rate calculation processing unit 403 calculates the blend rate r by the following formula (15 ) To calculate.

  In the formula (15), the seventh constant and the eighth constant are arbitrary constants. If r calculated by the equation (15) is larger than 1, it is clipped to 1. The seventh constant and the eighth constant are predetermined values determined empirically or experimentally, for example.

  The blend processing unit 405 blends the signals C and C ′ based on the blend rate, and calculates a new signal C ″ A. Hereinafter, four color components included in C ″ A are represented by R ″, G ″, B ″, and E ″. The blending method is represented by the following formulas (16) to (20).

R ″ = R × r + R ′ × (1-r) (16)
G ″ = G × r + G ′ × (1−r) (17)
B ″ = B × r + B ′ × (1-r) (18)
E ″ = E × r + E ′ × (1-r) (19)

  Thus, for G and E, when G is present in one horizontal line, from a mosaic image of colors picked up by a single-plate color imaging device having a filter array in which E does not exist in that line. The generation of false colors is suppressed, but the resolution is low and the reliability is low. The interpolated values of all the first colors and the false colors are likely to be generated, but the resolution is high and no false colors are generated. An interpolated value of the second all colors having high color reliability at the position can be obtained. Further, by comparing the interpolation values of all the two colors, it is possible to accurately determine the generation position of the false color, which has been difficult in the past.

  Further, it is a feature of the interpolation values of the first all colors (for example, the signal C generated by the all color interpolation processing unit 264) based on the saturation and the chromaticity difference obtained from the interpolation values of the two all colors. The high resolution and color reliability, which are the characteristics of the small number of false colors and the interpolation values of the second all colors (for example, the signal C ′ generated by the all color interpolation processing unit 265), are 2 It is possible to effectively synthesize the interpolation values of all the two colors, thereby enabling high-quality color component interpolation.

  This is different from the conventional process of making the generated false color close to an achromatic color, and the interpolation values of the first all colors closer to the true value are used for the false color suppression process than simply making the achromatic color. Therefore, a more natural interpolation image can be obtained.

  In this embodiment and other embodiments, the saturation calculation processing units 302 and 402 and the chromaticity difference calculation processing units 304 and 404 use the Euclidean distance (the sum of squares of the deviation) to calculate the saturation and chromaticity difference. Square value) is used, but Manhattan distance (sum of absolute values of deviations) may be used.

  In addition, the blend ratio r qualitatively increases as the chromaticity difference ΔC increases, and may be decreased as the saturation S increases. Therefore, the blend ratio r is not limited to Expression (9) or Expression (15), and Expression (20) You may make it use.

  r = ΔC × (9th constant) −S × (10th constant) (20)

  Of course, another formula showing the same behavior may be used.

  In the above example, the case of a four-color filter array as shown in FIG. 6 has been described, but the number of filter colors may be further increased. The essence of the present invention is that, for a certain color G, E, any one of G, E exists in all horizontal lines, and when G exists in one horizontal line, a filter in which E does not exist in that line. In the arrangement, two types of luminance are generated based on G and E. Considering that human vision is not so sensitive to luminance with respect to color components, as long as the luminance can be generated with the same accuracy, other colors can be interpolated without problems if there are enough samples to obtain the DC component. it can. For example, the present invention can be applied to FIG. 26 having a six-color arrangement.

  The above processing is summarized as follows.

  As a pixel interpolation method, a method of generating a luminance signal for all pixels and interpolating other color components using it as reference information is generally used. For example, “US Patent 4,642, 678“ Cok, David R. “Signal processing method and apparatus for producing interpolated chrominance values in a sampled color image signal” ”assumes that the color ratio in the local region is almost constant. Original, pixel interpolation method in which G is preliminarily arranged in all pixels, and the average of the ratio of R and G in the neighboring pixels and the average of the ratio of B and G are multiplied by G of the target pixel position to estimate the unknown color component Has been proposed.

  Therefore, in the present invention, an unknown color component is interpolated by multiplying the ratio of the DC component of the luminance signal and the DC component of the color signal at the target pixel position by the luminance signal. Hereinafter, this is referred to as color balance interpolation. The DC component can be easily obtained by averaging pixel values in the vicinity of the target pixel.

  When imaging is performed with thinning, if the distance between pixels used for interpolation of the luminance signal is widened, it causes a large increase in false color. In the Bayer array, G used as luminance exists in a checkered pattern. Therefore, even if thinning is performed for each horizontal line, a pixel position where G does not exist can be interpolated using G in the lateral vicinity, and the above problem is avoided. it can.

  However, the same method as the Bayer array cannot be used in an array in which a color component that is commonly used for all pixels as luminance cannot be generated by only pixels in the horizontal vicinity. Therefore, in the present embodiment, two types of luminance are generated, and the characteristics of the two interpolation values interpolated based on the two are combined to obtain the same effect.

  In the following, a case where color balance interpolation is used for color interpolation will be described using the four-color filter array shown in FIG. 6 as an example. Even after thinning, thinning is performed so that the arrangement in FIG. 6 is obtained, and thinning is performed at as even intervals as possible in order to suppress a reduction in resolution. For example, it is possible to consider a pattern in which 8 lines are set as one set and the fourth and seventh lines are thinned out to 1/4. Note that the arrangement and the interpolation method are not limited to those used in this description.

  First, in a line where G exists, G is interpolated using G adjacent to the left and right at a pixel position where G does not exist in the line, and in a line where E exists, a pixel where E does not exist in the line. E is interpolated using E adjacent to the left and right at the position. As a result, although G or E exists in all the pixels, they are regarded as one luminance without being distinguished. FIG. 27 shows a state where G and E are interpolated for each line. As an example, when the imaging target has only a DC component and the obtained mosaic signal value is R value 0.5, G value 1 and E value 0.8, R located at position 452 Consider the case of estimating.

  When the average value of the luminance of the neighboring 3 × 3 region 454 is obtained for color interpolation at the position 451, it is about 0.87. When the average value of the luminance of the neighboring 3 × 3 region 455 is obtained for color interpolation at the position 452, it is about 0.93. Since the average value of R is 0.5 in any region, when color balance interpolation is used, the R interpolation value R1 at the position 451 is calculated as shown in Expression (21), and the R value at the position 452 is calculated. The interpolation value R2 is calculated as shown in Expression (22).

  In this way, the color considered as luminance at the target pixel position is different for each line, and the average luminance is not the same, so simply performing color balance interpolation will change the interpolated color depending on the pixel position. become. Therefore, the interpolation results when the luminance of the target pixel position is G and E are combined to balance the pixels. In the thinning method used here, since the interpolation result when the luminance at the target pixel position is G and E is obtained for each line, the target pixel position and the color interpolated above and below it are used. . If the R interpolation value interpolated at the position 453 using the average value of the luminance of the region 456 is R3, the final R interpolation value R2 'at the position 452 is obtained by Expression (23).

  By using such a method, the interpolated pixels in which all the colors are aligned do not generate false colors because the neighboring pixels in the vertical direction with a wide sampling interval are not used for luminance generation. However, the addition of a plurality of interpolation values causes a reduction in resolution. In addition, since interpolation is performed without distinguishing between G and E, an error is included in the color estimation, and the reliability of the interpolated color is low. In order to increase the reliability, it is preferable that the spectral characteristics of G and E are as close as possible. Further, in order to consider that G and E are the same for an achromatic color, it is also possible to improve reliability by adjusting the level balance (white balance) at the mosaic signal stage.

  Subsequently, interpolation using another luminance is performed. First, G and E are aligned at all pixel positions by an appropriate interpolation method such as linear interpolation. Then, G and E on the same pixel are averaged to generate a new color component G + E. G + E is regarded as luminance, and unknown color components are interpolated by color balance interpolation. Since G + E uses vertical neighboring pixels with a wide sampling interval at the time of generation, false colors are likely to occur. However, for DC components to be imaged, since color estimation is performed correctly, pixel positions where no false colors are generated Then, the reliability of the interpolated color is high.

  By using the above two estimation methods, the occurrence of false colors is suppressed, but the first all-color interpolation values C (RF, GF, BF, EF) and the resolution are low and the color reliability is low. A second all-color interpolated value C ′ (RU, GU, BU, EU) is obtained which is easy to generate false colors, but has high resolution and high color reliability at pixel positions where no false colors appear. It is done. By synthesizing these two interpolated values of all colors, generation of false colors can be suppressed, and a new third interpolated value of all colors can be generated with high color reliability and high resolution. it can. As the method, first, the current color space is converted into a color space represented by luminance and chromaticity, and the first luminance and the first chromaticity are generated from the interpolation values of all the first colors. Next, second luminance and second chromaticity are generated from the interpolation values of the second all colors.

As a visual characteristic, a second luminance having a high resolution (for example, luminance signal C in FIG. 9) is used for luminance, which is sensitive to luminance resolution and insensitive to chromaticity resolution. ' _l ) is used as it is, and the chromaticity is synthesized. At the pixel position where the false color is generated, the reliability of the chromaticity is higher in the first chromaticity (for example, the chromaticity signal C _{c in} FIG. 9), and conversely, the pixel position where the false color is not generated. Then, based on the assumption that the second chromaticity is higher, the first and second chromaticities are compared, and if the chromaticity difference is large, it is determined that the color is false, and the first chromaticity is synthesized. By increasing the ratio and increasing the composite ratio of the first chromaticity with emphasis on the achromatic part where the false color is conspicuous, the chromaticity of the part where the false color does not appear is highly reliable. The synthesis is performed so that

  By the above processing, for example, for G and E, when G or E is arranged in all horizontal lines and G exists in one horizontal line, a single filter array having no E in that line is provided. The generation of false colors is suppressed from the mosaic image of the color picked up by the plate color image sensor, but the first color interpolated values and the false colors are low in resolution and low in reliability. Interpolated values of the second all colors, which are likely to occur but have high resolution and high color reliability at pixel positions where no false color appears, are obtained. By comparing the interpolation values of these two colors, it is possible to accurately determine the occurrence position of the false color.

  Further, based on the saturation and chromaticity difference obtained from the interpolation values of the two all colors, there are a small number of false colors, which are the characteristics of the interpolation values of the first all colors, and the characteristics of the interpolation values of the second all colors. It is possible to effectively combine the interpolation values of all the two colors by taking advantage of the high resolution and high color reliability, thereby enabling high-quality color component interpolation.

  This process is different from the conventional process of making the generated false color close to an achromatic color, and the interpolation values of the first all colors closer to the true value are used for the false color suppression process than simply making the achromatic color. Therefore, a more natural interpolation image can be obtained.

  In step S309 in FIG. 18, the predetermined value is divided by the sum W of the weighting factors wi. The weighting factor wi is changed according to the distance from the center position, and the sum is raised to a power of 2. Thus, a shift operation may be used instead of the process of dividing a predetermined value by the total sum W of the weighting factors wi (division using the number of pixels). This makes it possible to simplify the calculation.

  Further, the present invention can be applied to a solid-state image pickup device and an image pickup apparatus using the solid-state image pickup device (for example, a digital still camera) such as a digital camera. Specifically, an unknown color component can be estimated by applying the present invention to an imaging apparatus that obtains a color image with a single solid-state imaging device (single-plate color system).

  The series of processes described above can be executed by hardware, but can also be executed by software.

  In this case, the above-described functions are realized by executing the software by the DSP block 216. Further, for example, the demosaic processing unit 253 can be configured by a personal computer 501 as shown in FIG.

  In FIG. 28, a CPU (Central Processing Unit) 511 performs various processes according to a program stored in a ROM (Read Only Memory) 512 or a program loaded from a storage unit 518 to a RAM (Random Access Memory) 513. Execute. The RAM 513 also appropriately stores data necessary for the CPU 511 to execute various processes.

  The CPU 511, ROM 512, and RAM 513 are connected to each other via a bus 514. An input / output interface 515 is also connected to the bus 514.

  The input / output interface 515 includes an input unit 516 including a keyboard and a mouse, an output unit 517 including a display and a speaker, a storage unit 518 including a hard disk, and a communication unit 519 including a modem and a terminal adapter. It is connected. A communication unit 519 performs communication processing via a network including the Internet.

  A drive 520 is connected to the input / output interface 515 as necessary, and a magnetic disk 531, an optical disk 532, a magneto-optical disk 533, a semiconductor memory 534, or the like is appropriately mounted, and a computer program read from them is loaded. It is installed in the storage unit 518 as necessary.

  When a series of processing is executed by software, a program that configures the software is dedicated hardware (for example, DSP block 216 and demosaic processing unit 253, G, E interpolation processing included therein) Unit 262, G + E generation processing unit 263, all-color interpolation processing unit 264, all-color interpolation processing unit 265, or composition processing unit 266), or various programs can be installed by installing various programs. Is installed from a network or a recording medium into a general-purpose personal computer or the like.

  As shown in FIG. 28, this recording medium is distributed to supply a program to the user separately from the main body of the apparatus, and includes a magnetic disk 531 (including a floppy disk) on which the program is stored, an optical disk 532 ( CD-ROM (compact disk-read only memory), DVD (including digital versatile disk)), magneto-optical disk 533 (including MO (magnet optical disk)), or a package medium composed of semiconductor memory 534, etc. In addition, the program is configured by a ROM 512 storing a program supplied to the user in a state of being incorporated in the apparatus main body in advance, a hard disk included in the storage unit 518, and the like.

  In the present specification, the step of describing the program stored in the recording medium is not limited to the processing performed in chronological order in the order in which they are included, but is not necessarily processed in chronological order, either in parallel or individually. The process to be executed is also included.

It is a figure for demonstrating a Bayer arrangement | sequence. It is a figure for demonstrating G stripe RB perfect checkered color filter arrangement | sequence. It is a figure for demonstrating a RGB stripe color filter arrangement | sequence. It is a figure for demonstrating thinning sampling. It is a block diagram which shows the structure of the digital camera to which this invention is applied. It is a figure for demonstrating 4 color arrangement | sequence. It is a block diagram which shows the structure of the DSP block of FIG. It is a block diagram which shows the structure of the demosaic process part of FIG. It is a block diagram which shows the structure of the synthetic | combination process part of FIG. 6 is a flowchart for explaining image processing executed by the DSP block of FIG. 5. It is a flowchart for demonstrating the demosaic process which the demosaic process part of FIG. 8 performs. It is a flowchart for demonstrating G and E interpolation processing. It is a flowchart for demonstrating G and E interpolation processing. It is a flowchart for demonstrating a 1st all color interpolation process. It is a flowchart for demonstrating an estimated value calculation process. It is a flowchart for demonstrating a 1st interpolation process. It is a flowchart for demonstrating a 1st interpolation process. It is a flowchart for demonstrating the weighted average value _Mx calculation process. It is a figure for demonstrating the example of weighting. It is a flowchart for demonstrating a G + E production | generation process. It is a flowchart for demonstrating a 1st G + E production | generation process. It is a flowchart for demonstrating a 2nd G + E production | generation process. It is a flowchart for demonstrating a 2nd all color interpolation process. It is a flowchart for demonstrating a synthetic | combination process. It is a block diagram which shows the structure of the other synthetic | combination process part of FIG. It is a figure for demonstrating 6 color arrangement | sequence. It is a figure explaining the case where G and E are arranged for every line and it considers as a brightness | luminance. It is a block diagram which shows the structure of a personal computer.

Explanation of symbols

  216 DSP blocks, 251 white balance adjustment unit, 253 demosaic processing unit, 261 local region extraction unit, 262 G, E interpolation processing unit, 263 G + E generation processing unit, 264, 265 all color interpolation processing unit, 266 synthesis processing unit, 301 -1, 301-2 color space conversion processing unit, 302 saturation calculation processing unit, 303 blend rate calculation processing unit, 304 chromaticity difference calculation processing unit, 305 blend processing unit

Claims (6)

In an image processing device that generates a color image in which each pixel has a plurality of color components based on a mosaic image captured using a single-plate color image sensor,
The first color component is disposed on the first horizontal line, the second color component is disposed on the second horizontal line, and all the horizontal lines include the first color component and the second color component. Any one of the color components is arranged, and in the first horizontal line, only the pixels in the first horizontal line are used, and the interpolation value of the first color component in the pixel without the first color component The first interpolation value is calculated, and only the pixels in the second horizontal line are used in the second horizontal line, and the second color component in the pixel having no second color component is calculated. First interpolation value calculation means for calculating a second interpolation value that is an interpolation value;
Based on the first interpolation value and the first color component in a pixel in the vicinity of the target pixel of the mosaic image, and the second interpolation value and the second color component in a pixel in the vicinity of the target pixel. Second interpolation value calculation means for calculating interpolation values of all the first colors in the target pixel;
Generating means for generating a third color component composed of the first color component and the second color component in each pixel based on the pixel value of the mosaic image;
A third interpolation value for calculating an interpolated value of all the second colors in the target pixel based on the third color component in the pixel in the vicinity of the target pixel and the pixel value of the pixel in the vicinity of the target pixel; A calculation means;
4th interpolation value calculation means which synthesize | combines the interpolation value of said 1st all color and the interpolation value of said 2nd all color, and calculates the interpolation value of 3rd all color, It is characterized by the above-mentioned. Image processing device. The fourth interpolation value calculation means includes:
Saturation calculating means for calculating saturation based on the interpolation values of the first all colors;
Chromaticity difference calculating means for calculating a chromaticity difference based on the interpolation values of the first all colors and the interpolation values of the second all colors;
A blend rate calculation unit that calculates a blend rate based on the saturation calculated by the saturation calculation unit and the chromaticity difference calculated by the chromaticity difference calculation unit;
The image processing apparatus according to claim 1, further comprising: a combining unit configured to combine the interpolation values of the first all colors and the interpolation values of the second all colors using the blend ratio. The image processing apparatus according to claim 1, wherein the first color component and the second color component have spectral sensitivity highly correlated with each other. 2. The image according to claim 1, further comprising an adjusting unit that adjusts a white balance with respect to pixels of the mosaic image before calculating the first interpolation value and the second interpolation value. 3. Processing equipment. In an image processing method of an image processing apparatus that generates a color image in which each pixel has a plurality of color components based on a mosaic image captured using a single-plate color image sensor,
The first color component is disposed on the first horizontal line, the second color component is disposed on the second horizontal line, and all the horizontal lines include the first color component and the second color component. Any one of the color components is arranged, and in the first horizontal line, only the pixels in the first horizontal line are used, and the interpolation value of the first color component in the pixel without the first color component A first interpolation value calculating step for calculating a first interpolation value which is:
In the second horizontal line, only a pixel in the second horizontal line is used to calculate a second interpolation value that is an interpolation value of the second color component in a pixel having no second color component. A second interpolation value calculating step,
Based on the first interpolation value and the first color component in a pixel in the vicinity of the target pixel of the mosaic image, and the second interpolation value and the second color component in a pixel in the vicinity of the target pixel. A third interpolation value calculating step for calculating the interpolation values of all the first colors in the target pixel;
Generating a third color component composed of the first color component and the second color component in each pixel based on the pixel value of the mosaic image;
A fourth interpolation value for calculating an interpolated value of all the second colors in the target pixel based on the third color component in the pixel in the vicinity of the target pixel and the pixel value of the pixel in the vicinity of the target pixel; A calculation step;
And a fifth interpolation value calculating step of calculating an interpolation value of the third all color by combining the interpolation value of the first all color and the interpolation value of the second all color. Image processing method. A program for generating a color image in which each pixel has a plurality of color components based on a mosaic image captured using a single-plate color image sensor,
The first color component is disposed on the first horizontal line, the second color component is disposed on the second horizontal line, and all the horizontal lines include the first color component and the second color component. Any one of the color components is arranged, and in the first horizontal line, only the pixels in the first horizontal line are used, and the interpolation value of the first color component in the pixel without the first color component A first interpolation value calculating step for calculating a first interpolation value which is:
In the second horizontal line, only a pixel in the second horizontal line is used to calculate a second interpolation value that is an interpolation value of the second color component in a pixel having no second color component. A second interpolation value calculating step,
Based on the first interpolation value and the first color component in a pixel in the vicinity of the target pixel of the mosaic image, and the second interpolation value and the second color component in a pixel in the vicinity of the target pixel. A third interpolation value calculating step for calculating the interpolation values of all the first colors in the target pixel;
Generating a third color component composed of the first color component and the second color component in each pixel based on the pixel value of the mosaic image;
A fourth interpolation value for calculating an interpolated value of all the second colors in the target pixel based on the third color component in the pixel in the vicinity of the target pixel and the pixel value of the pixel in the vicinity of the target pixel; A calculation step;
A fifth interpolation value calculating step of combining the interpolation values for the first all colors and the interpolation values for the second all colors to calculate the interpolation values for the third all colors is executed on a computer. A program characterized by letting

Images