JP2005160044A - Image processing apparatus, image processing method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To more accurately interpolate a mosaic image. <P>SOLUTION: A texture-intensity computation section 334 computes a texture intensity T<SB>H</SB>and a texture intensity T<SB>V</SB>of an R pixel position by using interpolation values GA and EA. A G interpolation section 333-1 computes an interpolation value GF using a horizontally adjacent interpolation value GA and a signal R, and an E interpolation section 333-2 computes an interpolation value EF using a vertically adjacent interpolation value EA and the signal R. The G interpolation section 333-1 computes an interpolation value GU using the interpolation values GA, EA and EF, and the E interpolation section 333-2 computes an interpolation value EU using the interpolation values GA, EA and GF. A G+E generation section 335 combines the interpolation values GU and EF to compute (G+E)<SB>V</SB>. A G+E generation section 336 combines the interpolation values GF and EU to compute (G+E)<SB>H</SB>. A combining section 337 determines the signal G+E using (G+E)<SB>H</SB>, and (G+E)<SB>V</SB>, and the texture intensities T<SB>H</SB>and T<SB>V</SB>. The present invention may be applied to digital cameras. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to an image processing device, an image processing method, and a program, and more particularly, to an image processing device, an image processing method, and a program that can perform interpolation of a mosaic image with higher accuracy.

  When photographing with a photographing device using a single plate type fixed imaging element, only a single spectral sensitivity can be obtained. Therefore, in general, a method is used in which different color filters are attached to a large number of pixels and arranged in a specific pattern to perform color photographing. Since an image captured by this method can obtain only one color for each pixel, an image having a mosaic shape with respect to the color is generated. Therefore, by performing interpolation using color information obtained from surrounding pixels, it is possible to generate an image in which all colors (for example, RGB) are aligned in all pixels.

  Generally, a Bayer array as shown in FIG. 1 is used for the filter array. The Bayer array consists of two G filters that transmit only green (G) light, one R filter that transmits only red (R) light, and B that transmits only blue (B) light. One filter is configured with a total of four as a minimum unit. That is, the G filters are arranged in a checkered pattern and exist at twice the density of the R and B filters. For this reason, a method is often used in which information on other colors is aligned in each pixel by aligning colors for all pixels for G having a large amount of information and having a strong correlation with luminance, and performing pixel interpolation processing using that as reference information. .

  For example, in Patent Document 1, G is pre-aligned with all pixels based on the assumption that the color ratio in the local region is substantially constant, and the average ratio of R and G and the average ratio of B and G in neighboring pixels. A pixel interpolation method for multiplying G at the target pixel position to estimate an unknown color component has been proposed. In such a pixel interpolation method using G as reference information, there is a problem that the final result depends greatly on the interpolation accuracy when all pixels are aligned in G.

  Therefore, in Patent Document 2, a device is devised in which interpolation accuracy is increased by switching neighboring G pixels used for interpolation based on a G gradient value. Further, in Patent Document 3, not only the neighboring G pixels used for interpolation are switched, but also the horizontal and vertical correlation values of the interpolated pixel positions are obtained, and the processing suitable for the case where the horizontal correlation is strong, and the vertical correlation There has been proposed a method of mixing two interpolation values obtained by processing suitable for a case where the value is strong using a correlation value.

The various methods described above are premised on the characteristics of the Bayer arrangement in which G is arranged in a checkered pattern and has a higher sampling frequency than other colors, but the filters of a single-plate solid-state imaging device are in three colors. It is not limited and there are cases where four colors are used. In addition to the so-called complementary color filter arrangement, for example, Patent Document 4 discloses an arrangement of four colors by adding another color (for example, Y) to the three RGB colors in order to improve color reproducibility. It has been proposed to construct a filter. Also, for example, in the Bayer array, the spectral characteristics caused by the difference in the arrangement of R and B adjacent to G and the positional deviation of the filter in the manufacturing process, which are mentioned in Patent Documents 5 and 6, are also mentioned. Due to the difference and the gain error between the lines generated in the solid-state imaging device having different signal output paths in the even lines and the odd lines mentioned in Patent Document 7, Two Gs do not necessarily have the same characteristics, and may actually be an array of four colors.
US Patent, Cok, David R., USP 4,642,678 US Patent, Hibbard, Robert H., USP 5,382,976 Patent No. 2931520 JP 2002-271804 A JP 7-135302 A JP-A-8-191141 Japanese Patent Laid-Open No. 4-154281

  However, the method of interpolating G to all pixels using the G checkered arrangement in the Bayer array cannot be applied to signals obtained from such a four-color filter array. Therefore, there is a problem that the color interpolation method based on the premise that the color image cannot be used as it is.

  Even if G or a signal substituted for G is arranged in a checkered pattern by any method, the correlation values in the horizontal and vertical directions calculated at the target pixel position, as proposed in Patent Document 3, are directly interpolated. In the method to be used, if the correlation determination becomes wrong near the limit resolution due to the influence of noise or the like, the two interpolated values are mixed irregularly, and there is a problem that a very conspicuous artifact is generated.

  The present invention has been made in view of such a situation. Even in a four-color filter array, the present invention provides a process capable of accurately interpolating a luminance signal substituted for G into all pixels, and based on the signal. This makes it possible to perform interpolation processing of the color components.

  The image processing apparatus according to the present invention calculates the change intensity in the vertical direction and the change intensity in the horizontal direction at the target pixel based on the target pixel of the mosaic image and pixels in the vertical and horizontal directions of the target pixel. Based on the calculation means, the target pixel, and the pixels in the vertical direction of the target pixel, the first estimated value of the target pixel of the first color component is calculated, and the horizontal vicinity of the target pixel and the target pixel is calculated A first estimated value calculating means for calculating a second estimated value of the target pixel of the second color component based on the pixels of the vertical direction, a vertical variation strength, a horizontal variation strength, a first estimated value, And a second estimated value calculating means for calculating a third estimated value for the target pixel of the third color component composed of the first color component and the second color component based on the second estimated value. It is characterized by that.

  The change intensity calculation means interpolates the first interpolation values of the first color component and the second color component in the target pixel based on the pixel values of pixels in the vicinity of the target pixel, and positions the vertical direction of the target pixel. The second interpolation value of the first color component and the second color component in the pixel to be interpolated based on the pixel value of the pixel in the vicinity of the target pixel, and the first color component in the pixel located in the horizontal direction of the target pixel The third interpolation value of the color component and the second color component is interpolated based on the pixel value of the pixel in the vicinity of the target pixel, and the vertical interpolation is performed according to the set of differences between the first interpolation value and the second interpolation value. A gradient evaluation value is calculated, a lateral gradient evaluation value is calculated based on a set of differences between the first interpolation value and the third interpolation value, and a change intensity is determined based on the vertical gradient evaluation value and the horizontal gradient evaluation value. Calculating the deviation of the change intensity in the vicinity of the target pixel and the target pixel, and based on the deviation, Change the intensity of the eye pixels can be made to be corrected.

  The second estimated value calculating means calculates a fourth estimated value of the third color component in the target pixel based on the second color component of the pixel in the vicinity of the target pixel and the first estimated value. 4th estimation which calculates the 5th estimated value of the 3rd color component in a focused pixel based on the 1st color component and 2nd estimated value of the pixel of the vicinity of a focused pixel and the estimated value calculation means of this A fifth estimation value is calculated based on the value calculation means, the fourth estimated value, the fifth estimated value, the vertical change intensity at the target pixel, and the horizontal change intensity at the target pixel. And an estimated value calculating means.

  The first color component and the second color component can have spectral sensitivity highly correlated with each other.

  According to the image processing method of the present invention, the change intensity for calculating the vertical change intensity and the horizontal change intensity of the target pixel based on the target pixel of the mosaic image and pixels in the vertical and horizontal directions of the target pixel. A first estimated value calculating step for calculating a first estimated value for the target pixel of the first color component based on the calculation step, the target pixel, and a pixel in the vertical direction of the target pixel; A second estimated value calculating step of calculating a second estimated value of the target pixel of the second color component based on pixels in the horizontal direction of the target pixel; a vertical direction change intensity; and a horizontal direction change intensity. The third estimated value for the target pixel of the third color component composed of the first color component and the second color component is calculated based on the first estimated value and the second estimated value. And an estimated value calculating step. That.

  The program of the present invention is a change intensity calculation step for calculating a vertical change intensity and a horizontal change intensity of a target pixel based on the target pixel of the mosaic image and pixels in the vertical and horizontal directions of the target pixel. A first estimated value calculating step for calculating a first estimated value of the target pixel of the first color component based on the target pixel and a pixel in the vertical direction of the target pixel; and the target pixel and the target pixel A second estimated value calculating step of calculating a second estimated value of the target pixel of the second color component based on the neighboring pixels in the horizontal direction, a vertical change strength, a horizontal change strength, 3rd estimated value which calculates the 3rd estimated value in the attention pixel of the 3rd color component which consists of 1st color component and 2nd color component based on 1 estimated value and 2nd estimated value Computation process including calculation step Characterized in that to execute the data.

  According to the present invention, based on the target pixel of the mosaic image and the pixels in the vertical and horizontal directions of the target pixel, the vertical change strength and the horizontal change strength of the target pixel are calculated. The first estimated value for the target pixel of the first color component is calculated based on the pixels in the vertical direction of the target pixel, and based on the pixels in the horizontal direction of the target pixel and the target pixel, A second estimated value of the pixel of interest of the second color component is calculated, and the first color is based on the vertical direction change intensity, the horizontal direction change intensity, the first estimate value, and the second estimate value. A third estimated value of the target pixel of the third color component composed of the component and the second color component is calculated.

  According to the present invention, mosaic image interpolation can be performed with higher accuracy. In particular, according to the present invention, a stable interpolation result can be obtained even in the vicinity of the limit resolution where an error is likely to occur in direction detection.

  Embodiments of the present invention will be described below. The correspondence relationship between the invention described in this specification and the embodiments of the invention is exemplified as follows. This description is intended to confirm that the embodiments supporting the invention described in this specification are described in this specification. Therefore, even if there is an embodiment that is described in the embodiment of the invention but is not described here as corresponding to the invention, the fact that the embodiment is not It does not mean that it does not correspond to the invention. Conversely, even if an embodiment is described herein as corresponding to an invention, that means that the embodiment does not correspond to an invention other than the invention. Absent.

  Further, this description does not mean all the inventions described in this specification. In other words, this description is for the invention described in the present specification, which is not claimed in this application, that is, for the invention that will be applied for in the future or that will appear and be added by amendment. It does not deny existence.

According to the present invention, an image processing apparatus is provided. In this image processing apparatus (for example, the digital camera 201 in FIG. 2), the first color component (for example, G) is arranged horizontally and vertically every other line, and the second color component (for example, E) is horizontal. And each pixel based on a mosaic image picked up by a single-plate color image pickup device using color filters arranged vertically and every other line, and arranged in horizontal and vertical lines different from the first color component Is an image processing apparatus (for example, the digital camera 201 in FIG. 2) that generates a color image having a plurality of color components, and the pixel of interest of the mosaic image, and pixels in the vertical and horizontal directions of the pixel of interest based on the calculated vertical direction of the change intensity in the pixel of interest (e.g., the texture-intensity-T _V) and lateral change strength (e.g., texture strength T _H) change Degree calculation means (for example, the texture intensity calculation processing unit 334 in FIG. 6), the target pixel (for example, R), and a pixel in the vertical direction of the target pixel (for example, the vertical pixel G of R). Based on the first estimated value (eg, GF) of the target pixel of the first color component (eg, G), and the horizontal direction between the target pixel (eg, R) and the target pixel. A first estimated value (for example, EF) for the target pixel of the second color component (for example, E) is calculated based on a neighboring pixel (for example, a pixel E in the lateral vicinity of R). Estimated value calculation means (for example, the G interpolation processing unit 332-1 or E interpolation processing unit 332-2 in FIG. 6), the vertical change intensity, the horizontal change intensity, the first estimated value, And the first color component based on the second estimate Second estimated value calculation means (for example, G + E generation processing unit 335 in FIG. 6) that calculates a third estimated value (for example, G + E) of the target pixel of the third color component that is the second color component. A G + E generation processing unit 336 and a synthesis processing unit 337).

  The change intensity calculating means calculates the first interpolation values (for example, GA (j ′, k ′) and EA (j ′, k ′) (for example, (( j ′, k ′) interpolate the processing pixel position)) based on the pixel value of the pixel in the vicinity of the target pixel, and the first color component and the second color component in the pixel positioned in the vertical direction of the target pixel Second interpolation values (eg, GA (j ′, k′−1) and GA (j ′, k ′ + 1), and EA (j ′, k′−1) and EA (j ′, k ′ + 1) )) Is interpolated based on the pixel values of pixels in the vicinity of the target pixel, and third interpolation values (for example, GA) of the first color component and the second color component in the pixel located in the horizontal direction of the target pixel. (J′-1, k ′) and GA (j ′ + 1, k ′) and EA (j′−1, k ′) and EA (j ′ + 1, k ′)) Interpolation is performed based on the pixel values of the pixels in the vicinity of the eye pixel, and first interpolation values (for example, GA (j ′, k ′) and EA (j ′, k ′) at the target pixel position (j ′, k ′) are obtained. )) And second interpolated values (eg, GA (j ′, k′−1) and GA (j ′, k ′ + 1), and EA (j ′, k′−1) and EA (j ′, k ′ + 1)) is used to calculate a longitudinal gradient evaluation value (eg, dV) (eg, step S203 of FIG. 13), and a first interpolation value (eg, GA (j ′, k ′)) EA (j ′, k ′)) and third interpolation values (eg, GA (j′−1, k ′) and GA (j ′ + 1, k ′), and EA (j′−1, k ′) ) And EA (j ′ + 1, k ′)), a lateral gradient evaluation value (for example, dH) is calculated (for example, step S203 in FIG. 13), and the vertical gradient evaluation value and the lateral gradient are calculated. The change intensity is determined based on the distribution evaluation value (for example, step S211 in FIG. 13), the deviation of the change intensity in the vicinity of the target pixel and the target pixel is calculated, and the change intensity of the target pixel is corrected based on the deviation. (For example, step S154 in FIG. 12).

Based on the second color component (for example, E) and the first estimated value (for example, GF) of the pixel in the vicinity of the target pixel, the second estimated value calculation unit calculates the third color component (for example, the target pixel) , G + E) third estimated value calculating means (for example, G + E generation processing unit 336 in FIG. 6) for calculating a fourth estimated value (for example, (G + E) _H ), and first pixels of pixels in the vicinity of the target pixel. Fourth estimated value calculation for calculating a fifth estimated value (for example, (G + E) _V ) of the third color component in the pixel of interest based on the color component (for example, G) and the second estimated value (for example, EF). Based on the means (for example, the G + E generation processing unit 335 in FIG. 6), the fourth estimated value, the fifth estimated value, the vertical change strength of the target pixel, and the horizontal change strength of the target pixel, A fifth estimate that calculates an estimate of 3 (eg, (G + E)) Calculating means (for example, synthesis processing unit 337 of FIG. 6) can be made to and a.

  The first color component and the second color component can have spectral sensitivity highly correlated with each other.

According to the present invention, an image processing method is provided. In this image processing method, the first color component (for example, G) is arranged every other line in the horizontal and vertical directions, and the second color component (for example, E) is arranged every other line in the horizontal and vertical directions. And each pixel has a plurality of color components based on a mosaic image captured by a single-plate color image sensor using color filters arranged on horizontal and vertical lines different from the first color component An image processing method of an image processing apparatus (for example, the digital camera 201 in FIG. 2) that generates an image, wherein a target pixel is based on a target pixel of a mosaic image and pixels in the vertical and horizontal directions of the target pixel. vertical change intensity in (e.g., texture strength T _V) and lateral change strength (e.g., texture strength T _H) changes intensity calculating step of calculating (for example, 9, the first color component (for example, G) based on the target pixel (for example, R) and the pixel in the vicinity of the target pixel in the vertical direction (for example, the pixel G in the vertical vicinity of R). A first estimated value calculating step (for example, step S54 in FIG. 9) for calculating a first estimated value (for example, GF) in the target pixel of the target pixel, and the vicinity of the target pixel (for example, R) and the target pixel in the horizontal direction Second estimated value calculation for calculating a second estimated value (for example, EF) at the target pixel of the second color component (for example, E) based on the pixel (for example, a pixel E in the vicinity of R in the lateral direction). The first color component and the second color based on the step (for example, step S55 in FIG. 9), the vertical change strength, the horizontal change strength, the first estimated value, and the second estimated value. A third color component (for example, G + E) of the third pixel in the target pixel Estimate (e.g., G + E) and a third estimate calculation step of calculating (for example, step S58 to step S60 of FIG. 9).

According to the present invention, a program is provided. The program includes a first color component (eg, G) arranged horizontally and vertically every other line, and a second color component (eg, E) arranged horizontally and vertically every other line, In addition, a color image in which each pixel has a plurality of color components based on a mosaic image picked up by a single-plate color image pickup device using color filters arranged on horizontal and vertical lines different from the first color component. An image processing method of an image processing apparatus (for example, the digital camera 201 in FIG. 2) to be generated, in which a vertical direction at a target pixel is determined based on a target pixel of a mosaic image and pixels in the vertical and horizontal directions of the target pixel. direction change strength (e.g., texture strength T _V) and lateral change strength (e.g., texture strength T _H) changes intensity calculating step of calculating (for example, FIG. Step S53), the target pixel (for example, R), and the pixel in the vertical direction of the target pixel (for example, the pixel G in the vertical vicinity of R) of the first color component (for example, G) A first estimated value calculating step (for example, step S54 in FIG. 9) for calculating a first estimated value (for example, GF) in the target pixel, and a horizontal vicinity of the target pixel (for example, R) and the target pixel. A second estimated value calculating step of calculating a second estimated value (for example, EF) in the target pixel of the second color component (for example, E) based on the pixel (for example, a pixel E in the vicinity of R). (For example, step S55 in FIG. 9), the first color component and the second color component based on the vertical direction change intensity, the horizontal direction change intensity, the first estimated value, and the second estimated value. A third color component (for example, G + E) Value (e.g., G + E) third estimate calculation step of calculating (for example, step S58 to step S60 in FIG. 9) and causing the computer to execute.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 2 is a block diagram showing a configuration of a digital camera 201 to which the present invention is applied.

  As shown in FIG. 2, the digital camera 201 includes a lens 211, a diaphragm 212, a CCD (Charge Coupled Devices) image sensor 213, a correlated double sampling (CDS) circuit 214, an A / D converter 215, and a DSP (DSP). Digital Signal Processor) block 216, timing generator 217, D / A converter 218, video encoder 219, display unit 220, CODEC (CODEC: COmpression / DECompression) processing unit 221, memory 222, CPU (Central Processing Unit) 223, and The operation input unit 224 is configured.

  The CCD is a semiconductor element that converts light information into an electrical signal (photoelectric conversion). The CCD image sensor 213 has a plurality of light receiving elements (pixels) that convert light into electricity, and changes in light for each pixel. It converts to an electric signal independently. The correlated double sampling circuit 214 samples the reset noise, which is the main component of the noise included in the output signal of the CCD image sensor 213, from the sampled video signal period of the output pixel signals, and the reference period. Is a circuit that removes the signal by subtracting the sampled signal. The A / D converter 215 converts the supplied analog signal after noise removal into a digital signal.

  The DSP block 216 is a block having a signal processing processor and an image RAM (Random Access Memory), and the signal processing processor performs pre-programmed image processing on image data stored in the image RAM, or It performs image processing configured as arithmetic processing by hardware. The timing generator 217 is a logic circuit that generates various horizontal and vertical driving pulses necessary for driving the CCD and pulses used in the analog front processing in synchronization with the reference clock. The timing clock generated by the timing generator 217 is also supplied to the codec processing unit 221, the memory 222, and the CPU 223 via the bus 225.

  The D / A converter 218 converts the supplied digital signal into an analog signal and outputs the analog signal. The video encoder 219 encodes the supplied analog signal into video data in a format that can be displayed on the display unit 220. The display unit 220 is configured by, for example, an LCD (Liquid Crystal Display) and displays the video signal supplied from the video encoder 219.

  The codec processing unit 221 performs processing by digital video data compression or expansion algorithm such as MPEG (Moving Picture Coding Experts Group / Moving Picture Experts Group), for example. The memory 222 is configured by, for example, a semiconductor memory, a magnetic disk, a magneto-optical disk, or an optical disk, and stores supplied data or outputs stored data based on control of the CPU 223. . Note that the memory 222 may be detachable from the digital camera 201.

  The CPU 223 controls each unit of the digital camera 201 based on a user operation input supplied from the operation input unit 224 via the bus 225. The operation input unit 224 includes, for example, a jog dial, a key, a lever, a button, or a touch panel as well as a button for instructing recording, and receives an operation input by a user.

  Light input through the lens 211 and the diaphragm 212 is incident on the CCD image sensor 213, converted into an electrical signal by photoelectric conversion at the light receiving element, and supplied to the correlated double sampling circuit 214. The correlated double sampling circuit 214 removes noise by subtracting the sampled video signal period and the sampled reference period from the pixel signals output from the CCD image sensor 213, and A / D This is supplied to the converter 215. The A / D converter 215 converts the supplied analog signal after noise removal into a digital signal, and temporarily stores it in the image RAM of the DSP block 216.

  The timing generator 217 controls the CCD image sensor 213, the correlated double sampling circuit 214, the A / D converter 215, and the DSP block 216 so as to maintain image capture at a constant frame rate during imaging. .

  The DSP block 216 receives supply of pixel stream data at a constant rate, temporarily stores it in the image RAM, and executes image processing (to be described later) on the temporarily stored image data in the signal processing processor. When the image block is displayed on the display unit 220 after the image processing is completed, the DSP block 216 displays the code data in the D / A converter 218 and the codec processing unit 221 in the case where the image data is stored in the memory 222. Supply image data.

  The D / A converter 218 converts the digital image data supplied from the DSP block 216 into an analog signal and supplies the analog signal to the video encoder 219. The video encoder 219 converts the supplied analog image signal into a video signal, and outputs the video signal to the display unit 220 for display. That is, the display unit 220 plays the role of a camera finder in the digital camera 201. The codec processing unit 221 performs a predetermined encoding on the image data supplied from the DSP block 216 and supplies the encoded image data to the memory 222 for storage.

  Further, the codec processing unit 221 reads data designated by the user from among the data stored in the memory 222 based on the control of the CPU 223 that has received the user's operation input from the operation input unit 224, and performs predetermined decoding. The decoded signal is output to the DSP block 216. Thus, the decoded signal is supplied to the D / A converter 218 via the DSP block 216, converted into an analog signal, encoded by the video encoder 219, and displayed on the display unit 220.

  Incidentally, the on-chip color filter of the CCD image sensor 213 in FIG. 2 normally uses three or four kinds of colors, and these on-chip color filters have different colors alternately for each light receiving element. They are arranged in a mosaic. In the present invention, four colors obtained by adding a color (hereinafter referred to as E (Emerald)) that has a spectral characteristic close to G to three colors of RGB (Red, Green, Blue). An on-chip color filter with an array of is used. An example of this four-color mosaic arrangement is shown in FIG. In the present specification, the fourth color having spectral characteristics close to G is E, but this color is not limited to Emerald, and any color may be used as long as the spectral characteristics are close to G.

  As shown in FIG. 3, the four-color arrangement has one R filter that transmits only red (R) light, one G filter that transmits only green (G) light, and blue (B). One filter for transmitting only B light and one E filter for transmitting only light of emerald (E), a total of four is configured as a minimum unit. That is, the R, G, B, and E filters exist at the same density.

  As described above, when the four-color mosaic as shown in FIG. 3 is used for the on-chip color filter of the CCD image sensor 213, the image temporarily stored in the image RAM of the DSP block 216 is stored in each pixel. Both have only one color of R, G, B, and E. Therefore, the signal processing processor of the DSP block 216 processes this image by using an image processing program or hardware incorporated in advance, and generates image data having data of all colors in all pixels. Specifically, the signal processing processor of the DSP block 216 first generates a new color (hereinafter referred to as G + E) by combining G and E for all pixels, and G is set to all pixels in the three-color Bayer array. The interpolation method based on the premise of being aligned is also applicable to the mosaic arrangement of four colors.

  FIG. 4 is a block diagram showing a more detailed configuration of the DSP block 216 of FIG.

  As described above, the DSP block 216 includes the image RAM 241 and the signal processor 242. The signal processor 242 includes the white balance adjustment unit 251, the gamma correction unit 252, the demosaic processing unit 253, and the YC conversion. Part 254.

  The mosaic image converted into a digital signal by the A / D converter 215 is temporarily stored in the image RAM 241. The mosaic image is an intensity signal corresponding to any color of R, G, B, or E for each pixel, that is, an array defined by a color filter used in the CCD image sensor 213 (for example, using FIG. 3). This is composed of periodic pattern intensity signals of the four-color mosaic arrangement described.

  The white balance adjustment unit 251 performs processing (white balance adjustment processing) for applying an appropriate coefficient to the mosaic image according to the color of each pixel intensity so that the color balance of the achromatic subject region becomes an achromatic color. ). The gamma correction unit 252 performs gamma correction on each pixel intensity of the mosaic image whose white balance has been adjusted. A numerical value “gamma (γ)” is used to represent the response characteristics of the gradation of the image. The gamma correction is a correction process for correctly displaying the brightness and color saturation of the image displayed on the display unit 220. The signal output to the display unit 220 reproduces the brightness and color of the image by applying a specific voltage for each pixel. However, due to the characteristics (gamma value) of the display unit 220, the brightness and color of the actually displayed image does not double the brightness of the CRT even if the input voltage is doubled (has non-linearity). For this reason, the gamma correction unit 252 performs a correction process so that the brightness and color saturation of the image displayed on the display unit 220 are correctly displayed.

  The demosaic processing unit 253 statistically calculates the color distribution shape, thereby executing demosaic processing for aligning the intensity of the four colors R, G, B, and E at each pixel position of the mosaic image subjected to gamma correction. Therefore, the output signals from the demosaic processing unit 253 are four image signals corresponding to the four colors R, G, B, and E. The YC conversion unit 254 generates and outputs a Y image and a C image (YCbCr image signal) by subjecting the four-channel image of R, G, B, and E to matrix processing and band limitation for chroma components.

  In the signal processing processor 242 of the DSP block 216, the gamma correction unit 252 performs gamma correction before the demosaic processing by the demosaic processing unit 253. This is because the reliability of the demosaic processing of the demosaic processing unit 253 can be further improved by executing the demosaic operation in the non-linear pixel intensity space subjected to the gamma correction.

  For example, when the input image is a high-contrast contour region, the color distribution covers a very bright intensity region and a very dark intensity region. Physically, the object reflected light is obtained by multiplying the variation of the object surface by the incident light intensity from the illumination. Therefore, in the linear pixel intensity space proportional to the incident light intensity to the camera, the bright intensity region The distribution of the object color in the sparsely spreads (sparsely), and the distribution of the object color in the dark pixel intensity region does not spread so much and tends to be compact.

  In the demosaic processing unit 253, demosaic processing is executed by statistically calculating the color distribution shape. However, in the high-contrast contour region, the pixel intensity variation in the bright region and the pixel intensity variation in the dark region are greatly different, and it becomes difficult to apply statistical linear regression. Therefore, prior to demosaic processing in the demosaic processing unit 253, the input data is subjected to non-linear pixel intensity conversion such as gamma correction to raise the dark pixel intensity region (close to the bright pixel intensity region). ) By suppressing the dispersion of the pixel intensity to some extent, the reliability of the linear regression processing executed in the demosaic processing unit 253 can be improved.

  The nonlinear transformation applied for such a purpose is preferably a power transformation with an exponent smaller than 1 as in gamma correction, but a combination of a power portion and a linear portion as in sRGB gamma usually used in a color profile or the like. Even if it is such a conversion, any non-linear conversion may be used as long as it can be regarded as being substantially the same as a power function. Needless to say, even if the non-linear conversion is omitted, a non-linear conversion process such as gamma correction may be performed after the demosaic process.

  5 executes a demosaic process, which is a process of sequentially interpolating or estimating the intensity of a color that is not present at each pixel position so that there are four RGBE colors at all pixel positions. 4 is a block diagram showing a more detailed configuration of a demosaic processing unit 253. FIG.

  The demosaic processing unit 253 is provided with a local region extraction unit 281, a G + E calculation processing unit 282, and a pixel value interpolation unit 283. The local area extraction unit 281 cuts out a pixel of a local area having a predetermined size around the target pixel position from the gamma-corrected mosaic image. Here, the local region to be cut out is a 7 × 7 pixel rectangular region centered on the target pixel position. The G + E calculation processing unit 282 calculates G + E at the target pixel position using the pixels existing in the local region. The pixel value interpolation unit 283 interpolates four colors of RGBE for all pixels using G + E.

  FIG. 6 is a block diagram illustrating a detailed configuration example of the G + E calculation processing unit 282 of FIG. The G + E calculation processing unit 282 includes a G interpolation processing unit 331-1, an E interpolation processing unit 331-2, a G interpolation processing unit 332-1, an E interpolation processing unit 332-2, a G interpolation processing unit 333-1, and an E interpolation process. 333-2, texture intensity calculation processing unit 334, G + E generation processing unit 335, G + E generation processing unit 336, and composition processing unit 337. In this configuration, G + E is generated on R pixels using R, G, and E signals on the mosaic. Here, regression estimation is used to estimate the color component.

The G interpolation processing unit 331-1 calculates a simple G interpolation value GA for all neighboring pixel positions of the target R pixel, and similarly, the E interpolation processing unit 331-2 calculates an E interpolation value EA. The texture strength calculation processing unit 334 uses the interpolation values GA and EA obtained by the G interpolation processing unit 331-1 and the E interpolation processing unit 331-2 to use the texture strength T _H (change strength in the horizontal direction of the R pixel position). ) And the texture intensity T _V (change intensity) in the vertical direction. The G interpolation processing unit 332-1 calculates the second interpolation value GF of G using the horizontal neighboring interpolation value GA and the signal R, and the E interpolation processing unit 332-2 calculates the vertical neighboring interpolation value EA. And the second interpolation value EF of E is calculated using the signal R. The G interpolation processing unit 333-1 uses the interpolation values GA, EA, and EF to use the third interpolation value GU of G, and the E interpolation processing unit 333-2 uses the interpolation values GA, EA, and GF. A third interpolation value EU of E is calculated at the R pixel position.

The G + E generation processing unit 335 calculates (G + E) _V by combining the interpolation values GU and EF, and the G + E generation processing unit 336 calculates (G + E) _H by combining the interpolation values GF and EU. The composition processing unit 337 determines the signal G + E at the R pixel position by using two G + E values, that is, (G + E) _V and (G + E) _H and the texture intensities T _H and T _V.

  In the above description, the case where G + E is generated on the R pixel using R, G, E signals on the mosaic has been described, but the process of generating G + E on the B pixel is described in the above description. To B, G to E, and E to G. The method for generating G + E on the G and E pixels is a state in which the pixel value of either G or E is known in advance, and sufficient accuracy can be obtained by a simple interpolation method. Is omitted.

  Next, processing of the DSP block 216 in FIG. 4 will be described with reference to the flowchart in FIG.

  In step S1, the image RAM 241 is composed of periodic pattern intensity signals in an array (for example, the four-color array described with reference to FIG. 3) determined by the color filter used in the CCD image sensor 213. Obtain a mosaic image and save it temporarily.

  In step S2, the white balance adjustment unit 251 of the signal processing processor 242 applies an appropriate color according to the color of each pixel intensity so that the color balance of the achromatic subject region is achromatic with respect to the mosaic image. A white balance adjustment process, which is a process for applying a coefficient, is performed.

  In step S <b> 3, the gamma correction unit 252 performs gamma correction so that the brightness and color saturation of the image displayed on the display unit 220 are correctly displayed for each pixel intensity of the mosaic image with white balance. Do.

  In step S4, the demosaic processing unit 253 performs a demosaic process described later with reference to FIG.

  In step S5, the YC conversion unit 254 performs YC conversion by performing band limitation on the matrix processing and chroma components on the R, G, B, and E four-channel images that are output from the demosaic processing unit 253, A Y image and a C image are generated, and in step S6, the generated Y image and C image are output, and the process ends.

  Through such processing, the DSP block 216 performs various processing on the supplied mosaic image signal to generate a Y image and a C image, and displays the image data on the display unit 220 based on the control of the CPU 223. Is displayed in the D / A converter 218 and is stored in the memory 222, and is supplied to the codec processing unit 221.

  Next, the demosaic process executed in step S4 of FIG. 7 will be described with reference to the flowchart of FIG.

  In step S21, the local region extraction unit 281 sets one of the unprocessed pixels as a target pixel, and in step S22, selects a predetermined number of pixels around the target pixel position (the pixel region required for the subsequent processing). A sufficient number of pixels) are extracted as local regions and supplied to the G + E calculation processing unit 282 and the pixel value interpolation unit 283.

  In step S <b> 23, the G + E calculation processing unit 282 executes G + E calculation processing, which will be described later using the flowchart of FIG. 9, and supplies the calculated G + E to the pixel value interpolation unit 283.

  In step S24, the pixel value interpolation unit 283 interpolates the pixel values of each color using G + E.

  In step S25, the local region extraction unit 281 determines whether or not processing has been completed for all pixels. If it is determined in step S25 that the process has not been completed for all pixels, the process returns to step S21, and the subsequent processes are repeated. If it is determined in step S25 that the process has been completed for all pixels, the process proceeds to step S5 in FIG.

  In other words, each unit constituting the demosaic processing unit 253 executes the respective processes at the target pixel position when a certain target pixel position is determined, and the processes of steps S21 to S24 are completed for all the pixels. If so, the process is terminated.

  Next, the G + E calculation process executed in step S23 of FIG. 8 will be described with reference to the flowchart of FIG. This process is a process executed by the G + E calculation processing unit 282 in FIG.

  In step S51, the G interpolation processing unit 331-1 executes the first G interpolation processing. Details of this will be described later with reference to FIGS.

  In step S52, the E interpolation processing unit 331-2 executes the first E interpolation processing. Details of this will be described later with reference to FIGS.

  In step S53, the texture strength calculation processing unit 334 executes a corrected texture strength calculation process. Details of this will be described later with reference to FIG.

  In step S54, the G interpolation processing unit 332-1 executes the second G interpolation processing. Details of this will be described later with reference to FIG.

  In step S55, the E interpolation processing unit 332-2 executes a second E interpolation process. Details of this will be described later with reference to FIG.

  In step S56, the G interpolation processing unit 333-1 executes a third G interpolation process. Details of this will be described later with reference to FIG.

  In step S57, the E interpolation processing unit 333-2 executes a third E interpolation process. Details of this will be described later with reference to FIG.

  In step S58, the G + E generation processing unit 335 executes a first G + E generation process. Details of this will be described later with reference to FIG.

  In step S59, the G + E generation processing unit 336 executes a second G + E generation process. Details of this will be described later with reference to FIG.

  In step S60, the composition processing unit 337 executes composition processing. Details of this will be described later with reference to FIG.

  Through the process of FIG. 9, G + E is generated on the R pixel based on the mosaic R, G, and E signals.

  Next, the first interpolation processing (first G interpolation processing and first E interpolation processing) executed in step S51 or step S52 of FIG. 9 will be described with reference to the flowcharts of FIGS. . Since the same processing is executed for both G and E, the portions described as X in FIGS. 10 and 11 are appropriately read as G and E. Specifically, when the processing of FIGS. 10 and 11 is processing corresponding to step S51, it is processing executed by the G interpolation processing unit 331-1. In the case of the processing corresponding to step S52, E interpolation processing unit 331. -2 is a process to be executed.

  In step S101, the G interpolation processing unit 331-1 or the E interpolation processing unit 331-2 initializes the value j of the first register indicating the pixel position to be processed in the local region to j = −n + 1. In step S102, the value k of the second register indicating the pixel position to be processed in the local area is initialized to k = −n + 1. Here, processing is performed on a pixel group of (2n−1) × (2n−1) with the pixel (0, 0) as the center. Note that n is a positive integer, and (j, k) indicates a pixel position where the process is executed when the center (target pixel position) in the local region is (0, 0). Note that the value of n can be considered as an arbitrary constant required for processing. Therefore, when this embodiment is implemented, such detailed setting of register values can be omitted. In particular, if n is a sufficiently large value, it is considered that substantially the same result can be obtained. The same applies to n ′, n ″, and n ″ ″ described later.

  In step S103, the G interpolation processing unit 331-1 or the E interpolation processing unit 331-2 determines whether or not the pixel (j, k) is X. Here, X is set to G when the G interpolation processing unit 331-1 is executing this processing, and is set to E when the E interpolation processing unit 331-2 is executing this processing ( The same applies below). Since the above-described four-color filter array shown in FIG. 3 is known, in this process, the color can be determined using the pixel position.

  When it is determined in step S103 that the color of the pixel (j, k) is X, in step S104, the G interpolation processing unit 331-1 or E interpolation processing unit 331-2 performs interpolation of the pixel (j, k). The value XA is the color X of the pixel (j, k). That is, when it is determined that the color at the processing pixel position is X, no interpolation is required and the signal is used as it is.

  If it is determined in step S103 that the pixel (j, k) is not X, in step S105, the G interpolation processing unit 331-1 or E interpolation processing unit 331-2 determines the color of the pixel (j-1, k). Whether X is X or not.

  If it is determined in step S105 that the color of the pixel (j-1, k) is X, in step S106, the G interpolation processing unit 331-1 or E interpolation processing unit 331-2 determines that the pixel (j, k) Is an average value of the color X of the pixel (j−1, k) and the color X of (j + 1, k). That is, when it is determined that the color of the horizontally adjacent pixel is X, interpolation is performed from the left and right.

  If it is determined in step S105 that the color of the pixel (j-1, k) is not X, the G interpolation processing unit 331-1 or E interpolation processing unit 331-2 in step S107 determines that the pixel (j, k-1) If the color of the pixel (j, k−1) is determined to be X, the interpolated value XA of the pixel (j, k) is obtained from the pixel (j) in step S108. , K−1) and the average of the color of the pixel (j, k + 1). That is, when it is determined that the vertically adjacent pixel is X, vertical interpolation is performed.

  If it is determined in step S107 that the color of the pixel (j, k-1) is not X, in step S109, the G interpolation processing unit 331-1 or E interpolation processing unit 331-2 determines the pixel (j, k). Interpolation value XA is the average of the color of pixel (j-1, k-1), the color of pixel (j + 1, k-1), the color of pixel (j-1, k + 1), and the color of pixel (j + 1, k + 1). And That is, when it is determined that the pixel (j, k), the pixel (j-1, k), and the pixel (j, k-1) are not X, interpolation is performed from the upper left, upper right, lower left, and lower right.

  After step S104, step S106, step S108, or step S109, in step S110, the G interpolation processing unit 331-1 or E interpolation processing unit 331-2 refers to the value k of the second register, and k It is determined whether or not n−1. When it is determined in step S110 that k = n−1 is not satisfied, in step S111, the G interpolation processing unit 331-1 or the E interpolation processing unit 331-2 updates the value k of the second register to k = k + 1. Then, the process returns to step S103, and the subsequent processes are repeated.

  When it is determined in step S110 that k = n−1, in step S112, the G interpolation processing unit 331-1 or the E interpolation processing unit 331-2 stores the first register indicating the pixel position to be processed. With reference to the value j, it is determined whether j = n−1. When it is determined in step S112 that j = n−1 is not satisfied, in step S113, the G interpolation processing unit 331-1 or the E interpolation processing unit 331-2 updates the value j of the first register to j = j + 1. Then, the process returns to step S102, and the subsequent processes are repeated.

  When it is determined in step S112 that j = n−1, in step S114, the G interpolation processing unit 331-1 or the E interpolation processing unit 331-2 determines (2n−1) × (2n−1) pixel X. The set of interpolation values XA is output, and the process proceeds to step S52 or step S53 in FIG.

  10 and FIG. 11 are summarized, the G interpolation processing unit 331-1 or the E interpolation processing unit 331-2 determines the color of the processing pixel position or its neighboring pixel position, and performs linear interpolation by dividing each case. Thus, an interpolation value XA of X is calculated. That is, the interpolation value XA of X (G or E) can be calculated by the processing of FIGS. In other words, the G interpolation value GA and the E interpolation value EA are calculated by the processes of FIGS.

  Next, the corrected texture intensity calculation process executed in step S53 of FIG. 9 will be described with reference to the flowchart of FIG. This process is a process executed by the texture intensity calculation processing unit 334 in FIG.

Texture intensity calculation processing unit 334 at step S151 executes the texture strength T _H, the T _V calculation process. Details of this processing will be described later with reference to FIG.

In step S152, the texture intensity calculation processing unit 334 executes a weighted average value M _TH calculation process. Details of this processing will be described later with reference to FIG.

In step S153, the texture intensity calculation processing unit 334 executes a weighted average deviation _STH calculation process. Details of this processing will be described later with reference to FIG.

Texture intensity calculation processing unit 334 at step S154 executes the texture strength T _H, the T _V correction process. Details of this processing will be described later with reference to FIG. Thereafter, the processing proceeds to step S54 in FIG.

The corrected texture strengths T _H and T _V are calculated by the process of FIG.

Next, details of the texture intensity T _H and T _V calculation processing executed in step S151 of FIG. 12 will be described with reference to the flowchart of FIG. This process is a process for calculating the texture intensities T _H and T _V using the target pixel position and the total of GA and EA at five locations, that is, the top, bottom, left and right. Note that n ′ is a positive integer.

  In step S201, the texture intensity calculation processing unit 334 initializes the value j ′ of the first register indicating the pixel position where the processing is performed among the pixels of the supplied local region to j ′ = − n ′, In step S202, among the supplied pixels in the local region, the value k ′ of the second register indicating the pixel position for executing the process is initialized to k ′ = − n ′. Note that (j ′, k ′) indicates a pixel position where the process is executed when the center (target pixel position) in the local region is (0, 0).

  In step S203, the texture intensity calculation processing unit 334 calculates the lateral gradient evaluation value dH using the equation (1), and calculates the vertical gradient evaluation value dV using the equation (2). That is, the texture intensity calculation processing unit 334 calculates GA + EA at each pixel position, and calculates the sum of the absolute values of the gradient values in the horizontal direction obtained by performing subtraction between the target pixel position and the left and right sides thereof as the lateral gradient evaluation value. While calculating as dH, the sum of absolute values of the gradient values in the vertical direction obtained by performing subtraction between the target pixel position and the upper and lower sides thereof is calculated as the vertical gradient evaluation value dV.

dH = | GA (j′−1, k ′) + EA (j′−1, k ′) − (GA (j ′, k ′) + EA (j ′, k ′)) | + | GA (j ′ + 1 , K ′) + EA (j ′ + 1, k ′) − (GA (j ′, k ′) + EA (j ′, k ′)) | (1)
dV = | GA (j ′, k′−1) + EA (j ′, k′−1) − (GA (j ′, k ′) + EA (j ′, k ′)) | + | GA (j ′, k ′ + 1) + EA (j ′, k ′ + 1) − (GA (j ′, k ′) + EA (j ′, k ′)) |

In step S204, the texture strength calculation processing unit 334 determines whether or not the lateral gradient evaluation value dH> the vertical gradient evaluation value dV. If it is determined that dH> dV, the texture strength calculation processing unit in step S205. 334 sets the texture intensity T _H of the pixel (j ′, k ′) to 0 and the texture intensity T _V of the pixel (j ′, k ′) to 1.

If it is determined not dH> dV in step S204, the texture-intensity-calculation processing unit 334 in step S206, the texture-intensity T _H of the pixel (j ', k') as a 1, the pixel of the (j ', k') the texture strength T _V to 0.

  After the process of step S205 or step S206, the process proceeds to step S207, and the texture intensity calculation processing unit 334 refers to the value k ′ of the second register and determines whether k ′ = n ′. . When it is determined in step S207 that k ′ = n ′ is not satisfied, in step S208, the texture strength calculation processing unit 334 updates the value k ′ of the second register to k ′ = k ′ + 1, and the processing is performed in step S207. Returning to S203, the subsequent processing is repeated.

  When it is determined in step S207 that k ′ = n ′, in step S209, the texture intensity calculation processing unit 334 refers to the value j ′ of the first register indicating the pixel position where the process is performed, and j It is determined whether or not “= n”. If it is determined in step S209 that j ′ = n ′ is not satisfied, the texture intensity calculation processing unit 334 updates the value j ′ of the first register to j ′ = j ′ + 1 in step S210, and the processing is performed in step S210. Returning to S202, the subsequent processing is repeated.

If it is determined in step S209 that j ′ = n ′, the texture intensity calculation processing unit 334 in step S211 sets the texture intensity T _H and T _V of (2n ′ + 1) × (2n ′ + 1) pixels. And the process proceeds to step S152 of FIG.

  With the processing in FIG. 13, the texture intensity is calculated using the processing pixel position and the GA, EA in a total of five locations above, below, left, and right.

Next, details of the weighted average value M _TH calculation process executed in step S152 of FIG. 12 will be described with reference to the flowchart of FIG. This process is a process executed by the texture intensity calculation processing unit 334 in FIG.

In step S251, the texture intensity calculation processing unit 334 initializes the weighted average value M _TH to 0.

In step S252, the texture-intensity-calculation processing unit 334 obtains a texture intensity T _H. This texture strength _TH is a value calculated by the above-described processing of FIG.

  In step S253, the texture intensity calculation processing unit 334 initializes the value j ′ of the first register indicating the pixel position where the processing is performed among the pixels of the supplied local region to j ′ = − n ′, In step S254, among the pixels in the supplied local region, the value k ′ of the second register indicating the pixel position where the process is executed is initialized to k ′ = − n ′.

In step S255, the texture intensity calculation processing unit 334 sets M _TH = M _TH + (texture intensity T _{H of} the pixel (j ′, k ′) to which the weight coefficient wi is applied). That is, the pixel weighted is applied to the _{M TH (j ', k'} ) obtained by adding a texture intensity T _H of is the new M _TH.

  The weighting factor wi is a value set in advance using the distance from the position of the i-th data to the target pixel as an index. For example, a 7 × 7 local region can be set as shown in FIG. In the figure, the square indicates the pixel position of the local area, and the number in the square indicates the weighting coefficient of the data at the corresponding position. Needless to say, these weighting factors are not limited to those shown in FIG. 15, but as shown in FIG. 15, it is preferable that the weighting factors become larger as the pixel is closer to the target pixel.

  In step S256, the texture intensity calculation processing unit 334 refers to the value k ′ of the second register and determines whether k ′ = n ′. If it is determined in step S256 that k ′ = n ′ is not satisfied, in step S257, the texture intensity calculation processing unit 334 updates the value k ′ of the second register to k ′ = k ′ + 1, and the processing is performed. Returning to step S255, the subsequent processing is repeated.

  If it is determined in step S256 that k ′ = n ′, in step S258, the texture intensity calculation processing unit 334 refers to the value j ′ of the first register indicating the pixel position where the processing is performed, and j It is determined whether or not “= n”. If it is determined in step S258 that j ′ = n ′ is not satisfied, the texture intensity calculation processing unit 334 updates the value j ′ of the first register to j ′ = j ′ + 1 in step S259, and the processing is performed in step S259. Returning to S254, the subsequent processing is repeated.

If it is determined in step S258 that j ′ = n ′, the texture intensity calculation processing unit 334 sets M _TH = M _TH / W in step S260. This W is the total sum W of the weighting factors wi.

In step S261, the texture intensity calculation processing unit 334 outputs the weighted average value M _TH of the texture intensity T _H , and the process proceeds to step S153 in FIG.

The weighted average value M _TH of the texture intensity T _H is calculated by the process of FIG.

Next, details of the weighted average deviation S _TH calculation processing executed in step S153 of FIG. 12 will be described with reference to the flowchart of FIG. This process is a process executed by the texture intensity calculation processing unit 334 in FIG.

In step S301, the texture intensity calculation processing unit 334 initializes the weighted average deviation _STH to 0.

In step S302, the texture-intensity-calculation processing unit 334 obtains a texture intensity T _H. This texture strength _TH is a value calculated by the above-described processing of FIG.

In step S303, the texture intensity calculation processing unit 334 acquires the weighted average value M _TH . This weighted average value M _TH is a value calculated by the processing of FIG. 14 described above.

  In step S304, the texture intensity calculation processing unit 334 initializes the value j ′ of the first register indicating the pixel position where the process is performed among the pixels of the supplied local region to j ′ = − n ′, In step S305, the value k ′ of the second register indicating the pixel position to be processed among the supplied pixels in the local region is initialized to k ′ = − n ′.

In step S306, the texture intensity calculation processing unit 334 sets _STH = _STH + texture intensity T _{H of} pixel (j ′, k ′) − (M _TH ) | That is, the pixel (j ', k') to a value obtained by weighting taking the absolute value of the value obtained by subtracting the weighted average M _TH from the texture strength T _H of, those obtained by adding the S _TH is a new S _TH . This texture strength _TH is calculated by the above-described processing of FIG. This weight coefficient wi is the same as the weight coefficient wi in FIG. 14 described above.

  In step S307, the texture intensity calculation processing unit 334 refers to the value k ′ of the second register and determines whether k ′ = n ′. If it is determined in step S307 that k ′ = n ′ is not satisfied, in step S308, the texture intensity calculation processing unit 334 updates the value k ′ of the second register to k ′ = k ′ + 1, and the processing is performed. Returning to step S306, the subsequent processing is repeated.

  If it is determined in step S307 that k ′ = n ′, in step S309, the texture intensity calculation processing unit 334 refers to the value j ′ of the first register indicating the pixel position where the process is performed, and j It is determined whether or not “= n”. If it is determined in step S309 that j ′ = n ′ is not satisfied, the texture strength calculation processing unit 334 updates the value j ′ of the first register to j ′ = j ′ + 1 in step S310, and the processing is performed in step S309. Returning to S305, the subsequent processing is repeated.

If it is determined in step S309 that j ′ = n ′, in step S311, the texture intensity calculation processing unit 334 sets S _TH = S _TH / W. This W is the total sum W of the weighting factors wi as in the above-described processing of FIG.

In step S312, the texture intensity calculation processing unit 334 outputs the weighted average deviation _STH , and the process proceeds to step S154 in FIG.

The weighted average deviation S _TH of the texture intensity T _H is calculated by the process of FIG.

Next, details of the texture intensity T _H and T _V correction processing executed in step S154 of FIG. 12 will be described with reference to the flowchart of FIG. This process is a process executed by the texture intensity calculation processing unit 334 in FIG.

In step S351, the texture intensity calculation processing unit 334 acquires the weighted average deviation _STH . This weighted average deviation S _TH is calculated by the processing of FIG. 16 described above.

In step S352, the texture intensity calculation processing unit 334 calculates R as S _TH × (first constant). This first constant is a predetermined value determined empirically or experimentally, for example.

  In step S353, the texture intensity calculation processing unit 334 determines whether or not R> 1, and if it is determined that R> 1, R = 1 is set in step S354.

  If it is determined in step S353 that R> 1 is not satisfied, that is, R ≦ 1, the process of step S354 is skipped.

When it is determined in step S353 that R> 1 is not satisfied, or after the processing in step S354, in step S355, the texture intensity calculation processing unit 334 sets T _H = T _H × (1−R) + R / 2, and T _V = T _V × (1−R) + R / 2. That is, the texture strengths T _H and T _V are corrected. Thereafter, the process in step S154 in FIG. 12 is terminated, and the process proceeds to step S54 in FIG.

By the processing of FIG. 17, the values of the texture strengths T _H and T _V are corrected based on the weighted average deviation S _TH .

  Next, the second G interpolation process and the second E interpolation process executed in steps S54 and S55 of FIG. 9 will be described. The difference between the G interpolation processing unit 332-1 and the E interpolation processing unit 332-2 in FIG. 6 is the difference between the neighboring pixels used in the processing. Which of the neighboring pixels is used in the vertical direction and the horizontal direction by weighting? Can be switched. For example, if the weight in FIG. 18 is used, a sample in the vertical direction is used, and if the weight in FIG. 19 is used, a sample in the horizontal direction is used. In the present embodiment, a sample in the horizontal direction is used for the G interpolation processing unit 332-1, and a sample in the vertical direction is used for the E interpolation processing unit 332-2. It should be noted that just because the texture direction is the vertical direction, vertical neighboring pixels are not always used for estimation. For example, by applying linear regression to the color distribution of the local region used in the present embodiment and estimating the unknown color component, in order to obtain a color distribution suitable for calculating the regression line, In some cases, better estimation results can be obtained by using neighboring pixels in a direction perpendicular to the texture direction. Regardless of which estimation method is used, the estimated value with high reliability for the texture in the horizontal direction and the estimated value with high reliability for the texture in the vertical direction are determined depending on the difference in the neighboring pixels used for the estimation. It is only necessary to obtain two estimated values. In the following description, the portions described as X and dir are read as G and H in the process of the G interpolation processing unit 332-1 and appropriately read as E and V in the process of the E interpolation processing unit 332-2. H and V attached to represent the direction represent the directions of neighboring pixels used for the interpolation process.

  FIG. 20 is a flowchart illustrating the second interpolation process (second G interpolation process or second E interpolation process) executed in step S54 or step S55 of FIG. Since the same processing is executed for both G and E, the portions described as X and dir in FIG. 20 are appropriately read as G and H, and E and V. Specifically, when the process in FIG. 20 corresponds to the process in step S54, the process is executed by the G interpolation processing unit 332-1. Therefore, X and dir are read as G and H, and FIG. In the case of the process corresponding to the process of step S55, X and dir are read as E and V because the process is executed by the E interpolation processing unit 332-2.

In step S401, the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 executes an R weighted average value M _Rdir calculation process. Details of this processing will be described later with reference to FIG.

In step S402, the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 executes the XA weighted average value M _XAdir calculation processing. Details of this processing will be described later with reference to FIG.

In step S403, the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 performs an R weighted variance approximate value VA _RRdir calculation process. Details of this processing will be described later with reference to FIG.

In step S404, the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 executes R, XA weighted covariance approximate value VA _RXAdir calculation processing. Details of this processing will be described later with reference to FIG.

In step S405, the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 executes a regression line inclination K _dir calculation process. Details of this processing will be described later with reference to FIG.

  In step S406, the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 executes an X estimated value XF calculation process using a regression line. Details of this processing will be described later with reference to FIG. After the process of step S406, the process proceeds to step S56 of FIG.

Next, with reference to the flowchart of FIG. 21, the details of the R weighted average value M _Rdir calculation processing or the XA weighted average value M _XAdir calculation processing executed in step S401 or step S402 of FIG. explain. Since the same processing is executed for both R and XA, the portion described as Y in FIG. 21 is appropriately read as R and XA. Specifically, when the process in FIG. 21 is a process corresponding to step S54 in FIG. 9, the G interpolation processing unit 332-1 performs the process, and the process in FIG. 21 corresponds to step S55 in FIG. In this case, the process is executed by the E interpolation processing unit 332-2. Note that n ″ is a positive even number. Note that (j ″, k ″) indicates a pixel position where the process is executed when the center (target pixel position) in the local region is (0, 0).

In step S451, the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 initializes the weighted average value M _Ydir to 0.

  In step S452, the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 initially sets the value j ″ of the first register indicating the pixel position to be processed among the supplied pixels in the local region. J ″ = − n ″, and in step S453, among the supplied pixels in the local region, a value k ″ of the second register indicating the pixel position for executing the processing is initialized and k ″. = −n ″.

In step S454, the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 determines that M _Ydir = M _Ydir + (Y of the pixel (j ″, k ″) to which the weight coefficient _w i _dir is applied). To do. That, M pixel weighting is applied to _{Ydir (j '', k '} ') obtained by adding the Y of is the new M _YDIR.

  In step S455, the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 refers to the value k ″ of the second register and determines whether k ″ = n ″. When it is determined in step S455 that k ″ = n ″ is not satisfied, in step S456, the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 sets the value k ″ of the second register to k. After updating to “= k” +2, the processing returns to step S454, and the subsequent processing is repeated.

  If it is determined in step S455 that k ″ = n ″, in step S457, the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 indicates the first pixel position indicating the pixel to be processed. Referring to the register value j ″, it is determined whether j ″ = n ″. When it is determined in step S457 that j ″ = n ″ is not satisfied, in step S458, the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 sets the value j ″ of the first register to j ′. After updating to '= j' '+ 2, the process returns to step S453, and the subsequent processes are repeated.

If it is determined in step S457 that j ″ = n ″, in step S459, the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 sets M _Ydir = M _Ydir / W _dir . . The W _dir is the sum W _dir weighting coefficients wi _dir.

In step S460, the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 outputs the weighted average value _MYdir , and the process proceeds to step S402 or step S403 in FIG.

  By the processing in FIG. 21, the weighted average values of R and XA in the vicinity of the target pixel are calculated.

Next, details of the R weighted variance approximate value VA _RRdir calculation process executed in step S403 of FIG. 20 will be described with reference to the flowchart of FIG. This process is a process executed by the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 in FIG.

In step S501, the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 initializes the weighted variance approximate value VA _RRdir to 0.

In step S502, the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 acquires the weighted average M _Rdir . This weighted average M _Rdir is a value calculated by the above-described processing in FIG. 21 (processing in step S401 in FIG. 20).

  In step S503, the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 initially sets the value j ″ of the first register indicating the pixel position to be processed among the supplied local region pixels. J ″ = − n ″, and in step S504, the value k ″ of the second register indicating the pixel position for executing the processing is initialized among the supplied pixels in the local region, and k ″. = −n ″.

In step S505, the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 is subjected to VA _RRdir = VA _RRdir + weighting coefficient _w i _dir | (R of pixel (j ″, k ″)) − (M _Rdir ) | That is, a value _obtained by adding the VA _RRdir to the weighted value _obtained by subtracting the R weighted average value M _Rdir from the R of the pixel (j ″, k ″) is a new VA _RRdir . Is done.

  In step S506, the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 refers to the value k ″ of the second register and determines whether k ″ = n ″. When it is determined in step S506 that k ″ = n ″ is not satisfied, in step S507, the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 sets the value k ″ of the second register to k. After updating to "= k" +2, the process returns to step S505, and the subsequent processes are repeated.

  When it is determined in step S506 that k ″ = n ″, in step S508, the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 indicates the first pixel position indicating the pixel position to be processed. Referring to the register value j ″, it is determined whether j ″ = n ″. When it is determined in step S508 that j ″ = n ″ is not satisfied, in step S509, the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 sets the value j ″ of the first register to j. After updating to “= j” +2, the process returns to step S504, and the subsequent processes are repeated.

When it is determined in step S508 that j ″ = n ″, in step S510, the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 sets VA _RRdir = VA _RRdir / W _dir . . The W _dir is the sum W _dir weighting coefficients wi _dir.

In step S511, the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 outputs the R weighted variance approximate value VA _RRdir , and the process proceeds to step S404 in FIG.

The R weighted variance approximate value VA _RRdir is calculated by the process of FIG.

  Next, the R and XA weighted covariance approximation calculation processing executed in step S404 in FIG. 20 will be described with reference to the flowchart in FIG.

In step S521, the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 initializes the R, XA weighted covariance value VA _XAdir that is the output value to 0.

In step S522, the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 acquires the weighted average M _Rdir and the weighted average M _XAdir . The weighted average M _Rdir is a value calculated by the process of step S401 in FIG. 20 (process of FIG. 21), and the weighted average M _XAdir is calculated by the process of step S402 of FIG. 20 (process of FIG. 21). Value.

  In step S523, the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 initializes the value j ″ of the first register indicating the pixel position to be processed, and j ″ = − n ′. 'And

  In step S524, the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 initializes the value k ″ of the second register indicating the pixel position to be processed, and k ″ = − n ′. 'And

  In step S525, the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 performs an integration process which will be described later with reference to FIG.

  In step S526, the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 refers to the value k ″ of the second register indicating the pixel position to be processed, and k ″ = n ″. Judge whether there is.

  When it is determined in step S526 that k ″ = n ″ is not satisfied, in step S527, the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 sets the value k ″ of the second register. , K ″ = k ″ +2, and the process returns to step S525, and the subsequent processes are repeated.

  If it is determined in step S526 that k ″ = n ″, in step S528, the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 indicates the first pixel position indicating the pixel position to be processed. Referring to the register value j ″, it is determined whether or not j ″ = n ″.

  If it is determined in step S528 that j ″ = n ″ is not satisfied, in step S529, the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 sets the value j ″ of the first register. , J ″ = j ″ +2, and the process returns to step S524, and the subsequent processes are repeated.

If it is determined in step S528 that j ″ = n ″, in step S530, the G interpolation processing unit 332-1 or E interpolation processing unit 332-2 sets VA _RXAdir = VA _RXAdir / W _dir , This is output as a weighted covariance approximate value VA _RRdir of R and XA, and the process proceeds to step S405 in FIG.

  When calculating the slope of the regression line by a conventional method, it is necessary to perform an operation as shown in the following equation (3).

  However, when the calculation is performed using Expression (3), the problem is that the number of multiplications increases in proportion to the number N of data. Therefore, consider approximating this multiplication with a simple operation. If the data Xi and the data Yi are normalized to the interval [0, 1] for simplicity, 0 ≦ Mx and My ≦ 1 are satisfied, and −1 <Xi−Mx and Yi−My <1 are satisfied. To establish. Therefore, considering a multiplication pq of two variables p = Xi−Mx and q = Yi−My satisfying −1 <p, q <1, pq takes a range shown in FIG. As an example of a method for obtaining an approximate value based on the signs and absolute values of p and q, an example of an approximation method that can be realized by conditional calculation, absolute value calculation, and calculation only by addition / subtraction will be described.

That is,
When | p | ≧ | q | and p ≧ 0, pq is approximated by q,
When | p | ≧ | q | and p <0, approximate pq by −q,
When | p | <| q | and q ≧ 0, pq is approximated by p,
When | p | <| q | and q <0, the calculation can be simplified by approximating pq by −p.

  In this approximation method, when p = q, the square of p (or the square of q) is approximated to the absolute value of p (or q).

  FIG. 25 shows pq approximated using this approximation method, and FIG. 26 shows the square error due to the approximation calculation. It can be seen that there is no significant difference between the possible range of the pq value described with reference to FIG. 24 and the possible range of the pq value approximated using the approximation method shown in FIG. It can be seen from the square error by the approximation calculation shown in FIG. 26 that the difference is slight.

  Next, the integration process executed in step S525 of FIG. 23 will be described with reference to the flowchart of FIG. The integration process is a process executed when an approximate covariance value is calculated using the product approximation method described above, assuming that the value of the color component is normalized to [0, 1]. is there.

In step S551, the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 sets {R (j ″, k ″) − M _Rdir } to p.

In step S552, the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 sets {XA (j ″, k ″) − M _XAdir } to q.

  In step S553, an integration approximation process described later with reference to FIG. 28 is executed.

In step S554, the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 performs VA _XAdir = VA _XAdir + weighting coefficient _w i _dir (approximate value of pq). The process proceeds to S526.

  Next, the integration approximation process executed in step 553 in FIG. 27 will be described with reference to the flowchart in FIG.

  In step S601, the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 satisfies | p | ≧ | q | using the values p and q replaced in step S551 and step S552 in FIG. Determine whether or not.

  When it is determined in step S601 that | p | ≧ | q |, in step S602, the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 determines whether p ≧ 0. To do.

  When it is determined in step S602 that p ≧ 0 is not satisfied, in step S603, the G interpolation processing unit 332-1 or E interpolation processing unit 332-2 sets the approximate value of pq to −q, and the processing is as shown in FIG. The process proceeds to step S554.

  If it is determined in step S602 that p ≧ 0, in step S604, the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 sets the approximate value of pq to + q, and the processing is the step of FIG. The process proceeds to S554.

  If it is determined in step S601 that | p | ≧ | q | is not satisfied, in step S605, the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 determines whether q ≧ 0. To do.

  When it is determined in step S605 that q ≧ 0, in step S606, the G interpolation processing unit 332-1 or E interpolation processing unit 332-2 sets the approximate value of pq to + p, and the process is as shown in FIG. Proceed to step S554.

  When it is determined in step S605 that q ≧ 0 is not satisfied, in step S607, the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 sets the approximate value of pq to −p, and the processing is as shown in FIG. The process proceeds to step S554.

Next, details of the regression line inclination K _dir calculation processing executed in step S405 of FIG. 20 will be described with reference to the flowchart of FIG. This process is a process executed by the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 in FIG.

In step S651, the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 acquires the weighted variance approximate value VA _RRdir . This value is calculated by the process in step S403 (FIG. 22) in FIG.

In step S652, the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 acquires the weighted covariance approximation value VA _RXAdir . This value is calculated by the processing in step S404 (FIG. 23) of FIG.

In step S653, G interpolation processing section 332-1 or E interpolation processing section 332-2 determines whether the VA _RRdir <(second constant) is the VA _RRdir <(second constant) If it is determined, VA _RRdir = (second constant) is set in step S654. The second constant is a predetermined value determined empirically or experimentally, for example.

If it is determined in step S653 that VA _RRdir <(second constant) is not satisfied, that is, VA _RRdir ≥ (second constant), the process of step S654 is skipped.

If it is determined in step S653 that VA _RRdir <(second constant) is not satisfied, or after the processing in step S654, in step S655, the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 performs K _dir = VA _RXAdir / VA _RRdir . Thereafter, the processing proceeds to step S406 in FIG.

The slope K _dir of the regression line is calculated by the process of FIG.

  Next, the X estimated value XF calculation process by the regression line executed in step S406 of FIG. 20 will be described with reference to the flowchart of FIG.

In step S701, the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 acquires the weighted average M _Rdir . This value is calculated in step S401 of FIG. 20 described above.

In step S702, the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 acquires the weighted average M _XAdir . This value is calculated in step S402 in FIG.

In step S703, G interpolation processing section 332-1 or E interpolation processing section 332-2 obtains the inclination K _dir of the regression line. This value is calculated in step S405 of FIG.

In step S704, the G interpolation processing unit 332-1 or the E interpolation processing unit 332-2 estimates the estimated value XF of the pixel of interest XF = {K _dir (R-weighted average M _{Rdir of the} pixel of interest) + weighted average M _XAdir. }. That is, the regression line slope K _dir is multiplied by a value _obtained by subtracting the weighted average M _Rdir from the R of the pixel of interest, and the value _obtained by adding the weighted average M _XAdir to the value is the estimated value XF of the pixel of interest X. Is done. Thereafter, the process in FIG. 20 is terminated, and the process proceeds to step S56 in FIG.

  30, the G interpolation processing unit 332-1 calculates the estimated value GF of the target pixel G, and the E interpolation processing unit 332-2 calculates the estimated value EF of the target pixel E.

  Next, the third interpolation process (third G interpolation process or third E interpolation process) executed in step S56 or step S57 in FIG. 9 will be described with reference to FIG. Since the same processing is executed for both G and E, the portion described as X in FIG. 31 is appropriately read as G and E. Specifically, when the process of FIG. 31 is a process corresponding to the process of step S56 of FIG. 9, the process is executed by the G interpolation processing unit 333-1, so that X is read as G and the process of FIG. 9 is a process executed by the E interpolation processing unit 333-2 in the case of the process corresponding to the process of step S57 in FIG. Here, when X is G, Y is E, and when X is E, Y is G.

  In step S751, the G interpolation processing unit 333-1 or the E interpolation processing unit 333-2 acquires the YF of the target pixel. Specifically, when X is G, Y is E, and when X is E, Y is G. Therefore, the G interpolation processing unit 333-1 acquires EF, and the E interpolation processing unit 333-2 acquires GF. That is, the G interpolation processing unit 333-1 acquires the estimated value XF (EF) calculated by the E interpolation processing unit 332-2 in the process of FIG. 30, and the E interpolation processing unit 333-2 receives the process of FIG. To obtain the estimated value XF (GF) calculated by the G interpolation processing unit 332-1.

In step S752, the G interpolation processing unit 333-1 or the E interpolation processing unit 333-2 executes a weighted average M _XA calculation process. Details of this processing will be described later with reference to FIG.

In step S753, the G interpolation processing unit 333-1 or the E interpolation processing unit 333-2 executes a weighted average M _YA calculation process. Details of this processing will be described later with reference to FIG.

  In step S754, the G interpolation processing unit 333-1 or the E interpolation processing unit 333-2 executes an X interpolation value XU calculation process. Details of this processing will be described later with reference to FIG. After the process of step S754, the process of FIG. 31 is terminated, and the process proceeds to step S58 of FIG.

Next, the weighted average M _XA calculation process in step S752 of FIG. 31 will be described with reference to the flowchart of FIG. Note that n ′ ″ is a positive integer. In addition, (j ′ ″, k ′ ″) indicates a pixel position where the process is executed when the center (target pixel position) in the local region is (0, 0).

In step S801, the G interpolation processing unit 333-1 or the E interpolation processing unit 333-2 initializes the weighted average value M _XA to 0.

  In step S802, the G interpolation processing unit 333-1 or the E interpolation processing unit 333-2 sets the value j ′ ″ of the first register indicating the pixel position where the processing is performed among the supplied pixels in the local region. In step S803, the value k ′ ″ of the second register indicating the pixel position where the process is executed is initialized in step S803. K ′ ″ = − n ′ ″.

In step S804, the G interpolation processing unit 333-1 or the E interpolation processing unit 333-2 sets M _XA = M _XA + (pixel (j ′ ″, k ′ ″) to which the weight coefficient w′i is applied). Interpolated value XA). That is, the pixel weighting factor w'i has been subjected to _{M XA (j ''',}k''') obtained by adding the interpolated value XA of is the new M _XA.

  In step S805, the G interpolation processing unit 333-1 or the E interpolation processing unit 333-2 refers to the value k ″ ′ of the second register and determines whether k ′ ″ = n ′ ″. judge. If it is determined in step S805 that k ′ ″ = n ′ ″ is not satisfied, in step S806, the G interpolation processing unit 333-1 or the E interpolation processing unit 333-2 sets the value k ″ of the second register. 'Is updated to k' '' = k '' '+ 1, the process returns to step S804, and the subsequent processes are repeated.

  When it is determined in step S805 that k ′ ″ = n ′ ″, in step S807, the G interpolation processing unit 333-1 or the E interpolation processing unit 333-2 indicates a pixel position where the process is executed. With reference to the value j ′ ″ of the first register, it is determined whether j ′ ″ = n ′ ″. When it is determined in step S807 that j ′ ″ = n ′ ″ is not satisfied, in step S808, the G interpolation processing unit 333-1 or the E interpolation processing unit 333-2 determines the value j ′ ″ of the first register. Is updated to j ″ = j ′ ″ + 1, the process returns to step S803, and the subsequent processes are repeated.

If it is determined in step S807 that j ′ ″ = n ′ ″, in step S809, the G interpolation processing unit 333-1 or the E interpolation processing unit 333-2 determines that M _XA = M _XA / W ′. And This W ′ is the sum W ′ of the weighting factors w′i.

In step S810, the G interpolation processing unit 333-1 or the E interpolation processing unit 333-2 outputs the weighted average value M _XA , and the process proceeds to step S753 in FIG.

The weighted average value M _{XA of XA} is calculated by the processing of FIG.

In the process of FIG. 32, only the weighted average M _XA is described, but as described above, when the G interpolation processing unit 333-1 executes this process, X is read as G, and Y is When E is read as E and this process is executed by the E interpolation processing unit 333-2, X is read as E and Y is read as G. That is, M _XA and M _YA are calculated by executing the processing of FIG. 32 for G and E by the G interpolation processing unit 333-1 and the E interpolation processing unit 333-2. In FIG. 32, when the weighted average M _XA and the weighted average M _YA are calculated, the process proceeds to step S754 in FIG.

  Next, the X interpolation value XU calculation process executed in step S754 in FIG. 31 will be described with reference to the flowchart in FIG. Here, when X is G, Y is appropriately replaced with E, and when X is E, Y is appropriately replaced with G.

  In step S851, the G interpolation processing unit 333-1 or the E interpolation processing unit 333-2 acquires the interpolation value YF of the target pixel. This value is the same as that acquired in step S751 of FIG. 31 described above.

In step S852, the G interpolation processing unit 333-1 or the E interpolation processing unit 333-2 acquires the weighted average M _XA . This value is calculated by the process of step S752 (FIG. 32) of FIG.

In step S853, the G interpolation processing unit 333-1 or the E interpolation processing unit 333-2 acquires the weighted average M _YA . This value is calculated by the processing in step S753 (FIG. 32) of FIG.

In step S854, G interpolation processing section 333-1 or E interpolation processing section 333-2 determines whether an M _YA <(third constant) is the M _YA <(third constant) If it is determined, M ′ _YA = (third constant) is set in step S855. Note that the third constant is a predetermined value determined empirically or experimentally, for example.

If it is determined in step S854 that M _YA <(third constant) is not satisfied, that is, M _YA ≧ (third constant), the G interpolation processing unit 333-1 or the E interpolation processing unit 333 is determined in step S856. -2 is set as M ′ _YA = M _YA .

After the process of step S855 or the process of step S856, in step S857, the G interpolation processing unit 333-1 or the E interpolation processing unit 333-2 determines the X interpolation value XU = (M _XA / M ′ _YA ) (YF−M _{YA at the} target pixel position) + M _XA . Thereby, an interpolation value XU of X is calculated.

  In step S858, the G interpolation processing unit 333-1 or the E interpolation processing unit 333-2 outputs the X interpolation value XU of the target pixel position, and then the processing in FIG. 33 is terminated, and the processing is performed in step S58 in FIG. Proceed to

  Next, the first G + E generation process executed in step S58 of FIG. 9 will be described with reference to the flowchart of FIG. This process is a process executed by the G + E generation processing unit 335 in FIG.

  In step S901, the G + E generation processing unit 335 acquires the G interpolation value GU at the target pixel position calculated by the third interpolation processing. Specifically, the G interpolation value GU (a value corresponding to the interpolation value XU in FIG. 33) calculated by the G interpolation processing unit 333-1 is acquired in the process of step S56 in FIG.

  In step S902, the G + E generation processing unit 335 acquires the interpolation value EF of E at the target pixel position calculated by the second interpolation process. Specifically, the E interpolation value EF calculated by the E interpolation processing unit 332-2 (a value corresponding to the interpolation value XF in FIG. 30) is acquired in the process of step S55 of FIG.

In step S903, the G + E generation processing unit 335 sets (G + E) _V of the target pixel position = (GU of the target pixel position + EF of the target pixel position) / 2. That is, the average value of the GU at the target pixel position and the EF at the target pixel position is set to (G + E) _V of the target pixel position.

In step S904, the G + E generation processing unit 335 outputs (G + E) _V of the target pixel position, and the process proceeds to step S59 in FIG.

  Next, the second G + E generation process executed in step S59 of FIG. 9 will be described with reference to the flowchart of FIG. This process is a process executed by the G + E generation processing unit 336 in FIG.

  In step S951, the G + E generation processing unit 336 acquires the G interpolation value GF of the target pixel position calculated by the second interpolation process. Specifically, the G interpolation value GF (the value corresponding to the interpolation value XF in FIG. 30) calculated by the G interpolation processing unit 332-1 is acquired in the process of step S54 in FIG.

  In step S952, the G + E generation processing unit 336 acquires the interpolation value EU of E at the target pixel position calculated by the third interpolation process. Specifically, the E interpolation value EU (value corresponding to the interpolation value XU in FIG. 33) calculated by the E interpolation processing unit 333-2 is acquired in the process of step S57 in FIG.

In step S953, the G + E generation processing unit 336 sets (G + E) _H = (GF of the target pixel position + EU of the target pixel position) / 2 of the target pixel position. That is, the average value of GF at the target pixel position and EU at the target pixel position is set to (G + E) _{H of the} target pixel position.

In step S954, the G + E generation processing unit 336 outputs (G + E) _H of the target pixel position, and the process proceeds to step S60 in FIG.

  Next, the synthesis process executed in step S60 of FIG. 9 will be described with reference to the flowchart of FIG. This process is a process executed by the synthesis processing unit 337 in FIG.

In step S1001, the composition processing unit 337 acquires (G + E) _V of the target pixel position calculated by the first G + E generation process. Specifically, (G + E) _V calculated by the G + E generation processing unit 335 is acquired in the process of step S58 (FIG. 34) of FIG.

In step S1002, the composition processing unit 337 acquires (G + E) _H of the target pixel position calculated by the second G + E generation process. Specifically, (G + E) _H calculated by the G + E generation processing unit 336 is acquired in the process of step S59 (FIG. 35) of FIG.

In step S1003, the synthesis processing unit 337 acquires the corrected texture strengths T _H and T _V calculated by the corrected texture strength calculation processing. Specifically, the corrected texture strengths T _H and T _V calculated by the texture strength calculation processing unit 334 are acquired in the process of step S53 (FIG. 12) of FIG.

In step S1004, in the color interpolation by linear regression used in this embodiment, (G + E) _V created using the vertical sample is highly reliable for the horizontal texture, and the horizontal sample is used. Since (G + E) _H created in this way has high reliability in the texture in the vertical direction, the composition processing unit 337 has (G + E) = (G + E) _V × T _H + target pixel position of the target pixel position = (G + E) (G + E) _H × T _V ).

  In step S1005, the composition processing unit 337 outputs (G + E) of the target pixel position, the processing in FIG. 9 is terminated, and the processing proceeds to step S24 in FIG.

In the above example, the texture intensities T _H and T _V are interpolated by the G interpolation processor 331-1 (first G interpolation process) and the E interpolation processor 331-2 (first E interpolation process). However, it can also be directly performed directly from a mosaic image that is an original signal. FIG. 37 is a flowchart for describing processing for directly calculating the texture strengths T _H and T _V from the mosaic image instead of FIG. In the example of FIG. 37, a portion described as M represents a mosaic image signal.

  In step S1051, the texture intensity calculation processing unit 334 initializes the value j ′ of the first register indicating the pixel position where the process is executed among the pixels of the supplied local region to j ′ = − n ′, In step S1052, among the pixels in the supplied local region, the value k ′ of the second register indicating the pixel position for executing the process is initialized to k ′ = − n ′.

  In step S1053, the texture intensity calculation processing unit 334 calculates the lateral gradient evaluation value dH using Expression (4) and calculates the vertical gradient evaluation value dV using Expression (5).

dH = {(| M (j′−2, k ′) − M (j ′, k ′) | + | M (j ′ + 2, k ′) − M (j ′, k ′) |) / 2 + | M (j′−1, k ′) − M (j ′ + 1, k ′) |} / 2
... (4)
dV = {(| M (j ′, k′−2) −M (j ′, k ′) | + | M (j ′, k ′ + 2) −M (j ′, k ′) |) / 2 + | M (j ′, k′−1) −M (j ′, k ′ + 1) |} / 2
... (5)

  Since the processing from step S1054 to step S1061 is the same as the processing from step S204 to step S211 of FIG. 13 described above, the description thereof is omitted.

The texture strengths T _H and T _V can be easily calculated by the processing of FIG. The texture intensities T _H and T _V obtained by the process of FIG. 37 are affected by the difference in filter arrangement depending on whether the color of the pixel of interest is R or B, but the texture obtained by the process of FIG. Since results close to the intensities T _H and T _V are obtained, this processing may be applied.

  The above processing is summarized as follows.

  The texture intensity is a value (change intensity) representing the directionality of the texture calculated from the gradient value at the target pixel position. The estimation accuracy of the color component greatly depends on the selection of neighboring pixels used for estimation. Therefore, estimation using the neighboring pixels in the vertical direction and estimation using the neighboring pixels in the horizontal direction are performed, and appropriate estimation is performed by blending each estimation result based on the texture intensity.

A two-stage process is performed to calculate the texture intensity. First, a vertical gradient evaluation value and a horizontal gradient evaluation value are calculated based on gradient values in the vertical direction and horizontal direction of the target pixel position, and a direction in which the texture change is large is determined. Here, the texture strength in the direction in which the texture change is large is 0, and the texture strength in the small direction is 1. Here, the texture strength in the vertical direction is T _V and the texture strength in the horizontal direction is T _H. Next, the texture intensity obtained once is corrected in consideration of the state of the texture intensity obtained at the neighboring pixel position. If the texture direction is horizontal, the pixels T _H = 1 and T _V = 0 in all the local regions, and the pixels with T _H = 0 and T _V = 1 increase as the texture direction becomes vertical. That is, it is considered that the determination variation is the smallest when the texture direction is vertical or horizontal, and the variation is largest near 45 degrees. If this property is used to correct the texture intensity at the target pixel position, it can handle textures of all angles, and at the same time, an unnatural blend caused by variations in judgment results in areas where the texture direction near the limit resolution cannot be judged. Can be prevented. An example of the correction method is shown below. The horizontal texture intensity average deviation S _TH and the vertical texture intensity average deviation S _TV are calculated using the local regions T _H and T _V, and the correction formulas shown in the equations (6) to (8) are calculated. Thus, the texture strength is corrected.

R = S _TH × (first constant) (6)
T ′ _H (x, y) = T _H (x, y) × (1−R) + R / 2 (7)
T ′ _V (x, y) = T _V (x, y) × (1−R) + R / 2 (8)

Further, the blend result (G + E) of (G + E) _H and (G + E) _V is calculated by the equation (9) based on the corrected texture strengths T ′ _H and T ′ _V.

(G + E) = (G + E) _V × T ′ _H + (G + E) _H × T ′ _V (9)

Here, if R is greater than 1, it is clipped to 1. According to the equations (6) to (9), when the variation of the determination result is small, R approaches 0 and T _H and T _V are output as they are. Further, when the variation of the determination result is large, R becomes large and approaches T ′ _H = 0.5 and T ′ _V = 0.5. Further, according to the equations (6) to (9), the sum of T ′ _H and T ′ _V always becomes 1, and at the same time, the behavior of the values of T ′ _H and T ′ _V is controlled by the first constant. be able to.

In the case of regression estimation, it is possible to obtain an MTF (Modulation Transfer function) that is better not to blend (G + E) _H and (G + E) _V as much as possible, so the first constant is such that artifacts near the limit resolution are removed. By setting small, high MTF performance can be obtained. Here, the average deviation is used to evaluate the variation of the texture intensity in the local region including the target pixel position, but other statistics such as a standard deviation may be used. The calculation of R in equation (6) may be the following equation (10) using not only S _TH but also S _TV .

R = (S _TH + S _TV ) × (second constant) (10)

In the present invention, a new color component (G + E) is generated as a luminance component instead of G in the Bayer array (FIG. 1). G + E is obtained by averaging the G and E on the same pixel. Since G + E is made up of at least half of the G component, it can be expected that G + E has a high correlation with G, and if G and E have a spectral sensitivity highly correlated with each other, use G + E instead of G. Even if image processing is performed, a corresponding result can be expected. That is, when G and E each have spectral characteristics close to G in the Bayer array, it can be used as a color component representing luminance as in G + E. 2 calculates a G + E value for the texture in the horizontal direction (G + E) _V and a reliable estimate value (G + E) _H for the texture in the vertical direction. It can be said that this is a process of blending and aligning G + E adapted to the texture direction for all pixels.

  When the color of the target pixel position is either G or E, for example, G is interpolated at a pixel position where E is present, and E is interpolated at a pixel position where G is present (G + E). The reliability of the obtained G + E with respect to the texture direction can be switched depending on whether the neighboring pixels used for the interpolation are vertical or horizontal. If the color at the pixel position of interest is R or G, which is neither E nor G, then the result of interpolation of both E and G must be used, so some effort is needed to accurately interpolate G + E at this pixel position. It is.

In the present invention, when the color of the target pixel position is R or B, it is proposed to use a high-frequency component included in R or B for interpolation between E and G. That is, the E or G DC component obtained from the neighboring pixels is added to the R or B high frequency component at the target pixel position in the same way as the luminance high frequency component is added to the color DC component in the conventional interpolation method. In addition, E or G interpolation is performed. At this time, the neighboring pixels used for the interpolation are switched, and the interpolation value EF for E using the pixels near the vertical for interpolation and the interpolation value GF for G using the pixels near the horizontal for interpolation are calculated. Further, G at the target pixel position is interpolated using the high frequency component of EF and the DC component of G obtained from the neighboring pixels, and is added and averaged with EF to obtain (G + E) _V. Similarly, using the high frequency component of GF and the DC component of E obtained from the neighboring pixels, E at the target pixel position is interpolated and averaged with GF to obtain (G + E) _H.

  In the above example, the four-color filter array has been described. However, as shown in FIG. 38, two more colors are added to the four-color filter array. Even when added, G + E can be generated by using the same method as described above. Only the method of selecting neighboring pixels depending on the arrangement, that is, the conditions of the loops in FIGS. 21 to 23, 27 and 28 need only be changed. Considering that human vision is not so sensitive to luminance with respect to color components, if even G + E signals that have a strong correlation with luminance retain a high sampling frequency, the number of samples for other color components can be reduced. It does n’t matter so much. Therefore, if G is arranged horizontally and vertically every other line, and E is arranged every other line horizontally and vertically, it is arranged on a different line from E. A sequence using can be used.

  14, step S311 in FIG. 16, step S459 in FIG. 21, step S510 in FIG. 22, step S530 in FIG. 23, and step S809 in FIG. 32, the sum of weight coefficients (for example, w) ( For example, a predetermined value is divided by W), but the weighting factor w is changed according to the distance from the center position, and the sum is determined to be a power of 2 to obtain the sum of the weighting factors w. Instead of the process of dividing a predetermined value by W (division using the number of pixels), a shift operation may be used. This makes it possible to simplify division.

  Note that the present invention can be applied to a solid-state imaging device and an imaging device using the solid-state imaging device (for example, a digital still camera) including a digital camera. Specifically, an unknown color component can be estimated by applying the present invention to an imaging apparatus that obtains a color image with a single solid-state imaging device (single-plate color system).

  As described above, the texture intensity can be calculated as a value representing the texture direction from the mosaic image of the color imaged by the single-plate color imaging element having the four-color arrangement. In addition, the estimated value estimated using the neighboring pixels in the vertical direction and the estimated value estimated using the neighboring pixels in the horizontal direction can be blended based on the texture intensity, and the unknown color component can be estimated with high accuracy. .

  As a result, even in the vicinity of the limit resolution where an error is likely to occur in the direction detection, an appropriate correction process is applied to the texture intensity, so that a stable interpolation result can be obtained.

  The series of processes described above can be executed by hardware, but can also be executed by software.

  In this case, the above-described functions are realized by executing the software by the DSP block 216. Further, for example, the demosaic processing unit 253 or the G + E calculation processing unit 282 can be configured by a personal computer 501 as shown in FIG.

  39, a CPU (Central Processing Unit) 511 performs various processes according to a program stored in a ROM (Read Only Memory) 512 or a program loaded from a storage unit 518 to a RAM (Random Access Memory) 513. Execute. The RAM 513 also appropriately stores data necessary for the CPU 511 to execute various processes.

  The CPU 511, ROM 512, and RAM 513 are connected to each other via a bus 514. An input / output interface 515 is also connected to the bus 514.

  The input / output interface 515 includes an input unit 516 including a keyboard and a mouse, an output unit 517 including a display and a speaker, a storage unit 518 including a hard disk, and a communication unit 519 including a modem and a terminal adapter. It is connected. A communication unit 519 performs communication processing via a network including the Internet.

  A drive 520 is connected to the input / output interface 515 as necessary, and a magnetic disk 531, an optical disk 532, a magneto-optical disk 533, a semiconductor memory 534, or the like is appropriately mounted, and a computer program read from them is loaded. It is installed in the storage unit 518 as necessary.

  When a series of processing is executed by software, a program constituting the software is dedicated hardware (for example, the DSP block 216 or the demosaic processing unit 253 or the G + E calculation processing unit 282 included therein). The computer is installed from a network or a recording medium, for example, a general-purpose personal computer capable of executing various functions by installing various computers or various programs.

  As shown in FIG. 39, this recording medium is distributed to supply a program to the user separately from the apparatus main body, and includes a magnetic disk 531 (including a floppy disk) and an optical disk 532 (including a floppy disk) stored therein. CD-ROM (compact disk-read only memory), DVD (including digital versatile disk)), magneto-optical disk 533 (including MO (magnet optical disk)), or a package medium composed of semiconductor memory 534, etc. In addition, the program is configured by a ROM 512 storing a program supplied to the user in a state of being incorporated in the apparatus main body in advance, a hard disk included in the storage unit 518, and the like.

  In the present specification, the step of describing the program stored in the recording medium is not limited to the processing performed in chronological order in the order in which they are included, but is not necessarily processed in chronological order, either in parallel or individually. The process to be executed is also included.

It is a figure for demonstrating a Bayer arrangement | sequence. It is a block diagram which shows the structure of the digital camera to which this invention is applied. It is a figure for demonstrating the example of the mosaic arrangement | sequence of 4 colors of the on-chip color filter of a CCD image sensor. FIG. 3 is a block diagram illustrating a configuration of a DSP block in FIG. 2. It is a block diagram which shows the structure of the demosaic process part of FIG. It is a block diagram which shows the structure of the G + E calculation process part of FIG. 6 is a flowchart for explaining image processing executed by the DSP block of FIG. 4. It is a flowchart for demonstrating a demosaic process. It is a flowchart for demonstrating a G + E calculation process. It is a flowchart for demonstrating a 1st interpolation process. It is a flowchart for demonstrating a 1st interpolation process. It is a flowchart for demonstrating correction | amendment texture intensity | strength calculation processing. It is a flow chart for the texture strength T _H, the T _V calculation process 1 will be described. It is a flowchart for _{demonstrating a} weighted average value _MTH calculation process. It is a figure for demonstrating the example of a weighting coefficient. It is a flowchart for _{demonstrating} weighted average deviation _STH calculation processing. It is a flowchart for explaining the texture strength T _H, the T _V correction process. It is a figure for demonstrating the weighting in the case of using the sample of a vertical direction. It is a figure for demonstrating the weighting in the case of using a sample of a horizontal direction. It is a flowchart for demonstrating a 2nd interpolation process. It is a flowchart for _{demonstrating the} weighted average value _MYdir . It is a flowchart for _{demonstrating} the weighted dispersion _| distribution approximate value VA _RRdir calculation process of _R. It is a flowchart for demonstrating the weighted covariance approximation calculation process of R and XA. It is a figure for demonstrating the area | region which the product of p and q can take. It is a figure for demonstrating the area | region which the approximation value of pq can take when the integral approximation process is performed. It is a figure for demonstrating the square error of the approximate value of pq at the time of performing an integral approximation process. It is a flowchart for demonstrating an integrating | accumulating process. It is a flowchart for demonstrating an integral approximation process. It is a flowchart for _{demonstrating} the inclination _Kdir calculation process of a regression line. It is a flowchart for demonstrating the estimated value XF calculation process of X by a regression line. It is a flowchart for demonstrating a 3rd interpolation process. It is a flowchart for _{demonstrating a} weighted average value _MXA calculation process. It is a flowchart for demonstrating the interpolation value XU calculation process of X. It is a flowchart for demonstrating a 1st G + E production | generation process. It is a flowchart for demonstrating a 2nd G + E production | generation process. It is a flowchart for demonstrating a synthetic | combination process. . It is a flowchart for explaining the texture strength T _H, the T _V calculation process 2. It is a figure for demonstrating the case where two more colors are added to the filter arrangement of four colors. It is a block diagram which shows the structure of a personal computer.

Explanation of symbols

  201 digital camera, 216 DSP block, 251 white balance adjustment unit, 253 demosaic processing unit, 331-1 G interpolation processing unit, 331-2 E interpolation processing unit, 332-1 G interpolation processing unit, 332-2 E interpolation processing unit 3333-1 G interpolation processing unit, 333-2 E interpolation processing unit, 334 texture intensity calculation processing unit, 335,336 G + E generation processing unit, 337 synthesis processing unit

Claims (6)

The first color component is arranged every other line in the horizontal and vertical directions, and the second color component is arranged every other line in the horizontal and vertical directions, and is different from the first color component in the horizontal and vertical directions. In an image processing apparatus for generating a color image in which each pixel has a plurality of color components based on a mosaic image captured by a single-plate color image sensor using color filters arranged in a line,
A change intensity calculating means for calculating a vertical change intensity and a horizontal change intensity of the target pixel based on the target pixel of the mosaic image and pixels in the vertical and horizontal directions of the target pixel;
Based on the pixel of interest and pixels in the vertical direction of the pixel of interest, a first estimated value of the pixel of interest of the first color component is calculated, and the horizontal direction of the pixel of interest and the pixel of interest First estimated value calculating means for calculating a second estimated value of the target pixel of the second color component based on pixels in the vicinity of
Based on the vertical change intensity, the horizontal change intensity, the first estimated value, and the second estimated value, a third color component composed of the first color component and the second color component. An image processing apparatus comprising: a second estimated value calculating unit configured to calculate a third estimated value of the color component in the target pixel. The change intensity calculation means includes
Interpolating the first interpolation values of the first color component and the second color component in the target pixel based on pixel values of pixels in the vicinity of the target pixel;
Interpolating a second interpolation value of the first color component and the second color component in a pixel located in the vertical direction of the target pixel based on a pixel value of a pixel in the vicinity of the target pixel;
Interpolating a third interpolation value of the first color component and the second color component in a pixel located in a lateral direction of the target pixel based on a pixel value of a pixel in the vicinity of the target pixel;
A longitudinal gradient evaluation value is calculated from a set of differences between the first interpolation value and the second interpolation value,
A lateral gradient evaluation value is calculated by a set of differences between the first interpolation value and the third interpolation value,
Determining the intensity of change based on the longitudinal gradient evaluation value and the lateral gradient evaluation value;
The image processing apparatus according to claim 1, wherein a deviation of change intensity in the vicinity of the target pixel and the target pixel is calculated, and the change intensity of the target pixel is corrected based on the deviation. The second estimated value calculation means includes:
Third estimated value calculation for calculating a fourth estimated value of the third color component in the target pixel based on the second color component and the first estimated value of the pixel in the vicinity of the target pixel. Means,
Fourth estimated value calculation for calculating a fifth estimated value of the third color component in the target pixel based on the first color component and the second estimated value of the pixel in the vicinity of the target pixel. Means,
The third estimated value is calculated based on the fourth estimated value, the fifth estimated value, the vertical change intensity of the target pixel, and the horizontal change intensity of the target pixel. The image processing apparatus according to claim 1, further comprising: 5 estimated value calculation means. The image processing apparatus according to claim 3, wherein the first color component and the second color component have spectral sensitivity highly correlated with each other. The first color component is arranged every other line in the horizontal and vertical directions, and the second color component is arranged every other line in the horizontal and vertical directions, and is different from the first color component in the horizontal and vertical directions. In an image processing method of an image processing apparatus for generating a color image in which each pixel has a plurality of color components based on a mosaic image captured by a single-plate color image sensor using color filters arranged in a line,
A change intensity calculating step of calculating a vertical change intensity and a horizontal change intensity of the target pixel based on the target pixel of the mosaic image and pixels in the vertical and horizontal directions of the target pixel;
A first estimated value calculation step of calculating a first estimated value of the first color component in the target pixel based on the target pixel and a pixel in the vertical direction of the target pixel;
A second estimated value calculating step of calculating a second estimated value of the second color component in the target pixel based on the target pixel and a pixel in the horizontal direction of the target pixel;
Based on the vertical change intensity, the horizontal change intensity, the first estimated value, and the second estimated value, a third color component composed of the first color component and the second color component. And a third estimated value calculating step for calculating a third estimated value for the target pixel of the color component. The first color component is arranged every other line in the horizontal and vertical directions, and the second color component is arranged every other line in the horizontal and vertical directions, and is different from the first color component in the horizontal and vertical directions. A program for generating a color image in which each pixel has a plurality of color components based on a mosaic image captured by a single-plate color image sensor using a color filter arranged in a line,
A change intensity calculating step of calculating a vertical change intensity and a horizontal change intensity of the target pixel based on the target pixel of the mosaic image and pixels in the vertical and horizontal directions of the target pixel;
A first estimated value calculation step of calculating a first estimated value of the first color component in the target pixel based on the target pixel and a pixel in the vertical direction of the target pixel;
A second estimated value calculating step of calculating a second estimated value of the second color component in the target pixel based on the target pixel and a pixel in the horizontal direction of the target pixel;
Based on the vertical change intensity, the horizontal change intensity, the first estimated value, and the second estimated value, a third color component composed of the first color component and the second color component. A program causing a computer to execute a process including a third estimated value calculating step of calculating a third estimated value of the target pixel of a color component.

Images