JP2005210218A - Image processor, image processing method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To suppress noise such as false color at the time of all pixel reading and also decimation reading. <P>SOLUTION: Since G signals are arranged in checkered pattern and spatially distributed uniformly in every direction in a mosaic arrangement 231, interpolation of G can be carried out precisely in demosaic processing at the time of all pixel reading. A color arrangement being obtained when the decimation reading of every other row is performed become a G stripe of every other row, and such mosaic arrangements as R and B are mixed uniformly on the same horizontal line in a row other than G and arranged alternately on the same vertical line. When an on-chip color filter of such a mosaic arrangement 231 is employed in a CCD image sensor, noise such as false color is not generated easily at the time of decimation reading of every other row as well as at the time of all pixel reading, and a high quality low resolution image can be attained. The invention is applicable to a digital still camera and a digital video camera. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to an image processing device, an image processing method, and a program, and more particularly, to an image processing device, an image processing method, and a program that are suitable for performing thinning processing.

  In recent years, solid-state imaging devices having millions of pixels have been developed due to advances in semiconductor technology, and these solid-state imaging devices are used in, for example, digital cameras. An image captured by a digital camera using a multi-million-pixel solid-state image sensor has high image quality, but other than when capturing an image, for example, displaying an image in a state where a monitor of a digital camera is used as a viewfinder In displaying an image when determining shooting conditions such as exposure and white balance, it is necessary to increase the frame rate and speed up image processing, but a high-resolution image is not necessarily required. However, when a multi-million-pixel solid-state imaging device is used, it is difficult to read out information of all pixels at high speed.

  On the other hand, in a situation where readout of image information and high-speed processing are required, readout and processing are speeded up by thinning out and reading out signals from the solid-state imaging device and reducing the substantial number of pixels. A method has been proposed.

  Conventionally, there is a method of thinning out and reading out half the pixels of a readable image signal from an image signal obtained by a solid-state imaging device having a Bayer array color filter. Here, from an image signal obtained by a solid-state imaging device having a color filter with a Bayer array, half lines are thinned horizontally in units of one row (that is, reading is performed for each row and skipping is performed alternately), R Alternatively, one of the signals corresponding to the B color is completely lost. In this way, in order to prevent a specific color information from being lost, for example, as shown in FIG. 1, two rows are read out and two rows are skipped with respect to the horizontal line of the Bayer array 11. There is a technique in which thinning is performed by alternately performing the above, and after the thinning, an image signal of the mosaic array 12 is obtained. (For example, patent document 1). With this method, the Bayer array is maintained even in the mosaic image after thinning, and the subsequent processing can be used as it is.

JP 9-312849 A

  In addition, as shown in FIG. 2, an image signal (RGB stripe array 21, or an image signal obtained from a high-resolution solid-state imaging device having a color filter of a color array that includes all color information in one row of data, There is a technique of thinning a G stripe RB complete checkered array 22 mosaic image) and converting it into a low resolution image signal of the mosaic array 23 (for example, Patent Document 2).

Japanese Patent No. 3429061

  Indeed, in the technique described in Patent Document 1, as shown in FIG. 1, the color arrangement of the mosaic arrangement 12 after thinning is the same Bayer arrangement as the mosaic arrangement 11 before thinning. is there. However, because of the reading method in which two-line reading and two-line skip are alternately provided, the first and second lines of the mosaic array 12 after thinning are adjacent to each other in the mosaic array 11 before thinning. Since the second and third rows of the mosaic array 12 after this correspond to the second and fifth rows of the mosaic array 11 before thinning, they are not actually adjacent pixels. For this reason, in the RGB simultaneous processing (demosaic processing) using the image signal corresponding to the mosaic array 12 after thinning, when performing processing such as interpolation using adjacent lines up and down, the sampling interval is not constant. , Noise such as false colors is likely to occur in the vertical direction.

  In the technique described in Patent Document 2, as shown in FIG. 2, the mosaic arrangement 23 after the thinning-out process becomes an RGB stripe for each column, so that the spatial density of the RB in the horizontal direction is It becomes sparse compared to the vertical direction, and variations due to the resolution direction and false colors are likely to occur.

  The present invention has been made in view of such a situation, and when a process that does not require a frame rate is executed, all pixels can be read out, and when a process that requires a frame rate is executed, High-speed reading is realized by performing thinning readout that repeats reading and skipping for each row, and colors that make up the mosaic image are evenly distributed spatially at all pixel readout and thinning readout By realizing the filter arrangement, it is possible to obtain a high-quality image signal in which false colors and the like are suppressed as much as possible.

  The image processing apparatus of the present invention has three types of filters having different spectral sensitivities, and from the mosaic image signal obtained by the image sensor in which any one of the plurality of types of filters is used for each pixel, An image processing apparatus that generates a color image in which intensity information for each pixel position corresponding to three colors determined by spectral sensitivity is aligned in all pixels, and includes a control unit that controls reading of a signal from the image sensor, and an image sensor. Generation means for generating, from the obtained mosaic image signal, three single-color image data in which intensity information corresponding to each of the three types of colors determined by the spectral sensitivity of the filter is aligned in all pixels; One filter is arranged in a checkered pattern, the second filter is arranged in an oblique direction, and arranged every other pixel in the horizontal and vertical directions. The third filter has a filter arrangement in which the third filter continues in an oblique direction and is arranged every other pixel in the horizontal and vertical directions, and the control means performs all pixel readout and thinning readout of the signal from the image sensor. It is characterized by controlling.

  When the thinning readout from the image sensor is controlled by the control means, the horizontal direction corresponding to the single color image data based on the three single color image data generated by the generation means based on the mosaic image signal obtained by the thinning readout. Generation means for generating monochromatic image data reduced in size can be further provided.

  When the thinning readout from the image sensor is controlled by the control means, a generation means for generating a mosaic image signal reduced in the horizontal direction corresponding to the mosaic image signal based on the mosaic image signal obtained by the thinning readout is further provided. Can be provided.

  The control means can control readout of signals from the image sensor for each row and thinning readout by skipping.

  The image sensor can obtain image signals with a plurality of types of sensitivity such that at least one horizontal line has the same sensitivity.

  The control means can control the signal from the image sensor to read out a uniform number of rows in which all the types of sensitivities are all the same and to read out the data by skipping.

  According to the image processing method of the present invention, the intensity information corresponding to each of the three types of colors determined by the spectral sensitivity of the filter from the control step for controlling the readout of the signal from the image sensor and the mosaic image signal obtained by the image sensor. Generating three monochromatic image data in which all pixels are aligned, and the image sensor includes a first filter arranged in a checkered pattern, a second filter continuous in an oblique direction, and horizontal and horizontal It has a filter array that is arranged every other pixel in the vertical direction, and the third filter is arranged in the diagonal direction and arranged every other pixel in the horizontal and vertical directions. This is characterized in that all pixel readout and thinning readout of the above signal are controlled.

  The program of the present invention includes all of intensity information corresponding to each of the three types of colors determined by the spectral sensitivity of the filter from the control step for controlling the readout of the signal from the image sensor and the mosaic image signal obtained by the image sensor. A generation step of generating three monochromatic image data that is aligned with pixels, and the image sensor includes a first filter arranged in a checkered pattern, a second filter continuous in an oblique direction, and horizontal and vertical directions. Are arranged every other pixel, the third filter is arranged in a diagonal direction, and has a filter arrangement arranged every other pixel in the horizontal and vertical directions. The computer is caused to execute processing characterized by controlling all pixel readout and thinning readout.

  In the image processing apparatus, the image processing method, and the program of the present invention, readout of a signal from the image sensor is controlled, and each of the three types of colors determined by the spectral sensitivity of the filter is determined from the mosaic image signal obtained by the image sensor. Three monochrome image data in which the intensity information corresponding to each pixel is aligned is generated, and the filter arrangement of the image sensor is such that the first filter is arranged in a checkered pattern, the second filter is continuous in an oblique direction, and The third filter is arranged every other pixel in the horizontal and vertical directions, and the third filter is arranged in the diagonal direction and arranged every other pixel in the horizontal and vertical directions. Reading is controlled.

  According to the present invention, it is possible to generate a color image from a mosaic image signal, and in particular, to obtain a high-resolution image according to the readout speed and processing speed of the image signal and the resolution required for the obtained image. Pixel reading or thinning-out reading for obtaining a low resolution image is executed, and in any case, generation of a false color or the like can be prevented and a high quality image can be obtained.

  Embodiments of the present invention will be described below. The correspondence relationship between the invention described in this specification and the embodiments of the invention is exemplified as follows. This description is intended to confirm that the embodiments supporting the invention described in this specification are described in this specification. Therefore, even if there is an embodiment that is described in the embodiment of the invention but is not described here as corresponding to the invention, the fact that the embodiment is not It does not mean that it does not correspond to the invention. Conversely, even if an embodiment is described herein as corresponding to an invention, that means that the embodiment does not correspond to an invention other than the invention. Absent.

  Further, this description does not mean all the inventions described in this specification. In other words, this description is for the invention described in the present specification, which is not claimed in this application, that is, for the invention that will be applied for in the future or that will appear and be added by amendment. It does not deny existence.

  The image processing apparatus according to claim 1 has three types of filters (for example, color filters corresponding to three colors of RGB) having different spectral sensitivities, and any one of the plurality of types of filters is used for each pixel. A color image in which intensity information for each pixel position corresponding to the three colors determined by the spectral sensitivity of the filter is the same for all pixels is generated from the mosaic image signal obtained by the image sensor (for example, the CCD image sensor 213). 3 is determined by the spectral sensitivity of the filter from the mosaic image signal obtained by the control means (for example, the timing generator 217 in FIG. 3) and the image sensor obtained by controlling the readout of the signal from the image sensor. Generation means for generating three single-color image data in which intensity information corresponding to each type of color is aligned in all pixels ( For example, the image sensor includes the demosaic processing unit 253 in FIG. 7 or the demosaic processing unit 371 in FIG. 36, and the image sensor includes first filters (for example, color filters corresponding to G) arranged in a checkered pattern, Two filters (for example, a color filter corresponding to one of R or B) are arranged in an oblique direction and arranged every other pixel in the horizontal and vertical directions, and a third filter (for example, the other of R or B) Is a filter array corresponding to the mosaic array 231 in FIG. 4 or the mosaic array 361 in FIG. And the control means controls all pixel readout and thinning readout of the signal from the image sensor.

  When the thinning readout from the image sensor is controlled by the control means, the horizontal direction corresponding to the single color image data based on the three single color image data generated by the generation means based on the mosaic image signal obtained by the thinning readout. Generation means (for example, the horizontal direction reduction processing unit 255 in FIG. 6) for generating monochrome image data reduced in size can be further provided.

  When thinning readout from the image sensor is controlled by the control means, generation means (for example, generating a mosaic image signal reduced in the horizontal direction corresponding to the mosaic image signal based on the mosaic image signal obtained by thinning readout. 39, the horizontal reduction processing unit 398) of FIG.

  The image sensor can obtain image signals with a plurality of kinds of sensitivities such that at least one horizontal line has the same sensitivity (for example, obtain a mosaic image signal corresponding to the mosaic arrangement 361 in FIG. 34).

  The control means reads out the signal from the image sensor in a uniform number of rows (for example, every two rows for the mosaic image signal corresponding to the mosaic array 361 in FIG. 34) such that all the types of sensitivity are all aligned. Further, it is possible to control thinning readout by skipping.

  The image processing method of the present invention has three types of filters (for example, color filters corresponding to three colors of RGB) having different spectral sensitivities, and any one of a plurality of types of filters is used for each pixel. Image processing for generating, from a mosaic image signal obtained by an image sensor (for example, a CCD image sensor 213), a color image in which intensity information for each pixel position corresponding to three colors determined by the spectral sensitivity of the filter is aligned in all pixels. An image processing method of the apparatus, wherein a control step (for example, processing in step S2 or step S6 in FIG. 16) for controlling reading of a signal from the image sensor, and a mosaic image signal obtained by the image sensor, Three single-color image data in which intensity information corresponding to each of the three types of colors determined by the spectral sensitivity is the same for all pixels The image sensor includes a first filter (for example, a color filter corresponding to G) arranged in a checkered pattern, and a second generation step (for example, the process of step S3 or step S7 in FIG. 16). A filter (for example, a color filter corresponding to one of R or B) is arranged in an oblique direction and arranged every other pixel in the horizontal and vertical directions, and a third filter (for example, corresponding to the other of R or B) Filter array (for example, a filter array corresponding to the mosaic array 231 in FIG. 4 or the mosaic array 361 in FIG. 34) arranged in the diagonal direction and arranged every other pixel in the horizontal and vertical directions. In the control step processing, all pixel readout and thinning readout of the signal from the image sensor are controlled.

  Also in the program according to claim 8, the embodiment (however, an example) to which each step corresponds is the same as the information processing method according to claim 7.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 3 is a block diagram showing a configuration of a digital still camera 201 that executes arithmetic processing to which the present invention is applied.

  As shown in FIG. 3, a digital still camera 201 includes a lens 211, an aperture 212, a CCD (Charge Coupled Devices) image sensor 213, a correlated double sampling (CDS) circuit 214, an A / D converter 215, and a DSP. (Digital Signal Processor) From block 216, timing generator 217, D / A converter 218, video encoder 219, display unit 220, codec (CODEC: COmpression / DECompression) processing unit 221, memory 222, CPU 223, and operation input unit 224 Composed.

  The CCD is a semiconductor element that converts light information into an electrical signal (photoelectric conversion). The CCD image sensor 213 has a plurality of light receiving elements (pixels) that convert light into electricity, and changes in light for each pixel. It converts to an electric signal independently. The correlated double sampling circuit 214 samples the reset noise, which is the main component of the noise included in the output signal of the CCD image sensor 213, from the sampled video signal period of the output pixel signals, and the reference period. Is a circuit that removes the signal by subtracting the sampled signal. The A / D converter 215 converts the supplied analog signal after noise removal into a digital signal.

  The DSP block 216 is a block having a signal processing processor and an image RAM, and the signal processing processor performs pre-programmed image processing or hardware arithmetic processing on the image data stored in the image RAM. The image processing configured as follows is performed. The timing generator 217 is a logic circuit that generates various horizontal and vertical driving pulses necessary for driving the CCD and pulses used in the analog front processing in synchronization with the reference clock. The timing clock generated by the timing generator 217 is also supplied to the codec processing unit 221, the memory 222, and the CPU 223 via the bus 225.

  The D / A converter 218 converts the supplied digital signal into an analog signal and outputs the analog signal. The video encoder 219 encodes the supplied analog signal into video data in a format that can be displayed on the display unit 220. The display unit 220 is configured by, for example, an LCD (Liquid Crystal Display) and displays the video signal supplied from the video encoder 219.

  The codec processing unit 221 executes processing based on a compression or expansion algorithm for digital image data such as JPEG (Joint Picture Experts Group). The memory 222 is configured by, for example, a semiconductor memory, a magnetic disk, a magneto-optical disk, or an optical disk, and stores supplied data or outputs stored data based on control of the CPU 223. . Note that the memory 222 may be detachable from the digital still camera 201.

  The CPU 223 controls each unit of the digital still camera 201 based on a user operation input supplied from the operation input unit 224 via the bus 225. The operation input unit 224 includes, for example, a jog dial, a key, a lever, a button, or a touch panel as well as a button for instructing recording, and receives an operation input by a user.

  Light input through the lens 211 and the diaphragm 212 is incident on the CCD image sensor 213, converted into an electrical signal by photoelectric conversion at the light receiving element, and supplied to the correlated double sampling circuit 214. The correlated double sampling circuit 214 removes noise by subtracting the sampled video signal period and the sampled reference period from the pixel signals output from the CCD image sensor 213, and A / D This is supplied to the converter 215. The A / D converter 215 converts the supplied analog signal after noise removal into a digital signal, and temporarily stores it in the image RAM of the DSP block 216.

  The timing generator 217 controls the readout of the image by the CCD image sensor 213 based on the state during imaging whether or not the shutter button has been pressed, and maintains the image capture at a constant frame rate. The image sensor 213, the correlated double sampling circuit 214, the A / D converter 215, and the DSP block 216 are controlled.

  The DSP block 216 receives supply of pixel stream data at a constant rate, temporarily stores it in the image RAM, and executes image processing (to be described later) on the temporarily stored image data in the signal processing processor. When the image block is displayed on the display unit 220 after the image processing is completed, the DSP block 216 displays the code data in the D / A converter 218 and the codec processing unit 221 in the case where the image data is stored in the memory 222. Supply image data.

  The D / A converter 218 converts the digital image data supplied from the DSP block 216 into an analog signal and supplies the analog signal to the video encoder 219. The video encoder 219 converts the supplied analog image signal into a video signal, and outputs the video signal to the display unit 220 for display. That is, the display unit 220 plays a role of a camera finder in the digital still camera 201. The codec processing unit 221 performs a predetermined encoding on the image data supplied from the DSP block 216 and supplies the encoded image data to the memory 222 for storage.

  Further, the codec processing unit 221 reads data designated by the user from among the data stored in the memory 222 based on the control of the CPU 223 that has received the user's operation input from the operation input unit 224, and performs predetermined decoding. The decoded signal is output to the DSP block 216. Thus, the decoded signal is supplied to the D / A converter 218 via the DSP block 216, converted into an analog signal, encoded by the video encoder 219, and displayed on the display unit 220.

  That is, in the normal state (the state before the shutter button is pressed), the image signal thinned out from the CCD image sensor 213 is stored in the image RAM of the DSP block 216 in a certain frame by the control of the timing generator 217. It is constantly overwritten at the rate. The image signal processed by the DSP block 216 is supplied to the D / A converter 215, converted into an analog signal, converted into a video signal by the video encoder 219, and an image corresponding to the converted video signal is displayed on the display unit 220. Is displayed. Since the image displayed at this time is an image corresponding to the thinned image signal, the image has a resolution lower than that of the CCD image sensor 213. In this state, the display unit 220 serves as a finder for the CCD image sensor 213.

  Then, when the shutter button included in the operation input unit 224 is pressed by the user, the CPU 223 causes the timing generator 217 to read all the pixels from the CCD image sensor 213 based on the timing at which the shutter button is pressed. At the same time, the image RAM of the DSP block 216 is controlled so that new image data is not overwritten for a certain period, that is, the read image signal is processed and held. The image signal processed by the DSP block 216 is encoded by a codec processing unit 221 and stored in the memory 222.

  FIG. 4 shows a first example of a mosaic arrangement used in the on-chip color filter of the CCD image sensor 213 in FIG. 3, that is, a first example of a color arrangement of a mosaic image obtained by the CCD image sensor 213 in FIG. FIG. 5 shows the color array of the thinned mosaic image obtained when the thinning process is performed by repeating reading and skipping for each row with respect to the mosaic array of FIG.

  The mosaic arrangement 231 shown in FIG. 4 is arranged such that the G signals are arranged on the checkered pattern in the same manner as the Bayer arrangement, and are spatially evenly distributed in the vertical and horizontal directions. The RGB interpolation processing (demosaic processing) in the DSP block 216 at the time of all pixel readout makes it possible to perform G interpolation processing with high accuracy.

  Further, when the mosaic arrangement 231 shown in FIG. 4 is used for the on-chip color filter of the CCD image sensor 213, the CPU 223 controls the timing generator 217 during thinning-out reading, and thins out every other row. Read the signal. That is, among the image signals acquired by the CCD image sensor 213, the image signals are read out only in odd-numbered rows or even-numbered rows. Therefore, at the time of thinning readout, as shown in FIG. 5, the obtained color arrangement is every other G stripe, and R and B are uniformly mixed on the same horizontal line in non-G columns. And the mosaic arrangement 241 is arranged alternately on the same vertical line.

  Here, in the mosaic arrangement 241 of FIG. 5, when the positions in the mosaic arrangement 231 before the thinning of the G signals remaining after the thinning are confirmed, they are spatially evenly distributed. That is, the thinned image is equivalent to a resampled image of the all-pixel read image that is the original signal at equal intervals. Therefore, even in an image signal having the mosaic array 241 after thinning, variations due to the direction of G interpolation accuracy are unlikely to occur in the RGB synchronization processing (demosaic processing) in the DSP block 216. Further, for example, in the conventional method described with reference to FIG. 2, only one of R and B exists in the same column that is not the G stripe after thinning, but in the mosaic arrangement 241 in FIG. After thinning, R and B are mixed in the same column and distributed spatially evenly. Therefore, in the RGB synchronization processing (demosaic processing) in the DSP block 216, variations due to the direction of RB interpolation accuracy are unlikely to occur.

  From the above, by using the on-chip color filter shown in the mosaic array 231 in FIG. 4 for the CCD image sensor 213 in FIG. Even if it performs, it becomes possible to obtain a high-quality low-resolution image.

FIG. 6 is a block diagram showing a more detailed configuration of the DSP block 216 of FIG.

  As described above, the DSP block 216 includes the image RAM 241 and the signal processing processor 242, and the signal processing processor 242 includes the white balance adjustment unit 251, the gamma correction unit 252, the demosaic processing unit 253, the switch 254, the horizontal A direction reduction processing unit 255, a gradation conversion processing unit 256, and a YC conversion unit 257 are included.

  The mosaic image converted into a digital signal by the A / D converter 215 is temporarily stored in the image RAM 241. The mosaic image is an intensity signal corresponding to any color of R, G, or B for each pixel, that is, an array determined by the color filter used in the CCD image sensor 213 (the mosaic array described with reference to FIG. 4). 231) or a predetermined pattern (mosaic array 241 described with reference to FIG. 5) obtained by thinning out, is formed of periodic pattern intensity signals.

  The white balance adjustment unit 251 performs processing (white balance adjustment processing) for applying an appropriate coefficient to the mosaic image according to the color of each pixel intensity so that the color balance of the achromatic subject region becomes an achromatic color. ). The gamma correction unit 252 performs gamma correction on each pixel intensity of the mosaic image whose white balance has been adjusted. A numerical value “gamma (γ)” is used to represent the response characteristics of the gradation of the image. The gamma correction is a correction process for correctly displaying the brightness and color saturation of the image displayed on the display unit 220. The signal output to the display unit 220 reproduces the brightness and color of the image by applying a specific voltage for each pixel. However, due to the characteristics (gamma value) of the display unit 220, the brightness and color of the actually displayed image does not double the brightness of the CRT even if the input voltage is doubled (has non-linearity). For this reason, the gamma correction unit 252 performs a correction process so that the brightness and color saturation of the image displayed on the display unit 220 are correctly displayed.

  The demosaic processing unit 253 statistically calculates the color distribution shape, thereby executing a demosaic process for aligning all the R, G, and B intensities (intensity information) at each pixel position of the mosaic image subjected to gamma correction. . Therefore, the output signals from the demosaic processing unit 253 are three image signals corresponding to the three colors R, G, and B.

  Based on the control of the CPU 223, the switch 254 converts the supplied image signal into three colors R, G, and B that have been demosaiced using the image signal that has not been thinned out, that is, the image signal having the mosaic array 231. In the case of the corresponding three image signals, the supplied signal is supplied to the gradation conversion processing unit 256, and the supplied image signal is thinned, that is, demosaiced using the image signal having the mosaic array 241. In the case of three image signals corresponding to the processed three colors of R, G, and B, the supplied signals are supplied to the horizontal reduction processing unit 255.

  The horizontal direction reduction processing unit 255 is supplied with three image signals corresponding to the three colors R, G, and B that have been demosaiced using the image signal having the mosaic arrangement 241. In the image signal in which the horizontal line is thinned by 1/2, the number of vertical pixels is 1/2 that of the original image. Therefore, the horizontal reduction processing unit 255 performs a reduction process in the horizontal direction so that the aspect ratio of the thinned image signal becomes the same aspect ratio as that of the original image, and supplies it to the gradation conversion processing unit 256. .

  The gradation conversion processing unit 256 executes gradation conversion processing for compressing the gradation of the supplied image signal in accordance with the number of bits of the output image data, and the YC conversion unit 257 converts the gradation-converted image signal. To supply.

  The YC conversion unit 257 generates and outputs a Y image and a C image (YCbCr image signal) by subjecting the three-channel image of R, G, and B to band processing for matrix processing and chroma components.

  In the signal processing processor 242 of the DSP block 216, the gamma correction unit 252 performs gamma correction before the demosaic processing by the demosaic processing unit 253. This is because the reliability of the demosaic processing of the demosaic processing unit 253 can be further improved by executing the demosaic operation in the non-linear pixel intensity space subjected to the gamma correction.

  For example, when the input image is a high-contrast contour region, the color distribution covers a very bright intensity region and a very dark intensity region. Physically, the object reflected light is obtained by multiplying the variation of the object surface by the incident light intensity from the illumination. Therefore, in the linear pixel intensity space proportional to the incident light intensity to the camera, the bright intensity region The distribution of the object color in the sparsely spreads (sparsely), and the distribution of the object color in the dark pixel intensity region does not spread so much and tends to be compact.

  In the demosaic processing unit 253, demosaic processing is executed by statistically calculating the color distribution shape. However, in the high-contrast contour region, the pixel intensity variation in the bright region and the pixel intensity variation in the dark region are greatly different, and it becomes difficult to apply statistical linear regression. Therefore, prior to demosaic processing in the demosaic processing unit 253, the input data is subjected to non-linear pixel intensity conversion such as gamma correction to raise the dark pixel intensity region (close to the bright pixel intensity region). ) By suppressing the dispersion of the pixel intensity to some extent, the reliability of the linear regression processing executed in the demosaic processing unit 253 can be improved.

  The nonlinear transformation applied for such a purpose is preferably a power transformation with an exponent smaller than 1 as in gamma correction, but a combination of a power portion and a linear portion as in sRGB gamma usually used in a color profile or the like. Even if it is such a conversion, any non-linear conversion may be used as long as it can be regarded as being substantially the same as a power function. Needless to say, even if the non-linear conversion is omitted, a non-linear conversion process such as gamma correction may be performed after the demosaic process.

  Next, FIG. 7 executes a demosaic process that is a process of sequentially interpolating or estimating the intensity of a color that does not exist at each pixel position so that all RGB colors exist at all pixel positions. FIG. 7 is a block diagram showing a more detailed configuration of the demosaic processing unit 253 in FIG. 6.

  The demosaic processing unit 253 includes a local region extraction unit 281, a G intensity interpolation processing unit 282, a reliability calculation unit 283, a rough interpolation processing unit 284, statistic calculation units 285-1 to 285-3, and a regression calculation processing unit 286. -1 to 286-3.

  The local area extraction unit 281 cuts out a pixel of a local area having a predetermined size around the target pixel position from the gamma-corrected mosaic image. Here, in the image signal in which the local region to be cut out is not thinned out, an n × n pixel such as a 5 × 5 pixel, a 7 × 7 pixel, or a 9 × 9 pixel is used, for example. In the thinned image signal, a rectangular region of n × m pixels such as 5 × 3 pixels, 7 × 4 pixels, or 9 × 5 pixels, for example, with the pixel of interest at the center is used.

  The G intensity interpolation processing unit 282 calculates the G intensity at the target pixel position by interpolation processing using pixels existing in the local region. The reliability calculation unit 283 calculates the reliability using the pixels existing in the local region and the G intensity at the target pixel position calculated by the G intensity interpolation processing unit 282. Here, the reliability is a value that predicts the accuracy of estimation of the color distribution shape in the local region. How to calculate the reliability will be described later.

  The coarse interpolation processing unit 284 is configured to calculate R, G, and B pixel intensity estimates (hereinafter, rough values) by simple calculation so that a set of R, G, and B color intensities can be created at a plurality of pixel positions in the local region. Is calculated by using an interpolation method described later. In the present invention, in order to estimate the color distribution shape, it is necessary to calculate a second-order statistic or a value corresponding thereto, and for this reason, the intensity of each color at the same pixel position can be taken as a set. Need to be. In order to generate this intensity set, the coarse interpolation processing unit 284 has a plurality of pixels in the local region (here, the pixel of interest is centered on the local region of n × m pixels (n−2)). R (G−2) pixels, and when n = m, the rough estimated values of R, G, and B in (n−2) × (n−2) pixels) are calculated.

  In the conventional demosaic technique, there is no processing corresponding to such a rough interpolation processing unit 284, and thus there is a problem that information for performing accurate color distribution shape estimation cannot be calculated. On the other hand, in the demosaic processing unit 253 to which the present invention is applied, a rough estimation value can be obtained as information for performing accurate color distribution shape estimation using the rough interpolation processing unit 284. Has been made.

  The statistic calculators 285-1 to 285-3 calculate a statistic between two colors using a set of R, G, and B intensities at each pixel in the local area calculated by the coarse interpolation processor 284. . In any of the statistic calculation units 285-1 to 285-3, the first color is G. The second color supplied to the statistic calculation unit 285-1 is R, and the second color supplied to the statistic calculation unit 285-2 is G, like the first color. The second color supplied to the amount calculation unit 285-3 is B. Then, the statistic calculation units 285-1 to 285-3 calculate the average value of the first color, the average value of the second color, the dispersion value of the first color, and the first color and the second color. The covariance value of is computed and output. The demosaic processing unit 253 includes three statistical amount calculation units 285-1 to 285-3 depending on the color of the input signal, but their operations are all the same. It shall be called the quantity calculation part 285.

  The regression calculation processing units 286-1 to 286-3 correspond to one of the G intensity at the target pixel position calculated by the G intensity interpolation processing unit 282 and the statistic calculation units 285-1 to 285-3. Based on the calculated statistic and the reliability calculated by the reliability calculation unit 283, linear regression calculation of the color distribution shape is performed, and an intensity estimation value at the target pixel position is calculated. Specifically, in addition to the G intensity of the target pixel position and the reliability calculated by the reliability calculation unit 283, the regression calculation processing unit 286-1 includes the average value of each of G and R, the variance value of G, and , The covariance value of G and R is supplied, and the R intensity of the pixel of interest is estimated and output. In addition to the G intensity at the target pixel position and the reliability calculated by the reliability calculation unit 283, the regression calculation processing unit 286-2 receives the average value of G and the variance value of G, and receives the G of the target pixel. Estimate and output the intensity. Then, in addition to the G intensity at the target pixel position and the reliability calculated by the reliability calculation unit 283, the regression calculation processing unit 286-3 adds the average value of each of G and B, the variance value of G, and G and Upon receiving the covariance value with B, the B intensity of the pixel of interest is estimated and output. The demosaic processing unit 253 includes three regression calculation processing units 286-1 to 286-3 depending on the color of the input signal, but their operations are all the same. It is assumed to be called an arithmetic processing unit 286.

  Note that the demosaic processing unit 253 omits the statistic calculation unit 285-2 and the regression calculation processing unit 286-2 for calculating the G intensity estimation value in order to perform the demosaic processing, and the G intensity interpolation processing unit. The G intensity at the target pixel position calculated by 282 may be used. However, as shown in FIG. 7, by using the statistic calculation unit 285-2 and the regression calculation processing unit 286-2 to perform regression calculation on G and recalculate the estimated value, the intensity of G Therefore, the statistical amount calculation unit 285-2 and the regression calculation processing unit 286-2 for calculating the G intensity estimation value are provided without being omitted. It is preferable to improve the quality of the obtained image.

  Next, a more detailed operation of each unit of the demosaic processing unit 253 will be described.

  The local area extraction unit 281 acquires pixel information in a predetermined rectangular area near the target pixel position. The pixel information acquisition method may be any method. For example, when the present invention is implemented as software, pixel values in a rectangular area near the target pixel position are stored in an internal memory as an array. In the case of realization by hardware, it is only necessary to obtain a pixel signal adjacent to the upper and lower sides of the pixel of interest using a delay line. Usually, in a signal processing system of an imaging apparatus, hardware is often mounted so that a signal supplied from the CCD image sensor 213 is a one-dimensional series of pixel intensities in a horizontal line sequence. A delay line capable of holding the pixel intensity for one horizontal line is used to ensure access to the pixels of the horizontal lines adjacent vertically. Therefore, in order to acquire an image signal of a 9 × 9 rectangular area, it is sufficient to prepare at least eight delay lines.

  The G intensity interpolation processing unit 282 uses the pixels in the rectangular region of n × n or n × m (for example, 9 × 9 or 9 × 5) pixels extracted by the local region extraction unit 281 to use the G intensity at the target pixel position. The value is calculated by interpolation. Any method may be used for the G intensity interpolation processing unit 282 to calculate the G intensity using pixels around the target pixel. For example, “Murata, Mori, Maenaka, Okada, and Chihara,“ PS ” -Correlation discriminating color separation method in CCD ", Journal of the Institute of Image Information and Television Engineers, Vol.55, No.1, pp.120-132, 2001" (Non-patent Document 1) can be used. .

  The reliability calculation unit 283 includes n × n or n × m (for example, 9 × 9 or 9 × 5) pixels in the rectangular local region extracted by the local region extraction unit 281, and the G intensity interpolation processing unit 282. The reliability of color distribution shape estimation at the target pixel position is calculated using the calculated G intensity at the target pixel position. In the color distribution shape estimation in the demosaic process to which the present invention is applied, it is assumed that there are two different color regions in the extracted local region. Since this assumption is often applied to a normal object contour region, sufficiently accurate contour color reproduction is possible with this assumption. However, the assumption is not true in the image, and there may be cases where three different colors exist in the local region. Typically, this is a case of a thin striped texture. In such a texture, many colors are likely to be included in the local region. Therefore, the reliability calculation unit 283 detects such a thin striped texture and outputs a reliability value based on the intensity of the texture.

  The regression calculation processing unit 286 that receives the supply of the reliability value performs correction based on the reliability value with respect to the statistic related to the color distribution shape calculated by the statistic calculation unit 285. It is possible to suppress an estimation error of color intensity due to deviation from the assumption that there are two different color areas in the local area.

  FIG. 8 is a block diagram illustrating a configuration of the reliability calculation unit 283. Specifically, the reliability calculation unit 283 calculates a low-frequency filter for each direction with respect to the G intensity around the target pixel position in order to detect a thin striped texture, and determines the G intensity at the target pixel position. In the case where there is a luminance change such as “bright-dark-bright” or “dark-bright-dark” around the target pixel position by performing high-frequency extraction by performing the difference calculation process of The reliability is calculated by integrating the detection results for each direction. The reliability calculation unit 283 includes high frequency extraction units 321-1 to 321-6, an addition processing unit 322, and a clip processing unit 323. The six high-frequency extraction units 321-1 to 321-6 perform high-frequency extraction in six different directions around the target pixel and output the intensity.

  A specific example of direction-specific high-frequency extraction by the frequency extraction units 321-1 to 321-6 of the reliability calculation unit 283 when pixel signals that have not been thinned are supplied will be described with reference to FIG. FIG. 9 shows a 7 × 7 pixel local region extracted by the local region extraction unit 281. FIG. 9A shows a case where the target pixel is G, and FIG. 9B shows a case where the target pixel is R. Indicates. Since only the G pixel is used for the calculation of the reliability, when the target pixel is B, the same processing as that in the case of R is executed.

  First, 12 positions from positions close to the target pixel (positions indicated by circles in the figure) that are shifted from the pixel by exactly ½ in the horizontal and vertical directions (that is, surrounded by four pixels). ) G intensity is calculated. Specifically, since the G pixels are diagonally adjacent to each other at the 12 positions shown in FIG. 9, the average value of the two G intensities is used as the G intensity at each of the 12 positions. . Here, the direction of adjacent G is different between the case where the target pixel is G and the case where the target pixel is R or B. In either case, the direction in which G exists at each position is The determination can be made based on whether the pixel of interest is G or R or B.

  A specific example of the direction-specific high-frequency extraction by the frequency extraction units 321-1 to 321-6 of the reliability calculation unit 283 when the thinned pixel signal is supplied will be described with reference to FIG. FIG. 10 shows a 7 × 5 pixel local region extracted by the local region extraction unit 281. For explanation, FIG. 10A shows a case where the target pixel is G, and FIG. 10B shows a case where the target pixel is R. Since only the G pixel is used for the calculation of the reliability, when the target pixel is B, the same processing as that in the case of R is executed.

  First, 12 positions from positions close to the target pixel (positions indicated by circles in the figure) that are shifted from the pixel by exactly ½ in the horizontal and vertical directions (that is, surrounded by four pixels). ) G intensity is calculated. Specifically, since the G pixels are diagonally adjacent to each other at the 12 positions shown in FIG. 10, the average value of the two G intensities is used as the G intensity at each of the 12 positions. . Here, the direction of adjacent G is different between the case where the target pixel is G and the case where the target pixel is R or B. In either case, the direction in which G exists at each position is The determination can be made based on whether the pixel of interest is G or R or B.

  In this way, after the G intensity is calculated at 12 locations, a low frequency filter in six directions is calculated using them. The direction of the filter will be described with reference to FIG. In FIG. 11, only 5 × 5 pixels in the vicinity of the target pixel are shown in the local region.

  The calculation positions of the 12 G intensities are arranged in a point-symmetric manner around the target pixel position in both the pixel signals that are not thinned out and the pixel signals that are thinned out, and the G intensities that are in symmetrical positions with respect to each other. In the figure, a pair of G intensity indicated by A, a group of G intensity indicated by D, a group of G intensity indicated by VA, a group of G intensity indicated by VD, and a G intensity indicated by HA A total of 6 sets can be made, and a set of G intensity indicated by HD. If a filter that calculates an average value is used for each of these six G intensity pairs, a low-frequency filter in six directions is calculated. Further, if the difference between the output of the low frequency filter and the G intensity at the target pixel position is taken, high frequency extraction can be performed for each direction.

  Then, the addition processing unit 322 outputs a value obtained by adding the six high-frequency extraction results with an appropriate balance. The clip processing unit 323 performs clipping processing on the addition processing result by the addition processing unit 322, and outputs the added value of the clipped high-frequency extraction as a reliability value.

  The calculated reliability value increases as the strength of the thin striped texture increases, so the smaller the reliability value is, the more likely it is that the estimation of the color distribution shape by statistics is more certain. It will be.

  Next, the coarse interpolation processing unit 284 uses a method that is not a complicated calculation method so that a set of intensities of R, G, and B colors can be created at a plurality of positions in the local region. The estimated value) is calculated by interpolation. For example, R, G, and B interpolation values can be calculated from 8 pixels around the pixel of interest for both the pixel signal that has not been thinned and the pixel signal that has been thinned.

  Specifically, since the pixel signal that has not been thinned has the mosaic arrangement 231 described with reference to FIG. 4, the pixel pair located above and to the left of the target pixel, the right and the bottom R, G, B rough using a pair of pixels, a pair of pixels located diagonally up and down right, a pair of pixels located diagonally up and down left, and the pixel of interest itself At least one correct estimate can be created. Further, since the thinned image has the mosaic arrangement 241 described with reference to FIG. 5, a pair of pixels located above and below the diagonal pixel, diagonally upper left, diagonally left, and diagonally left Create at least one rough estimate of R, G, and B using the three pixels located below, the three pixels located diagonally right above, left, and diagonally below right, and the pixel of interest itself can do. In this way, the rough interpolation processing unit 284 can calculate rough estimated values of R, G, and B at each pixel position.

  Here, the estimated values of R, G, and B obtained by the coarse interpolation processing unit 284 are only required to obtain a set of estimated values of R, G, and B at the same position. It doesn't have to be above. For example, an interpolation value may be obtained for a position shifted from the pixel by exactly ½ in the horizontal and vertical directions (that is, a position surrounded by four pixels). Since this position is a position deviated by ½ pixel horizontally and vertically from the surrounding pixels, in the thinned pixel signal, one R is always present in the four surrounding pixels, two Gs, and one B. Exists. Therefore, Nearest Neighbor interpolation (nearest neighbor interpolation) is performed for R and B, an average value of two neighboring pixels is calculated for G, and rough estimated values for the three colors R, G, and B are calculated at each pixel position. You can do that.

  Next, FIG. 12 is a block diagram illustrating a configuration of the statistic calculation unit 285. Here, the case where the statistics of G and R are calculated, that is, the process of the statistic calculator 285-1 in the demosaic processor 253 will be described as an example.

  The statistic calculation unit 285 includes an average value calculation unit 331, an average value calculation unit 332, a variance calculation unit 333, and a covariance calculation unit 334. The average value calculation unit 331 receives supply of a rough set of G intensity interpolation values in the local region calculated by the rough interpolation processing unit 284, and calculates the average value. Similarly, the average value calculation unit 332 receives a set of rough R intensity interpolation values in the local region calculated by the coarse interpolation processing unit 284, and calculates the average value. The variance calculation unit 333 receives the G average value calculated by the average value calculation unit 331 and a set of rough G intensity interpolation values in the local region, and calculates the G variance value. The covariance calculation unit 334 includes a set of an average value of G calculated by the average value calculation unit 331 and a rough G intensity interpolation value, and an average value of R calculated by the average value calculation unit 332 and a rough R intensity interpolation value. And a covariance value of G and R is calculated.

  The average value calculation unit 331 and the average value calculation unit 332 calculate the average value Mc using, for example, the following equation (1).

・・・（１）

... (1)

In Expression (1), C _roughi is i (i = 1 to N) th data of the rough intensity interpolation value of the supplied color C (R, G, or C). The color C is G in the average value calculation unit 331 and R in the average value calculation unit 332. W _i is a weight value (weighting coefficient) for the i-th data.

The weight value w _i is a value set in advance using the distance from the position of the i-th data to the target pixel position as an index. 13 and 14 show examples of the weighting coefficient w _i . FIG. 13 shows the weight value w _i of the 5 × 5 area for the image signal that has not been subjected to the thinning process. In FIG. 14, the 5 × 3 area weight value for the image signal that has been subjected to the thinning process. An example of the weighting coefficient w _i is shown. It goes without saying that the absolute values of these weight values are not limited to those shown in FIG. 13 or FIG. 14, but as shown in these figures, the weight values are set so as to increase as they are closer to the target pixel position. It is better to be done.

The variance calculation unit 332 calculates the variance value V _C1C1 of the first signal (here, G) using, for example, the calculation formula shown in the following formula (2).

・・・（２）

... (2)

In this equation, C _1roughi is the i-th data of the rough intensity interpolation value of the color C1 (here, G), M _C1 is the average value of the color C1, and w _i is a weight value for the i-th data. The weight value w _i may be the same as that used by the average value calculation unit 331 and the average value calculation unit 333.

The covariance calculation unit 334 calculates the covariance value V _C1C2 of the first signal (here, G) and the second signal (here, R), for example, by the calculation formula shown in the following formula (3). .

In this equation, C _2roughi is the i-th data of the rough intensity interpolation value of the color C2 (here, R), and M _C2 is the average value of the color C2.

・・・（３）

... (3)

  When the calculation of the variance value described using the equation (2) and the calculation of the covariance described using the equation (3) are executed, accurate statistics as defined can be obtained. However, when the calculations of Formula (2) and Formula (3) are executed, a large amount of integration appears, so that the calculation time becomes long, and the circuit scale when this calculation is realized by hardware increases. Therefore, the variance and covariance may be calculated by simpler calculation.

  For example, the following equation (4) may be used for the covariance calculation.

・・・（４）

... (4)

  Here, the function sgn (a, b) is for checking the coincidence of the signs of the two variables a and b. Since the output of sgn (a, b) is one of {1, 0, -1}, the operation of sgn {a, b} does not actually require integration. The calculation of equation (4) replaces the i-th deviation of G and R in the calculation of equation (3) with the sum of absolute deviations of G and R. Even if the integration is replaced with summation using equation (4), it is possible to calculate an approximate value having sufficient accuracy for estimating the color distribution shape of the local region targeted by the present invention.

  Further, for example, the following equation (5) may be used for the covariance calculation.

・・・（５）

... (5)

  Here, the calculation of Expression (5) is obtained by replacing the integration of deviations of each i-th G and R in the calculation of Expression (3) with the process of selecting the minimum value of G and R. Note that the calculation of equation (5) has better approximation accuracy to the covariance calculation in equation (3) than the calculation of equation (4).

The expression (4) or the expression (5) is used to explain that the approximation operation for the covariance operation shown in the expression (3) is possible, but the dispersion operation described using the expression (2) is Since it is equivalent to the covariance when the two inputs are equal, the variance calculation described using equation (2) can be similarly approximated using equation (4) or equation (5). Needless to say. Specifically, when the variance calculation is approximated, (C _1roughi −M _C1 ) ² is approximated to (C _1roughi −M _C1 ) _{regardless of which} of the equations (4) and (5) is used.

  Next, FIG. 15 is a block diagram illustrating a configuration of the regression calculation processing unit 286. In FIG. 15, the regression calculation processing unit 286 (that is, the regression calculation processing unit 286-1) related to R is shown, but the operation related to G and B only has to replace R with G or B. The same process is executed.

  The regression calculation processing unit 286 includes an inclination calculation unit 351, an inclination calculation unit 352, an inclination synthesis unit 353, and a pixel intensity estimation unit 354.

The inclination calculation unit 351 calculates an inclination Km of the GR color distribution based on the average value MR of _R and the average value MG of _G. Specifically, the inclination calculation unit 351 calculates the ratio of the two average values as shown in the equation (6). Here, C1 is G and C2 is R.

・・・（６）

... (6)

M _threshold is a constant for avoiding the value to diverge by dividing by zero, and a sufficiently small positive value is set in advance.

The gradient calculating unit 352 calculates the gradient Ks of the C1-C2 color distribution based on the covariance values V _{C1, C2 of C1 and C2} and the variance values V _{C1, C1} of C1. Here, C1 is G and C2 is R. Specifically, the slope calculation unit 352 calculates the ratio of the variance and covariance values as shown in Expression (7).

・・・（７）

... (7)

V _threshold is a constant for avoiding the value to diverge by dividing by zero, and a sufficiently small positive value is set in advance. Using V _threshold in equation (7) to clip the variance of C1 in the denominator, division by zero can be avoided, but clipping using V _threshold further reduces noise in the flat part of the image. Can be used to reduce.

  That is, the variance of C1 as the denominator is a value reflecting the variation in luminance of the local region, and a small value is synonymous with the fact that the local region is flat. Since the noise of a solid-state image sensor stands out better as the image becomes flat, if noise reduction processing is performed only on the flat part of the image, the noticeable noise is effectively reduced without degrading the overall image quality. This can be reduced and is preferable. Also, by clipping the denominator of the gradient Ks of the color distribution so that the denominator does not become any smaller, the absolute value of Ks can be suppressed to be smaller than the original value. By suppressing the absolute value of the slope Ks to be small, the rate of change of C2 with respect to C1 is reduced, and as a result, the effect that the amplitude of C2 in the local region can be suppressed occurs.

  As described above, the clipping process for the denominator of the slope Ks has the effect of reducing noise, as in the case where both the process of determining whether the image is flat and the process of suppressing the output amplitude are executed. Thus, by using the color estimation method of the present invention, it is possible to suppress noticeable noise in a flat portion of an image without adding a separate noise reduction process.

  The inclination combining unit 353 combines the two inclination estimation values Km and Ks based on the reliability h, and calculates the inclination estimation value K. As described above, the inclination estimation value Ks based on the variance and covariance cannot always be correctly estimated in a region having a thin striped texture. Therefore, the inclination synthesis unit 353 uses the reliability h that reflects the strength of the thin striped texture and combines the inclination estimation value Km based on the average value using, for example, Equation (8) to estimate the inclination. The value K is calculated.

・・・（８）

... (8)

Then, the pixel intensity estimating means 354 applies the obtained gradient estimated value K, the two average values M _G and M _R , and the G intensity at the target pixel position to the linear regression equation of the following equation (9). Thus, an estimated value of the R intensity at the target pixel position is calculated.

・・・（９）

... (9)

Here, C _1center and C _2center are respectively the intensity of the color C1 (that is, G) corresponding to the first signal at the target pixel position and the color C2 (corresponding to the second signal at the target pixel position). That is, the estimated intensity value of R).

  Further, the pixel intensity estimating means 354 may calculate the intensity estimated value using a regression equation different from the equation (9). For example, as shown in Expression (10), the linear regression calculation may be performed by multiplying the slope estimation value K by an appropriate constant u.

・・・（１０）

... (10)

  In Equation (10), the first term can be considered as a high frequency component of the color C2 (ie, R) intensity corresponding to the second signal, and the second term can be considered as a low frequency component. In the equation (10), the same effect as when appropriate high-frequency correction for R is performed can be obtained by slightly enhancing the high-frequency component with an appropriate constant u. By doing so, it is possible to obtain an image in which the high frequency is corrected without adding a separate high frequency correction process.

In the above-described conventional third technique, the estimated value of the intensity of the color C2 corresponding to the second signal is obtained by calculation of C2 = M _C2 + β × (C1−M _C1 ). Looks like it corresponds to u. However, in the third conventional technique, β only emphasizes the high-frequency component of (C1−M _C1 ), that is, the C1 intensity, so that the high frequency band of the color C2 corresponding to the second signal is high. The component cannot be corrected correctly.

  In the above description, a case has been described in which the regression calculation processing unit 286 calculates the slope K of the regression line based on the average value, the variance value, and the covariance value calculated by the statistic calculation unit 285. The slope of the regression line may be calculated using a method other than this.

  For example, a definition formula for calculating the slope of the regression line using the standard deviation and the correlation coefficient is represented by the following formula (11).

・・・（１１）

(11)

  The standard deviations Sx and Sy are statistics indicating how much the data values are distributed around the average, and are values close to dx and dy representing the variation in the xy direction of two variables. Conceivable. You may make it obtain | require a regression line using this Formula (11).

  In particular, when data is distributed on a straight line having a positive slope and Rxy is 1, Sx and Sy are equivalent to dx and dy. That is, instead of using a complicated standard deviation of calculation, if there is another statistic that represents the data distribution width and is easier to calculate, Sx and Sy may be replaced with other statistic that represents the data distribution width. The slope of the regression line can be expected to show close behavior.

  Therefore, as another statistic representing the data distribution width, an average deviation used to represent the data distribution width along with the standard deviation is used instead. The definition of the average deviation Ax of x is shown by the following equation (12).

・・・（１２）

(12)

  Similarly, the average deviation Ay of y is also expressed by the following equation (13).

・・・（１３）

... (13)

  When Rxy is rewritten using the average deviations Ax and Ay, the following equation (14) is obtained.

・・・（１４）

(14)

  The average deviation can be calculated with a small amount of calculation compared to the fact that a square root or multiplication is necessary for the calculation using the standard deviation. Further, approximation of Rxy can be calculated at high speed by performing approximate calculation of multiplication used to calculate Vxy and multiplication of Ax and Ay.

  In this way, instead of the process of calculating the slope K of the regression line in the regression calculation processing unit 286 based on the average value, the variance value, and the covariance value calculated in the statistic calculation unit 285, the standard deviation And using the correlation function to calculate the slope of the regression line in the two-dimensional distribution of the two systems of data, or after approximating the deviation and correlation, respectively, the slope of the regression line in the two-dimensional distribution of the two systems of data May be calculated.

  Next, imaging processing executed by the digital still camera 201 will be described with reference to the flowchart of FIG.

  In step S <b> 1, the CPU 223 determines whether the user has pressed the shutter button based on a signal supplied from the operation input unit 224 via the bus 225.

  If it is determined in step S1 that the shutter button has not been pressed, in step S2, the CPU 223 controls the timing generator 217 to thin out the image signal acquired by the CCD image sensor 213 at a constant frame rate. Read is executed. The image signal read out is subjected to noise removal by the correlated double sampling circuit 214, converted into a digital signal by the A / D converter 215, and stored in the image RAM of the DSP block 216.

  In step S3, image processing to be described later with reference to FIG. 17 or 41 is executed.

  In step S4, the D / A converter 218 converts the digital image signal synchronized by the DSP block 216 into an analog signal, and the video encoder 219 converts the analog image signal supplied from the D / A converter 218 into video. The signal is converted into a signal and output to the display unit 220.

  In step S5, the display unit 220 displays the supplied video signal. That is, the display unit 220 plays a role of a camera finder in the digital still camera 201.

  If it is determined in step S1 that the shutter button has been pressed, in step S6, the CPU 223 controls the timing generator 217 to read out all pixels at a constant frame rate of the image signal acquired by the CCD image sensor 213. Is executed. The image signal read from all pixels is denoised by the correlated double sampling circuit 214, converted to a digital signal by the A / D converter 215, and stored in the image RAM of the DSP block 216.

  In step S7, image processing to be described later with reference to FIG. 17 or 41 is executed.

  In step S <b> 8, the codec processing unit 221 performs a predetermined format encoding process on the image data supplied from the DSP block 216.

  In step S9, the memory 222 receives the encoded image data and records the image captured by the user.

  After step S5 or step S9 is completed, in step S10, the CPU 223 determines whether the user has turned off the power based on the signal supplied from the operation input unit 224. If it is determined in step S10 that the power is not turned off, the process returns to step S1, and the subsequent processes are repeated. If it is determined in step S10 that the power has been turned off, the process ends.

  By such processing, when the shutter button is not pressed by the user, thinning readout is executed so that the image reading and processing can be executed at high speed, and the processed image is converted into video data and displayed. Therefore, the display unit 220 can play the role of a camera finder in the digital still camera 201. When the shutter button is pressed by the user, all the pixels are acquired in order to obtain a high-resolution image. Reading is performed and the processed image is encoded and recorded as a captured image.

  Next, image processing 1 executed by the DS2P block 216 in FIG. 6 in step S3 or step S7 in FIG. 16 will be described with reference to the flowchart in FIG.

  In step S <b> 21, the image RAM 241 is arranged with an array (for example, the mosaic array 231 described with reference to FIG. 4 or the mosaic array 241 described with reference to FIG. 5) determined by the color filter used in the CCD image sensor 213. ) Is acquired and temporarily stored.

  In step S22, the white balance adjustment unit 251 of the signal processor 242 applies an appropriate color according to the color of each pixel intensity so that the color balance of the achromatic subject region is achromatic with respect to the mosaic image. A white balance adjustment process, which is a process for applying a coefficient, is performed.

  In step S23, the gamma correction unit 252 performs gamma correction so that the brightness and color saturation of the image displayed on the display unit 220 are correctly displayed for each pixel intensity of the mosaic image with the white balance. Do.

  In step S24, the demosaic processing unit 253 performs a demosaic process described later with reference to FIG.

  In step S25, the switch 254 determines whether or not the supplied RGB image data is a thinned image. If it is determined in step S25 that the image is not a thinned image, the switch 254 supplies the supplied image to the gradation conversion processing unit 256, and thus the process proceeds to step S27.

  If it is determined in step S25 that the image is a thinned image, the switch 254 supplies the supplied image to the horizontal direction reduction processing unit 255 in step S26. The horizontal direction reduction processing unit 255 reduces the image in the horizontal direction so that the aspect ratio of the supplied image is the same as that of the non-thinned image.

  If it is determined in step S25 that the image is not a thinned image, or after the process of step S26 is completed, in step S27, the gradation conversion processing unit 256 compresses the gradation of the supplied image signal. The tone conversion processing is executed, and the tone-converted image signal is supplied to the YC conversion unit 257.

  In step S28, the YC conversion unit 257 performs YC conversion on the R, G, and B three-channel images that are output from the gradation conversion processing unit 257 by performing band processing on matrix processing and chroma components, A Y image and a C image are generated. In step S29, the generated Y image and C image are output, and the process is terminated.

  Through such processing, the DS2P block 216 performs various processing on the supplied mosaic image signal to generate a Y image and a C image, and displays the image data on the display unit 220 based on the control of the CPU 223. Is displayed in the D / A converter 218 and is stored in the memory 222, and is supplied to the codec processing unit 221.

  Next, the demosaic process 1 executed in step S24 of FIG. 17 will be described with reference to the flowchart of FIG.

  In step S41, the local region extraction unit 281 sets one of unprocessed pixels as a target pixel.

  In step S42, the local region extraction unit 281 determines whether or not the supplied mosaic image data is a thinned image. If it is determined in step S42 that the image is not a thinned image, the process proceeds to step S45.

  If it is determined in step S42 that the image is a thinned image, in step S43, the local region extraction unit 281 extracts a predetermined number (n × n) of pixels around the target pixel position as a local region, This is supplied to the G intensity interpolation processing unit 282, the reliability calculation unit 283, and the coarse interpolation processing unit 284.

  In step S44, coarse interpolation processing 1 described later using the flowchart of FIG. 19 is executed.

  If it is determined in step S42 that the image is not a thinned image, in step S45, the local region extraction unit 281 extracts a predetermined number (n × m) of pixels around the target pixel position as a local region, This is supplied to the G intensity interpolation processing unit 282, the reliability calculation unit 283, and the coarse interpolation processing unit 284.

  In step S46, coarse interpolation processing 2 described later using the flowchart of FIG. 20 is executed.

  After step S44 or step S46 is completed, in step S47, the G intensity interpolation processing unit 282 uses the pixels in the local area extracted by the local area extraction unit 281 to calculate the G intensity at the target pixel position. , The reliability calculation unit 283, and the regression calculation processing units 286-1 to 286-3

  In step S48, a reliability value calculation process to be described later is executed using the flowchart of FIG.

  In step S49, a statistic calculation process for a two-color distribution shape, which will be described later using the flowchart of FIG. 22, is executed in parallel in the statistic calculation units 285-1 to 285-3.

  In step S50, interpolation pixel estimation processing, which will be described later using the flowchart of FIG. 32, is executed in parallel in the regression calculation processing units 286-1 to 286-3.

  In step S51, the local region extraction unit 281 determines whether or not processing has been completed for all pixels. If it is determined in step S51 that the process has not been completed for all pixels, the process returns to step S41, and the subsequent processes are repeated. If it is determined in step S51 that the process has been completed for all pixels, the process proceeds to step S25 in FIG.

  In other words, each unit constituting the demosaic processing unit 253 executes the respective processes at the target pixel position when a certain target pixel position is determined, and the processes of step S41 to step S50 are completed for all the pixels. If so, the process is terminated.

  By such processing, the mosaic image obtained based on the color filter array of the CCD image sensor 213 is demosaiced (color interpolation or synchronization), and each color constituting the color filter is interpolated in each pixel. Image data can be obtained.

  Next, the rough interpolation process 1 executed in step S44 of FIG. 18 will be described with reference to the flowchart of FIG.

  In step S61, the coarse interpolation processing unit 284 initializes the value s of the first register indicating the pixel position where the processing is performed among the pixels of the supplied local region, and sets s = 2. Among the supplied local area pixels, the value t of the second register indicating the pixel position where the process is executed is initialized to t = 2.

  In step S63, the coarse interpolation processing unit 284 sets the pixel intensity of the pixel (s, t) as the interpolation value α1.

  In step S64, the coarse interpolation processing unit 284 is a pixel (s-1, t), which is a pixel adjacent to the left of the pixel (s, t), and a pixel above the pixel (s, t). The average intensity with the pixel (s, t−1) is set as an interpolation value α2.

  When the mosaic arrangement 231 described with reference to FIG. 4 is used, for example, when the target pixel (s, t) is a pixel indicating G intensity, the pixel (s−1, t) and the pixel (s, t-1) indicates R intensity or B intensity, and when the target pixel (s, t) is a pixel indicating R intensity or B intensity, the pixel (s-1, t) and the pixel (s, t) -1) indicates the G intensity.

  In step S65, the coarse interpolation processing unit 284 interpolates the average intensity of the pixel (s + 1, t) immediately adjacent to the pixel (s, t) and the pixel (s, t + 1) below the pixel (s, t). Let it be the value α3.

  When the mosaic arrangement 231 described with reference to FIG. 4 is used, for example, when the target pixel (s, t) is a pixel indicating G intensity, the pixel (s + 1, t) and the pixel (s, t + 1) Indicates R intensity or B intensity (however, a color different from the color corresponding to the average intensity of the pixels obtained in step S64), and the pixel of interest (s, t) is a pixel indicating R intensity or B intensity. At this time, the pixel (s + 1, t) and the pixel (s, t + 1) indicate G intensity.

  In step S66, the coarse interpolation processing unit 284 averages the pixel (s−1, t−1) and the pixel (s + 1, t + 1) that are in the upper left direction and the lower right direction with respect to the target pixel. The intensity is an interpolation value α4.

  When the mosaic arrangement 231 described with reference to FIG. 4 is used, for example, when the target pixel (s, t) is a pixel indicating G intensity, the pixel (s−1, t−1) and the pixel ( s + 1, t + 1) similarly indicates G intensity. For example, when the target pixel (s, t) is a pixel indicating R intensity, the pixel (s-1, t-1) and the pixel (s + 1, t + 1) Indicates the R intensity. For example, when the target pixel (s, t) is a pixel indicating the B intensity, the pixel (s−1, t−1) and the pixel (s + 1, t + 1) B intensity is shown.

  In step S67, the coarse interpolation processing unit 284 averages the pixel (s-1, t + 1) and the pixel (s + 1, t-1) that are in the diagonally lower left direction and the diagonally upper right direction with respect to the target pixel. The intensity is an interpolation value α5.

  When the mosaic arrangement 231 described with reference to FIG. 4 is used, for example, when the target pixel (s, t) is a pixel indicating G intensity, the pixel (s−1, t + 1) and the pixel (s + 1, t-1) similarly indicates G intensity. For example, when the target pixel (s, t) is a pixel indicating R intensity, the pixel (s-1, t + 1) and the pixel (s + 1, t-1) Indicates the B intensity. For example, when the target pixel (s, t) is a pixel indicating the B intensity, the pixel (s−1, t + 1) and the pixel (s + 1, t−1) have the R intensity. Indicates.

  In step S68, the coarse interpolation processing unit 284 determines which of RGB corresponds to the interpolation values α1 to α5 calculated by the processing of steps S63 to S66.

  In step S69, the coarse interpolation processing unit 284 sets the average value of the two interpolation values corresponding to G as the G interpolation value of the pixel (s, t).

  In step S70, the coarse interpolation processing unit 284 sets the interpolation value corresponding to R (the average value if there are two) as the R interpolation value of the pixel (s, t).

  In step S71, the coarse interpolation processing unit 284 sets the interpolation value corresponding to B (the average value if there are two) as the B interpolation value of the pixel (s, t).

  In step S72, the coarse interpolation processing unit 284 refers to the value t of the second register indicating the pixel position where the process is executed, and determines whether t = n−1.

  When it is determined in step S72 that t = n−1 is not satisfied, in step S73, the coarse interpolation processing unit 284 sets the value t of the second register indicating the pixel position to be processed to t = t + 1, The process returns to step S63, and the subsequent processes are repeated.

  If it is determined in step S72 that t = n−1, in step S74, the coarse interpolation processing unit 284 refers to the value s of the first register indicating the pixel position to be processed, and s = n It is determined whether or not -1.

  If it is determined in step S74 that s = n−1 is not satisfied, in step S75, the coarse interpolation processing unit 284 sets the value s of the first register indicating the pixel position to be processed to s = s + 1, The process returns to step S62, and the subsequent processes are repeated.

  If it is determined in step S74 that s = n−1, in step S76, the coarse interpolation processing unit 284 sets a set of RGB interpolation values of (n−2) square pixels near the target pixel. Is output to each of the statistic calculation units 285-1 to 285-3, and the process proceeds to step S47 in FIG.

  By such processing, the RGB interpolated values at the (n−2) × (n−2) pixel positions centered on the target pixel are obtained with respect to the n × n pixels extracted as the local region. Calculated.

  However, in the rough interpolation process described with reference to FIG. 19, the processes in steps S63 to S67 may be interchanged with each other, and the processes in steps S69 to S71 may be interchanged with each other. It doesn't matter.

  Next, the rough interpolation process 2 executed in step S46 of FIG. 18 will be described with reference to the flowchart of FIG.

  In step S91, the coarse interpolation processing unit 284 initializes the value s of the first register indicating the pixel position where the process is performed among the pixels of the supplied local region, sets s = 2, and in step S92, Among the supplied local area pixels, the value t of the second register indicating the pixel position where the process is executed is initialized to t = 2.

  In step S93, the coarse interpolation processing unit 284 sets the pixel intensity of the pixel (s, t) as the interpolation value α1.

  In step S94, the coarse interpolation processing unit 284 sets the average intensity of the pixel (s, t−1), which is a pixel above and below the pixel (s, t), and the pixel (s, t + 1) as the interpolation value α2.

  When the mosaic arrangement 241 described with reference to FIG. 5 is used, for example, when the target pixel (s, t) is a pixel indicating G intensity, the pixel (s, t−1) and the pixel (s , T + 1) indicates the G intensity, and when the target pixel (s, t) is a pixel indicating the R intensity, the pixel (s, t−1) and the pixel (s, t + 1) have the B intensity. When the target pixel (s, t) is a pixel indicating B intensity, the pixel (s, t−1) and the pixel (s, t + 1) indicate R intensity.

  In step S95, the coarse interpolation processing unit 284 is a pixel (s + 1, t-1) that is an upper right pixel of the pixel (s, t), and a pixel (s-) that is the pixel to the left of the pixel (s, t). 1, t) and the average intensity of the pixel (s, t) and the pixel (s + 1, t + 1), which is a pixel diagonally lower right, is an interpolation value α3.

  When the mosaic arrangement 241 described with reference to FIG. 5 is used, for example, when the target pixel (s, t) is a pixel indicating G intensity, the pixel (s + 1, t−1), the pixel (s− 1, t) and pixel (s + 1, t + 1) all indicate R intensity, or both indicate B intensity. For example, a pixel of interest (s, t) indicates R intensity or B intensity. The pixel (s + 1, t−1), the pixel (s−1, t), and the pixel (s + 1, t + 1) all exhibit G intensity.

  In step S <b> 96, the coarse interpolation processing unit 284 performs the pixel (s−1, t−1) and pixel (s−1, t + 1) that are the upper left, lower left, and right pixels with respect to the target pixel. ) And the average intensity of the pixel (s + 1, t) is an interpolation value α4.

  When the mosaic arrangement 241 described with reference to FIG. 5 is used, for example, when the target pixel (s, t) is a pixel indicating G intensity, the pixel (s + 1, t−1), the pixel (s− 1, t) and pixel (s + 1, t + 1) all indicate R intensity, or both indicate B intensity. For example, a pixel of interest (s, t) indicates R intensity or B intensity. The pixel (s + 1, t−1), the pixel (s−1, t), and the pixel (s + 1, t + 1) all exhibit G intensity.

  In step S97, the coarse interpolation processing unit 284 determines which of RGB corresponds to the interpolation values α1 to α4 calculated by the processing of steps S93 to S96.

  In step S98, the coarse interpolation processing unit 284 sets the average value of the two interpolation values corresponding to G as the G interpolation value of the pixel (s, t).

  In step S99, the coarse interpolation processing unit 284 sets the interpolation value corresponding to R as the R interpolation value of the pixel (s, t).

  In step S100, the coarse interpolation processing unit 284 sets the interpolation value corresponding to B as the B interpolation value of the pixel (s, t).

  In step S101, the coarse interpolation processing unit 284 refers to the value t of the second register indicating the pixel position where the process is executed, and determines whether t = m−1.

  If it is determined in step S101 that t = m−1 is not satisfied, in step S102, the coarse interpolation processing unit 284 sets the value t of the second register indicating the pixel position to be processed to t = t + 1, The process returns to step S93, and the subsequent processes are repeated.

  When it is determined in step S101 that t = m−1, in step S103, the coarse interpolation processing unit 284 refers to the value s of the first register indicating the pixel position where the process is performed, and s = n It is determined whether or not -1.

  If it is determined in step S103 that s = n−1 is not satisfied, in step S104, the coarse interpolation processing unit 284 sets the value s of the first register indicating the pixel position to be processed to s = s + 1, The process returns to step S92, and the subsequent processes are repeated.

  When it is determined in step S103 that s = n−1, in step S105, the coarse interpolation processing unit 284 determines the RGB interpolation values of the (n−2) and (m−2) pixels near the target pixel. The set is output to each of the statistic calculators 285-1 to 285-3, and the process proceeds to step S47 in FIG.

  By such processing, RGB interpolation values at each of (n−2) and (m−2) pixel positions centered on the target pixel are calculated for n × n pixels extracted as the local region. Is done.

  However, in the coarse interpolation process described with reference to FIG. 20, the order of steps S93 to S96 may be interchanged, and the order of steps S98 to S100 may also be interchanged. It doesn't matter.

  Next, the reliability calculation processing executed by the reliability calculation unit 283 in step S48 of FIG. 18 will be described with reference to the flowchart of FIG.

  In step S121, the high-frequency extraction units 321-1 to 321-6 of the reliability calculation unit 283 explain the G interpolation values at the predetermined 12 positions near the target pixel described with reference to FIG. The G interpolation value of the pixel position necessary for obtaining the high frequency extracted by itself is calculated.

  In step S122, the high-frequency extraction units 321-1 to 321-6 extract and add high-frequency components in six directions centered on the pixel of interest based on the G interpolation value calculated in the processing of step S121. The data is supplied to the processing unit 322.

  In step S123, the addition processing unit 322 adds the six high frequency components supplied from the high frequency extraction units 321-1 to 321-6 and outputs the result to the clipping processing unit 323.

  In step S124, the clipping processing unit 323 clips the addition result supplied from the addition processing unit 322, and outputs the value as a reliability value to the regression calculation processing units 286-1 to 286-3. Proceed to step S49 of FIG.

  Through such processing, a reliability value indicating whether or not a high-frequency component exists in the vicinity of the target pixel is calculated and output to the regression calculation processing units 286-1 to 286-3. The calculated reliability value increases as the strength of the thin striped texture increases, so the smaller the reliability value, the higher the probability that the color distribution shape is estimated by the statistics.

  Next, the two-color distribution shape statistic calculation process executed in the statistic calculation units 285-1 to 285-3 in step S49 of FIG. 18 will be described with reference to the flowchart of FIG.

  Each of the statistic calculation units 285-1 to 285-3 receives supply of rough interpolation values of two colors of RGB (the statistic calculation unit 285-2 converts the rough interpolation value of G to 2). Since the statistics can be calculated by the same processing, the two-color coarse interpolation values supplied to the statistics calculation unit 285 are represented in the flowchart of FIG. These will be described as a first signal and a second signal, respectively.

  In step S131, the average value calculation unit 331 of the statistic calculation unit 285 includes the first signal (statistic calculation units 285-1 to 285-3) among the rough interpolation values supplied from the rough interpolation value processing unit 284. In either case, the coarse interpolation value of G) is acquired.

  In step S132, the average value calculation unit 331 executes an average value calculation process to be described later using the flowchart of FIG. 23 or FIG.

  In step S133, the average value calculation unit 333 is the second signal (in the statistical value calculation unit 285-1, the rough interpolation value of R of the rough interpolation values supplied from the rough interpolation value processing unit 284). The coarse interpolation value processing unit 285-2 obtains the G coarse interpolation value, and the coarse interpolation value processing unit 285-3 obtains the B coarse interpolation value).

  In step S134, the average value calculation unit 333 executes an average value calculation process to be described later using the flowchart of FIG. 23 or FIG.

  In step S135, the variance calculation unit 332 executes a variance calculation process, which will be described later using the flowchart of FIG.

  In step S136, the covariance calculation unit 334 executes a covariance calculation process, which will be described later using the flowchart of FIG. 26, and the process returns to step S50 of FIG.

  Next, the average value calculation process 1 executed in step S132 or step S134 in FIG. 22 will be described with reference to the flowchart in FIG.

  In step S141, the average value calculation unit 331 or the average value calculation unit 333 calculates the average value Mx as the calculation result (the subscript x of the average value Mx is R-interpolated when the average value of the G interpolation value is calculated). When the average value of the values is calculated, R is replaced with R when the average value of the B interpolation values is calculated.

  In step S142, the average value calculation unit 331 or the average value calculation unit 333 sums the acquired interpolation values (here, (n−2) square interpolation values).

  In step S143, the average value calculation unit 331 or the average value calculation unit 333 divides the total value calculated in step S142 by the number of acquired values (here, the square of (n−2)).

  In step S144, the average value calculation unit 331 or the average value calculation unit 333 outputs the division result calculated by the process of step S143, and the process proceeds to step S133 or step S135 of FIG.

  Further, when obtaining the average value, each of the intensity of RGB subjected to coarse interpolation is weighted by weighting the distance from the target pixel as described with reference to FIG. 13 or FIG. 14, for example. The average value may be obtained based on the RGB intensity.

  The average value calculation process 2 executed in step S132 or step S134 of FIG. 22 will be described with reference to the flowchart of FIG.

  In step S151, the average value calculation unit 331 or the average value calculation unit 333 calculates the average value Mx (the subscript x of the average value Mx is the result of the calculation when the average value of the G interpolation value is calculated. When the average value of the values is calculated, R is replaced with R when the average value of the B interpolation values is calculated.

  In step S152, the average value calculation unit 331 or the average value calculation unit 333 uses, for example, FIG. 13 or FIG. 14 as the acquired interpolation value (here, (n−2) square interpolation values). Apply the weights as described and sum the weighted values.

  In step S153, the average value calculation unit 331 or the average value calculation unit 333 calculates the sum of weights.

  In step S154, the average value calculation unit 331 or the average value calculation unit 333 calculates the above-described equation (1). That is, in step S154, the average value calculation unit 331 or the average value calculation unit 333 calculates the total value calculated in step S152 as the number of acquired values (here, the square of (n−2)), Divide by the product of the weighted sum calculated in step S153.

  In step S155, the average value calculation unit 331 or the average value calculation unit 333 outputs the division result calculated by the process of step S154, and the process proceeds to step S133 of FIG. 22 or step S135.

  Next, the distributed calculation process 1 executed in step S135 of FIG. 22 will be described with reference to the flowchart of FIG.

  In step S171, the variance calculation unit 332 of the statistic calculation unit 285 outputs the variance value Vxx (assuming that the first signal is all the G intensity because the first signal is all G intensity, and therefore the variance value of the G interpolation value is Therefore, the subscript x of Vxx is replaced with G).

  In step S172, the variance calculation unit 332 calculates the average value Mx of the first signal calculated by the average value calculation unit 331 (here, since all the first signals have G intensity, the subscript x of Mx is , G).

  In step S173, the variance calculation unit 332 receives the rough interpolation value of a part of the local region (range of (n−2) × (n−2) centered on the target pixel), Among them, the value s ′ of the first register indicating the pixel position to be processed is initialized to s ′ = 1, and in step S174, a part of the local region (centered on the target pixel (n −2) × (n−2) range) of pixels, the second register value t ′ indicating the pixel position to be processed is initialized, and t ′ = 1.

In step S175, the variance calculation unit 332 subtracts the average value Mx acquired in step S172 from the intensity X (here, G intensity) of the pixel (s ′, t ′), and squares the subtraction result. The value added to the current variance value Vxx is updated as the variance value Vxx. That is, the variance calculation unit 332 calculates Vxx = Vxx + (| intensity X−average value Mx |) ² of | pixel (s ′, t ′). In this calculation, the variance calculation unit 332 may weight the intensity X of the pixel (s ′, t ′) as described with reference to FIG. 13 or FIG.

  In step S176, the variance calculation unit 332 refers to the value t ′ of the second register indicating the pixel position for executing the process, and determines whether t ′ = n−2 or m−2 (not thinned out). It is determined whether or not t ′ = n−2 for the image signal and t ′ = m−2 for the thinned image signal).

  When it is determined in step S176 that t ′ = n−2 or m−2 is not satisfied, in step S177, the variance calculation unit 332 updates the value t ′ of the second register to t ′ = t ′ + 1. Then, the process returns to step S175, and the subsequent processes are repeated.

  When it is determined in step S176 that t ′ = n−2 or m−2, in step S178, the variance calculation unit 332 refers to the value s ′ of the first register indicating the pixel position where the process is executed. Then, it is determined whether or not s ′ = n−2.

  When it is determined in step S178 that s ′ = n−2 is not satisfied, in step S179, the variance calculation unit 332 updates the value s ′ of the first register to s ′ = s ′ + 1 to perform processing. Returns to step S174, and the subsequent processing is repeated.

  If it is determined in step S178 that s ′ = n−2, in step S180, the variance calculation unit 332 selects a part of the local region ((n−2) × ( The variance value Vxx of the coarse interpolation value in the range n-2) is output, and the process returns to step S136 in FIG.

For the calculation of the variance value, by definition, it is necessary to divide (| intensity X of pixel (s ′, t ′) − average value Mx |) ² by the number of pixels used for calculating the variance value. Since the calculation result of the value is used to divide the calculation result of the covariance value by a process described later, when the covariance value is divided by the same number, the division process is performed by calculating the variance value and covariance. It can be omitted in both the calculation of values.

In addition, in the calculation of the variance value, when weighting is performed, as shown in the definition of the variance value described using Expression (2), (| intensity X−average value of pixels (s ′, t ′)] The variance value can be calculated by dividing Mx |) ² by the sum of the weighting factors. However, since the calculation result of the variance value is used to divide the calculation result of the covariance value by the processing described later, when weighting is also applied in the calculation of the covariance value, the division by the sum of the weighting factors is performed. The processing can be omitted in both the variance value calculation and the covariance value calculation.

  In addition, in the distributed calculation process 1 described with reference to FIG. 25, since there is a square calculation, there is a problem that the calculation process takes time and the scale of hardware increases. Therefore, the square calculation may be omitted by approximate calculation.

  Next, the variance calculation process 2 executed in step S135 in FIG. 22 when the square calculation is omitted by the approximate calculation will be described with reference to the flowchart in FIG.

  In step S191 to step S194, the same processing as step S171 to step S174 described with reference to FIG. 25 is executed.

  In step S195, the variance calculation unit 332 subtracts the average value Mx acquired in step S142 from the intensity X (here, G intensity) of the pixel (s ′, t ′), and squares the subtraction result. The value added to the current variance value Vxx is updated as the variance value Vxx. That is, the variance calculation unit 332 calculates Vxx = Vxx + (| intensity X−average value Mx |) of pixel (s ′, t ′). In this calculation, the variance calculation unit 332 may weight the intensity X of the pixel (s ′, t ′) as described with reference to FIG. 13 or FIG. 14, for example.

When the value | p | is normalized and 0 ≦ | P | <1, p ² can be approximated by | p |. This is similar to the fact that the approximation calculation for the covariance calculation shown in Expression (3) is possible using Expression (4) or Expression (5).

  In steps S196 to S200, the same processes as those in steps S176 to S180 described with reference to FIG. 25 are executed, and the process returns to step S136 in FIG.

  In this way, by using the approximate calculation processing similar to the case of performing the approximate calculation for the covariance calculation shown in Expression (3) using Expression (4) or Expression (5), 2 necessary for calculating the variance value is used. It is possible to obtain an approximate value of the variance value by an approximation operation that omits the calculation of the power.

  Next, the covariance calculation process executed in step S136 of FIG. 22 will be described with reference to the flowchart of FIG.

  In step S211, the covariance calculation unit 334 outputs a covariance value Vxy that is an output value (here, since all the first signals have G intensity, the subscript x of Vxy is replaced with G. Then, the subscript y of Vxy is replaced with R when the second signal has R intensity, and the subscript y of Vxy is replaced with G when the second signal has G intensity).

  In step S212, the covariance calculation unit 334 calculates the average value Mx of the first signal calculated by the process of step S92 of FIG. 22 by the average value calculation unit 331, and step S94 of FIG. 22 by the average value calculation unit 333. The second average value My calculated by the process is acquired.

  In step S213, the covariance calculation unit 334 receives a rough interpolation value of a part of the local region (range (n−2) × (n−2) centered on the target pixel). Among them, the value s ′ of the first register indicating the pixel position where the process is executed is initialized, and s ′ = 1.

  In step S214, the covariance calculation unit 334 performs processing among coarse interpolation values of pixels of a part of the local region (range of (n−2) × (n−2) centered on the target pixel). Initialize the value t ′ of the second register indicating the pixel position to execute t ′ = 1.

  In step S215, integration processing described later with reference to FIG. 28 or FIG. 29 is executed.

  In step S216, the covariance calculation unit 334 refers to the value t ′ of the second register indicating the pixel position to be processed, and determines whether t ′ = n−2 or m−2 (decimated). It is determined whether or not t ′ = n−2 for a non-image signal, and t ′ = m−2 for a thinned image signal).

  When it is determined in step S216 that t ′ = n−2 or m−2 is not satisfied, in step S217, the covariance calculation unit 334 sets the value t ′ of the second register to t ′ = t ′ + 1. After updating, the process returns to step S215, and the subsequent processes are repeated.

  When it is determined in step S216 that t ′ = n−2 or m−2, in step S218, the covariance calculation unit 334 sets the value s ′ of the first register indicating the pixel position where the process is performed. It is determined whether or not s ′ = n−2.

  If it is determined in step S218 that s ′ = n−2 is not satisfied, in step S219, the covariance calculation unit 334 updates the value s ′ of the first register to s ′ = s ′ + 1, The processing returns to step S144, and the subsequent processing is repeated.

  When it is determined in step S218 that s ′ = n−2, in step S220, the covariance calculation unit 334 outputs the covariance value Vxy, and the process proceeds to step S50 in FIG.

  The definition of the covariance value is defined by (| intensity X of pixel (s ′, t ′) − average value Mx |) (| intensity y of pixel (s ′, t ′) − average value My |). , It is necessary to divide by the number of pixels used in the calculation of the covariance value, but the calculation result of the covariance value is divided by the calculation result of the dispersion value by the process described later. In the case of division by the same number, the division process can be omitted in both the variance value calculation and the covariance value calculation.

  In addition, in the calculation of the covariance value, when weighting is performed, as shown in the definition of the dispersion value described using Expression (3), the intensity X−average of (| pixel (s ′, t ′) The covariance value can be calculated by dividing the value Mx |) (| intensity y of pixels (s ′, t ′) − average value My |) by the sum of the weighting coefficients. However, since the calculation result of the covariance value is divided by the calculation result of the dispersion value by a process described later, when weighting is applied also in the calculation of the dispersion value, the division process by the sum of the weighting coefficients is performed. It is possible to omit both the calculation of the variance value and the calculation of the covariance value.

  The integration process executed in step S215 of the above-described covariance calculation process of FIG. 27 includes the integration process 1 executed when the covariance value is calculated as defined using the above-described equation (3). There are two types of processing: integration processing 2 executed when the covariance value is calculated using approximate calculation.

  Next, the integration process 1 executed in step S215 of FIG. 27 will be described with reference to the flowchart of FIG. The integration process 1 is a process executed when a covariance value is calculated as defined using the above-described equation (3).

  In step S231, the covariance calculation unit 334 (the intensity X of the pixel (s ′, t ′ of the first signal) −the average value Mx of the first signal) and the pixel (s ′ of the second signal (s ′ , T ′), the product of intensity Y−second signal average value My).

  In step S232, the covariance calculation unit 334 sets Vxy = Vxy + wi (integration result), and the process proceeds to step S216 in FIG. Here, wi is a weighting value in the pixel (s ′, t ′).

  By executing such integration processing, a covariance value is calculated as defined using the above-described equation (3). When integration process 1 is executed, the calculation is as defined, so the calculation result is very accurate. Instead, the calculation takes time or the number of gates in the hardware implementation increases. Will be invited.

  Next, the integration process 2 executed in step S215 of FIG. 27 will be described with reference to the flowchart of FIG. The integration process 2 is a process executed when an approximate covariance value is calculated using the above-described equation (4) or equation (5).

  In step S241, the covariance calculation unit 334 sets {the intensity X of the pixel (s ′, t ′) of the first signal−the average value Mx of the first signal} to p.

  In step S242, the covariance calculation unit 334 sets {the intensity Y of the pixel (s ′, t ′) of the second signal−the average value My of the second signal} to q.

  In step S243, an integration approximation process described later with reference to FIG. 30 or FIG. 31 is executed.

  In step S244, the covariance calculation unit 334 sets Vxy = Vxy + (approximate value of pq), and the process proceeds to step S216 in FIG.

  Next, with reference to the flowchart of FIG. 30, the integration approximation process 1 executed in step S243 of FIG. 29 will be described. The integrated approximation process 1 is a process executed when an approximate covariance value is calculated using the above-described equation (4).

  In step S261, the covariance calculation unit 334 determines whether or not | p | ≧ | q | using the values p and q replaced in steps S241 and S242 of FIG.

  If it is determined in step S261 that | p | ≧ | q |, in step S262, the covariance calculation unit 334 determines whether p ≧ 0.

  If it is determined in step S262 that p ≧ 0, in step S263, the covariance calculation unit 334 sets pq approximate value = + q, and the process returns to step S244 in FIG.

  If it is determined in step S262 that p ≧ 0 is not satisfied, in step S264, the covariance calculation unit 334 sets an approximate value of pq = −q, and the process returns to step S244 in FIG.

  When it is determined in step S261 that | p | ≧ | q | is not satisfied, in step S265, the covariance calculation unit 334 determines whether q ≧ 0.

  If it is determined in step S265 that q ≧ 0, in step S266, the covariance calculation unit 334 sets pq approximate value = + p, and the process returns to step S244 in FIG.

  If it is determined in step S265 that q ≧ 0 is not satisfied, in step S267, the covariance calculation unit 334 sets an approximate value of pq = −p, and the process returns to step S244 in FIG.

  In this process, when q or p is 0, the approximate value of pq is always 0. Specifically, when q is 0, | p | ≧ | q | always holds, so that the approximate value of pq is 0 regardless of the value of p. Also, when p is 0, | p | ≧ | q | does not always hold, so the approximate value of pq is 0 regardless of the value of q.

  With the processing described with reference to FIG. 30, the covariance value can be approximated using the above-described equation (4).

  Next, the integration approximation process 2 executed in step S243 in FIG. 29 will be described with reference to the flowchart in FIG. The integrated approximation process 1 is a process executed when an approximate covariance value is calculated using the above-described equation (5).

  In step S281, the covariance calculation unit 334 determines whether p or q is 0 using the values p and q replaced in steps S241 and S242 of FIG.

  If it is determined in step S281 that p or q is 0, in step S281, the covariance calculation unit 334 sets the approximate value of pq to 0, and the process returns to step S244 in FIG.

  If it is determined in step S281 that p and q are not 0, in step S283, the covariance calculation unit 334 determines that the relationship between p and q is p> 0 and q> 0, or p <0. In addition, it is determined whether any one of q <0 is satisfied.

  If it is determined in step S283 that the relationship between p and q is either p> 0 and q> 0, or p <0 and q <0, in step S284, the covariance calculation unit In step 334, the approximate value of pq is set to (| p | + | q |) / 2, and the process returns to step S244 in FIG.

  In step S283, when it is determined that the relationship between p and q is neither p> 0 and q> 0, or p <0 and q <0, the covariance calculation unit 334 determines in step S285. , Pq is set to (− | p | − | q |) / 2, and the process returns to step S244 in FIG.

  Note that it is very rare that the value p or q replaced in step S241 and step S242 in FIG. 29 takes a value of 0. Therefore, the processing in step S281 and step S282 may be omitted. By doing so, it is possible to increase the calculation processing speed and reduce the hardware implementation scale.

  By the processing described with reference to FIG. 31, the covariance value can be approximated using the above-described equation (5).

  By executing the integration process described with reference to FIGS. 29 to 31, the covariance value can be approximated using the above-described equation (4) or equation (5). In such a case, compared with the case where the calculation of the covariance value as defined in Expression (3) is executed, the calculation speed is increased or the hardware gate implemented for the calculation is used. Advantages such as being able to reduce the number occur.

  Next, the interpolation pixel estimation process executed in step S27 of FIG. 18 will be described with reference to the flowchart of FIG.

  In step S <b> 321, the inclination calculation unit 351 calculates the estimated inclination value Km based on the average value using the above-described equation (6), and outputs it to the inclination synthesis unit 353.

  In step S322, the inclination calculation unit 352 calculates the inclination estimation value Ks based on the variance value and the covariance value using the above-described equation (7), and outputs the inclination estimation value Ks to the inclination synthesis unit 353.

  Here, the inclination calculation unit 352 performs clipping processing on the denominator of the inclination Ks as shown in the above-described equation (7). This clipping process has the effect of reducing noise, as in the case where both the determination of whether an image is flat and the process of suppressing the output amplitude are executed. By executing this process, it is possible to suppress noise that is noticeable in a flat portion of an image without adding a separate noise reduction process.

  In step S323, the inclination synthesis unit 353 uses the inclination estimation value Km calculated in step S321 and the inclination estimation value Ks calculated in step S322 based on the reliability h using the above-described equation (8). Are combined to calculate an estimated inclination value K and output the result to the pixel intensity estimation unit 354.

In step S324, the elementary intensity estimation unit 354, based on the estimated slope value K calculated in step S323, the two average values M _G and M _R (or M _B ), and the G intensity of the target pixel position, The estimated value of the R intensity (or B intensity) of the target pixel position is calculated using the linear regression expression of the above-described Expression (9) or Expression (10), and the process returns to Step S28 in FIG.

  Through such processing, the estimated intensity of each color at the target pixel position can be calculated. In particular, the high frequency component is slightly enhanced by multiplying the slope K by an appropriate constant u greater than 1 using the above-described equation (10) in the linear regression equation for calculating the estimated intensity, An effect similar to that obtained when appropriate high-frequency correction for R is executed can be obtained. By doing so, it is possible to obtain an image in which the high frequency is corrected without adding a separate high frequency correction process.

  As described above, the mosaic arrangement 231 (all pixel readout) described with reference to FIG. 4 is such that the G signal is arranged on the checkered pattern, is evenly distributed in both the horizontal and vertical directions, and The R signal and the B signal are evenly distributed in both the horizontal and vertical directions. Further, in the mosaic arrangement 241 (decimation readout) described with reference to FIG. 5, the obtained color arrangement is every other column. In the non-G column, R and B are uniformly mixed on the same horizontal line and alternately arranged on the same vertical line. In other words, the digital still camera 201 of the present invention using the filter arrangement described with reference to FIGS. 4 and 5 is capable of reading out all pixels and comparing the image signal acquired with the conventionally used filter arrangement. In any of the thinning-out readings, it is possible to perform the G interpolation process with high accuracy, and furthermore, since it is difficult for the RB interpolation accuracy to vary depending on the direction, false colors and the like can be suppressed.

  In the above description, the RGB synchronization processing has been described as being realized by obtaining a linear regression line. However, the RGB synchronization processing other than obtaining a linear regression line, for example, the following method You may implement | achieve by calculating by.

First method: Assuming that there is little change in color within the local region, calculation is performed by weighted average using only the information of the color to be obtained.
Second method: Calculation is performed on the assumption that the ratio of the low frequency color and the ratio of the high frequency color in the local region are substantially equal.
Third method: Calculation is performed on the assumption that the correlation between the G intensity signal in the local region and the R intensity signal or the B intensity signal is high.

  Next, the specific method will be described with reference to FIG. 33 as an example of calculating R corresponding to the position of G5 which is the center pixel of the thinned 5 × 3 mosaic array. Note that the weighting coefficient described with reference to FIG. 14 is used as the weighting coefficient corresponding to FIG.

  In the first method described above, R at the position of G5 is represented by the following equation (15).

      R = (R1 × 2 + R2 × 6 + R3 × 2) / 10 (15)

  In the second method described above, R at the position of G5 is expressed by the following equation (16).

      R = G5 × LPF (R) / LPF (G) (16)

  In the equation (16), LPF (R) = (R1 × 2 + R2 × 6 + R3 × 2) / 10 and LPF (G) = (G1 + G3 + G7 + G9 + (G2 + G4 + G6 + G8) × 3 + G5 × 9) / 25.

  And in the 3rd method mentioned above, R of the position of G5 is shown by following Formula (17).

  R = k × (G5−LPF (G)) + LPF (R) (17)

  In Expression (17), LPF (R) = (R1 × 2 + R2 × 6 + R3 × 2) / 10, LPF (G) = (G1 + G3 + G7 + G9 + (G2 + G4 + G6 + G8) × 3 + G5 × 9) / 25, and k is It is the slope of the regression line calculated from the R and G statistics within the local region.

  As described above, when the color filter having the mosaic arrangement 231 of FIG. 4 to which the present invention is applied is used, the RGB distribution is not biased in a specific direction as compared with the conventional case in both the all pixel readout and the thinning readout. Since it is uniform, the deterioration of resolution in a specific direction does not occur, and variations in resolution due to false colors and directions can be suppressed, and a high-quality image can be obtained.

  In addition to the mosaic arrangement 231 of the first example described with reference to FIG. 4, the mosaic arrangement used for the on-chip color filter of the CCD image sensor 213 includes, for example, as shown in FIG. Although the color filter array is the same as that of the first example, a mosaic array 361 that alternately applies different sensitivities for each row may be used.

  In FIG. 34, in the numbers added to the color information, it is assumed that 1 indicates bright sensitivity and 0 indicates dark sensitivity.

  When the mosaic arrangement 361 shown in FIG. 34 is used for the on-chip color filter of the CCD image sensor 213, the mosaic arrangement 231 of the first example is used for the on-chip color filter of the CCD image sensor 213, and In contrast, if the thinning process is performed every other line, only one of the color intensity signals having the sensitivity remains in the mosaic image after the thinning. Therefore, the CPU 223 controls the timing generator 217 to read out the image signal acquired by the CCD image sensor 213 every two rows so that skipping is executed alternately.

  FIG. 35 shows a thinned mosaic array obtained by such thinning-out reading every two rows. In the mosaic array 362 of FIG. 35, it is possible to obtain a bright-sensitive image signal and a dark-sensitive image signal for each row. Therefore, the digital still camera 201 using the mosaic arrangement on-chip color filter shown in the second example can perform demosaic processing using the color intensity information of both sensitivities in the DSP block 216. In addition to preventing the occurrence of noise such as color, it is possible to prevent overexposure and underexposure and obtain a higher quality image.

  When the mosaic arrangement 361 shown in FIG. 34 is used for the on-chip color filter of the CCD image sensor 213, the DSP block 216 of the digital still camera 201 is replaced with the demosaic processing unit 253 described with reference to FIG. The demosaic processing unit 371 in FIG. 36 is used.

  Note that portions corresponding to those in FIG. 7 are denoted by the same reference numerals, and description thereof is omitted as appropriate. That is, the demosaic processing unit 371 in FIG. 36 is basically the same except that a sensitivity compensation unit 375 necessary for processing the mosaic image data having the mosaic arrangement described with reference to FIGS. 34 and 35 is newly provided. Further, it has the same configuration as the demosaic processing unit 253 of FIG.

  The sensitivity compensation unit 375 receives the input of the mosaic image of the local region extracted by the local region extraction unit 281, and further receives the high sensitivity of the mosaic array 361 used for the on-chip color filter of the CCD image sensor 213 from the CPU 223. If the signal intensity of each pixel of the mosaic image is an effective value (ie, a value that does not cause overexposure or underexposure), the color component signal of the low sensitivity pixel is received. Then, compensation is performed so as to correspond to a high-sensitivity photographed image, and the obtained sensitivity-compensated local color mosaic image is converted into a G intensity interpolation processing unit 282, a reliability calculation unit 283, and rough interpolation processing. Supplied to the unit 284.

  FIG. 37 is a block diagram showing the configuration of the sensitivity compensation unit 375.

  The multiplication unit 381 of the sensitivity compensation unit 375 multiplies each pixel of the mosaic image supplied from the local region extraction unit 281 by the sensitivity ratio between the pixel photographed with high sensitivity and the pixel photographed with low sensitivity. Output to the selector 382. The selector 382 receives the mosaic image and the sensitivity-compensated mosaic image output from the multiplication unit 381, and based on the information indicating the sensitivity arrangement of the mosaic image stored in advance, the color intensity signal at the pixel position with high sensitivity. For the low-sensitivity pixel position, the mosaic intensity color intensity signal supplied from the multiplication section 381 is output as the validity of the mosaic image color intensity signal. The data is output to the determination unit 383.

  The validity determination 383 is a pixel value (color component value) that is input from the selector 382 and has a predetermined pixel value (color component value) among pixels constituting a local mosaic image in which the sensitivity at the time of pixel shooting is unified. A pixel whose noise level is below (ie, blackout) or whose pixel value is above a predetermined saturation level (ie, overexposure) is determined to be an invalid pixel, and the pixel value is determined to be an invalid pixel. The value is substituted with a value indicating that it is present (for example, a negative value), and is output to the G intensity interpolation processing unit 282, the reliability calculation unit 283, and the coarse interpolation processing unit 284.

  Next, the demosaic process 2 executed by the demosaic processor 371 of FIG. 36 will be described with reference to the flowchart of FIG.

  In step S351, the local region extraction unit 281 sets one of unprocessed pixels as a target pixel.

  In step S352, the sensitivity compensation unit 375 performs sensitivity compensation processing on the supplied mosaic image data. Specifically, as described with reference to FIG. 37, the multiplication unit 381 captures the pixels of the mosaic image supplied from the local region extraction unit 281 with the pixels captured with high sensitivity and the low sensitivity. The selector 382 multiplies the sensitivity ratio of the selected pixel, and the selector 382 validates the color intensity signal of the mosaic image with respect to the color intensity signal of the highly sensitive pixel position based on the information indicating the sensitivity array of the mosaic image stored in advance. The output is output as it is to the sex determination unit 383, and the sensitivity-compensated mosaic image color intensity signal supplied from the multiplication unit 381 is output to the validity determination unit 383 for the low sensitivity pixel position. The validity determination 383 determines that the pixel value is equal to or lower than a predetermined noise level or the pixel value equal to or higher than the predetermined saturation level as an invalid pixel, and the pixel value is an invalid pixel. This value is replaced with a value indicating this, and output to the G intensity interpolation processing unit 282, the reliability calculation unit 283, and the coarse interpolation processing unit 284.

  In steps S353 to S362, processing equivalent to steps S42 to S51 of FIG. 18 is executed. That is, it is determined whether or not the image has been thinned out, and for all pixel readout images that have not been thinned out, a predetermined number (n × n) of pixels around the target pixel position are extracted as local regions. Then, the rough interpolation process 1 described with reference to the flowchart of FIG. 19 is executed.

  When it is determined that the image is a thinned image, a predetermined number (n × m) of pixels around the target pixel position are extracted as local regions, and the rough interpolation process 2 described with reference to the flowchart of FIG. 20 is performed. Executed.

  After the rough interpolation process 1 or the rough interpolation process 2, the G intensity at the target pixel position is calculated, the value calculation process described with reference to the flowchart of FIG. 21 is executed, and the two colors described with reference to the flowchart of FIG. The distribution shape statistic calculation processing is executed, and the interpolation pixel estimation processing described with reference to the flowchart of FIG. 32 is executed.

  Then, it is determined whether or not processing has been completed for all pixels. Returning to step S351, the subsequent processing is repeated. If it is determined that the processing has been completed for all the pixels, the processing returns to step S25 in FIG.

  By such processing, the mosaic image obtained based on the color filter array of the CCD image sensor 213 is demosaiced (color interpolation or synchronization), and each color constituting the color filter is interpolated in each pixel. In addition, it is possible to obtain image data that is free from blackout and color skipping.

  Further, the digital still camera 201 to which the present invention is applied may include a DSP block 391 shown in FIG. 39 instead of the DSP block 216 described with reference to FIG.

  Note that portions corresponding to those in FIG. 6 are denoted by the same reference numerals, and description thereof is omitted as appropriate. That is, the DSP block 391 in FIG. 39 is basically the same configuration as the DSP block 216 described with reference to FIG. 6 except that a signal processor 395 is provided instead of the signal processor 242. In the signal processing processor 395, a switch 254 is provided at the subsequent stage of the gamma correction unit 252, and a horizontal direction reduction processing unit 398 is provided instead of the horizontal direction reduction processing unit 255, and one of the switches 254 is provided. And the output of the horizontal reduction processing unit 398 are basically the same as the signal processing processor 242 described with reference to FIG. 6, except that the demosaic processing unit 253 is supplied. It is.

  The horizontal direction reduction processing unit 398 performs horizontal reduction processing so that the aspect ratio of the thinned mosaic image supplied from the switch 254 is the same as that of the mosaic image that has not been thinned. Here, the horizontal direction reduction processing unit 398 does not change the mosaic arrangement of the thinned mosaic image supplied from the switch 254, but changes the aspect ratio to be the same as that of the non-decimated mosaic arrangement.

  Specifically, the horizontal direction reduction processing unit 398 thins the supplied mosaic array 241 or mosaic array 362 every four columns or uses a predetermined filter to reduce the same array in the horizontal direction. Create mosaic image data.

  An example of a filter used for horizontal thinning will be described with reference to FIG.

  In order to reduce the number of pixels in the horizontal direction to ½ without changing the mosaic arrangement, the horizontal reduction processing unit 398 uses the color intensity signal of the pixels arranged in one row to change the original mosaic arrangement. A G intensity signal is generated at the position of the pixel having the B intensity signal, and an R or B intensity signal is generated at the position of the pixel having the R intensity signal of the original mosaic arrangement.

  At this time, the horizontal reduction processing unit 398 calculates the values of pixels having the respective color intensities of RGB by weighted average calculation using color signals of the same color in the vicinity of the same column as the generated intensity signal. .

  For example, the horizontal reduction processing unit 398 uses the four pixels G1, G2, G3, and G4 in the vicinity of the pixel interpolation position to generate a pixel signal having the G color intensity, and uses the following equation (18): Is used to calculate a color intensity and create a pixel signal having an R color intensity, and using the five pixels R1, R2, R3, R4, and R5 in the vicinity of the pixel interpolation position, 19) is used to calculate the color intensity and create a pixel signal having the B color intensity, using the four pixels B1, B2, B3, and B4 in the vicinity of the pixel interpolation position, 20) can be used to calculate the color intensity.

G = (G1 + G2 × 3 + G3 × 3 + G4) / 8 (18)
R = (R1 + R2 × 4 + R4 × 6 + R4 × 4 + R5) / 16 (19)
B = (B1 + B2 × 3 + B3 × 3 + B4) / 8 (20)

  In this way, the horizontal reduction processing unit 398 reduces the horizontal direction in order to make the aspect ratio of the pixel signal the same as that of the non-decimated mosaic image without changing the mosaic arrangement of the supplied mosaic image. Processing can be executed.

  Next, image processing 2 executed by the DSP block 391 shown in FIG. 39 will be described with reference to the flowchart of FIG.

  In steps S401 to S403, the same processing as in steps S21 to S23 of FIG. 17 is executed. In other words, the image RAM 241 acquires a mosaic image composed of intensity signals of a periodic pattern having an arrangement determined by the color filter used in the CCD image sensor 213, temporarily stores it, and the white balance adjustment unit 251. Performs a white balance adjustment process that applies an appropriate coefficient according to the color of each pixel intensity so that the color balance of the achromatic subject area is achromatic for the mosaic image, and gamma correction The unit 252 performs gamma correction on each pixel intensity of the mosaic image with white balance so that the brightness and color saturation of the image displayed on the display unit 220 are correctly displayed.

  In step S404, the switch 254 determines whether or not the supplied RGB image data is a thinned image. If it is determined in step S404 that the image is not a thinned image, the switch 254 supplies the supplied image to the demosaic processing unit 253, and thus the process proceeds to step S406.

  If it is determined in step S404 that the image is a thinned image, the switch 254 supplies the supplied image to the horizontal reduction processing unit 398 in step S405. The horizontal direction reduction processing unit 398 reduces the image in the horizontal direction so that the aspect ratio of the supplied image is the same as that of the non-thinned image.

  If it is determined in step S404 that the image is not a thinned image, or after the process of step S405 is completed, the demosaic processing unit 253 performs the demosaic process described with reference to FIG. 18 in step S406.

  In steps S407 to S409, processing similar to that in steps S27 to S29 in FIG. 17 is executed. That is, the gradation conversion processing unit 256 executes gradation conversion processing for compressing the gradation of the supplied image signal, and supplies the image signal subjected to gradation conversion to the YC conversion unit 257. Then, the YC conversion unit 257 performs YC conversion on the R, G, and B three-channel images by performing matrix processing and band limitation on chroma components, and generates and outputs Y images and C images. Is terminated.

  Through such processing, the DSP block 391 performs various processing on the supplied mosaic image signal to generate a Y image and a C image, and displays the image data on the display unit 220 under the control of the CPU 223. Can be displayed in the D / A converter 218, and can be supplied to the codec processing unit 221 when stored in the memory 222.

  As described above, according to the present invention, the color line of the color filter array shown in FIG. 4 in which the horizontal line is composed of three colors of RGB, G is checkered, and RB is diagonally arranged. In the digital still camera 201 using the image pickup device, for example, in the still image pickup mode, since there is little variation in resolution depending on the direction, a high-quality high-resolution image can be obtained, and also in the color arrangement after thinning Since the R intensity signal and the B intensity signal are more evenly distributed as compared with the prior art, reading and skipping are repeated by reading each line and repeating skipping in an imaging mode such as monitor display. Image processing can be speeded up, and it is possible to obtain a high-quality low-resolution image in which false colors and the like are suppressed. Further, the color filter shown in FIG. 22 is configured such that the horizontal line is composed of three colors of RGB, G is a checkered arrangement, RB is arranged in an oblique direction, and the sensitivity is different every other row. Similarly, in the digital still camera 201 using the array of color solid-state imaging devices, it is possible to obtain a high-quality low-resolution image in which false colors and the like are suppressed as compared with the conventional one, and the image is captured. It is possible to prevent overexposure and underexposure of an image. Thus, the digital still camera 201 to which the present invention is applied can cope with both high-quality and high-speed low-resolution image output and high-quality high-resolution image output.

  Further, since none of the conventional demosaic techniques has means for generating a pair of C1 and C2 intensities like the rough interpolation processing unit 284 of FIG. 7 or FIG. For example, in the case where does not have a positive correlation, the correct color intensity cannot be estimated. On the other hand, in the demosaic processing unit 253 to which the present invention is applied, regardless of whether the color distribution is descending to the right or to the right (that is, the color has a positive correlation or an uncorrelated correlation). Whether or not) can make a correct estimation.

  Furthermore, in the present invention, a statistical calculation such as variance, covariance, correlation coefficient, etc., which has a large calculation cost, can be replaced with a simple calculation to perform an approximate calculation. The simplification of these statistic calculations used in the present invention has sufficient accuracy to obtain a reasonable approximate value in estimating the correlation between two colors in demosaic processing.

  Conventionally, a process or circuit for performing high-frequency correction and noise reduction processing has been added separately. However, when the present invention is used, a process or circuit for performing high-frequency correction and noise reduction processing is separately provided. Without adding, it is possible to obtain the effects of high-frequency correction and noise reduction at the same time as estimating and interpolating colors by demosaic processing. Therefore, by applying the present invention, it is possible to construct a simple camera system having performances such as high-frequency correction and noise reduction processing.

  In the above-described processing, the case where arithmetic approximation is performed in the image processing in the digital still camera 201 using the three-color array has been described. However, the present invention also applies to a digital video camera that can capture a moving image. Is applicable. When the present invention is applied to a digital video camera, the codec processing unit 221 executes processing by digital image data compression or decompression algorithm such as MPEG (Moving Picture Coding Experts Group / Moving Picture Experts Group).

  In addition to the digital still camera 201 using a CCD image sensor as an image pickup device, the present invention can read out all pixels and perform line-by-line readout as an image pickup device, and picks up a mosaic image with a predetermined color arrangement. Needless to say, any image sensor can be used as long as it can be used, for example, a CMOS or other image sensor.

  The series of processes described above can be executed by hardware, but can also be executed by software.

  In this case, the above-described functions are realized by executing the software by the DSP block 216. Further, for example, a part of the processing of the digital still camera 201 can be executed by a personal computer 401 as shown in FIG.

  42, a CPU (Central Processing Unit) 411 performs various processes according to a program stored in a ROM (Read Only Memory) 412 or a program loaded from a storage unit 418 to a RAM (Random Access Memory) 413. Execute. The RAM 413 also appropriately stores data necessary for the CPU 411 to execute various processes.

  The CPU 411, ROM 412, and RAM 413 are connected to each other via a bus 414. An input / output interface 415 is also connected to the bus 414.

  The input / output interface 415 includes an input unit 416 including a keyboard and a mouse, an output unit 417 including a display and a speaker, a storage unit 418 including a hard disk, and a communication unit 419 including a modem and a terminal adapter. It is connected. The communication unit 419 performs communication processing via a network including the Internet.

  A drive 420 is connected to the input / output interface 415 as necessary, and a magnetic disk 431, an optical disk 432, a magneto-optical disk 433, a semiconductor memory 434, or the like is appropriately mounted, and a computer program read from these is loaded. The storage unit 418 is installed as necessary.

  When a series of processing is executed by software, a program constituting the software is incorporated in dedicated hardware (for example, the DSP block 216 and the demosaic processing unit 253 included therein). For example, a general-purpose personal computer that can execute various functions by installing a computer or various programs is installed from a network or a recording medium.

  As shown in FIG. 42, this recording medium is distributed to supply a program to the user separately from the apparatus main body, and includes a magnetic disk 431 (including a floppy disk) on which a program is stored, an optical disk 432 ( CD-ROM (including Compact Disk-Read Only Memory), DVD (including Digital Versatile Disk)), magneto-optical disk 433 (including MD (Mini-Disk) (trademark)), or semiconductor memory 434 or other package media In addition to being configured, it is configured by a ROM 412 in which a program is stored and a hard disk included in the storage unit 418, which is supplied to the user in a state of being pre-installed in the apparatus main body.

  In the present specification, the step of describing the program stored in the recording medium is not limited to the processing performed in chronological order in the order in which they are included, but is not necessarily processed in chronological order, either in parallel or individually. The process to be executed is also included.

It is a figure for demonstrating the mosaic arrangement | sequence thinned out when the Bayer arrangement | sequence used conventionally and a Bayer arrangement | sequence are used. It is a figure for demonstrating the arrangement | sequence of the color filter of the color arrangement | sequence which contains the information of all the colors in the data of 1 line, and the mosaic arrangement | sequence thinned out in that case. It is a block diagram which shows the structure of the digital still camera to which this invention is applied. It is a figure for demonstrating the 1st example of the mosaic arrangement | sequence of the color filter used with the digital still camera to which this invention is applied. It is a figure for demonstrating a mosaic arrangement | sequence at the time of thinning out the mosaic arrangement | sequence of FIG. It is a block diagram which shows the structure of the DSP block of FIG. It is a block diagram which shows the structure of the demosaic process part of FIG. It is a block diagram which shows the structure of the reliability calculation part of FIG. It is a figure for demonstrating the extraction position of the pixel intensity | strength in a reliability calculation part. It is a figure for demonstrating the extraction position of the pixel intensity | strength in a reliability calculation part. It is a figure for demonstrating the group of the pixel intensity | strength for extracting a high frequency component in a reliability calculation part. It is a block diagram which shows the structure of the statistic calculation part of FIG. It is a figure for demonstrating the example of a weighting coefficient. It is a figure for demonstrating the example of a weighting coefficient. It is a block diagram which shows the structure of the regression calculation process part of FIG. It is a flowchart for demonstrating an imaging process. It is a flowchart for demonstrating about the image process 1 which the DSP block of FIG. 6 performs. 10 is a flowchart for explaining demosaic processing 1; 6 is a flowchart for explaining rough interpolation processing 1; 10 is a flowchart for explaining rough interpolation processing 2; It is a flowchart for demonstrating reliability calculation processing. It is a flowchart for demonstrating the statistic amount calculation process of 2 color distribution shape. 6 is a flowchart for explaining an average value calculation process 1; 10 is a flowchart for explaining average value calculation processing 2; 10 is a flowchart for explaining distributed calculation processing 1; 10 is a flowchart for explaining distributed calculation processing 2; It is a flowchart for demonstrating a covariance calculation process. 5 is a flowchart for explaining an integration process 1; 5 is a flowchart for explaining an integration process 2; 5 is a flowchart for explaining integration approximation processing 1; 10 is a flowchart for explaining integration approximation processing 2; It is a flowchart for demonstrating an interpolation pixel estimation process. It is a figure for demonstrating a weighted average calculation. It is a figure for demonstrating the 2nd example of the mosaic arrangement | sequence of the color filter used with the digital still camera to which this invention is applied. It is a figure for demonstrating a mosaic arrangement | sequence at the time of thinning out the mosaic arrangement | sequence of FIG. It is a block diagram which shows a different structure of the demosaic process part of FIG. It is a block diagram which shows the structure of the sensitivity compensation part of FIG. 10 is a flowchart for explaining demosaic processing 2; FIG. 4 is a block diagram showing a different configuration of the DSP block of FIG. 3. It is a figure for demonstrating the process of the horizontal direction reduction process part of FIG. It is a flowchart for demonstrating about the image process 2 which the DSP block of FIG. 39 performs. It is a block diagram which shows the structure of a personal computer.

Explanation of symbols

  201 Digital Still Camera, 216 DSP Block, 231 Mosaic Array, 241 Mosaic Array, 242 Signal Processing Processor, 253 Demosaic Processor, 254 Switch, 255 Horizontal Reduction Unit, 281 Local Area Extractor, 282 G Intensity Interpolator , 283 reliability calculation unit, 284 coarse interpolation processing unit, 285 statistics calculation unit, 286 regression calculation processing unit, 321 high frequency extraction unit, 322 addition processing unit, 323 clip processing unit, 331 average value calculation unit, 332 variance value calculation Unit, 333 average value calculation unit, 334 covariance calculation unit, 351, 352 inclination calculation unit, 353 inclination combination unit, 354 pixel intensity estimation unit, 361, 362 mosaic arrangement, 375 sensitivity compensation unit, 391 signal processing processor, 39 8 Horizontal reduction processor

Claims (8)

3 types determined by the spectral sensitivity of the filter from a mosaic image signal obtained by an image sensor having three types of filters with different spectral sensitivities, and any one of the plurality of types of filters is used for each pixel. In an image processing apparatus for generating a color image in which intensity information for each pixel position corresponding to one color is aligned with all pixels,
Control means for controlling readout of signals from the image sensor;
Generating means for generating, from a mosaic image signal obtained by the image sensor, three single-color image data in which intensity information corresponding to each of the three types of colors determined by the spectral sensitivity of the filter is aligned in all pixels; ,
The image sensor
The first filter is arranged in a checkered pattern,
The second filter is continuous in an oblique direction and arranged every other pixel in the horizontal and vertical directions;
The third filter has a filter array that is arranged in an oblique direction and arranged every other pixel in the horizontal and vertical directions;
The image processing apparatus, wherein the control unit controls reading of all pixels and thinning-out reading of a signal from the image sensor. When thinning readout from the image sensor is controlled by the control means, the single color is based on the three monochrome image data generated by the generation means based on the mosaic image signal obtained by the thinning readout. The image processing apparatus according to claim 1, further comprising generating means for generating monochrome image data reduced in the horizontal direction corresponding to the image data. When thinning readout from the image sensor is controlled by the control means, a mosaic image signal reduced in the horizontal direction corresponding to the mosaic image signal is generated based on the mosaic image signal obtained by the thinning readout. The image processing apparatus according to claim 1, further comprising: generating means for performing the processing. The image processing apparatus according to claim 1, wherein the control unit controls reading of the signal from the image sensor for each line and skipping by skipping. The image processing apparatus according to claim 1, wherein the image sensor obtains image signals with a plurality of types of sensitivity such that at least one horizontal line has the same sensitivity. 6. The image according to claim 5, wherein the control unit controls reading of the signals from the image sensor with a uniform number of rows in which the plurality of types of sensitivities are all aligned and skipping by skipping. Processing equipment. 3 types determined by the spectral sensitivity of the filter from a mosaic image signal obtained by an image sensor having three types of filters with different spectral sensitivities, and any one of the plurality of types of filters is used for each pixel. In an image processing method of an image processing apparatus for generating a color image in which intensity information for each pixel position corresponding to one color is aligned in all pixels,
A control step for controlling reading of a signal from the image sensor;
Generating three monochromatic image data in which intensity information corresponding to each of the three types of colors determined by the spectral sensitivity of the filter is aligned in all pixels from the mosaic image signal obtained by the image sensor. ,
The image sensor
The first filter is arranged in a checkered pattern,
The second filter is continuous in an oblique direction and arranged every other pixel in the horizontal and vertical directions;
The third filter has a filter array that is arranged in an oblique direction and arranged every other pixel in the horizontal and vertical directions;
In the process of the control step, all pixel readout and thinning readout of a signal from the image sensor are controlled. 3 types determined by the spectral sensitivity of the filter from a mosaic image signal obtained by an image sensor having three types of filters with different spectral sensitivities, and any one of the plurality of types of filters is used for each pixel. A program for causing a computer to execute a process of generating a color image in which intensity information for each pixel position corresponding to one color is aligned in all pixels,
A control step for controlling reading of a signal from the image sensor;
Generating three monochromatic image data in which intensity information corresponding to each of the three types of colors determined by the spectral sensitivity of the filter is aligned in all pixels from the mosaic image signal obtained by the image sensor. ,
The image sensor
The first filter is arranged in a checkered pattern,
The second filter is continuous in an oblique direction and arranged every other pixel in the horizontal and vertical directions;
The third filter has a filter array that is arranged in an oblique direction and arranged every other pixel in the horizontal and vertical directions;
In the process of the control step, all pixel readout and thinning readout of a signal from the image sensor are controlled.

Images