JPH11215512A - Image processing unit - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a image processing nit, where artifacts hardly take place at all parts of an image and missing color signals are restored to generate an image with high quality. SOLUTION: An area extract section 20 extracts plural image area from each pixel of an input image, a means variance evaluation section 24 and a parameter calculation section 28 estimate color correlation on each extracted image area to obtain a parameter. Then a parameter evaluation section 32 evaluates the reliability of each parameter estimated by the means variance evaluation section 24 and the parameter calculation section 28 and gives only the parameters with high reliability to a pixel restoring section 34. The pixel restoring section 34 restores a color signal value of each pixel of the input image, based on only the parameters relating to the pixel among received parameters.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus for outputting a color digital image in which the color of each pixel is represented by a plurality of types of color signal values, and more particularly, to a single-chip imaging system or a two-chip imaging system. Input a digital image that is missing one or more color signal values for each pixel
The present invention relates to an image processing apparatus that outputs a color digital image in which a missing color signal value in each pixel is supplemented.

[0002]

2. Description of the Related Art A single-chip image pickup system using an image pickup device in which color filters are arranged on a light receiving surface in a mosaic form, one is a G (green) filter only, and the other is an R (red) and B (blue) filter. A two-chip imaging system that performs imaging with two CCDs provided with a mosaic filter has an advantage that it can be configured at a lower cost than a three-chip imaging system using a spectral system and three imaging elements. However, in these imaging systems, since at least one or more color signal values are missing in each pixel of the obtained image, interpolation processing for estimating and supplementing the color signal values is required.

Conventionally, as this interpolation processing, simple linear interpolation processing in which a missing color signal value of a certain pixel is supplemented with an average value of the same non-missing color signal values around the pixel is conventionally performed. However, the processing results include a phenomenon that degrades the image quality, such as a false color according to the color filter arrangement, and unevenness or coloring of the edge portion. Therefore, a new method of restoring a missing color signal value that suppresses the occurrence of these phenomena and improves the image quality has been proposed. Above all, a method of compensating for missing color signal values using a relationship between color signal values in a local region of a color image, so-called color correlation, has a high image quality improving effect.

For example, as a first example, Japanese Patent Application Laid-Open No.
Japanese Patent Publication No. 02096 discloses an interpolation method for a low sampling density color signal in an image signal in which a luminance signal is obtained at a high sampling density and other color signals are obtained at a relatively low sampling density. In this method, a low sampling density chrominance signal is calculated using a high sampling density luminance signal. For this reason, it is assumed that a sample value of the high sampling density signal S1 always exists at the sampling position of the low sampling density signal S2. Signal S2
The difference between the sample values of S1 at the sampling position is compared with a predetermined threshold value. If the difference is smaller than the threshold value, a simple linear interpolation process using only the sample values of the signal S2 is performed. If the threshold value is exceeded, it is determined that there is an edge between the sampling positions of the signal S2, and it is assumed that S1 and S2 have a linear relationship of S2 = a * S1 + b, and the coefficients a and b are set to S1 at the sampling position of S2. And the value of S2.

[0005] As a second example, Japanese Patent Application Laid-Open No. 5-564 is disclosed.
No. 46, suppose that a change in color is more gradual than a change in brightness in a local area of an image, and performs two-dimensional low-pass processing on each of N types of color signals obtained by a color video camera, The missing K color signal value (K is any one of 1 to N) is calculated from the L color signal value (L is any one of 1 to N) of the pixel P by the following equation: K = (Klow / Llow)
) × L. Klow and Llow are values of the signals obtained by subjecting the K color signal and the L color signal to low-pass processing at the pixel P, respectively.

[0006]

Any of the prior arts utilizing color correlation makes a certain assumption about the nature of the color correlation of a general image and performs processing on the assumption that the assumption holds near each pixel. It is.

However, in these prior arts, there are pixels in the image having a neighborhood where the assumption provided does not hold, and artifacts (image characteristics that do not originally exist) are generated in the processing results of such a part. No mention is made of degradation of image quality.

[0008] Further, in the prior art, for a pixel for which an assumption regarding color correlation is established in the vicinity, only one neighborhood is taken for each pixel, and the missing color signal value is restored using the color correlation in the vicinity. Was. Therefore, there is no description that the restoration result is weak against noise.

The present invention has been made in view of the above points, and it is determined whether or not an assumption made regarding color correlation is established in a certain area, and only information of an area determined to be satisfied is used by using information of the area determined to be established. The object of the present invention is to provide an image processing apparatus capable of restoring a missing color signal value by performing restoration of a missing color signal value so that artifacts hardly occur in all parts of the image and generating a high-quality image. .

The present invention also provides an image processing apparatus capable of restoring a missing color signal value that is resistant to noise by performing processing on a certain pixel by integrating a plurality of neighboring color correlations related to the pixel. The purpose is to:

[0011]

In order to achieve the above object, an image processing apparatus according to the present invention comprises at least one type of color signal value for each pixel obtained by a single-chip imaging system or a two-chip imaging system. An image processing apparatus for inputting a digital image having missing color, estimating a missing color signal value of each pixel, and outputting a color digital image, comprising: A region extracting unit for extracting a plurality of different image regions, and a color correlation parameter representing a color correlation on the image region for each image region extracted by the region extracting unit is present on the image region. A color correlation parameter estimating means for estimating based on the color signal value; and a reliability of each color correlation parameter estimated by the color correlation parameter estimating means. Parameter selection means for evaluating based on the color signal value on the image area corresponding to the color correlation parameter and selecting only the color correlation parameter with high reliability, and for each pixel of the digital image lacking the color signal value On the other hand, the missing color signal value of the pixel is reconstructed based on the color signal value present in the pixel and the color correlation parameter selected by the color correlation parameter selection means, the corresponding image area including the pixel. And a restoring means for performing the restoration.

That is, according to the image processing apparatus of the present invention, the area extracting means extracts a plurality of image areas for each pixel of the input image, and performs color correlation on the extracted image areas with respect to each of the extracted image areas. The color correlation parameter estimating means estimates and parameterizes. Then, the parameter selection unit evaluates the reliability of each parameter estimated by the color correlation parameter estimation unit, and sends only the highly reliable parameter to the restoration unit. The restoring unit restores the color signal value of each pixel of the input image based on the parameter related to that pixel among the parameters sent.

Therefore, a plurality of image areas are taken for each pixel of the input image, and only highly reliable color correlation parameters are selected from the image areas. Can be suppressed. In addition, since the restoration of the missing color signal value of each pixel is performed from a plurality of color correlation parameters related to the pixel, it is possible to achieve a strong restoration against noise.

Further, the image processing apparatus according to the present invention inputs a digital image obtained by a single-chip imaging system or a two-chip imaging system, in which one or more types of color signal values are missing for each pixel. An image processing apparatus for estimating a missing color signal value and outputting a color digital image, wherein a plurality of different image areas are extracted for each pixel from the digital image having the missing color signal value. , For each image region extracted by the region extraction means,
A color correlation parameter estimating means for estimating a color correlation parameter representing a color correlation on the image area based on a color signal value present on the image area; and a missing color signal value on each image area in the image area. Area restoring means for restoring from a corresponding color correlation parameter and a color signal value present in each pixel to generate a restored image area; and for each restored image area generated by the area restoring means, the restored image area A restoration area selection unit that evaluates the reliability of the restoration result based on the above color signal values and selects a restoration image area evaluated as having high reliability, and a digital image in which the color signal values are missing. Pixel restoring means for restoring a missing color signal value of a pixel on the basis of a pixel including the pixel in the restored image area selected by the restoring area selecting means. .

That is, according to the image processing apparatus of the present invention, the region extracting means extracts a plurality of image regions for each pixel of the input image, and performs color correlation on the extracted image regions with respect to each of the extracted image regions. The color correlation parameter estimating means estimates and parameterizes. Then, the region restoring unit restores the missing color signal value at each pixel on the corresponding image region for each of the estimated parameters, and generates a color restored image region. This processing is performed based on each color correlation parameter and a color signal value already existing on the corresponding image area. When the restored image area is thus generated for each color correlation parameter, the restored area selection means evaluates the reliability of the restored result based on the color signal values on each restored image area, and the restored image that has been evaluated to be highly reliable Select only areas. The pixel restoring unit restores a color signal value of each pixel of the input image based on a pixel including the pixel in the restored image area selected by the restoration area selecting unit.

Therefore, a plurality of image areas are taken for each pixel of the input image, and the missing color signal value on each image area is restored based on the color correlation parameter on the image area, and the reliability is high. Since only the determined restoration result is selected and used for restoring the missing color signal value of each pixel, it is possible to suppress artifacts caused by using the unreliable restoration result. In addition, since the restoration of the missing color signal value of each pixel is performed based on a plurality of restoration results related to the pixel, it is possible to perform a strong restoration against noise. Furthermore, since the restoration result of each image area is directly used to determine the reliability of restoration, the reliability evaluation can be performed more reliably and efficiently.

[0017]

Embodiments of the present invention will be described below with reference to the drawings.

[First Embodiment] FIG. 1 is a diagram showing the configuration of an electronic camera system to which an image processing apparatus according to a first embodiment of the present invention is applied.

This electronic camera system includes an electronic camera 1
0 and a processing apparatus main body 12 connected to the electronic camera 10 via a cable.

In the electronic camera 10, a single-plate primary color CCD 1 is provided.
4 and an A / D converter 16. The output of the A / D converter 16 is connected to an input image buffer 18 provided in the main body 12.

In the main body 12, the input image buffer 18 is provided.
In addition, an area extraction unit 20, an area buffer 22, an average / variance evaluation unit 24, a filter arrangement holding ROM 26, a parameter calculation unit 28, a parameter buffer 30, a parameter evaluation unit 32, a pixel restoration unit 34, and an output image buffer 36 are arranged. Have been. Here, the output of the input image buffer 18 is input to the area extraction unit 20, and the output of the area extraction unit 20 is input to the area buffer 22. The output of the area buffer 22 is input to the average / variance evaluation unit 24 and the pixel restoration unit 34. The filter arrangement holding ROM 26 has a single-plate primary color C.
The arrangement information of the mosaic colors on the CD 14 is stored,
The output of the filter arrangement holding ROM 26 is equal to the average
The data is input to the variance evaluator 24 and the parameter calculator 28. The output of the average / variance evaluation unit 24 is
The data is input to the parameter calculator 28. The output of the parameter calculation unit 28 is input to a parameter buffer 30, and the output of the parameter buffer 30 is input to a parameter evaluation unit 32. The output of the parameter evaluation unit 32 is output to the pixel restoration unit 34. The output of the pixel restoring unit 34 is input to the output image buffer 36, and the output of the output image buffer 36 is led out of the processing device main body 12.

A compression processing section for performing image compression processing such as JPEG and a storage medium such as a memory card are arranged outside the processing apparatus body 12, and a color digital image output from the output image buffer 36 is provided. It is normal to compress and save.

In this electronic camera system, the electronic camera 10 and the processing device main body 12 are configured separately, but as shown by a broken line in FIG. It goes without saying that the processing section and the storage medium may be integrated into one housing to constitute an electronic camera device.

Next, the operation in the configuration as shown in FIG. 1 will be described.

When an image is taken by the electronic camera 10,
The signal from the single-plate primary color CCD 14 is supplied to the A / D converter 16.
, And transferred to the input image buffer 18 of the main body 12 as a digital single-plate image. Original RGB obtained when the same scene is imaged by the three-panel imaging system
While the image has three color signal values of R, G, and B for each pixel, this digital single-chip image has one color signal value for each pixel in accordance with the color filter arrangement of the single-plate primary color CCD 14. And the other two types of color signal values are missing.
Therefore, in the main body 12, a process of estimating a missing color signal value at each pixel of the digital single-chip image and restoring an original RGB image obtained when the same scene is imaged by the three-chip imaging system is performed. This restoration processing is achieved by sequentially performing the following processing on each pixel of the digital single-chip image in the input image buffer 18.

In the following description, the pixel currently subjected to the restoration processing is referred to as a target pixel. Further, in a certain area of the image, the S color signal value (S is R, G,
B) is called an S-color signal in that area.

First, the region extracting unit 20 sequentially reads out the images in all the rectangular regions having the size of 5 × 5 pixels including the target pixel from the input image buffer 18 and reads out the read images. Is written to the area buffer 22. When writing of images in all rectangular areas is completed, the average
The variance evaluator 24 reads the images in each rectangular area in parallel from the area buffer 22 and performs the following processing on each image in parallel.

That is, the average / variance evaluator 24 obtains the R color signal value in the rectangular area from the read position of the rectangular area and the arrangement information of the color filters stored in the filter arrangement holding ROM 26. It specifies a pixel position, calculates an average value a _r and variance V _r of the color signal values at those pixel positions. Similarly, for the G and B color signals, the average values A _g and A _b
And the variances V _g and V _b are calculated. Here, r, g, and b are suffixes indicating the types of color signals. And the variance V _c (c
Is any of r, g, and b), and predicts the reliability of the parameter calculated by the parameter calculation unit 28 using the following equation (1). Only when V _c satisfies Equation (1), A _c and V _c (c is any of r, g, and b) are transferred to the parameter calculation unit 28. The relationship between equation (1) and the reliability of the parameters will be described later.

[0029]

(Equation 1)

Here, max (p, q) indicates the maximum value of p and q, and min (p, q) indicates the minimum value of p and q. Th
1, Th2 is an appropriate threshold.

The parameter calculator 28 calculates the calculated values A _c ,
When _Vc is transferred, first, the type S (S is any of R, G, B) of the color signal value present in the target pixel is determined by referring to the color filter layout information stored in the filter layout holding ROM 26. It is specified from the position of the rectangular area. Then, based on the transferred average value and variance, a color signal S and another color signal T (T is not R among R, G, and B) in the original RGB image in the rectangular area where they are calculated. Estimate the relationship that should hold. This relationship is hereinafter referred to as a color correlation between the color signals S and T. The color correlation between the estimated color signals S and T is expressed by the following equation (2) including the parameters α _st and β _st .

T = α _st × S + β _st (2) (s and t are suffixes corresponding to S and T and are any of r, g and b) Here, parameters α _st and β _st are color correlation parameters and It is called and calculated from the average value A _c and the variance V _c .

Here, the color correlation in the original RGB image and the meaning of equation (2) will be described. Generally, when an object having a diffuse reflection characteristic is imaged under a natural illumination environment in which there is one dominant illumination light, a C color signal value at a pixel P corresponding to the object surface in the original RGB image C
_p (C is any one of R, G, and B) is L (λ) representing the spectral characteristic of the dominant illumination light, and Ref is the spectral reflectance of the object surface.
(Λ), the spectral sensitivity characteristic of the image sensor is expressed as F _c (λ) (c is C
Any of r, g, and b)

(Equation 2)

## EQU1 ## In this equation (3), n (P) is an amount determined by the direction of the illumination light and the direction of the normal to the object surface corresponding to pixel P, and A (λ) is the multiple reflection of the illumination light and multiple objects. It is the spectral characteristics of the resulting ambient light. The integration is performed in the wavelength range where the spectral sensitivity of the image sensor is positive. From this equation (3), the color signals C and C ′ in the vicinity of the pixel P are obtained.
It can be seen that a linear relational expression of C = α × C ′ + β (4) is established between (C, C ′ is any of R, G, B) using α and β as parameters.

When the object has both specular reflection characteristics and diffuse reflection characteristics, it is known that the color signals C and C 'in the region corresponding to the surface of the object over a wide range have the relationship shown in FIG. Have been. FIG. 2 shows the original RG in that region.
For each pixel of the B image, the C color signal value is plotted on the horizontal axis, and the C ′ color signal value is plotted on the vertical axis, showing the relationship between the color signals in the area. Hereinafter, such a graph is called a color correlation diagram. Here, the relationship between the color signals C and C ′ in a very small neighborhood of the pixel P can be considered to be obtained by extracting a very small part of this graph, but this can be sufficiently approximated by Expression (4). Even under more complicated lighting conditions and reflection characteristics of an object, in a very small image area corresponding to the surface of an object made of a single material, the relationship between the color signals C and C ′ can be approximated by the straight line of Expression (4). This is the reason for using equation (2) for color correlation.

Next, a method of estimating the color correlation parameters α _st and β _st from the calculated values A _c and V _c will be described with reference to FIG.

In the color correlation diagram shown on the right side in FIG. 3, light shaded portions show the color signals S and T of the color signals 38 in the original RGB image, and the color correlation between the original two color signals is shown. Is shown. However, since the color signal 40 of the single-plate image in the rectangular area has one color signal value per pixel, FIG.
Are obtained by projecting the thin hatched portions on the respective axes. Therefore, in the present embodiment, the estimated approximation line 42 approximating the original color correlation is used.
Is estimated from the hatched portion in FIG. 3 as follows.

First, the inclination of the estimation approximation line 42 is estimated as the ratio of the variation of each color signal of the single-plate image. Next, the passing point is determined by the average value of each color signal of the single-plate image. When the standard deviation (square root of the variance) of each color signal is used as the variation, the parameters α _st and β _st of the estimation approximation line 42 are determined by the following equation (5).

[0039]

(Equation 3)

[0040] It should be noted _{here, A s, V s, A} t, V t is,
The average and variance of the color signals S and T in a rectangular area. The parameter calculation unit 28 calculates four color correlation parameters α _st , β _st , α _st ′, β _st ′ (t and t ′ are two types of color signals other than s) based on the above equation (5), It is transferred to the parameter buffer 30 as a parameter set.

When the parameter calculation section 28 completes the processing for all the transferred A _c and V _c , the parameter evaluation section 32 reads out the parameter sets transferred to the parameter buffer 30 one by one and determines the reliability of the parameter sets. To evaluate.

Here, the importance of parameter reliability and its evaluation method will be described. The color correlation parameter calculated based on the above equation (5) is the color signal S in the rectangular area.
And T are obtained by assuming that equation (2) holds for the color correlation. However, at the boundary between the two object surfaces having different hues and the surface brightness is changed by the shape of the surface or the like, the color signal 38 of the three-plate image in the color correlation diagram shown on the right side in FIG. As shown in (1), the color correlation is composed of a plurality of straight lines, and cannot be approximated well by a single straight line. Hereinafter, this portion will be referred to as a color edge region 44, whereas a portion in which the color correlation can be approximated by a single straight line as shown in FIG. 3 will be referred to as a material uniform region 46.

In the color edge region 44, when a missing color signal is estimated based on the calculated color correlation parameter, a large estimation error occurs. Therefore, it is necessary to avoid using the color correlation parameter. However, since the graph shown in FIG. 4 cannot be obtained from the color signal 40 of the single-plate image, it is not directly known whether the approximation is good or bad, and it is necessary to estimate the reliability. In the present embodiment, the average / variance evaluator 24 first detects a situation where the reliability may be lowered based on the variance _Vc , and the parameter evaluator 32 estimates the approximation accuracy based on the value of the color correlation parameter itself. Is performed in two stages.

In general, the variance of each color signal tends to be larger in the color edge region 44 than in the material uniform region 46. However, in the achromatic region, a linear relationship between the color signals is established even if the variance of each color signal is large. The average / variance evaluator 24 detects and avoids this situation by using the equation (1), thereby preventing the parameter calculator 28 from calculating a low-reliability color correlation parameter. Further, as for the value of the color correlation parameter itself, the probability that the value of the parameter β has a large absolute value in the color edge region 44 is high. On the other hand, in the material uniform region 46, the parameter β takes a value close to “0”. In the material uniform region 46 as well, the parameter β can take a large value in the case where the spectral characteristics of the ambient light and the direct light are significantly different from each other or the periphery of the highlight as suggested by the equation (3). Relatively few in.

Therefore, the parameter evaluation unit 32 compares the absolute value | β _st | of β _st in each parameter set with a predetermined threshold Th3, and only the parameter sets having | β _st | equal to or smaller than the threshold have high reliability. Pixel restoration unit 3
Transfer to 4. As a result, as shown in FIG. 5, the parameter set 50 of the color edge area 44 is excluded from the parameters approximating the color correlation in the rectangular area including the target pixel 48, and the parameter set 52 of the material uniform area 46 is selected. Is transferred to the pixel restoration unit 34.

The pixel reconstruction unit 34 reads a color signal value S _p which is present in the pixel of interest from the region buffer 22, each parameter set alpha _st transferred, missing color signal values at the pixel of interest with respect to beta _st T _p ( T is R, G, B but not S)
Is estimated by the following equation (6).

T _p = α _st × S _p + β _st
(6) This is shown in FIG. 5 as a process of obtaining the missing color signal 56 of the target pixel from the color signal 54 of the target pixel. Then, the average value of T _p obtained by each parameter set is calculated and used as the value of the missing color signal of the pixel of interest to generate a three-color signal value of the pixel of interest. If no parameter set is sent, the pixel restoring unit 34 selects and reads a rectangular area centered on the pixel of interest from the area buffer 22 and calculates the average value of the color signal T inside the area. Let it be a signal value. Finally, the three-color signal values of the target pixel are written to the corresponding positions in the output image buffer 36, and the processing for the target pixel ends.

As a result of the above-described processing being performed on each pixel, an original RGB image with missing color signals is obtained in the output image buffer 36.

In the present embodiment, the region extracting unit 20 selects a plurality of rectangular regions for each pixel, and the parameter calculating unit 28 calculates a color correlation parameter for each rectangular region, so that A plurality of estimated values of the color correlation parameter in the vicinity are obtained. Then, only those having high reliability are selected from the estimated values by the average / variance evaluator 24 and the parameter evaluator 32, and the missing color of the pixel of interest is determined by the pixel restoring unit 34 based on the selected color correlation parameters. The signal value is supplemented. Therefore, it is possible to reduce or eliminate artifacts occurring near the color edge region 44 in the conventional missing color restoration method based on color correlation, and to obtain an original RGB image with higher image quality in the output image buffer 36 and to obtain a uniform material. In the area 46 as well, by integrating the estimated values of a plurality of color correlation parameters, it is possible to improve the image quality that is resistant to noise.

Further, since the color correlation is represented by the equation (2) including two parameters, the missing color of a scene having various illumination conditions and object surface characteristics can be more accurately restored.

Also in the estimation of the color correlation parameter, since the color correlation parameter is calculated from the average value and the variance of the local area, the color correlation parameter is flexible enough to cope with a single-chip imaging system having various color filter arrangements.

Various modifications and changes can be made to this embodiment. The area extracted by the area extracting unit 20 is not limited to a rectangular area having a size of 5 × 5 pixels as long as it is small to some extent, and may be an area having an arbitrary shape and size optimal for mounting. In particular, a method of determining a region to be extracted based on a feature amount calculated from a single-plate image as described in a second embodiment described below is effective. In this case, the region extraction unit 20 calculates a feature amount such as luminance for the entire image before extracting each pixel. Then, in the extraction for each pixel, a plurality of neighborhoods are taken for each pixel, each neighborhood is divided into regions based on the feature amount, and each divided region is extracted.

The arrangement of the color filters of the single-plate primary color CCD 14 may be arbitrary, and the type is not limited to the primary color system. The average / variance evaluator 24 calculates the maximum value and the minimum value of each color signal in the rectangular area in addition to the average / variance, and transfers them to the parameter calculator 28. The parameter calculator 28 calculates the transferred maximum value and minimum value. The color correlation parameter may be calculated based on the color correlation parameter. In this case, the parameters α _st and β _st are the maximum value MAX_c and the minimum value MIN of each color signal transferred instead of the equation (5).
_C (c is one of r, g, and b) is calculated based on the following equation. This method has a high processing speed and is particularly effective for document images.

[0054]

(Equation 4)

It is also possible to use another expression for the relational expression (2) representing the color correlation. For example, in equation (2), β _st = 0
Using the equation described above, only α _st may be calculated according to equation (2). In this case, the parameter evaluation unit 32 becomes unnecessary, and the processing speed can be improved by estimating α _st as the ratio of the average value or the intermediate value of the color signals on the rectangular area. If the object to be imaged is limited and the basic equation (3) for parameterizing the color correlation can be described in more detail, a more advanced color correlation model may be adopted. Various modifications are also conceivable for the final missing color signal value calculation method in the pixel restoration unit 34. It is also effective not to simply average the estimated value T _p of the missing color signal value for each parameter set, but to weight it with the reliability of each parameter. If there is enough processing time, each parameter set is regarded as a four-dimensional vector, clustering is performed, and the final missing color signal value is calculated using the representative value of the largest cluster as a parameter, thereby estimating the reliability of the parameter. There are ways to mitigate the effects of failure.

Various methods other than equation (1) can be considered for the method of predicting the reliability of the color correlation parameter in the average / variance evaluator 24. For example, taking the difference between the variances of the two color signals,
A method of predicting that the reliability is low when the difference exceeds a certain threshold value is more preferable than Expression (1) in that the calculation can be performed at high speed.

Regarding the method of region extraction, a plurality of regions are not extracted each time each pixel is processed, but color correlation parameters relating to a rectangular region of a predetermined size included in a single-plate image are first calculated and stored in the parameter buffer 30. In any case, the speed of processing can be increased by using only a highly reliable color correlation parameter among the color correlation parameters of the rectangular area including the pixel to restore the missing color of each pixel.

[Second Embodiment] FIG. 6 is a diagram showing a configuration of an electronic camera system to which an image processing apparatus according to a second embodiment of the present invention is applied. In the figure,
The same parts as those in the first embodiment are denoted by the same reference numerals as those in FIG.

This electronic camera system includes an electronic camera 1
0 and a processing apparatus main body 12 connected to the electronic camera 10 via a cable. In the present embodiment, the input image buffer 18, the reference image calculation unit 58, the reference image buffer 60, the block extraction unit 6
2. Block division unit 64, block buffer 66, average / variance evaluation unit 24a, filter arrangement holding ROM 26, parameter calculation unit 28a, block restoration unit 68, artifact evaluation unit 70, linear interpolation unit 72, integrated average unit 7
4. An output image buffer 36a is provided.

Here, the output of the input image buffer 18 is
It is input to the reference image calculation unit 58, the block extraction unit 62, and the linear interpolation unit 72. The output of the reference image calculation unit 58 is input to the reference image buffer 60, and the reference image buffer 6
The output of 0 is output to the block division unit 64. The output of the block extracting unit 62 is also input to the block dividing unit 64. The output of the block division unit 64 is input to the block buffer 66, and the output of the block buffer 66 is input to the average / variance evaluation unit 24a and the block restoration unit 68. The average / variance evaluator 24a has a filter arrangement holding R
The output of the OM 26 is also input, and the average / variance evaluator 24a
Is input to the parameter calculator 28a. The output of the parameter calculation unit 28a is input to the block restoration unit 68, and the output of the filter arrangement holding ROM 26 is further input to the block restoration unit 68.

The output of the block restoration section 68 is input to an artifact evaluation section 70, and the output of the artifact evaluation section 70 is input to a cumulative average section 74. The outputs of the linear interpolation unit 72 and the output image buffer 36a are further input to the integration averaging unit 74, and the output of the integration averaging unit 74 is input to the linear interpolation unit 72 and the output image buffer 36a. The output image buffer 36a holds a three-color image having the same number of pixels as the image in the input image buffer 18,
A counter is held for each pixel. This counter is provided to hold the number of times writing has been performed at each pixel position in the buffer. In the present embodiment, during processing, estimated values of a plurality of color signals corresponding to each pixel position of the input image buffer 18 are calculated, and they are output to the output image buffer 36.
a, and the integrated value is finally averaged by the integrated averaging unit 74 to obtain a final estimated value of the color signal. This process requires the number of times of writing.

The output of the output image buffer 36a is
It is led out of the processing device main body 12 and, like the first embodiment described above, is provided outside the processing device main body 12.
For example, a compression processing unit for performing image compression processing such as JPEG and a storage medium such as a memory card are provided, and the color digital image output from the output image buffer 36a is compressed and stored. As in the first embodiment, as shown by the broken line in FIG. 6, the electronic camera 10 and the processing device main body 12, and the compression processing unit (not shown) and the storage medium are integrated into one housing. Of course, it may be configured as an electronic camera device.

Next, the operation in the configuration shown in FIG. 6 will be described.

When an image is taken by the electronic camera 10,
The signal from the single-plate primary color CCD 14 is supplied to the A / D converter 1
The digitalized image is transferred to an input image buffer 18 of the main body 12 as a digital single-plate image. Original RGB obtained when the same scene is imaged by the three-panel imaging system
While the image has three color signal values of R, G, and B for each pixel, this digital single-chip image has only one type of color signal value for each pixel in accordance with the color filter arrangement of the single-plate primary color CCD 14. No, the other two types of color signal values are missing. Therefore, in the main body 12, a process of estimating a missing color signal value at each pixel of the digital single-chip image and restoring an original RGB image obtained when the same scene is imaged by the three-chip imaging system is performed. This restoration processing is achieved by performing the following processing on the digital single-plate image in the input image buffer 18.

When the transfer of the image from the electronic camera 10 to the main body 12 is completed, first, the reference image calculation unit 58 converts the digital single-chip image in the input image buffer 18 into a 3 × 3 image having the following equation (8). A low-pass filter is applied to the pixels, and the result is written to the reference image buffer 60 as a reference image.

0.0625 0.125 0.0625 0.125 0.25 0.125 (8) 0.0625 0.125 0.0625 The image directly calculated from the veneer image by the low-pass filter is a luminance component of the original RGB image corresponding to the veneer image, 0.25R + 0.5G + 0. .25B is reflected.

Next, the pixel value and counter of each pixel in the output image buffer 36a are initialized to “0”, and all 5 × 5 pixel size rectangles included in the digital single-chip image in the input image buffer 18 are set. The following processing is performed on the area. Hereinafter, an image formed by extracting an S color signal value (S is one of R, G, and B) from each pixel in a certain region of the image will be referred to as an S color signal in that region.

The position of the rectangular area is shifted one pixel at a time from the upper left corner of the digital single-chip image in the input image buffer 18 to the right and then downward by the block extracting section 62, and the images in the rectangular area are sequentially shifted. Is read. The image that has been read by the block extraction unit 62 and has begun processing is called the block of interest. The following processing is slightly different depending on whether the target block belongs to the color edge area 44 or the material uniform area 46. FIG. 7 shows the processing for the color edge area 44 and FIG. 8 shows the processing for the material uniform area 46. .

Hereinafter, the processing will be mainly described with reference to FIG. The block of interest 76 is first transferred to the block division unit 64. The block dividing unit 64 includes the reference image buffer 6
The partial reference image 78 corresponding to the rectangular area of the block of interest is read from the reference image in 0, and the maximum value RefMAX and the minimum value RefMIN in the partial reference image 78 are calculated.
Then, from the partial reference image 78, (RefMAX + Ref
A binary image 80 having a pixel value of “1” or “0” is generated using MIN) / 2 as a threshold value. Next, attention block 76
Is divided into two, and a block A 82 corresponding to a region having a pixel value “1” on the binarized image 80 and a block B 84 corresponding to a region having a pixel value “0” on the binarized image 80
Generate And block A 82 and block B
Write 84 to the block buffer 66.

As described above, the partial reference image 78 which is a part of the reference image corresponds to the luminance component of the original RGB image. Therefore, the original RG is included in the binarized image 80 of the partial reference image 78.
The edge structure of the B image is reflected, and blocks A 82 and B 84 correspond to the left and right regions of the edge. First
In the color edge area 44 shown in FIG. 4 used in the description of this embodiment, the left and right edges are the material uniform areas 46.
Corresponds to the straight line portion included in the color signal 38 of the original RGB image. Therefore, even if the target block belongs to the color edge area 44 and the color correlation parameter in the block cannot be estimated correctly, the block A 82 and the block B 8
In 4, the color correlation can be approximated by a straight line, and there is a high probability that the color correlation parameter can be obtained correctly.

When the two images are written in the block buffer 66, the average / variance evaluator 24a reads out the color filter arrangement information stored in the filter arrangement holding ROM 26, and outputs the color signals in the blocks A 82 and B 84, respectively. The positions where R, G, and B are obtained are specified. The number N _c (A), N _{c of} color signals C obtained inside each of the blocks A 82 and B 84
(B), mean A _c (A), A _c (B), and variance V _c
(A), V _c (B) (C is any of R, G, B. c is a subscript representing the type of the color signal C by any of r, g, b). Further, from these values, block A
82, the average value A _c (M) and the variance V _{c of the} entire block of interest 76 obtained by merging the block B 84 (Merge).
(M) is calculated by the following equation (9).

[0072]

(Equation 5)

Then, the average value and the variance are transferred to the parameter calculating section 28a. Hereinafter, the character in parentheses is M
Indicates that the quantity is related to the entire block of interest in which blocks A 82 and B 84 are combined, and if the character represents A or B, it indicates that the quantity is related to the corresponding block. .

The parameter calculating section 28a calculates the target block 76 and the block A 8 from the transferred average value and variance.
2 and the color correlation parameters in block B 84 are calculated. Each block A 82, B 84 and block 7 of interest
In the whole, as in Expression (2) in the first embodiment, the following Expression (10) is assumed between the color signals S and T.

T = α _st (X) × S + β _st (X) (10) Then, similarly to the equation (5), the color correlation parameter is calculated by the following equation (11). Note that this equation (10), (1
In 1), S and T are different from each other in any of R, G and B. S and t are subscripts corresponding to S and T, respectively, and X is any of A, B and M.

[0076]

(Equation 6)

Then, the parameter calculation unit 28a compares the color correlation parameters of the block A 82 and the block B 84. If the difference between all the color signal combinations does not exceed a certain threshold, the color correlation parameter α
_st (M) and β _st (M) are transferred to the block restoration unit 68. Otherwise, the color correlation parameter α _st (X is either A or B) for the block X (X is either A or B) with the larger total number of color signals N _r (X) + N _g (X) + N _b (X) in the block X) and β _st (X) are transferred to the block restoration unit 68.

Here, the color correlation parameters transferred from the parameter calculator 28a will be described with reference to FIGS. The attention block is the attention block 76 in FIG.
In the case of belonging to the color edge region 44 as described above, by performing block division based on the partial reference image 78 which is a luminance component, the reliability of the color correlation parameter of the divided block is improved. However, the attention block is the attention block 8 in FIG.
In the case of belonging to the material uniform region 46 as in 6, the color correlation parameter for the entire block of interest can be calculated with high reliability. Conversely, when the block is divided by the partial reference image 88, the block A 90, the block B
At 92, the number of pixels is small, so the reliability of the color correlation parameter is rather reduced. Therefore, the parameter calculation unit 28
a compares the color correlation parameters of the divided blocks,
If they are similar, the color correlation parameter for the entire block of interest is calculated again.

The block restoration section 68 performs the following processing according to the type of the transferred color correlation parameters α _st and β _st . The transferred color correlation parameters are α _st (M), β
_In the case of _st (M), two blocks are read out from the block buffer 66 together. On the other hand, if the color correlation parameter relates to any block, the block buffer 6
The corresponding block is read from 6. Then, for each pixel of the read image, the type S (S is any of R, G, B) of the color signal value existing in the pixel is specified from the color filter arrangement information stored in the filter arrangement holding ROM 26. The color signal value T _p (T is a type different from S) missing from the color signal value _Sp and the transferred color correlation parameter is obtained from the equation (6) as in the first embodiment.
Then, by compensating for the lack of all the color signals in the block related to the transferred color correlation parameter, a color restoration block in which each pixel has three color signal values is generated. If the transferred color correlation parameters are α _st (M) and β _st (M), the restoration block 94 in which the lack of the color signal of the entire block of interest 86 has been restored, otherwise, the block A 82 and the block B 84 Are generated. The restoration block is transferred to the artifact evaluation unit 70, and it is determined whether the restoration result includes an artifact.

The artifact that occurs in the color edge area 44 is that the color signal value at a pixel position where a specific type of color filter exists is significantly different from the surrounding color signal values. The artifact evaluation unit 70 detects and evaluates this artifact included in the restored block as follows.

The detection is performed for each color signal S (S is any of R, G, B). First, each pixel P of the restored block
For, by comparing the color signal value S _p of the color signal value and the pixel P of the 3 × 3 pixel neighborhood, the color signal value S _p is checked whether the maximum or minimum in the neighborhood. Then, if that is the maximum or minimum, average A _p s and standard deviation sigma _p s of the color signals of 3 × 3 pixel neighborhood (s subscripts corresponding to S) was calculated, and the pixel P The value is given by the following equation (1) for a predetermined constant k.
Check whether 2) is satisfied.

[0082] S _p <A _p s -k×σ _p s MatawaS _{p> A p s + k ×} σ p s ... (12) Equation (12) pixels which are satisfied, it is determined that artifact generation pixel. After checking all the color signals in this way, if the total number of detected artifact-occurring pixels is less than a predetermined threshold, it is determined that the restored block does not include the artifact, and the restored block is integrated with the integrated averaging unit 74. Transfer to Otherwise, it is assumed that the restored block contains an artifact and no transfer is performed.

The averaging unit 74 performs the following processing only when the restored block is transferred from the artifact evaluation unit 70.

First, a restored block is read from the block restoring section 68. Then, in each pixel P of the restoration block, the color signal value of the pixel P of the restoration block 96 is added to the integrated value at the position corresponding to the pixel P in the output image buffer 36a. Then, the counter corresponding to the pixel P in the output image buffer 36a is increased by "1".

After the above processing has been performed on all the rectangular areas of 5 × 5 pixels included in the digital single-chip image in the input image buffer 18, the integrating and averaging unit 74 sets each of the rectangular areas in the output image buffer 36 a. The following process is performed by reading the integrated value of the pixel P and the counter.

First, when the counter is “0”, the position of the pixel P is notified to the linear interpolation unit 72. The linear interpolation unit 72 converts the 3 × 3 pixel area around the pixel P into the input image buffer 1
8, the color signal existing in the pixel P is used as it is, and the missing color signal value is supplemented by the average value of the similar color signal values in the vicinity to generate three color signal values of the pixel. Then, the three color signal values are transferred to the integration and averaging unit 74. The integrated averaging unit 74 outputs the transferred restoration value to the output image buffer 3
Write to 6a. If the counter is not "0", the integrated averaging unit 74 divides the integrated value by the counter to obtain an average value of a plurality of restored values obtained during processing, and writes this value to the output image buffer 36a. When the processing up to this point is completed, the output image buffer 36a stores the original RGB data in which the missing color signals have been compensated.
An image is obtained.

In this embodiment, the input image buffer 18
The block extracting unit 62 reads out the digital veneer image while shifting the rectangular region, and the block dividing unit 64 divides the block into the material uniform region 46 based on the reference image calculated from the veneer image by the reference image calculating unit 58. As a result, the parameter calculation unit 28a
Can calculate a highly reliable color correlation parameter. Then, the block restoring unit 68 restores all the missing color signals of the block based on the color correlation parameter to generate a restored block, and the integrated averaging unit 74 performs the restoration based on the evaluation of the artifact evaluating unit 70 without the artifact. By integrating only the output image buffer 36a with the output image buffer 36a, a missing restoration value is obtained for each pixel of the output image buffer 36a using the color correlation parameter relating to a plurality of material uniform regions 46 including the pixel. The finally calculated missing restoration values are integrated, averaged, and output to the output image buffer 36a, so that the artifacts that occur near the color edge in the conventional missing color restoration method based on color correlation can be reduced. Original RGB images that can be reduced and removed and have higher image quality can be obtained in the output image buffer 36a. Further, in addition to the same image quality improving effect as in the first embodiment, the shape of the region for calculating the color correlation parameter becomes variable by performing region division.
There is an effect that the color correlation parameter is more correctly calculated and the image quality of a complicated scene can be easily improved. Further, there is an effect that the artifact can be more reliably removed by directly evaluating the artifact of the restored block.

Further, in this embodiment, various modifications and changes are possible as in the first embodiment. The size of the block extracted by the block extracting unit 62, the primary color C
The color filter array of the CD 14 and the calculation method in the parameter calculation unit 28a can all be modified as described in the first embodiment. Further, as in the first embodiment, the reliability is predicted before calculating the color correlation parameter in the average / variance evaluator 24a, and if the reliability is low, the subsequent processing is not performed. Can also.

The reference image calculated by the reference image calculation unit 58 may be any image whose difference in pixel value reflects the difference in color correlation parameters in the original RGB image. May be used as a reference image by calculating a color difference component after linear interpolation. The artifact evaluation method in the artifact evaluation unit 70 is also arbitrary as long as it detects a pixel having a color signal value that is particularly different from surrounding pixels. For example, an edge detection filter may be applied to the restoration block, and the determination may be made from the pattern of the output result.

The area dividing method in the block dividing unit is not limited to binarization, and various labeling methods can be applied. If the area division is limited so that the color correlation parameter can be estimated with high reliability on the divided area, it is more effective in suppressing false colors.

[Third Embodiment] FIG. 9 is a diagram showing a configuration of an electronic camera system to which an image processing apparatus according to a third embodiment of the present invention is applied. In the figure,
Parts similar to those in the first and second embodiments are given the same reference numerals as in FIGS.

This electronic camera system includes an electronic camera 1
0 and a processing apparatus main body 12 connected to the electronic camera 10 via a cable. In the present embodiment, the electronic camera 10 has a two-chip imaging system, in its interior, decor GCCD 14 ₁ and the R filter and B filter arranged G filter in a checkered pattern for each pixel in all the pixels R /
The output of the B checkered CCD 14 ₂ via the A / D converter 16, and is configured to be input to the input image buffer 18 of the cable connected processing apparatus main body 12.
On the other hand, the block extraction unit 62,
A block buffer 66, an average / variance evaluator 24b, a parameter calculator 28b, a parameter evaluator 32b, a block reconstructor 68b, an averaging unit 74, a linear interpolator 72, and an output image buffer 36a are provided.

Here, the output of the input image buffer 18 is
It is input to the block extraction unit 62 and the linear interpolation unit 72. The output of the block extraction unit 62 is input to the block buffer 66, and the output of the block buffer 66 is input to the average / variance evaluation unit 24b, the parameter evaluation unit 32b, and the block restoration unit 68b. The output of the average / variance evaluator 24b is input to the parameter calculator 28b, and the output of the parameter calculator 28b is input to the parameter evaluator 32b. The output of the parameter evaluation unit 32b is output from the block restoration unit 6
8b, and the output of the block restoring unit 68b is input to the integrated averaging unit 74.

The integrated averaging unit 74 further includes a linear interpolation unit 7
2 and the output of the output image buffer 36a are input, and the output of the integrated averaging unit 74 is input to the linear interpolation unit 72 and the output image buffer 36a. Output image buffer 36
a holds a three-color image having the same number of pixels as the image in the input image buffer 18 and also holds a counter for each pixel.
This counter is provided to hold the number of times writing has been performed at each pixel position in the buffer. In the present embodiment, a plurality of color signal estimated values corresponding to each pixel of the input image buffer 18 are calculated during the processing, and these are accumulated in the output image buffer 36a. Are performed to obtain a final color signal estimated value by averaging. This process requires the number of times of writing.

The output of the output image buffer 36a is
It is led out of the processing device main body 12 and, like the first embodiment described above, is provided outside the processing device main body 12.
For example, a compression processing unit for performing image compression processing such as JPEG and a storage medium such as a memory card are provided, and the color digital image output from the output image buffer 36a is compressed and stored. As in the case of the first embodiment, as shown by the broken line in FIG. 9, the electronic camera 10 and the processing device main body 12, and the compression processing unit (not shown) and the storage medium are integrated into one housing. Of course, it may be configured as an electronic camera device.

Next, the operation in the configuration as shown in FIG. 9 will be described.

When an image is taken by the electronic camera 10,
Signal from GCCD 14 ₁ and R / B checkered CCD 14 ₂ is digitized through an A / D converter 16,
The image is transferred to the input image buffer 18 of the main body 12 as a digital two-plate image. Original RGB images obtained when the same scene is imaged by the three-panel imaging system have R, G, B
Three types of contrast with the color signal value, the digital second plate images two kinds of color signals G and R, or G and B according to the color filter arrangement of R / B checkered CCD 14 ₂ to each pixel of the Only one value, and the other one of the color signal values is missing. Therefore, in the main body 12, the color signal value missing at each pixel of the digital image in the input image buffer 18 is estimated,
A process of restoring an original RGB image obtained when the same scene is imaged by the three-chip imaging system is performed. This restoration process
This is achieved by sequentially performing the following processing on the digital two-plate image in the input image buffer 18.

That is, when the transfer of the image from the electronic camera 10 to the main body 12 is completed, first, the pixel value and the counter of each pixel in the output image buffer 36a are initialized to "0".
The following processing is performed on all the 5 × 5 pixel size rectangular areas included in the digital two-plate image in the input image buffer 18. Hereinafter, an image formed by extracting an S color signal value (S is one of R, G, and B) of each pixel in a certain region of the image will be referred to as an S color signal in that region.

The block extracting section 62 shifts the position of the rectangular area one pixel at a time from the upper left corner of the digital image in the input image buffer 18 rightward and then downward, and sequentially reads the images in the rectangular area. . The image that has been read by the block extraction unit 62 and has begun processing is called the block of interest. The read block of interest is transferred to the block buffer 66.

[0100] Then, the mean-variance evaluation unit 24b reads the target block from the block buffer 66, calculates the average value A _g and dispersion V _g of the G color signals present in all the pixels. Next, the type (either R or B) of the color signal value existing in each pixel of the target block is determined by the R / B checkered CCD 1.
4 ₂ is calculated using the regularity of the color filter arrangement. Then, the average value _Ar and the variance V of the R color signal in the block of interest are calculated.
_r, and calculates an average value A _b and variance V _b B color signals.
(R, g, b are subscripts indicating the type of color signal) Then, the average value A _c and the variance V _c (c is any of r, g, b) are transferred to the parameter calculation unit 28b.

The parameter calculator 28b performs the following processing based on the transferred average value A _c and variance V _c .
First, the variance V _g of the G color signal is compared with a predetermined threshold. _If the variance V _g is equal to or larger than the threshold, the color correlation parameters α _gr , β _gr and α _gb , β
_gb is calculated by the above equation (5) described in the first embodiment. On the other hand, when V _g is equal to or smaller than the threshold, the color correlation parameters α _br , β _br and α _rb , β _rb are similarly calculated by the above equation (5). Then, the calculated color correlation parameter is transferred to the parameter evaluation unit 32b.

The parameter evaluator 32b evaluates the reliability of the transferred parameters as follows. First, when the transferred parameters are α _gr , β _gr and α _gb , β _gb , the block of interest is read from the block buffer 66. Then, in each pixel P of the target block, R / B
Color signal T other than G existing in the pixels P by utilizing the regularity of the color filter arrangement checkered CCD 14 ₂ (T is R, one of B) to identify, the color signal value T _p and G color signals The value G _p ,
And color correlation parameter alpha _gt, the estimation error of beta _gt (t is in response to T r, either b) actual color signal value and the color signal estimate based on the color correlation parameter from e _p = T _p - (α _gt × G _p + β _gt ) (13) Next, the sum of the absolute values of the errors in all the pixels in the block of interest is calculated. If the sum does not exceed a predetermined threshold, the color correlation parameters α _gr , β _gr and α _gb ,
It is determined that the reliability of β _gb is high, and the block restoring unit 68b
, The color correlation parameters α _gr and β _gr and α _gb and β _gb are transferred.

[0103] Figure 10 is a pixel p obtained the R color signal as an example in a color edge region 44, illustrates the meaning of the error e _p. Since two types of color signals are obtained for each pixel in the two-chip imaging system, a color correlation diagram can be partially illustrated as shown as a color signal 98 of the two-chip image in FIG. is there. Error e _p is the color correlation parameter alpha _gr, beta
The difference between the R color signal estimated value 102 estimated from the G color signal value 100 based on the estimated approximation line 42 represented by _gr and the actual R color signal value 104, and the color correlation is estimated by the estimated approximation line 4
The lower the reliability approximated by 2, the larger the value.

On the other hand, when the transferred color correlation parameters are α _br , β _br and α _rb , β _rb , the parameter β _br
The absolute value | beta _br | and beta absolute value of _rb | beta _rb | is checked whether or not exceed the predetermined threshold value, if neither exceed the color correlation parameter alpha _br, beta _br and alpha _rb, beta _rb Is determined to be highly reliable, and the color correlation parameters α _br , β _br and α _rb , β _rb are transferred to the block restoration unit 68b.

When the color correlation parameter is transferred from the parameter evaluation section 32b, the block restoring section 68b reads the block of interest from the block buffer 66, and the type T of the color signal missing at each pixel p of the block of interest (T is R, identified based one) of B to the regularity of the R / B checkered CCD 14 ₂ color filter array. Then, when the transferred parameters are α _gr , β _gr and α _gb , β _gb , from the G color signal value G _p of each pixel P in the block of interest,
The color signal value T _p missing at each pixel is estimated by the following equation.

T _p = α _gt × G _p + β _gt (14) Further, the color correlation parameter α is obtained from the parameter evaluation unit 32b.
_{When br} , _βbr and _αrb , _βrb are transferred, the color signal values Sp (S is not T among R and B) existing in each pixel P in the target block other than the G color signal are obtained. The color signal value Tp missing at each pixel is estimated by the following equation.

T _p = α _st × S _p + β _st (15) In each case, a restored block having three color signal values for each pixel is generated. The restored block generated in this way is transferred to the integrated averaging unit 74. On the other hand, when the color correlation parameter is not transferred from the parameter evaluation unit 32b, neither generation nor transfer of the restored block is performed.

The integrating and averaging unit 74 includes a block restoring unit 68b
Only when the restoration block has been transferred from, for each pixel P of the restoration block, the color signal value at the pixel P of the restoration block is added to the integrated value at the position corresponding to the pixel P in the output image buffer 36a. And the output image buffer 3
The counter corresponding to the pixel P in 6a is incremented by "1".

After the above processing has been performed on all the rectangular areas of 5 × 5 pixels included in the digital two-plate image in the input image buffer 18, the accumulation averaging unit 74 sets each pixel in the output image buffer 36a. The integrated value of P and the counter are read and the following processing is performed.

First, when the counter is "0", the position of the pixel P is notified to the linear interpolation section 72. The linear interpolation unit 72
A 3 × 3 pixel area centered on the pixel P is read from the input image buffer 18, the color signal value existing in the pixel P is used as it is, and the missing color signal value is the average value of the same kind of color signal value in the vicinity. To generate a three-color signal value of the pixel. Then, the three color signal values are transferred to the integration and averaging unit 74. The integration and averaging unit 74 writes the transferred restoration value into the output image buffer 36a. If the counter is not "0",
The integrated averaging unit 74 divides the integrated value by a counter to obtain an average value of a plurality of restored values obtained during processing, and writes this value to the output image buffer 36a.

When the processing up to this point is completed, an original RGB image in which missing color signals have been compensated for in the output image buffer 36a is obtained.

As described above, in the present embodiment, the block extracting section 62 reads the digital two-plate image in the input image buffer 18 while shifting the rectangular area, and the parameter calculating section 28b calculates the color correlation parameter. Next, the parameter evaluation unit 32b determines the reliability of the color correlation parameter by calculating the restoration error of the color signal from the two types of color signal values present in each pixel of the two-plate image, and determines only the highly reliable parameter. To the block restoration unit 68b. Then, the block restoring unit 68b restores all missing color signals of the block based on the color correlation parameter to generate a restored block, and the integrating and averaging unit 74 integrates the restored block in the output image buffer 36a, thereby outputting the image. Image buffer 3
In each pixel 6a, an integrated value of a plurality of missing restoration values calculated based on only highly reliable color correlation parameters is obtained. Finally, the average value of them is output image buffer 3
By outputting the original RGB image to the output image buffer 36a, the image is output to the output image buffer 36a with a small amount of artifacts occurring near the color edge portion as in the first embodiment. In the present embodiment, the reliability of the color correlation parameter is reliably evaluated by using the fact that the input image is a two-plate image, so that the occurrence of artifacts can be more effectively achieved than in the first embodiment. It is possible to suppress. Also, by switching which of the two types of color signals obtained at each pixel is used for missing color restoration in accordance with the local variance of the G color signal, the G signal is small and the restoration from only the G signal reduces noise. Good restoration results can be maintained even in the case of being easily affected.

Note that various modifications and changes can be made to this embodiment as in the case of the first embodiment. The blocks extracted by the block extracting section 62 size, R / B color checkered CCD 14 ₂ filter array and a parameter calculating section 2
In the calculation method in 8b, all the modifications described in the first embodiment are possible.

For example, the average / variance evaluator 24b predicts the reliability of the color correlation parameter based on the average and variance values, and if the reliability is low, does not perform the subsequent processing. The same is possible as in the first embodiment. In the case of a two-chip imaging system, since a color correlation diagram in a block of interest can be formed, various methods can be considered for calculating the reliability of the color correlation parameter. For example, when two straight lines are fitted to the color correlation diagram, and the fitting error is smaller than when a single straight line is fitted, it can be determined that the reliability of the color correlation parameter is low. Further, it is easy to increase the reliability of parameters by performing area division as in the second embodiment. In this case, the average / variance evaluator 24b extracts a straight line from the color correlation diagram by using a straight line extraction method such as Hough transform, and the region corresponding to the best-fit straight line portion may be processed as follows. As a simpler method of dividing the area, the parameter evaluation unit 32
At b, the target block is divided into two partial blocks according to the sign of the estimated error e _p calculated at each pixel P in the target block, and is written back to the block buffer 66. A method of calculating a parameter and using a highly reliable partial block for subsequent processing may be used. Also, similar to the second embodiment, even when the area is divided based on the feature amount,
If the G component existing in all pixels is used as the feature amount, more accurate missing color restoration can be performed.

Various modifications can be made to the processing method in the integration averaging unit 74 and the block restoration unit 68b. Originally G
Since the color signals are obtained without loss, the integration for the G color signals does not have to be performed. Further, the parameter evaluation unit 32b collectively performs the reliability evaluation of the R and G color correlation parameters and the B and G color correlation parameters. If the correlation parameters are independently transferred, even if the color correlation between R and G can be approximated by a straight line, but the color correlation between B and G cannot be approximated, it is possible to restore the missing B signal.

Although the present invention has been described based on the embodiments, the present invention is not limited to the above-described embodiments, and various modifications and applications are possible within the scope of the present invention. . Here, the summary of the present invention is as follows.

(1) A digital image in which one or more types of color signal values are missing for each pixel obtained by a single-chip imaging system or a two-chip imaging system is input, and a missing color signal value of each pixel is estimated. An image processing device for outputting a color digital image by extracting a plurality of different image regions for each pixel from the digital image lacking the color signal value; A color correlation parameter estimating unit for estimating a color correlation parameter representing a color correlation on the image region based on a color signal value existing on the image region; The reliability of each estimated color correlation parameter is evaluated based on the color correlation parameter and the color signal value on the image area corresponding to the color correlation parameter. Parameter selection means for selecting only highly reliable color correlation parameters, and for each pixel of the digital image lacking the color signal value, selection by the color signal value existing in the pixel and the color correlation parameter selection means And a restoring means for restoring the missing color signal value of the pixel based on the color correlation parameter corresponding to the image area including the pixel.

This structure corresponds to the first structure shown in FIGS.
And the third embodiment shown in FIGS. 9 and 10 correspond to the third embodiment. That is, the area extracting means
The first embodiment corresponds to the region extracting unit 20, and the third embodiment corresponds to the block extracting unit 62. The color correlation parameter estimating means comprises an average / variance evaluator 24, 24b
And the parameter calculation units 28 and 28b. The parameter selection means corresponds to the parameter evaluation units 32 and 32b. The restoration unit is a pixel restoration unit 3 in the first embodiment.
4, the block restoration unit 68 according to the third embodiment
b and the integrated averaging unit 74.

According to the above arrangement, the area extracting means extracts a plurality of image areas for each pixel of the input image, and calculates the color correlation on the image area for each of the extracted image areas by using the color correlation parameter. Estimation means estimates and parameterizes. Then, the parameter selection unit evaluates the reliability of each parameter estimated by the color correlation parameter estimation unit, and sends only the highly reliable parameter to the restoration unit.
The restoring unit restores the color signal value of each pixel of the input image based on the parameter related to that pixel among the parameters sent.

Therefore, a plurality of image areas are taken for each pixel of the input image, and only the highly reliable color correlation parameters in each image area are selected. Can be suppressed. In addition, since the restoration of the missing color signal value of each pixel is performed from a plurality of color correlation parameters related to the pixel, it is possible to achieve a strong restoration against noise.

(2) A digital image in which one or more types of color signal values are missing for each pixel obtained by a single-chip imaging system or a two-chip imaging system is input, and a missing color signal value of each pixel is estimated. An image processing device for outputting a color digital image by extracting a plurality of different image regions for each pixel from the digital image lacking the color signal value; A color correlation parameter estimating means for estimating a color correlation parameter representing a color correlation on the image area based on a color signal value existing on the image area; and a missing color on each image area. Area restoring means for restoring a signal value from a color correlation parameter corresponding to the image area and a color signal value present in each pixel to generate a restored image area; For each restored image area
Restoration area selection means for evaluating the reliability of the restoration result based on the color signal values on the restored image area and selecting the restored image area evaluated as having high reliability, and the color signal value is missing. Pixel restoring means for restoring a missing color signal value of each pixel of the digital image based on the restored image area selected by the restoring area selecting means based on a pixel including the pixel. Characteristic image processing device.

This structure corresponds to the second structure shown in FIGS.
Corresponds to this embodiment. That is, the area extracting unit corresponds to the reference image calculation unit 58, the reference image buffer 60, the block extraction unit 62, and the block division unit 64. The color correlation parameter estimating means corresponds to the average / variance evaluator 24a and the parameter calculator 28a. The area restoring means corresponds to the block restoring unit 68, and the restored area selecting means corresponds to the artifact evaluating unit 70. And the pixel restoration means is
This corresponds to the integrated averaging unit 74.

According to the above arrangement, the region extracting means extracts a plurality of image regions for each pixel of the input image, and calculates a color correlation on the image region for each extracted image region by using a color correlation parameter. Estimation means estimates and parameterizes. Then, the region restoring unit restores the missing color signal value at each pixel on the corresponding image region for each of the estimated parameters, and generates a color restored image region. This processing is performed based on each color correlation parameter and a color signal value already existing on the corresponding image area. When the restored image area is thus generated for each color correlation parameter, the restored area selection means evaluates the reliability of the restored result based on the color signal values on each restored image area, and the restored image that has been evaluated to be highly reliable Select only areas. The pixel restoring unit restores a color signal value of each pixel of the input image based on a pixel including the pixel in the restored image area selected by the restoration area selecting unit.

Therefore, a plurality of image areas are taken for each pixel of the input image, and the missing color signal value on each image area is restored based on the color correlation parameter on the image area, and if the reliability is high. Since only the determined restoration result is selected and used for restoring the missing color signal value of each pixel, it is possible to suppress artifacts caused by using the unreliable restoration result. In addition, since the restoration of the missing color signal value of each pixel is performed based on a plurality of restoration results related to the pixel, it is possible to perform a strong restoration against noise. Furthermore, since the restoration result of each image area is directly used to determine the reliability of restoration, the reliability evaluation can be performed more reliably and efficiently.

(3) The area extracting means calculates a predetermined characteristic amount related to a color correlation parameter in each pixel of the digital image based on a color signal value existing near the pixel. Dividing means for dividing the vicinity of each pixel of the digital image into a plurality of image areas based on the feature amounts calculated by the feature amount calculating means, and extracting each of the divided image areas. The image processing apparatus according to the above (1) or (2).

This structure is similar to the first structure shown in FIGS.
This embodiment also corresponds to the third embodiment shown in FIGS. 9 and 10, but mainly corresponds to the second embodiment shown in FIGS. That is, in the second embodiment, the area extracting means is the reference image calculating unit 5
8, corresponding to the reference image buffer 60, the block extracting unit 62, and the block dividing unit 64, and thereafter, the feature amount calculating unit corresponds to the reference image calculating unit 58, and the dividing unit corresponds to the block dividing unit 64. In the first embodiment, the region extracting unit, the feature amount calculating unit, and the dividing unit all correspond to the region extracting unit 20. In the third embodiment, the area extracting unit, the feature amount calculating unit, and the dividing unit all correspond to the block extracting unit 62.

According to the above configuration, in the region extracting means, the characteristic amount calculating means calculates, for each pixel of the input image, a characteristic amount related to a color correlation parameter representing a color correlation at the pixel. Then, the dividing unit divides the neighborhood of each pixel into regions based on the feature amount, and extracts each of the divided partial regions as an image region related to the pixel.

Therefore, in extracting an image region, a feature amount closely related to a color correlation parameter near each pixel is calculated from an input image, and an image region to be extracted is determined based on the feature amount. The color correlation parameters on the image area become highly reliable in advance, and the occurrence of artifacts can be suppressed more reliably.

That is, in the above-described configuration, utilizing the features that can be calculated from the input image such as the luminance, only the region having a high possibility of satisfying the assumption made regarding the color correlation is positively extracted, and information on the region is used. By restoring the missing color signal value, it is possible to provide a method of restoring a missing color signal value that is less likely to cause artifacts in all parts of the image and can generate a high-quality image.

(4) The color correlation parameter estimating means determines the reliability of the color correlation parameter based on the color signal value existing on the image area corresponding to the color correlation parameter before estimating the color correlation parameter. The image processing according to any one of (1) to (3), further including a reliability prediction unit for predicting, and estimating the color correlation parameter only when the reliability is determined to be high. apparatus.

This structure corresponds to the second structure shown in FIGS.
This embodiment also corresponds to the third embodiment shown in FIGS. 9 and 10, but mainly corresponds to the first embodiment shown in FIGS. That is, the average / variance evaluator 24 corresponds to the reliability prediction means.

According to the above configuration, in the parameter estimation by the color correlation parameter estimating means, the reliability estimating means predicts the reliability of the color correlation parameter from the color signal value on the target image area, and Is calculated to be low, the calculation of the color correlation parameter is not performed. If it is predicted to be high, an estimate is made.

Therefore, the calculation of the color correlation parameter having a clearly low reliability can be prevented, so that the occurrence of the artifact is more reliably suppressed, and the processing speed is improved.

(5) The image according to (3), wherein the characteristic amount calculating means calculates an average luminance value or hue in the vicinity of each pixel of the digital image to obtain a characteristic amount. Processing equipment.

This structure corresponds to the second structure shown in FIGS.
Corresponds to this embodiment. That is, the feature amount calculation unit corresponds to the reference image calculation unit 58.

According to the above arrangement, the characteristic amount calculating means calculates the average luminance value or the average luminance value in the vicinity of each pixel as the characteristic amount related to the color correlation parameter indicating the color correlation at each pixel of the input image. Calculate hue.

Therefore, the reliability of the color correlation parameter for each area divided by the dividing means is improved by using a luminance value which can be stably calculated in the vicinity of each pixel and has a small influence of moire. I do. Further, in an edge portion having the same luminance value but different color, the hue is used as a feature value instead of the luminance value, so that the reliability of the color correlation parameter for each region similarly divided by the dividing means is improved.

That is, in the above-described configuration, by utilizing features that can be calculated from an input image, such as luminance, only an area having a high possibility that the assumption made regarding color correlation is satisfied is positively extracted, and information on the area is used. By restoring the missing color signal value, it is possible to provide a method of restoring a missing color signal value that is less likely to cause artifacts in all parts of the image and can generate a high-quality image.

(6) The digital image is obtained by a two-chip image pickup system, and the parameter selecting means uses the color correlation parameter estimated by the color correlation parameter estimating means to calculate the color correlation parameter. An estimation error between an estimated value when one of two types of color signal values already existing at each pixel on the corresponding image area is estimated from the other and an actual color signal value is calculated, and the color error is calculated based on the estimated error. The image processing apparatus according to (1), wherein the reliability of the correlation parameter is evaluated.

This configuration corresponds to the third configuration shown in FIGS.
Corresponds to this embodiment.

According to the above configuration, when an image obtained by the two-chip imaging system is input, two types of color signal values are obtained for each pixel of the input image. In evaluating the reliability of the color correlation parameter, the parameter selection unit first calculates an estimated value of a color signal value existing in each pixel of the corresponding image region using the color correlation parameter. At this time, of the two types of color signal values existing in each pixel, the other color signal value is estimated from one color signal value. Then, an error between the estimated value of each pixel and the color signal value of the same type already existing in the pixel is calculated.

Then, the reliability of the color correlation parameter is evaluated based on the error, and it is determined that the larger the error, the lower the reliability.

That is, unlike an image obtained by the single-chip imaging system, an image obtained by the two-chip imaging system has two types of color signal values for each pixel. Therefore, the parameter selection unit can intentionally estimate the color signal value already existing at each pixel using the color correlation parameter, and examine the error of the estimation result. As a result, the reliability of the color correlation parameter can be reliably evaluated.

(7) The parameter selecting means evaluates the reliability of the color correlation parameter based on whether or not the value of the color correlation parameter is included in a predetermined range. The image processing device according to (1).

This configuration is similar to the first configuration shown in FIGS.
And the third embodiment shown in FIGS. 9 and 10 correspond to the third embodiment. That is, the parameter selection unit corresponds to the parameter evaluation units 32 and 32b.

According to the above configuration, the parameter selection means estimates the reliability of the color correlation parameter from the parameter value itself. The parameter value range is set in advance to a range in which the tendency of the value of the color correlation parameter when the reliability of the color correlation parameter is low is determined, and the color correlation parameter is empirically determined to be sufficiently reliable. I have. The parameter selection means checks whether or not the value of the color correlation parameter falls within this range, and if not, determines that the reliability is low, and if it does, determines that the reliability is high.

Therefore, since other data is not required for evaluating the reliability of the color correlation parameter, the reliability evaluation can be performed at high speed.
It can be done easily.

(8) The restoration area selection means detects a pixel having a color signal value that is significantly different from the color signal values of all neighboring pixels from the restoration image area, and based on the detection result, detects the restoration image area. The image processing apparatus according to (2), wherein reliability is determined.

This structure corresponds to the second structure shown in FIGS.
Corresponds to this embodiment. That is, the restoration area selection means
Corresponds to the artifact evaluation unit 70.

Here, in the restored image area restored by the area restoring means, a specific artifact appears when the reliability of the restored result is poor. This artifact has a feature that, at a specific pixel, its color signal value takes a value far apart from the color signal values of any surrounding pixels. The restoration area selection means detects such a pixel from the restoration image area, and determines the reliability of the color correlation parameter according to the detection result.

Therefore, the reliability of the restoration result can be reliably evaluated by explicitly detecting the feature of the artifact from the restored image.

(9) The color correlation parameter estimating means assumes that the color correlation satisfies the equation T = α × S + β for two types of color signal values S and T, and sets the parameters α and β to S The image processing apparatus according to any one of (1) to (8) above, wherein the color correlation parameter relating to T is set.

This configuration is similar to the first configuration shown in FIGS.
, The second embodiment shown in FIGS. 6 to 8, and the third embodiment shown in FIGS. 9 and 10 correspond to the third embodiment.

According to the above configuration, the color correlation parameter estimating means assumes that the color correlation for any two types of color signal values S and T satisfies the equation T = α × S + β, and the parameters α and β Is estimated as a color correlation parameter for S and T.

Therefore, when the image area extracted by the area extracting means is sufficiently local, the input image is expressed by a plurality of light sources or specular reflection by using a linear expression having two parameters as an expression expressing the color correlation. A color correlation in a complicated case including an object having characteristics can be expressed. as a result,
It is possible to reduce the range of the region where the reliability of the color correlation parameter decreases and the artifact occurs.

In other words, in the above configuration, a general-purpose color correlation as a color correlation of a local region of an image is expressed as a parameter, and a calculation method for restoring a missing color signal value by using the statistical properties of the local region of the image. Therefore, even if the input image is a complex one including a plurality of light sources and objects having various surface reflection characteristics, there are few artefacts, and the flexibility of being able to cope with various imaging methods and mosaic color filter arrangements. A method for restoring a certain missing color signal value can be provided.

(10) The color correlation parameter estimating means calculates the color correlation parameter as a statistic of color signal values existing on an image area corresponding to the color correlation parameter, in particular, an average value and a variance, or a maximum value. The image processing apparatus according to (9), wherein the estimation is performed based on the minimum value.

This configuration corresponds to the first configuration shown in FIGS.
, The second embodiment shown in FIGS. 6 to 8, and the third embodiment shown in FIGS. 9 and 10 correspond to the third embodiment.

According to the above arrangement, the color correlation parameter estimating means classifies the color signal values on the image area for which parameters are to be estimated by type, and calculates the average and variance or the maximum and minimum values for each type. Is calculated. Next, the square root of the variance or the difference between the maximum value and the minimum value is calculated for each type, and the magnitude of the variation of the color signal value is estimated. Then, α of the color correlation parameters is calculated from the ratio of the magnitudes of the fluctuations and is used as an estimated value.
Next, β of the color correlation parameters is calculated from the calculated α and the average value or the intermediate value between the maximum value and the minimum value, and is set as an estimated value.

Therefore, by estimating the color correlation parameter based on the statistics related to the color signal values on the corresponding image area, in particular, the average value and the variance or the maximum value and the minimum value, the image pickup system used to obtain the input image is obtained. The arrangement of the color filters does not need to be limited, and the color correlation parameter is less likely to be affected by noise, so that a missing color signal value that is strong against noise can be restored.

[0161]

As described in detail above, according to the present invention,
It is determined whether or not the assumption provided for the color correlation is established within a certain area, and the missing color signal value is restored using only the information of the area determined to hold the assumption. It is possible to provide an image processing apparatus capable of restoring a missing color signal value that is less likely to cause artifacts and can generate a high-quality image.

Further, according to the present invention, an image processing apparatus capable of restoring a missing color signal value that is resistant to noise by performing processing on a pixel by integrating a plurality of nearby color correlations relating to the pixel. Can be provided.

[Brief description of the drawings]

FIG. 1 is a diagram illustrating a configuration of an electronic camera system to which an image processing device according to a first embodiment of the present invention is applied.

FIG. 2 is a diagram for explaining color correlation characteristics.

FIG. 3 is a diagram for explaining a color correlation parameter and a calculation method thereof.

FIG. 4 is a diagram for explaining the reliability of a color correlation parameter.

FIG. 5 is a diagram for explaining an effect of selecting a color correlation parameter.

FIG. 6 is a diagram illustrating a configuration of an electronic camera system to which an image processing device according to a second embodiment of the present invention is applied.

FIG. 7 is a diagram for explaining processing in a color edge area.

FIG. 8 is a diagram for explaining processing in a material uniform region.

FIG. 9 is a diagram illustrating a configuration of an electronic camera system to which an image processing device according to a third embodiment of the present invention is applied.

FIG. 10 is a diagram for explaining reliability evaluation of a color correlation parameter.

[Explanation of symbols]

10 an electronic camera 12 processor body 14 veneer primary _{CCD 14 1 GCCD 14 2 R /} B checkered CCD 16 A / D converter 18 the input image buffer 20 area extracting unit 22 region buffer 24, 24a, 24b mean-variance evaluation unit 26 filters Arrangement holding ROM 28, 28a, 28b Parameter calculation unit 30 Parameter buffer 32, 32b Parameter evaluation unit 34 Pixel restoration unit 36, 36a Output image buffer 58 Reference image calculation unit 60 Reference image buffer 62 Block extraction unit 64 Block division unit 66 Block buffer 68, 68b Block restoration unit 70 Artifact evaluation unit 72 Linear interpolation unit 74 Integration average unit

Claims (3)

[Claims] 1. A digital image obtained by a single-chip imaging system or a two-chip imaging system, in which one or more types of color signal values are missing for each pixel, is input, and a missing color signal value of each pixel is estimated. An image processing apparatus for outputting a color digital image by extracting a plurality of different image regions for each pixel from the digital image lacking the color signal value; Color correlation parameter estimating means for estimating a color correlation parameter representing a color correlation on the image area based on color signal values existing on the image area; The reliability of each color correlation parameter obtained is evaluated based on the color correlation parameter and the color signal value on the image area corresponding to the color correlation parameter, and the reliability is evaluated. Parameter selection means for selecting only highly correlated color correlation parameters, and for each pixel of the digital image lacking the color signal value, the color signal value existing in the pixel and the color selection parameter selection means are selected. And a restoring means for restoring the missing color signal value of the pixel based on the color correlation parameter in which the corresponding image area includes the pixel. 2. A digital image obtained by a single-chip imaging system or a two-chip imaging system, in which one or more types of color signal values are missing for each pixel, and a missing color signal value of each pixel is estimated. An image processing device for outputting a color digital image by extracting a plurality of different image regions for each pixel from the digital image lacking the color signal value; A color correlation parameter estimating means for estimating a color correlation parameter representing a color correlation on the image area based on a color signal value existing on the image area; and a missing color signal on each image area. Area restoring means for restoring a value from a color correlation parameter corresponding to the image area and a color signal value present in each pixel to generate a restored image area; For each of the restored image areas, the reliability of the restoration result is evaluated based on the color signal values on the restored image area, and restoration area selection means for selecting a restoration image area evaluated as having high reliability. For each pixel of the digital image that lacks the color signal value, the missing color signal value of the pixel is restored based on the restored image area selected by the restoration area selection unit based on the pixel including the pixel. An image processing apparatus comprising: a pixel restoration unit. 3. A feature value calculating unit that calculates a predetermined feature value related to a color correlation parameter in each pixel of the digital image based on a color signal value existing near the pixel. Dividing means for dividing the vicinity of each pixel of the digital image into a plurality of image areas based on the feature amount calculated by the feature amount calculating means, and extracting each of the divided image areas. The image processing device according to claim 1.