JP5153842B2 - Image processing apparatus and image processing program - Google Patents

Description

  The present invention relates to an image processing device that processes an image output from a single-plate image pickup system, a two-plate image pickup system, or a three-plate pixel shift image pickup system, and generates a color digital image having three color component values for each pixel. It relates to a processing program.

  A single-plate image pickup system that is often used in digital cameras, etc. uses a single-plate image pickup device equipped with a different color filter for each pixel, and an output image from the image pickup device has one kind of color for each pixel. It has only component values. Therefore, in order to generate a color digital image, it is necessary to perform an interpolation process to compensate for missing color component values in each pixel. The interpolation processing for compensating for such missing color component values is also necessary in the case of a device using a two-plate imaging system or a three-plate pixel shifting imaging system.

  If this interpolation process is not devised, the color image finally obtained may be deteriorated such as blur or false color. For this reason, interpolation processing by various methods has been proposed in the past, and can be roughly classified into those based on the edge detection method and those based on the color correlation method.

  As a technique based on edge detection, for example, a technique described in JP-A-8-298669 can be cited. The technique described in this publication will be described with reference to FIG. FIG. 32 is a diagram for explaining a conventional interpolation process based on edge detection.

Assume that the single-plate image pickup device is a single-plate Bayer array image pickup device including color filters of a three-primary-color Bayer array as shown in FIG. At this time, as shown in FIG. 32B, an interpolation value Gh for G in the horizontal direction with respect to the target pixel is obtained by taking a cross-shaped neighborhood along the vertical and horizontal directions around B5 as the target pixel. The interpolation value Gv for G in the vertical direction is estimated as shown in the following equation 1.
[Equation 1]
Gh = (G4 + G6) / 2 + (2 × B5-B3-B7) / 4
Gv = (G2 + G8) / 2 + (2 × B5-B1-B9) / 4
Next, evaluation values dH and dV indicating whether there are many steps in the horizontal direction or the vertical direction are expressed by the following Equation 2,
[Equation 2]
dH = | G4-G6 | + | B3-2 × B5 + B7 |
dV = | G2-G8 | + | B1-2 × B5 + B9 |
And calculate. Here, the symbol | x | represents the absolute value of x. Of the evaluation values dH and dV calculated in this way, the interpolation value Gh or the interpolation value Gv in the direction in which the value is determined to be smaller and flatter is used.

  On the other hand, as a technique based on color correlation, for example, a technique described in JP-A-11-215512 can be cited. The technique described in this publication will be described with reference to FIG. FIG. 33 is a diagram for explaining a conventional interpolation process based on color correlation.

As shown in FIG. 33, first, in the vicinity of the target pixel X, any one of the color components of the color filter configured in the image sensor (for example, r, g, b in the case of a primary color filter). The following Equation 3 expressed using coefficients αqp and βqp between the component values Vp and Vq of the two color components p and q,
[Equation 3]
Vq = αqp × Vp + βqp
It is assumed that a linear correlation such as Here, when the color filter configured in the image sensor is, for example, a primary color filter, p and q are any of r, g, and b. The component values Vp and Vq corresponding to the color components p and q are values in pixels near the target pixel X, respectively.

Then, in the neighborhood U set around the pixel of interest X, the pixels in the neighborhood U are classified according to the types of the obtained color components r, g, b to form three subsets Ur, Ug, Ub. Then, the coefficients αqp and βqp are obtained by calculating the average Ac of pixel values in each subset (here, c is r, g, or b) and the standard deviation Sc (here, c is any of r, g, or b). ) Based on the following equation (4).
[Equation 4]
αqp = Sq / Sp, βqp = Aq−αqpAp
The ratio of the distribution width (standard deviation Sq) of the color component q to the distribution width (standard deviation Sp) of the color component p gives the linear correlation slope αqp shown in Equation 3 above. The straight line with the inclination αqp is based on the idea that it passes through the points plotted by the average value Ap of the color component p and the average value Aq of the color component q.

Then, based on the estimated αqp and βqp, the value Xm of the missing color component m in the target pixel X is expressed by the following equation 5 from the value Xe of the color component e obtained in the target pixel X. Ask for it.
[Equation 5]
Xm = αme × Xe + βme
Here, m and e are either r, g, or b, respectively.

Thus, the prior art based on color correlation is characterized by estimating the correlation of pixel values between different color components in the vicinity of the pixel of interest.
JP-A-8-298669 JP-A-11-215512

  By the way, in an actual imaging system, there is a difference in gain between lines, or even the pixels having the same color filter may have slightly different spectral characteristics of the filters. Sensitivity variation may occur between pixels that should be obtained.

  However, all of the above-described conventional techniques use the color component itself obtained at the target pixel as it is as a result of the interpolation process without any correction. However, the information of the target pixel itself is reflected in the result as it is. Normally, it is more efficient to use the color component information originally obtained for each pixel as it is for the result of the interpolation process without change, but the sensitivity is different for pixels that can obtain the same color component. In such a case, when such a process is performed, the sensitivity variation between pixels is reflected in the result of the interpolation process as it is, and there is a side effect such as a slight check of the grid pattern.

  The present invention has been made in view of the above circumstances, and provides an image processing apparatus and an image processing program capable of performing high-precision interpolation processing while reducing the influence even when the sensitivity between pixels varies. It is an object.

  In order to achieve the above object, an image processing apparatus according to a first aspect of the present invention is one for each pixel obtained by capturing an optical image with a single-plate imaging system, a two-plate imaging system, or a three-plate pixel shifting imaging system. An image processing apparatus that processes an image in which more than one type of color component is missing, and the sensitivity between pixels for which the color component is obtained for a specified type of color component in a predetermined vicinity of the pixel of interest Determining means for determining the magnitude of the difference, and interpolating means for performing processing to compensate for missing color components in the pixel of interest and capable of changing the processing content according to the determination result by the determining means when performing the processing And.

  Furthermore, in the image processing device according to the second invention, in the image processing device according to the first invention, the interpolation means causes the color component to be within the vicinity of the color component obtained at the pixel of interest. A pixel value of the color component that the target pixel should take when a reference pixel is determined from the obtained pixels and the sensitivity of the target pixel to the color component is matched with the sensitivity of the reference pixel to the color component Is calculated as a correction value, and the pixel value of the color component of the target pixel is changed to the correction value according to the magnitude of the sensitivity difference determined by the determination means.

  An image processing apparatus according to a third invention is the image processing apparatus according to the first invention, wherein the interpolation means includes weighted average means for weighted average of the pixel values in the predetermined neighborhood. The weighted average means changes the weight according to the determination result of the determination means and calculates the pixel value of the target pixel by weighted average.

  An image processing apparatus according to a fourth invention is the image processing apparatus according to the third invention, wherein the weighted averaging means obtains the weight and the type of color component determined to have a large sensitivity difference by the determination means. The pixel is changed so as to be smaller with respect to the pixel being set.

  An image processing apparatus according to a fifth invention is the image processing apparatus according to the third invention, wherein the weighted averaging means uses the weights and pixel values between pixels in which the same kind of color components are obtained in the vicinity. When the determination means determines that the sensitivity difference is large with respect to a specific color component, the function expression is calculated as an effect of the pixel value difference related to the color component. Is changed so as to be smaller.

  An image processing apparatus according to a sixth aspect of the present invention is the image processing apparatus according to the third aspect, wherein the weighted average means obtains the type of color component determined to have a sensitivity difference in the determination means. On the other hand, the weight is changed so as to approach the pixels equally.

  An image processing apparatus according to a seventh aspect is the image processing apparatus according to the first aspect, wherein the determination means includes a feature quantity calculation means for calculating a predetermined feature quantity from a pixel value within a predetermined vicinity of the target pixel. And the magnitude of the sensitivity difference is determined based on the magnitude of the feature amount.

  An image processing apparatus according to an eighth invention is the image processing apparatus according to the seventh invention, wherein the determination means further has a table in which the feature amount related to the designated color component is associated with the amount of the sensitivity difference. The size of the sensitivity difference is determined by referring to the table from the feature amount.

  An image processing apparatus according to a ninth invention is the image processing apparatus according to the seventh or eighth invention, wherein the determination means uses a predetermined method to identify pixels in which the designated color component is obtained within the vicinity. Classification is performed, and the feature amount is calculated based on the average value and the variation amount of the pixel value in each classification.

  An image processing device according to a tenth invention is the image processing device according to the seventh or eighth invention, wherein the determination means is an average of pixel values of pixels for which a designated color component is obtained in the vicinity. The feature amount is calculated based on the value and the average value of pixel values of pixels from which a predetermined color component different from the designated color component is obtained.

  An image processing apparatus according to an eleventh aspect of the invention is the image processing apparatus according to the first aspect of the invention, wherein the interpolation means can generate a color image obtained by reducing the image size obtained by the imaging system at a predetermined reduction rate. The determination means does not perform sensitivity difference determination when the reduction ratio is equal to or less than a predetermined threshold when the interpolation means generates a reduced color image. .

  An image processing program according to a twelfth aspect of the invention includes a computer having at least one color component for each pixel obtained by capturing an optical image with a single-plate imaging system, a two-plate imaging system, or a three-plate pixel shifting imaging system. An image processing program for processing a missing image, wherein the computer detects sensitivity between pixels for which the color component is obtained with respect to a specified type of color component in a predetermined vicinity of the pixel of interest. A determination unit for determining the magnitude of the difference, an interpolation unit for performing a process of compensating for a missing color component in the target pixel, and capable of changing a processing content according to a determination result by the determination unit when performing the process; Is an image processing program for functioning as

  According to the image processing apparatus and the image processing program of the present invention, even if there is variation in sensitivity between pixels, it is possible to reduce the influence and perform highly accurate interpolation processing.

1 is a block diagram showing a configuration of a digital camera in Embodiment 1 of the present invention. FIG. 5 is a block diagram showing a configuration of a digital camera in Embodiment 2 of the present invention. 7 is a flowchart showing processing of a G level difference detection circuit in the second embodiment. The figure which shows the classification | category of the pixel in the G level | step difference detection circuit and G interpolation circuit in the said Example 2. FIG. 9 is a flowchart showing processing of a G interpolation circuit in the second embodiment. The figure for demonstrating the process of step S16 in the process of the G interpolation circuit shown in the said FIG. The figure which shows the direction of the weight calculation by the weight calculation circuit in the said Example 2. FIG. 9 is a flowchart showing processing of an RB interpolation circuit in the second embodiment. The figure for demonstrating the pixel selection of step S31 in the process of the RB interpolation circuit shown in the said FIG. 7 is a flowchart illustrating software processing performed by a computer in the second embodiment. 9 is a flowchart showing G interpolation software processing performed by a computer in the second embodiment. 9 is a flowchart showing RB interpolation software processing performed by a computer in the second embodiment. FIG. 9 is a block diagram illustrating a configuration of a digital camera according to Embodiment 3 of the present invention. FIG. 10 is a diagram illustrating pixel numbers used when processing is performed by the correction circuit according to the third embodiment. 10 is a flowchart showing processing of a correction circuit in the third embodiment. 7 is a flowchart illustrating software processing performed by a computer in the third embodiment. FIG. 6 is a block diagram illustrating a configuration of a digital camera according to Embodiment 4 of the present invention. FIG. 6 is a block diagram illustrating a configuration example of a step correction circuit according to the fourth embodiment. The figure for demonstrating the G level | step difference in the said Example 4. FIG. 10 is a flowchart showing processing of a level difference correction circuit according to the fourth embodiment. The figure which shows the example of the vicinity at the time of performing the process in the level | step difference correction circuit of the said Example 4. FIG. In the said Example 4, the diagram which shows the example of the function for calculating | requiring correction amount (DELTA). FIG. 6 is a block diagram showing another configuration example of the step correction circuit in the fourth embodiment. The figure for demonstrating the combination average calculated by the combination average calculation circuit in the said Example 4. FIG. 10 is a flowchart illustrating software processing performed by a computer in the fourth embodiment. FIG. 10 is a block diagram illustrating a configuration of a digital camera according to Embodiment 5 of the present invention. The block diagram which shows one structural example of the level | step difference smoothing circuit in the said Example 5. FIG. 10 is a flowchart showing processing of a step smoothing circuit in the fifth embodiment. In the said Example 5, the diagram which shows the example of the function for calculating coefficient mixture ratio (alpha). FIG. 10 is a diagram for explaining frequency characteristics of filter coefficients set by the coefficient setting circuit according to the fifth embodiment. FIG. 10 is a block diagram illustrating another configuration example of the step smoothing circuit according to the fifth embodiment. The figure for demonstrating the conventional interpolation process based on edge detection. The figure for demonstrating the conventional interpolation process based on a color correlation.

  Embodiments of the present invention will be described below with reference to the drawings.

  FIG. 1 shows a first embodiment of the present invention and is a block diagram showing a configuration of a digital camera.

  In the first embodiment, the image processing apparatus of the present invention is applied to a digital camera.

  As shown in FIG. 1, the digital camera 1 includes an optical system 2, a single-plate Bayer array CCD 3, a single-plate image buffer 4, a line correction circuit 5, a WB correction circuit 6, an interpolation circuit 7, and a color image. A buffer 8, an image quality adjustment circuit 9, a recording circuit 10, and a control circuit 11 are included.

  The optical system 2 is for condensing the subject luminous flux.

  The single-plate Bayer array CCD 3 photoelectrically converts a subject image formed by the optical system 2 and outputs an imaging signal.

  The single plate image buffer 4 temporarily stores image data output from the single plate Bayer array CCD 3 and digitized by an A / D conversion circuit (not shown).

  The line correction circuit 5 is a sensitivity correction unit that corrects a difference in gain between lines for an image stored in the single-plate image buffer 4.

  The WB correction circuit 6 performs white balance processing on the image data output from the line correction circuit 5.

  The interpolating circuit 7 is an interpolating unit that generates color image data by interpolating the color components missing in each pixel in the single-plate image data output from the WB correction circuit 6 from information on neighboring pixels. is there.

  The color image buffer 8 is for temporarily storing the three-plate color image data interpolated by the interpolation circuit 7.

  The image quality adjustment circuit 9 performs image quality adjustment processing such as color conversion and edge enhancement on the color image stored in the color image buffer 8.

  The recording circuit 10 records three-plate color image data adjusted in image quality by the image quality adjusting circuit 9.

  The control circuit 11 is a control means for comprehensively controlling the digital camera 1 including the circuits as described above.

  The operation of the digital camera 1 is as follows.

  When a shutter button (not shown) provided in the digital camera 1 is pressed by the user, first, an optical image by the optical system 2 is picked up by the single-plate Bayer array CCD 3, and there is only one type of color component for each pixel. An image in a single plate state is stored in the single plate image buffer 4.

  Next, the WB correction circuit 6 operates to calculate the ratio Crg between the R component and the G component and the ratio Cbg between the B component and the G component for the white subject, respectively, to obtain a white balance coefficient. In the present embodiment, pixel values are read from the single-plate image buffer 4 for each block of a predetermined size, and within this block, processing is performed to average the pixel values for each type of color component obtained at each pixel. .

  As a result, an RGB value of the average color of each block is obtained. Further, the ratio between the R component and the G component and the ratio between the B component and the G component are calculated, and these ratios satisfy a predetermined relationship. Select only the blocks that are present. Then, the final white balance coefficients Crg and Cbg are obtained by further averaging the ratio between the R component and the G component and the ratio between the B component and the G component of the selected block. This method itself is known. However, during this calculation, when the pixel value is read from the single-panel image buffer 4, the line correction circuit 5 operates to correct the pixel value.

That is, the line correction circuit 5 corrects a gain step between even lines and odd lines that occur when signals are read from the single-plate Bayer array CCD 3, and the circuit includes coefficients (a, b) is retained. This coefficient is the average of the pixel values of the pixels where the G component is obtained in the odd lines with respect to the pixel values in the single-panel image buffer 4 when the uniform gray subject is photographed, and the G component in the even lines. V2 is the average of the pixel values of the pixels from which
[Equation 6]
V2 = a × V1 + b
It is calculated in advance so that Using this coefficient, the line correction circuit 5 corrects the pixel value v of the pixel for which the G component is obtained in the odd line as shown in the following Equation 7 in accordance with the sensitivity of the even line,
[Equation 7]
v ′ = a × v + b
The corrected pixel value (correction value) v ′ is output to the WB correction circuit 6.

  When the processing of the WB correction circuit 6 as described above is completed, the white balance coefficients Crg and Cbg are held in the WB correction circuit 6.

  Next, the interpolation circuit 7 performs an interpolation process on each pixel in the single-panel image buffer 4. The interpolation processing algorithm performed by the interpolation circuit 7 is the same as that described in the above-mentioned Japanese Patent Application Laid-Open No. 8-298669. However, in this embodiment, the pixels are obtained from the single-panel image buffer 4 by five lines. The data is read out and stored in an internal buffer (not shown) in the interpolation circuit 7, and interpolation processing is performed on the data in the buffer.

  Then, when data is transferred from the single-panel image buffer 4 to the internal buffer, the line correction circuit 5 and the WB correction circuit 6 operate to change the pixel values read from the single-panel image buffer 4.

In this case, first, the correction result (correction value) v ′ with respect to the pixel value v read out from the single-panel image buffer 4 by the line correction circuit 5 is expressed by [Equation 8].
v ′ = a × v + b (when the read pixel is an odd line and the G component is obtained)
v ′ = v (other cases)
Further, the WB correction circuit 6 outputs the output v ″ to the correction result (correction value) v ′ of the line correction circuit 5,
[Equation 9]
v ″ = v ′ (when G component is obtained in the readout pixel)
v ″ = Crg × v ′ (when R component is obtained in the readout pixel)
v ″ = Cbg × v ′ (when B component is obtained in readout pixel)
And the process of writing v ″ into the internal buffer of the interpolation circuit 7 is performed.

  Thereafter, the color image output to the color image buffer 8 by the interpolation circuit 7 is subjected to processing such as color conversion, edge enhancement and gradation conversion by the image quality adjustment circuit 9, and the recording circuit 10 displays a recording medium (not shown). To be recorded. In this way, the imaging operation of the digital camera 1 is completed.

  The first embodiment describes a basic configuration and processing, and many variations can be considered in this embodiment. The processing by the interpolation circuit 7 may be anything as long as the pixel value is used as it is as the value of the G component after the interpolation processing in the pixel from which the color component G is obtained.

  In this embodiment, a single-plate imaging system is used as an example, but the same processing can be applied to a two-plate imaging system or a three-plate pixel shift imaging system.

  For example, in the case of a two-plate imaging system, the G component is obtained from all the pixels from one of the two imaging elements, and the R component and the B component are obtained from the other in a checkered pattern. For example, for the correction of the sensitivity difference with respect to the R component, the same technique as the sensitivity correction described in this embodiment can be applied to the G component obtained in a checkered pattern in the single-plate Bayer array CCD 3.

  Further, in the case of a three-plate pixel shifted imaging system, the G component is acquired by two of the three image sensors, the R and B components are acquired in a checkered pattern by the remaining one, and from these information Some types generate high-definition luminance components. Also in this case, the sensitivity correction of the R and B components obtained in a checkered pattern can be performed in the same manner as described above.

  According to the first embodiment, even if the sensitivity is different among pixels from which the same color component is obtained, particularly between lines, the correction is performed by the line correction circuit serving as the sensitivity correction unit, and then the interpolation circuit serving as the interpolation unit. Since the processing is performed, it is possible to prevent the difference in sensitivity from affecting the interpolated image and obtain a higher quality image.

  FIGS. 2 to 12 show a second embodiment of the present invention. FIG. 2 is a block diagram showing the configuration of the digital camera. FIG. 3 is a flowchart showing the processing of the G step detection circuit. FIG. 5 is a flowchart showing processing of the G interpolation circuit, and FIG. 6 is a diagram for explaining the processing of step S16 in the processing of the G interpolation circuit shown in FIG. FIG. 7 is a diagram showing the direction of weight calculation by the weight calculation circuit, FIG. 8 is a flowchart showing the processing of the RB interpolation circuit, and FIG. 9 explains pixel selection in step S31 in the processing of the RB interpolation circuit shown in FIG. FIG. 10 is a flowchart showing software processing performed by the computer, FIG. 11 is a flowchart showing G interpolation software processing performed by the computer, and FIG. Is a flow chart showing the RB interpolation software processing performed by a computer.

  In the second embodiment, the same parts as those of the first embodiment are denoted by the same reference numerals and the description thereof is omitted, and only different points will be mainly described.

  In the second embodiment, the image processing apparatus is applied to a digital camera as in the first embodiment.

  In the digital camera 21 of the second embodiment, a G step detection circuit 22 as a determination unit and a feature amount calculation unit is added to the digital camera 1 of the first embodiment, and an interpolation unit is used instead of the interpolation circuit 7. The difference is that a G interpolation circuit 23, a G image buffer 26, and an RB interpolation circuit 27 as interpolation means are provided.

  The G interpolation circuit 23 is calculated by a weight calculation circuit 24 serving as a weighted average means for calculating a weight related to a pixel to be interpolated based on an output from the G step detection circuit 22, and the weight calculation circuit 24. And an averaging circuit 25 which is a weighted average means for calculating a weighted average value based on the weights and outputting it to the G image buffer 26 as a pixel value of a pixel to be interpolated.

  Next, the operation of the digital camera 21 will be described mainly with respect to differences from the first embodiment.

  When the shutter button is pressed by the user, an optical image by the optical system 2 is picked up by the single-plate Bayer array CCD 3 as in the first embodiment, and an image in a single-plate state having only one kind of color component for each pixel. Is stored in the single-panel image buffer 4.

  Next, the line correction circuit 5 and the WB correction circuit 6 operate in the same manner as in the first embodiment, and the white balance coefficients Crg and Cbg are calculated.

  Next, the G interpolation circuit 23 operates. At this time, first, a block of 5 × 5 pixels around each pixel related to the image data stored in the single-panel image buffer 4 is read into an internal buffer provided in the G interpolation circuit 23. At that time, as in the first embodiment, the line correction circuit 5 and the WB correction circuit 6 are operated to correct the pixel value, and the G step detection circuit 22 is further operated to obtain the G component. An estimate of the step between the even and odd lines of the pixel being calculated is calculated.

  The step estimation by the G step detection circuit 22 will be described with reference to the flowchart of FIG.

  The G level difference detection circuit 22 classifies each pixel in the block of 5 × 5 pixels read into the internal buffer for each obtained color component, and calculates an average value of pixel values for each classification. At that time, as shown in FIG. 4, the G component is G1 when the color component obtained from the pixel adjacent in the horizontal direction is R, and the color component obtained from the pixel adjacent in the horizontal direction is B as shown in FIG. The case is further classified into two types as G2. Accordingly, there are four types of average values to be calculated: the average value of the R component is Ar, the average value of the B component is Ab, the average value of the G1 component is Ag1, and the average value of the G2 component is Ag2. For the pixels classified as G1 or G2, the variances Vg1 and Vg2 of the pixel values within each classification are also calculated (step S1).

  Next, it is determined whether or not the variances Vg1 and Vg2 are both below a certain threshold T (step S2).

  Here, if both are equal to or less than the threshold value T and it is determined that the block is relatively flat, the step difference estimation value d is set as the difference Ag1−Ag2 between the average values of the classifications G1 and G2 (step S3). ).

  In step S2, if at least one of the variances Vg1 and Vg2 exceeds the threshold value T, it is unknown whether the block is flat. Therefore, the difference between the R component and the B component in the block is determined by the average value Ar of the pixels from which the R component is obtained and the average value Ab of the pixels from which the B component is obtained. Evaluation is made based on whether or not the difference | Ar−Ab | is larger than a predetermined threshold value Trb (step S4).

  Here, when the difference | Ar−Ab | is larger than the threshold value Trb, the following processing is performed.

  First, in the present embodiment, a table obtained by the G step detection circuit 22 taking a subject of various colors and measuring how many steps d occur in the G component between lines in each image. Td is held in advance. This is because, even if the line correction circuit 5 performs the sensitivity correction between the lines related to the image data before the processing by the WB correction circuit 6, the pixel of the pixel from which the G component is obtained according to the even / odd of the line This is because there may be a step in the value.

  Then, the difference p1 (feature value) between the average value Ar of the pixels from which the R component is obtained and the average value Ab of the pixels from which the B component is obtained, and the average of the G component including both G1 and G2 For the two indexes of size p2 (feature amount), the G step amount d that can be generated is obtained by referring to a table such as d = Td (p1, p2). Here, when this table is referred to and an approximate estimated value d of the G step amount is obtained, this G step detection process is terminated (step S5).

  In step S4, if the difference | Ar−Ab | is equal to or smaller than the threshold value Trb, it is possible to estimate from the data in the block how much the G step is present even if it is generated. Since this is not possible, the G step amount d is set to −255 indicating impossibility of estimation, and the process ends (step S6).

  When the reading of the block of 5 × 5 pixels and the estimated value d of the step amount of the G component between the lines are obtained in this way, the G interpolation circuit 23 calculates the G component of the central pixel of the block. Then, the G image buffer 26 is written. Such processing will be described with reference to the flowchart of FIG.

  When the process is started, it is determined whether or not the classification m of the central pixel of the block is G1 in the classification shown in FIG. 4 (step S11). As illustrated in FIG. 4, the classification m is any one of four types R, G1, G2, and B depending on the type of the color component that is obtained and the type of the horizontal adjacent pixel.

  Here, when the classification m is G1, the value V of the G component of the central pixel is set as the pixel value U0 of the central pixel of the block (step S12), and the process ends.

  If the classification m of the central pixel of the block is not G1 in step S11, it is further determined whether or not the classification m is G2 in the classification shown in FIG. 4 (step S13).

  Here, when the classification m of the central pixel of the block is G2, it is determined whether or not the G step estimated value d is −255, that is, whether or not estimation is disabled (step S14).

  If the estimated G step value d is not −255, the G component value V of the central pixel is estimated as V = U0 + d based on the pixel value U0 of the block central pixel and the estimated G step value d (step S15). ),finish.

On the other hand, when the estimated G level difference d is −255 in step S14, the neighborhood of 3 × 3 pixels with respect to the central pixel (pixel having a pixel value U0) is taken as shown in FIG. The difference duj between the pixel values U1 to U4 of the pixels for which G components at the four corners in the vicinity are obtained and the pixel value U0 of the central pixel is calculated as follows.
duj = | U0−Uj |
Here, j is an integer of 1 to 4. Of these differences, for j giving the smallest difference, V = (U0 + Uj) / 2 is set as the value of the G component of the central pixel (step S16), and the process ends.

  In step S13, if the classification m of the central pixel of the block is not G2, it is determined whether or not the G step estimated value d is -255, that is, whether or not estimation is set (step S17). .

  If the estimated G level difference d is not −255, the pixel value Uk of the pixel k classified as G2 in the block is corrected as Uk = Uk + d (step S18).

  If the estimated G level difference d is −255 in step S17, the parameter α used in the calculation in step S20, which will be described later, is halved with respect to a preset value, and the step S20 The parameter β used in the above calculation is doubled with respect to a preset value (step S19).

When step S18 or step S19 is completed, the weight calculation circuit 24 performs weights Wj (j is an integer from 1 to 4) in the vertical and horizontal directions of the central pixel as shown in FIGS. 7 (A) to 7 (D). ) Is calculated as shown in Equation 10 below.
[Equation 10]
E1 = | B5-B7 |
+ Α × (| G4-G6 | -min (| G4-G6 |, | G2-G8 |))
+ Β
E2 = | B5-B9 |
+ Α × (| G2-G8 | -min (| G4-G6 |, | G2-G8 |))
+ Β
E3 = | B5-B3 |
+ Α × (| G4-G6 | -min (| G4-G6 |, | G2-G8 |))
+ Β
E4 = | B5-B1 |
+ Α × (| G2-G8 | -min (| G4-G6 |, | G2-G8 |))
+ Β
Wj = (1 / Ej) / (Σ (1 / Ek))
Α and β in Equation 10 are constants and take predetermined values or values changed in step S19. In the calculation of W, j is an integer from 1 to 4 as described above, and the sum of the denominators is an integer k from 1 to 4. This calculation formula is for the arrangement of FIGS. 7A to 7D, and is an example when the classification of the central pixel is B, but when the classification of the central pixel is R May be calculated by replacing B and R (step S20).

Next, the averaging circuit 25 weights the pixel values of the pixels in each direction using the weight Wj calculated in step S20, and calculates the final G component of the center pixel by the following equation (11) ( Step S21).
[Equation 11]
Vg = W1 * G6 + W2 * G8 + W3 * G4 + W4 * G2
Vb = W1 * B7 + W2 * B9 + W3 * B3 + W4 * B1
V = Vg + (Vb-B5) / 2
When such a series of processing is completed for all the pixels of the image in the single-plate image buffer 4, an image in which the lack of the G component is compensated for in the G image buffer 26 is obtained. Then, the RB interpolation circuit 27 operates to generate a color image having no missing color component at all pixels and store it in the color image buffer 8.

  That is, as with the G interpolation circuit 23, the RB interpolation circuit 27 first reads, for each pixel in the single-panel image buffer 4, a block of 5 × 5 pixels centered on the pixel. At the time of reading, the line correction circuit 5, the WB correction circuit 6, and the G level difference detection circuit 22 operate in the same manner as the G interpolation circuit 23, and the pixel value after the change of the block of 5 × 5 pixels is the RB interpolation. In addition to being obtained in an internal buffer (not shown) of the circuit 27, an estimated G step value d in the block is also obtained.

  Next, the RB interpolation circuit 27 reads the value of the G component at the pixel position corresponding to the read block from the G image buffer 26 and similarly stores it in an internal buffer (not shown).

  Thereafter, the RB interpolation circuit 27 obtains the missing color component of the central pixel by a process as shown in FIG. In the processing shown in FIG. 8, the value of the designated color component c for the central pixel of the block is calculated from the data in the internal buffer by weighted averaging.

  At this time, since the missing color component varies depending on the pixel position, the RB interpolation circuit 27 sets c = B when the classification of the central pixel of the block is R, and c = R when the classification is B. Alternatively, in the case of G2, the process shown in FIG. 8 is executed for each designation of c = R and c = B.

  Below, the process shown in FIG. 8 is demonstrated in order.

  When the processing is started, pixels P1 to Pn (n is an integer) from which the designated color component c is obtained are obtained in the block (step S31). FIG. 9B shows the result for c = R in the block arranged as shown in FIG. 9A. In this case, since the pixels P1 to P6 are obtained, n is 6.

  Next, it is determined whether or not the estimated G level difference value d in the block is −255, that is, estimation is impossible (step S32).

  Here, when the estimated G level difference d is −255, the parameter γ used in the calculation in step S34 described later is doubled in advance, and the parameter δ used in the calculation in step S35 described later. Is set to half of a preset value (step S33).

When the process of step S33 ends, or when the G step estimated value d is not −255 in step S32, the weight Wj assigned to the pixel Pj is calculated as shown in the following equation 12 (step S34). ).
[Equation 12]
Ej = | Gj−G0 | + γ
Wj = (1 / Ej) / (Σ (1 / Ek))
Here, j in the calculation of W is an integer from 1 to 4, and the sum of the denominators is from an integer k from 1 to 4. Γ is a constant and takes a predetermined value or a value changed in step S33.

Next, the value V of the missing color component c in the center pixel is calculated as shown in the following Equation 13. In Expression 13, Cj and Gj are values of the color components c and G obtained in the pixel Pj (step S35).
[Equation 13]
Vc = ΣWj × Cj (j is an integer from 1 to n)
Vg = G0-.SIGMA.Wj.times.Gj (j is an integer from 1 to n)
V = Vc + δ × Vg
Here, δ is a constant and takes a predetermined value or a value changed in step S33.

  When the processing so far is completed for all the pixels of the image in the single-plate image buffer 4, a color image in which the missing color component is restored in all pixels, that is, a so-called three-plate color image is stored in the color image buffer 8. can get.

  The subsequent operation is the same as that of the first embodiment described above, and the obtained color image is subjected to processing such as color conversion, edge enhancement, gradation conversion and the like by the image quality adjustment circuit 9, and is applied to a recording medium (not shown) by the recording circuit 10. This is recorded, and the imaging operation of the digital camera 21 is completed.

  In the above description, processing is performed by hardware (circuit or the like) inside the digital camera 21 that is an image processing apparatus. Such processing is performed on a computer such as a PC (personal computer). It is also possible to do this programmatically. 10 to 12 are flowcharts showing examples of software processing performed by the computer.

  This software processing processes the image stored in the memory InImg, that is, the image obtained only by the single-plate Bayer array CCD3 and in which only one color component is obtained for each pixel, and the memory OutImg 3 color images are output. However, the white balance coefficients Cr and Cb and the gain correction coefficients a and b between the lines are given as parameters.

  Below, each step of the flowchart of FIGS. 10-12 is demonstrated.

  First, FIG. 10 shows an outline of the flow of software processing.

  That is, when the process is started, the G component for all the pixels is first calculated in the middle of this process, so a memory Gimg for storing the calculation result is secured (step S41).

  Next, G interpolation software processing as will be described later with reference to FIG. 11 is executed (step S42).

  Further, RB interpolation software processing as will be described later with reference to FIG. 12 is executed (step S43).

  Thereafter, the secured memory Gimg is released (step S44), and this software process is terminated.

  Next, the details of the G interpolation software processing in step S42 will be described with reference to FIG.

  When the process is started, it is determined whether or not an unprocessed pixel exists in the memory InImg (step S51).

  Here, when an unprocessed pixel exists in the memory InImg, one of the unprocessed pixels is determined as a processing target pixel, and 5 × 5 pixels centered on the processing target pixel. Is read out (step S52).

  Then, line-to-line correction is performed by calculating Formula 8 and WB correction is performed by calculating Formula 9 (step S53).

  Further, the step difference estimation value d is calculated by performing the G step estimation process as shown in FIG. 3 (step S54).

  Based on the G step estimation result, G interpolation processing as shown in FIG. 5 is performed (step S55), and the processed G interpolation result V is recorded as a pixel value at a position corresponding to the processing target pixel in the memory Gimg. (Step S56), the process returns to Step S51.

  In step S51, if all the processing of the image in the memory InImg has been completed, this processing ends.

  Next, the details of the RB interpolation software processing in step S43 will be described with reference to FIG.

  When the process is started, it is determined whether or not an unprocessed pixel exists in the memory InImg (step S61).

  Here, when an unprocessed pixel exists in the memory InImg, one of the unprocessed pixels is determined as a processing target pixel, and 5 × 5 pixels centered on the processing target pixel. Are read from the memory InImg and the memory Gimg, respectively (step S62).

  Then, line-to-line correction is performed by calculating Formula 8 and WB correction is performed by calculating Formula 9 (step S63).

  Further, the step difference estimation value d is calculated by performing the G step estimation process as shown in FIG. 3 (step S64).

  Thereafter, it is determined whether or not the type of the central pixel of the block is R (step S65). If it is not R, RB interpolation processing as shown in FIG. 8 is performed after c = R, and the interpolation is performed. The result is recorded as the R component of the corresponding pixel in the memory OutImg (step S66).

  When step S66 ends or when the type of central pixel is R in step S65, it is determined whether or not the type of central pixel of the block is B (step S67).

  Here, when the type of the central pixel is B, the process directly returns to step S61. On the other hand, when the type of the central pixel is not B, RB interpolation as shown in FIG. Processing is performed, and the interpolation result is recorded as the B component of the corresponding pixel in the memory OutImg (step S68), and the process returns to step S61.

  In step S61, if all the processing of the image in the memory InImg has been completed, this processing ends.

  This embodiment can be modified in several ways. One preferred modification is one in which the G interpolation circuit 23 and the RB interpolation circuit 27 have a function of generating an image having a smaller number of pixels than the image held in the single-panel image buffer 4, and the functions are used. In this case, the processing of the G step detection circuit 22 is skipped. A mode of outputting with a reduced number of pixels is provided as standard in a digital camera. In this case, the G interpolation circuit 23 and the RB interpolation circuit 27 thin out pixels from the single-plate image buffer 4. Processing is performed while reading. Then, even if there is a difference in sensitivity between the G pixels, the influence on the processing result is reduced. Therefore, the processing of the G level difference detection circuit is stopped according to the thinning rate (that is, when the reduction rate is equal to or less than a predetermined threshold). Then, by outputting d = −255 as the step difference estimated value, it becomes possible to improve the processing efficiency and reduce the power consumption.

  In this embodiment, a single-plate imaging system is used as an example, but the same processing can be applied to a two-plate imaging system or a three-plate pixel shift imaging system.

  For example, in the case of a two-plate imaging system, the G component is obtained from all the pixels from one of the two imaging elements, and the R component and the B component are obtained from the other in a checkered pattern. At this time, the processing from step S1 to step S3 in FIG. 3 is changed to detect the difference in average value between the R pixels in the even lines and the R pixels in the odd lines, and the sensitivity difference according to the detected amount. What is necessary is just to correct | amend.

  Further, in the case of a three-plate pixel shifted imaging system, the G component is acquired by two of the three image sensors, the R and B components are acquired in a checkered pattern by the remaining one, and from these information Some types generate high-definition luminance components. Also in this case, the sensitivity correction of the R and B components obtained in a checkered pattern can be performed in the same manner as in the case of the two-plate imaging system.

  When the sensitivity correction is performed for such R and B components other than the G component, the above table is of course a table in which the feature amount for the designated color component is associated with the amount of sensitivity difference.

  Further, the weight calculation method in the weight calculation circuit 24 and the average circuit 25 as the weighted average means obtains the type of color component whose weight is determined to be large in the G step detection circuit 22 as the determination means. Any other method may be used as long as it is smaller than the pixel being used. For example, in addition to Expression 10, the weight calculation expression may be a function expression of another form for the pixel value difference between pixels in which the same kind of color component is obtained in the vicinity. However, when the G level difference detection circuit 22 determines that the sensitivity difference is large with respect to a specific color component, the function formula is changed so that the influence of the pixel value difference with respect to the color component becomes small. It is desirable. Further, the calculation of the weights for the pixels from which the type of color component determined to have a difference in sensitivity in the G level difference detection circuit 22 is finally calculated so that the weights between the G pixels approach equally. It is desirable that it can be realized by adjusting β in Expression 10, but other calculation expressions may be used as a matter of course.

  According to the second embodiment, as in the first embodiment described above, even if pixels having the same color component have different sensitivities, the correction processing is performed after the correction by the sensitivity correction device. Therefore, it is possible to prevent a difference in sensitivity from affecting the interpolated image and obtain a higher quality image.

  Moreover, according to the second embodiment, before performing the processing by the interpolation unit, the determination unit determines whether or not there is a large sensitivity difference between pixels from which the same color component is obtained. If it is determined, the processing of the interpolation means has been changed to processing that is not easily affected by the sensitivity difference, that is, the value of the color component that has been originally obtained is set to the predetermined reference sensitivity. Since the processing is corrected to a value that should be taken in the case of the above and used for the interpolation result, the image quality of the finally obtained image can be improved without significantly reducing the processing efficiency.

  Furthermore, since the weight at the time of the interpolation process is changed according to the presence or absence of the sensitivity difference, a natural process in which the image quality between the part judged to have a sensitivity difference and the part not so is not largely switched, It becomes possible to carry out with a simple processing circuit.

  At this time, since the calculation is performed so that the weights of pixels having different sensitivities are equal to each other, side effects can be reduced as a result.

  In this interpolation processing, the value of the missing color component is estimated using information on a plurality of color components. At this time, the ratio of use of information with respect to color components including pixels having different sensitivities is reduced. As a result, side effects such as the appearance of plaid can be made difficult to occur.

  Furthermore, the weight calculation is performed based on the difference of the same color pixel. However, in the calculation, the influence of the difference of the same color pixel having a low sensitivity and a difference in sensitivity is reduced. Side effects can be suppressed.

  In addition, there are various causes for the sensitivity difference. However, when the pixel value pattern in the vicinity of the target pixel satisfies a specific condition, the sensitivity difference is likely to occur. Is determined using the feature amount, and the higher the rate at which the pattern is determined, the greater the sensitivity difference. It is possible to process so that there is no.

  At that time, regarding the neighboring pixel value patterns that are likely to cause a sensitivity difference, the sensitivity difference generated with respect to the pattern and the feature amount of the pattern are stored as a table in association with each other, thereby determining the magnitude of the sensitivity difference more reliably. can do.

  In addition, using the fact that the positional relationship of the pixels causing the sensitivity difference is known, statistics are taken separately for each position, and the results are compared with each other, so it is certain whether there is a sensitivity difference. Can be judged. For example, in the case of a Bayer array, a sensitivity difference may occur in G between lines due to the leakage of charges from adjacent R and B, but such a step can be detected.

  Furthermore, when recording the results with a reduced number of pixels, the sensitivity difference determination process is skipped by taking advantage of the fact that side effects due to sensitivity differences will eventually become inconspicuous, thus improving the processing efficiency. I am letting.

  FIGS. 13 to 16 show a third embodiment of the present invention, FIG. 13 is a block diagram showing the configuration of a digital camera, and FIG. 14 is a diagram showing pixel numbers used when processing is performed by a correction circuit. FIG. 15 is a flowchart showing processing of the correction circuit, and FIG. 16 is a flowchart showing software processing performed by the computer.

  In the third embodiment, the same parts as those in the first and second embodiments are denoted by the same reference numerals, description thereof is omitted, and only different points will be mainly described.

  In the third embodiment, as in the first embodiment described above, the image processing apparatus is applied to a digital camera.

  In the digital camera 31 of the third embodiment, a correction circuit 32 as post-interpolation correction means is added between the color image buffer 8 and the image quality adjustment circuit 9 with respect to the digital camera 1 of the first embodiment. ing.

  Next, the operation of the digital camera 31 will be described mainly with respect to differences from the first embodiment.

  When the shutter button is pressed by the user, an optical image by the optical system 2 is picked up by the single-plate Bayer array CCD 3 as in the first embodiment, and an image in a single-plate state having only one kind of color component for each pixel. Is stored in the single-panel image buffer 4.

  Next, the line correction circuit 5 and the WB correction circuit 6 operate. The operation is the same as in the first embodiment, and the white balance coefficients Crg and Cbg are calculated and held while performing sensitivity correction between lines.

  Subsequently, the interpolation circuit 7 performs an interpolation process on each pixel in the single-plate image buffer 4, which is also the same as in the first embodiment, and a color image in which the missing color component of each pixel is compensated is Obtained in the color image buffer 8.

  Thereafter, the correction circuit 32 performs processing. This correction circuit 32 scans each pixel in the color image buffer 8 in order, reads the neighborhood of each pixel consisting of 3 × 3 pixels, and performs the process shown in FIG. The value is output to the image quality adjustment circuit 9.

  Below, each step of the flowchart of FIG. 15 is demonstrated.

When the process is started, the luminance value Y is calculated from the RGB components for each pixel in the vicinity of 3 × 3 pixels (step S71). This brightness value is calculated using the following standard definition formula:
Yj = 0.3 × Rj + 0.6 × Gj + 0.1 × Bj
Is used. Each pixel is assigned a number as shown in FIG. Below, the value regarding each pixel will be represented as j = 0-8 etc. using the horizontal letter of the pixel number.

Next, the luminance difference between the central pixel and the peripheral pixels (this indicates the correlation between the pixel value of the pixel of interest and the pixel values of neighboring pixels).
dYj = | Y0-Yj |
(Step S72). Here, j is an integer from 1 to 8.

And the following conditions (1), (2),
(1) dYk1 is the smallest among dY0 to dY8. (2) Integers k1 and k2 are searched such that dYk2 satisfies the second smallest among dY0 to dY8 (step S73).

  Next, it is checked whether or not dYk1 is larger than T1 and dYk2−dYk1 is smaller than T2 with respect to predetermined threshold values T1 and T2 (step S74). This condition is a condition indicating that, when the condition is satisfied, the probability that the central pixel is perceived as an isolated point is high.

Here, when the condition is satisfied and it is determined as an isolated point, the correction value (R0 ′, G0 ′, B0 ′) for the central pixel is calculated by the following formula 14,
[Formula 14]
R0 '= (Rk1 + Rk2) / 2
G0 '= (Gk1 + Gk2) / 2
B0 '= (Bk1 + Bk2) / 2
The calculation result is output and the process ends (step S75).

  On the other hand, if the condition is not satisfied in step S74, the RGB pixel value of the central pixel is output without correction (step S76), and the process ends.

  The corrected RGB values output to the image quality adjustment circuit 9 in this way are adjusted by the image quality adjustment circuit 9 and then recorded by the recording circuit 10 as in the first embodiment, and all the processes are completed.

  In the above description, processing is performed by hardware (circuit or the like) inside the digital camera 31 that is an image processing apparatus. Such processing is performed on a computer such as a PC (personal computer). It is also possible to do this programmatically.

  This software is the same as that described with reference to FIG. 10 to FIG. 12 in the above-described second embodiment, and each of the images stored in the memory InImg, that is, immediately after being obtained by the single-plate Bayer array CCD3. An image in which only one type of color component is obtained by a pixel is processed, and a three-color image is output to the memory OutImg. However, the white balance coefficients Cr and Cb and the gain correction coefficients a and b between the lines are given as parameters.

  Hereinafter, each step of the flowchart of FIG. 16 will be described. FIG. 16 shows the flow of software processing.

  When this processing is started, first, a memory tmpimg for holding an intermediate result in the interpolation processing is secured (step S81).

  Next, it is determined whether or not an unprocessed pixel exists in the memory InImg (step S82).

  Here, when there is an unprocessed pixel, one of the unprocessed pixels is determined as a processing target pixel, and a block of 5 × 5 pixels centering on the processing target pixel is stored in the memory. Read from InImg (step S83).

  Then, the inter-line correction is performed on the pixels in the block read from the memory InImg by performing the calculation according to the above formula 8, and the WB correction is performed by performing the calculation according to the above formula 9 (step S84).

  Subsequently, interpolation processing is performed by a known method to generate RGB three-color components for each pixel (step S85).

  Thereafter, the result of the interpolation process is recorded at the corresponding pixel position in tmpimg (step S86), and the process returns to step S82 to shift to the process of another pixel.

  On the other hand, if there is no unprocessed pixel in the memory InImg in step S82, it is further determined whether or not there is a pixel that has not undergone correction processing in the memory tmpimg (step S87).

  Here, if there is a pixel that has not been subjected to correction processing, neighboring data consisting of 3 × 3 pixels centered on the uncorrected pixel is read from the memory tmpimg (step S88), and 15 is performed (step S89), the result of this correction process is recorded in the corresponding pixel position of the memory OutImg (step S90), and the process returns to step S87 to return to the next unprocessed pixel. Transition to processing.

  In step S87, if there is no pixel that has not undergone correction processing in the memory tmpimg, the memory area secured as tmpimg is released (step S91), and the process ends.

  This embodiment can be modified in several ways. One preferred modification is one in which the interpolation circuit 7 has a function of generating an image having a smaller number of pixels than the image held in the single-plate image buffer 4, and the correction circuit 32 is used when the function is used. Is to skip the process. As described above, the mode for outputting with a reduced number of pixels is provided in a digital camera as standard, and in this case, the interpolation circuit 7 reads out the pixels from the single-plate image buffer 4 and reads them out. Will be processed. Then, even if there is a difference in sensitivity between the G pixels, the influence on the processing result is reduced. Therefore, the processing of the correction circuit 32 is stopped according to the thinning rate (that is, when the reduction rate is equal to or less than a predetermined threshold). By doing so, it is possible to improve processing efficiency and reduce power consumption without affecting image quality.

  In this embodiment, a single-plate imaging system is used as an example. However, for a two-plate imaging system and a three-plate pixel shifted imaging system, the same correction is performed after obtaining a color image by interpolation processing according to each method. Processing can be performed.

  According to the third embodiment, the effects similar to those of the first and second embodiments described above are obtained, and the side effects that may be caused by the sensitivity difference between the pixels are changed according to the sensitivity difference. However, since the side effect pattern that may occur after the interpolation process is directly corrected, the image quality can be improved more easily.

  In addition, by examining whether or not the neighboring pixel and the target pixel are in harmony with each other by correlation, the pixel value of the neighboring pixel having a high degree of harmony is mixed with the pixel value of the target pixel, so side effects are good. Can be reduced.

  Furthermore, when the result is recorded with the number of pixels reduced, the side effect caused by the sensitivity difference is finally inconspicuous, and the correction processing after interpolation is skipped, so the processing efficiency is improved. Can be improved.

  FIGS. 17 to 25 show a fourth embodiment of the present invention, FIG. 17 is a block diagram showing the configuration of a digital camera, FIG. 18 is a block diagram showing a configuration example of a step correction circuit, and FIG. FIG. 20 is a flowchart illustrating the process of the step correction circuit, FIG. 21 is a diagram illustrating an example of the vicinity when performing the process in the step correction circuit, and FIG. 22 is a function for obtaining the correction amount Δ. FIG. 23 is a block diagram showing another configuration example of the step correction circuit, FIG. 24 is a diagram for explaining the combination average calculated by the combination average calculation circuit, and FIG. 25 is performed by a computer. It is a flowchart which shows a software process.

  In the fourth embodiment, parts similar to those of the first to third embodiments are denoted by the same reference numerals, description thereof is omitted, and only different points will be mainly described.

  In the fourth embodiment, as in the first embodiment described above, the image processing apparatus is applied to a digital camera.

  As shown in FIG. 17, the digital camera 41 according to the fourth embodiment is provided with a step correction circuit 43 and a noise reduction circuit 44 instead of the line correction circuit 5 with respect to the digital camera 1 according to the first embodiment. It has become. Further, between the single-plate Bayer array CCD 3 and the single-plate image buffer 4, an analog video signal output from the single-plate Bayer array CCD 3 is gain-adjusted and then converted into a digital video signal. Although the D conversion circuit 42 is clearly provided, this is similarly provided in the digital cameras of the first to third embodiments described above, and is simply omitted from illustration. . The A / D conversion circuit 42, the step correction circuit 43, and the noise reduction circuit 44 are controlled by the control circuit 11.

  The step correction circuit 43 is for correcting the sensitivity difference of the G component with respect to the image stored in the single-plate image buffer 4.

  The noise reduction circuit 44 is for performing noise reduction processing on the signal output from the step correction circuit 43.

  More specifically, as shown in FIG. 18, the step correction circuit 43 includes a correction circuit 51, an average calculation circuit 52, a step amount estimation circuit 53, and a correction amount calculation circuit 54. .

  The average calculation circuit 52 reads image data in units of a predetermined size block from the single-plate image buffer 4 and calculates an average value of G components other than the target pixel.

  The step amount estimation circuit 53 estimates the amount of sensitivity difference by calculating the difference between the average value calculated by the average calculation circuit 52 and the G component value of the target pixel.

  The correction amount calculation circuit 54 calculates a feature amount from the block-unit image data read from the single-panel image buffer 4, obtains a threshold value based on the feature amount, and the threshold value and the step amount estimation circuit 53 Based on the estimated sensitivity difference amount of the G component, the sensitivity difference correction amount of the image data read out from the single-panel image buffer 4 in units of blocks is calculated.

  The correction circuit 51 corrects the sensitivity difference of the G component value of the pixel of interest based on the sensitivity difference correction amount calculated by the correction amount calculation circuit 54 and outputs it to the noise reduction circuit 44.

  Regarding the operation of such a digital camera 41, a different part from each Example mentioned above is demonstrated.

  The level difference correction circuit 43 reads a neighboring 3 × 3 block area for each pixel of the single-panel image buffer 4. Then, the pixel located at the center of the read 3 × 3 size block area is set as the target pixel. The step correction circuit 43 performs the step correction processing as described below only when the color component of the target pixel is the G component.

  Here, the G step will be described with reference to FIG. The single-plate Bayer array CCD3 has a color filter array as shown in FIG. 19A. However, due to the characteristics of the sensor, the G pixels (denoted as Gr) in the odd-numbered rows in the figure under certain shooting conditions. ) And G pixels (denoted as Gb) in even-numbered rows, there may be a difference in sensitivity even though the same G component acquisition is performed. That is, for example, even when a flat subject that captures an output value of “128” is obtained for all pixels, a pixel value of “180” is obtained from Gr, as shown in FIG. A pixel value of “100” is obtained from Gb, and there may be a difference between the pixel values of Gr and Gb. Such a difference is hereinafter referred to as a G step.

  The level difference correction circuit 43 corrects this G level difference along the flow shown in FIG. 20 and outputs a corrected pixel value.

  That is, when the process shown in FIG. 20 is started, the level difference correction circuit 43 first determines whether or not the central pixel of the read neighboring block is either Gr or Gb (S101).

Here, when the central pixel is either Gr or Gb, the average calculation circuit 52 is the G pixel G0 located at the four corners of the pixel array as shown in FIG. The average value Ag (feature value) of .about.G3 is calculated and output to the step amount estimation circuit 53.
[Equation 15]
Ag = (G0 + G1 + G2 + G3) / 4
The step amount estimation circuit 53 calculates the difference between the average value Ag calculated by the average calculation circuit 52 and the value of the central pixel G4 of the neighboring block as a step amount d as an amount of sensitivity difference as shown in the following Expression 16. The calculated step amount d is output to the correction amount calculation circuit 54 (S102).
[Equation 16]
d = Ag-G4
First, the correction amount calculation circuit 54 calculates an average value R ′ (feature amount) of R pixels and an average value B ′ (feature amount) of B pixels included in the neighboring block. The correction amount calculation circuit 54 refers to a prediction table (lookup table: LUT) stored in the internal ROM of the correction amount calculation circuit 54 using the average values R ′ and B ′ as indexes. Thus, a correction amount threshold F (amount of sensitivity difference) for performing appropriate step correction is obtained (S103). The prediction table includes an upper limit value of a step amount generated between the Gr pixel and the Gb pixel when shooting various subjects under various illumination conditions, surrounding R pixel values and B pixel values, These relationships are created in advance.

  Next, the correction amount calculation circuit 54 actually corrects the G step based on the correction amount threshold F calculated as described above and the step amount d estimated by the step amount estimation circuit 53. An appropriate correction amount Δ (sensitivity difference correction amount) is calculated (S104). This calculation is also performed by referring to a table stored in the internal ROM of the correction amount calculation circuit 54. The table stored in the internal ROM is, for example, a table created by discrete approximation of a function f (predetermined function) as shown in FIG. This function f (x; p) is a function of the independent variable x having p as a parameter. When | x | is less than or equal to p, f (x; p) = x, and | x | is equal to or greater than αp (threshold). It is a function that becomes 0 at the time. When | x | is larger than p and smaller than αp (threshold), the function shape is such that these are connected continuously (for example, connected by a straight line). Here, | x | indicates the absolute value of x, and α is a predetermined constant of 1 or more. The correction amount calculation circuit 54 obtains the correction amount Δ substantially by setting Δ = f (d; F) by referring to the table corresponding to the function f as x = d, p = F. The correction amount Δ is output to the correction circuit 51.

Subsequently, the correction circuit 51 subtracts half of the correction amount Δ obtained from the correction amount calculation circuit 54 from the central pixel value G4, as shown in the following Expression 17, to obtain a final step correction value G4 ′. To calculate
[Equation 17]
G4 ′ = G4-Δ / 2
Thereafter, the value G4 ′ is output as the central pixel value to the noise reduction circuit 44 (S105).

  If the subject is flat and the neighborhoods G0 to G3 in the block shown in FIG. 21 have the same value V1, and the central pixel G4 has the value V2, the step amount d calculated in S102 is V2-V1. When the absolute value of V2-V1 is equal to or smaller than the threshold value F calculated in S103, the correction amount Δ calculated in S105 matches V2-V1. Therefore, the value of G4 ′ calculated in S105 is V2− (V2−V1) / 2 = (V1 + V2) / 2. As described above, when the step amount is smaller than the threshold value F, the correction result is corrected so as to approach the average value of the pixel value of the Gr pixel and the pixel value of the Gb pixel. This is a feature of correction processing.

  If it is determined in S101 that the central pixel is neither Gr nor Gb, the pixel value of the central pixel is output to the noise reduction circuit 44 as it is (S106).

  Thus, when the process of S105 or S106 is performed, the G level difference correction process is terminated.

  When the above-described processing is performed by the step correction circuit 43, the pixel values after performing the G step correction on each pixel of the single-plate image buffer 4 are sequentially input to the noise reduction circuit 44. The noise reduction circuit 44 performs noise reduction processing due to the characteristics of the single-plate Bayer array CCD 3 and outputs the result to the WB correction circuit 6.

  Subsequent processing is the same as that in the first embodiment.

  In addition, many modifications can be considered for this embodiment. For example, the processing order of the WB correction circuit 6, the noise reduction circuit 44, and the step correction circuit 43 is not limited to that described above, and can be changed to an arbitrary order.

  As a modification for simplifying the circuit, in step S103 by the step correction circuit 43, the threshold value F may be set to a fixed value instead of being predicted based on the feature amount of the neighboring pixels. In this case, a fixed value is stored in the internal ROM of the correction amount calculation circuit 54 instead of the table. Depending on the characteristics of the single-plate Bayer array CCD 3, even if such a means is used, it is within the practical range.

  Alternatively, the step correction circuit 43 may be configured as shown in FIG. In the configuration example shown in FIG. 23, the level difference correction circuit 43 includes a correction circuit 51, a combination average calculation circuit 56, a difference selection circuit 57, and a correction amount calculation circuit 54. That is, the configuration shown in FIG. 23 is a configuration in which a combination average calculation circuit 56 and a difference selection circuit 57 are provided instead of the average calculation circuit 52 and the step amount estimation circuit 53 in the configuration shown in FIG. .

The combination average calculation circuit 56 includes a selector that selects a specific combination of G pixel pairs included in the vicinity of 3 × 3, and a circuit that calculates the average of each pair sequentially selected by the selector. Has been. The combination pair selected by the combination average calculation circuit 56 and the average value thereof are, for example, A0 to A5 (features) as shown in FIG. 24, that is, A0 to A5 as shown in the following Expression 18.
[Equation 18]
A0 = (G0 + G1) / 2
A1 = (G0 + G2) / 2
A2 = (G1 + G3) / 2
A3 = (G2 + G3) / 2
A4 = (G1 + G2) / 2
A5 = (G0 + G3) / 2
It becomes. The average values A0 to A5 calculated by the combination average calculation circuit 56 are output to the difference selection circuit 57.

  The difference selection circuit 57 calculates the difference Di = G4-Ai (i is an integer from 0 to 5) between the average values A0 to A5 and the central pixel value G4 received from the combination average calculation circuit 56, respectively. Di having the smallest absolute value is output to the correction amount calculation circuit 54 as an amount corresponding to the step amount d in S102 of FIG. The subsequent processing of the correction amount calculation circuit 54 and the processing of the correction circuit 51 are the same as described above.

  In the above description, processing is performed by hardware (circuit or the like) inside the digital camera 41 that is an image processing apparatus. However, such processing is performed on a computer such as a PC (personal computer). It is also possible to do this programmatically.

  This software performs development processing on a RAW data file obtained by converting an analog output from the single-plate Bayer array CCD 3 into a digital signal.

  Below, each step of the flowchart of FIG. 25 is demonstrated. FIG. 25 shows the flow of software processing.

  In performing this process, it is assumed that, for example, a first buffer and a second buffer are secured as working storage areas in the memory area of the PC.

  When this process is started, first, image data in a single plate state is read from the RAW data file and stored as a two-dimensional array in the first buffer (S111).

  Next, the first buffer is first scanned in the row direction, and then in the column direction to determine whether or not there is an unprocessed pixel in the image data (S112).

  Here, if there is an unprocessed pixel, the neighborhood 3 × 3 block of the selected unprocessed pixel is read from the first buffer (S113), and the step correction process as shown in FIG. 20 is performed (S114). .

  Then, the step correction result is written in the address corresponding to the unprocessed pixel in the second buffer secured separately from the first buffer (S115), and the process returns to S112 to make a determination for the next unprocessed pixel. .

  On the other hand, if it is determined in S112 that there is no unprocessed pixel in the first buffer, noise reduction processing is performed on the image in the second buffer, and the processing result is output to the first buffer (S116). ).

  Next, processing for correcting white balance (WB) is performed on the image in the first buffer, and the result is output again to the first buffer (S117).

  Further, the missing color component interpolation process is performed on the image of the first buffer, and the result is output to the second buffer (S118).

  Subsequently, image quality adjustment processing such as edge enhancement is performed on the image in the second buffer, and the result is output to the first buffer (S119).

  Then, the image in the first buffer is output to the designated file (S120), and this process ends.

  According to the fourth embodiment, as in the first to third embodiments, the difference in sensitivity between pixels from which the same kind of color component can be obtained is corrected before the missing pixel interpolation process is performed. Therefore, the interpolation of missing pixels can be performed satisfactorily while reducing the influence of the difference in sensitivity.

  In addition, since the degree of sensitivity difference between the target pixel and the pixel of the same color component as the target pixel is estimated, optimum sensitivity correction according to the sensitivity difference can be performed. At this time, since the degree of sensitivity difference is estimated based on the feature amount extracted from the data in the vicinity of the target pixel, optimal sensitivity correction according to the sensitivity difference can be performed.

  Since the correspondence relationship between the feature amount and the sensitivity difference amount is previously investigated and tabulated, high-precision sensitivity correction can be performed at high speed.

  Furthermore, since the sensitivity difference correction amount is calculated from the amount of sensitivity difference using a predetermined function, and the calculated sensitivity difference correction amount is added to the pixel value of the target pixel to perform sensitivity correction, the circuit configuration is simplified. In addition, sensitivity correction can be performed without causing much blur.

  At this time, since the sensitivity difference correction amount is calculated using a function that becomes 0 when the absolute value of the sensitivity difference amount is equal to or greater than a certain threshold value, a very large estimated value of sensitivity difference is obtained. The correction can be stopped. As a result, it is possible to appropriately prevent the edge from being blurred as it is corrected.

  In addition, since the feature amount is calculated from the pixel values of pixels of a color component different from the pixel of interest, it is possible to cope with a case where a sensitivity difference occurs due to factors of color components other than the color component of interest. Is possible.

  Furthermore, a difference that minimizes the absolute value of the difference between the N feature values and the pixel value of the target pixel is obtained by calculating N average values of pixel values of pixels of the same color component as the target pixel. Therefore, the amount of sensitivity difference can be estimated with higher accuracy.

  FIGS. 26 to 31 show Embodiment 5 of the present invention, FIG. 26 is a block diagram showing the configuration of a digital camera, FIG. 27 is a block diagram showing an example of the configuration of a step smoothing circuit, and FIG. FIG. 29 is a flowchart showing an example of a function for calculating the coefficient mixture ratio α, and FIG. 30 is a diagram for explaining the frequency characteristics of the filter coefficient set by the coefficient setting circuit. FIG. 31 is a block diagram showing another configuration example of the step smoothing circuit.

  In the fifth embodiment, parts similar to those in the first to fourth embodiments described above are denoted by the same reference numerals, description thereof is omitted, and only different points will be mainly described.

  In the fifth embodiment, the image processing apparatus is applied to a digital camera as in the first embodiment.

  As shown in FIG. 26, the digital camera 61 of the fifth embodiment is provided with a step smoothing circuit 62 instead of the step correction circuit 43 with respect to the digital camera 41 of the fourth embodiment. The step smoothing circuit 62 is controlled by the control circuit 11.

  More specifically, the step smoothing circuit 62 includes a filter circuit 71, a correction amount calculation circuit 72, and a coefficient setting circuit 73, as shown in FIG.

  The correction amount calculation circuit 72 calculates the average value R ′ (feature amount) of R pixels and B for the center 3 × 3 pixels of the image data read from the single-panel image buffer 4 in units of 7 × 7 pixels. An average value B ′ (feature amount) of the pixel is calculated, and an appropriate step correction amount threshold F (sensitivity) is calculated from these feature amounts with reference to a table stored in the internal ROM of the correction amount calculation circuit 72. The amount of difference) is obtained, and the coefficient mixture ratio α is calculated as a function of the threshold value F.

  The coefficient setting circuit 73 calculates a filter coefficient used for the calculation performed by the filter circuit 71 based on the coefficient mixture ratio α calculated by the correction amount calculation circuit 72.

  The filter circuit 71 performs a predetermined calculation on the image data read from the single-plate image buffer 4 using the filter coefficient calculated by the coefficient setting circuit 73 to smooth the G step.

  Such processing of the level difference smoothing circuit 62 will be described with reference to FIG. In FIG. 28, the same reference numerals are given to the portions that perform the same processing as in FIG. 20 of the fourth embodiment. Further, as described above, the step smoothing circuit 62 reads out a block area of 7 × 7 size for each pixel of the single-panel image buffer 4 and performs the following processing. Yes.

  When this process is started, first, the process of S101 is performed to check whether the central pixel is Gr or Gb.

  Here, when the center pixel is Gr or Gb, the correction amount calculation circuit 72 firstly calculates the average value R ′ of the R pixels and the average value B of the B pixels included in the center 3 × 3 pixels in the neighboring block. 'And calculate. Then, the correction amount calculation circuit 72 refers to the prediction table stored in the internal ROM using the calculated average values R ′ and B ′ as indexes, and obtains an appropriate threshold correction amount threshold value F (S <b> 131). This prediction table is the same as the prediction table described in S103 of FIG.

  Next, the correction amount calculation circuit 72 calculates a coefficient mixture ratio α for determining a filter to be applied by the subsequent filter circuit 71 as a function of the threshold value F of the step correction amount (S132). As an example of a preferable function, there is a function as shown in FIG. 29 in which α is 1 when F is 0, and α is 0 when F is a certain value T or more. The meaning of the coefficient mixture ratio will be described later.

  Subsequently, the coefficient setting circuit 73 calculates a filter coefficient used for the calculation performed by the filter circuit 71 based on the coefficient mixture ratio α calculated by the correction amount calculation circuit 72. That is, the filter circuit 71 first performs linear filtering on the 7 × 7 neighboring block read from the single-panel image buffer 4 after setting the data at the position where the G component is not obtained to 0. The value output from the coefficient setting circuit 73 is used as the filter coefficient.

  The coefficient setting circuit 73 holds the coefficients of two types of 7 × 7 filters in advance in order to set filter coefficients used in the filter circuit 71. One of the two types of filter coefficients is a coefficient P_i, j (i and j are integers from 1 to 7, respectively). In this coefficient P_i, j, only P_4,4 corresponding to the center of the filter is 1, and other coefficients are 0. The other one of the two types of filter coefficients is a coefficient Q_i, j (i and j are integers from 1 to 7, respectively). This coefficient Q_i, j is a coefficient whose sum is 2 and whose horizontal or vertical characteristics are set to be 0 at the Nyquist frequency as shown by a curve L0 shown by a solid line in FIG. It has become.

  Therefore, when the vicinity of 7 × 7 read from the single-panel image buffer 4 is filtered using the coefficient P_i, j, the value of the center pixel that is not corrected is output from the filter circuit 71. .

  On the other hand, when the 7 × 7 neighborhood read from the single-panel image buffer 4 is filtered using the coefficient Q_i, j, only the frequency component near the Nyquist frequency is suddenly dropped, and the other frequency components are not significantly affected. Smoothing that is not given is performed. When there is a step between the Gr pixel and the Gb pixel, the influence mainly appears near the Nyquist frequency. Therefore, as a result of filtering, the effect of the step is smoothed and removed, but the G component value of the central pixel is obtained such that the blur of the edge portion does not occur so much.

The coefficient setting circuit 73 newly mixes the coefficient P_i, j and the coefficient Q_i, j as described above by using the coefficient mixture ratio α acquired from the correction amount calculation circuit 72 as shown in Equation 19, thereby obtaining a new value. A simple filter coefficient C_i, j is generated and output to the filter circuit 71 (S133).
[Equation 19]
C_i, j = α × P_i, j + (1−α) × Q_i, j
According to the equation 19 and the coefficient mixture ratio α as shown in FIG. 29, when a large step is generated in the neighboring block, the characteristic close to the coefficient Q_i, j from which the step is removed. A filter coefficient such as a curve L1 indicated by a dotted line in FIG. 30 is output to the filter circuit 71. On the other hand, when almost no step is generated in the neighboring block, a filter coefficient such as a curve L2 indicated by a one-dot chain line in FIG. 30 having characteristics close to the coefficient P_i, j is output to the filter circuit 71. By mixing the filter coefficients in this way, it is possible to always perform processing in which band degradation and step correction are optimally balanced.

こうしてフィルタ回路７１により計算されたフィルタリング結果ｖが、ノイズ低減回路４４へ出力される。 As described above, the filter circuit 71 performs linear filtering on neighboring blocks based on the filter coefficient C_i, j output from the coefficient setting circuit 73. Specifically, the following processing is performed. Here, the pixel value at the position (i, j) (i, j is an integer from 1 to 7) when the coordinate of the upper left corner of the 7 × 7 neighboring block is (1, 1) is x_i, j. . First, the pixel value x_i, j at a position where the G component is not obtained is set to 0. Thereafter, the filtering result v is calculated using the following Expression 20.
[Equation 20]

Thus, the filtering result v calculated by the filter circuit 71 is output to the noise reduction circuit 44.

  In this embodiment, various modifications can be made. For example, after mixing the coefficients, instead of filtering using the mixed coefficients, filtering may be performed using two kinds of coefficients, and the filtered results may be mixed. FIG. 31 shows a configuration example of a circuit for performing such processing.

  That is, the step smoothing circuit 62 shown in FIG. 31 is different from the step smoothing circuit 62 shown in FIG. 27 in that two filter circuits 71 a and 71 b and a mixing circuit are used instead of the filter circuit 71 and the coefficient setting circuit 73. 75, and the output of the correction amount calculation circuit 72 is connected to the mixing circuit 75.

  The filter circuit 71a performs filtering by the coefficient P_i, j and outputs a result V1.

  The filter circuit 71b performs filtering using the coefficient Q_i, j and outputs a result V2.

The mixing circuit 75 receives the coefficient mixture ratio α from the correction amount calculation circuit 72, and finally uses the output V1 from the filter circuit 71a and the output V2 from the filter circuit 71b to calculate the final value by the following equation (21). The result V3 is calculated, and the calculated result V3 is output to the noise reduction circuit 44.
[Equation 21]
V3 = α × V1 + (1−α) × V2
Further, in any of the configuration shown in FIG. 27 and the configuration shown in FIG. 31, the number of filters to be mixed is not limited to two, and may be three or more. In this case, the correction amount calculation circuit 72 calculates a plurality of mixture ratios as a function of the threshold value F of the step correction amount.

  As a simpler circuit, in the configuration shown in FIG. 31, it is also possible to provide a switching circuit instead of the mixing circuit and switch the outputs of the filter circuit 71a and the filter circuit 71b in accordance with the coefficient mixing ratio α. It is.

  In addition, in the process of FIG. 25, as the level difference correction process of S114, instead of performing the process as shown in FIG. 20, the process shown in FIG. Processing can be realized.

  According to the fifth embodiment, the same effect as that of the fourth embodiment described above can be obtained, and a linear filter can be applied to a pixel having the same color component as the target pixel by a filter coefficient set based on the amount of sensitivity difference. As a result, the sensitivity difference can be corrected more reliably even when the estimation of the sensitivity difference is wrong, compared to the case where the sensitivity difference correction amount is added to the pixel value.

  In addition, this invention is not limited to the Example mentioned above, Of course, a various deformation | transformation and application are possible within the range which does not deviate from the main point of invention.

  The present invention relates to an image processing device that processes an image output from a single-plate image pickup system, a two-plate image pickup system, or a three-plate pixel shift image pickup system, and generates a color digital image having three color component values for each pixel. It can utilize suitably for a processing program.

1, 21, 31, 41, 61 ... Digital camera 2 ... Optical system 3 ... Single plate Bayer array CCD
4 ... Single image buffer 5 ... Line correction circuit (sensitivity correction means)
6 ... WB correction circuit 7 ... Interpolation circuit (interpolation means)
8 ... Color image buffer 9 ... Image quality adjustment circuit 10 ... Recording circuit 11 ... Control circuit 22 ... G step detection circuit (determination means, feature amount calculation means)
23 ... G interpolation circuit (interpolation means)
24 ... Weight calculation circuit (weighted averaging means)
25 ... Average circuit (weighted average means)
26: G image buffer 27 ... RB interpolation circuit (interpolation means)
32. Correction circuit (correction means after interpolation)
42 ... A / D circuit 43 ... Step correction circuit (sensitivity correction means)
44 ... Noise reduction circuit 51 ... Correction circuit 52 ... Average calculation circuit (feature calculation means)
53. Step amount estimation circuit (estimating means)
54 ... Correction amount calculating circuit (estimating means, feature calculating means)
56. Combination average calculation circuit (feature calculation means)
57 ... Difference selection circuit 62 ... Step smoothing circuit (sensitivity correction means)
71, 71a, 71b ... Filter circuit (filter means)
72. Correction amount calculation circuit (estimation means, feature calculation means)
73 ... Coefficient setting circuit (coefficient setting means)
75 ... Mixed circuit

Claims (12)

An image processing apparatus that processes an image obtained by capturing an optical image with a single-plate imaging system, a two-plate imaging system, or a three-plate pixel shifting imaging system and lacking one or more types of color components for each pixel Because
A determination unit that determines, for a specified type of color component in a predetermined vicinity of the pixel of interest, a sensitivity difference between pixels from which the color component is obtained;
Interpolating means that compensates for missing color components in the pixel of interest and that can change the processing content according to the determination result by the determining means when performing the processing;
An image processing apparatus comprising:   The interpolation means determines a reference pixel from among pixels in which the color component is obtained in the vicinity of the color component obtained from the pixel of interest, and sets the sensitivity of the pixel of interest to the color component. The pixel value of the color component that should be taken by the pixel of interest when matched with the sensitivity of the reference pixel to the color component is calculated as a correction value, and depending on the sensitivity difference determined by the determination unit, The image processing apparatus according to claim 1, wherein the pixel value of the color component of the target pixel is changed to the correction value.   The interpolation means includes weighted average means for weighted average of pixel values in the predetermined neighborhood, and the weighted average means changes the weight according to the determination result of the determination means. The image processing apparatus according to claim 1, wherein the pixel value of the target pixel is calculated by weighted averaging.   The weighted averaging means is characterized in that the weight is changed so as to be small with respect to a pixel from which a kind of color component determined to have a large sensitivity difference in the determining means. The image processing apparatus according to claim 3.   The weighted average means calculates the weight as a function expression of a pixel value difference between pixels in which the same kind of color component is obtained in the vicinity, and the weight is averaged by the determination means. 4. The method according to claim 3, wherein when it is determined that the sensitivity difference is large, the function formula is changed so that the influence of the pixel value difference on the color component is reduced. Image processing device.   The weighted average unit is configured to change the weights so that the pixels are evenly approached with respect to pixels from which the color component of the type determined to have a sensitivity difference is obtained in the determination unit. The image processing apparatus according to claim 3.   The determination unit includes a feature amount calculation unit that calculates a predetermined feature amount from pixel values within a predetermined vicinity of the pixel of interest, and the magnitude of the sensitivity difference is determined based on the size of the feature amount. The image processing apparatus according to claim 1, wherein:   The determination unit further includes a table in which the feature amount related to the specified color component is associated with the amount of the sensitivity difference, and the sensitivity difference is referred to by referring to the table from the feature amount. The image processing apparatus according to claim 7, wherein the size of the image processing apparatus is determined.   The determination means classifies pixels for which a designated color component is obtained within the vicinity by a predetermined method, and calculates the feature amount based on an average value and a variation amount of pixel values in each classification. The image processing apparatus according to claim 7, wherein the image processing apparatus is an image processing apparatus.   The determination means includes an average value of pixel values of pixels in which the designated color component is obtained and a pixel value of pixels in which a predetermined color component different from the designated color component is obtained in the vicinity. 9. The image processing apparatus according to claim 7, wherein the feature amount is calculated based on an average value. The interpolation means is configured to generate a color image obtained by reducing the image size obtained by the imaging system at a predetermined reduction rate,
The determination means does not perform a sensitivity difference determination when the interpolation means generates a reduced color image when the reduction ratio is equal to or less than a predetermined threshold. The image processing apparatus according to 1. Let a computer process an image in which one or more color components are missing for each pixel, which is obtained by capturing an optical image with a single-plate imaging system, a two-plate imaging system, or a three-plate pixel shifting imaging system. An image processing program for
The computer
Determination means for determining the magnitude of a sensitivity difference between pixels for which the color component is obtained for a specified type of color component in a predetermined vicinity of the pixel of interest;
Interpolating means that compensates for missing color components in the pixel of interest and that can change the processing content according to the determination result by the determining means when performing the processing;
An image processing program that functions as an image processing program.

Images