JP2009010847A - Color component interpolation apparatus, and method thereof - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a color component interpolation apparatus which prevents color slippage to flexibly deal with a pixel array change or color filter change and to also deal with digital zooming. <P>SOLUTION: A color component interpolation apparatus includes: an image pickup data input section 101 for capturing image pickup data consisting of positions, colors and a signal amount of a plurality of pixels constituting an imaging element; a parameter estimation section 102 for estimating parameters of two or more color component functions with a matched derived function from the image pickup data; and a color component estimation section 103 for determining color components at respective points using the color component functions, wherein at least one unknown parameter of the color component functions is estimated from the image pickup data of a pixel at an attention position and the image pickup data of a plurality of pixels around the attention position in such a manner as to match derived functions of the color component functions with each other for each color component, and the color component function for each color component and the estimated parameter for each color component are used to estimate a signal amount of the missed color component at the attention position. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a color component interpolation apparatus and method used for a digital camera or the like.

  A small digital camera obtains image data by narrowing the image sensor to one and applying a color filter such as a Bayer array shown in FIG. 7 to each pixel of the element. Since this imaging data holds only one color information for each point, a demosaicing process for generating a color image by interpolating the color components of neighboring points to compensate for the missing color information is necessary.

  The demosaicing method disclosed in Patent Document 1 suppresses color misregistration by correlating the color components of the output image.

  However, this is a method dedicated to the Bayer array, and cannot cope with image data of a color filter (see FIG. 8) as disclosed in Patent Document 2, for example.

  In addition, this method can only estimate the color component at the point where the pixel of the image sensor is present. For example, when obtaining a color image in which pixels are arranged in a square lattice from the image sensor as shown in FIGS. It can't cope with a kind of digital zoom.

  On the other hand, the interpolation method disclosed in Non-Patent Document 1 can easily obtain an interpolation filter corresponding to the arrangement even if the pixel arrangement is changed.

However, it cannot be applied as it is to an image having a plurality of color components. Although this method can be applied to each color component to generate a color image, the position of the edge for each color component cannot be aligned, and color misregistration occurs.
US Pat. No. 5,629,734 JP 2006-211631 A H. Takeda et al., “Kernel Regression for Image Processing and Reconstruction,” Trans. On IP, Vol.16, No.2,2007

  As described above, the conventional demosaicing method has the following problems.

  The first problem cannot flexibly cope with a change in pixel arrangement or a change in color filter.

  The second problem is that the color component of the pixel gap cannot be estimated.

  A third problem is that when a conventional interpolation method is individually applied to color components, color misregistration occurs.

  Accordingly, the present invention has been made to solve the above-described problems, and is a color component that suppresses color misregistration, flexibly responds to changes in pixel arrangement and color filters, and is compatible with digital zoom. It is an object of the present invention to provide an interpolation apparatus and method.

  The present invention relates to an image data input unit that acquires image data having a pixel position indicating a position of each pixel on the image sensor, a color of each pixel, and a signal amount of each pixel, and an arbitrary image on the image sensor. The respective parameters of the plurality of color component functions for estimating the signal amounts of the plurality of colors at the position of interest are arranged around the position of interest so that the derivatives of the plurality of color component functions are similar to each other. A parameter estimation unit that estimates using imaging data of a plurality of pixels; and a color component estimation unit that estimates a signal amount of an arbitrary color at the target position using the color component function and the parameter. This is a color component interpolation device.

  According to the present invention, the color shift of the output image can be suppressed, it is possible to flexibly cope with the change in the pixel arrangement and the change in the color filter, and it is also possible to deal with the digital zoom.

  Hereinafter, a color component interpolation apparatus 100 according to an embodiment of the present invention will be described with reference to the drawings.

(First embodiment)
The color component interpolation apparatus 100 according to the first embodiment of the present invention will be described below with reference to FIGS. 1 to 2, 11, and 12.

(1) Configuration of Color Component Interpolation Device 100 FIG. 1 is a block diagram showing a color component interpolation device 100 according to this embodiment.

  The color component interpolation apparatus 100 includes an imaging data input unit 101 that acquires imaging data composed of a combination of positions, colors, and signal amounts of a plurality of pixels that form a single-plate imaging device, and two that have derivatives that match the imaging data. A parameter estimation unit 102 that estimates the parameters of the above color component function and a color component estimation unit 103 that obtains a color component at each point using the color component function are provided.

  The color component interpolation apparatus 100 can also be realized by using, for example, a general-purpose computer apparatus as basic hardware. That is, it can be realized by causing a processor mounted on a computer to execute a program. At this time, the color component interpolation apparatus 100 may be realized by installing the above program in a computer in advance, or may be stored in a storage medium such as a CD-ROM, or distributed through the network. Then, this program may be realized by appropriately installing it in a computer. Further, it can be realized by appropriately using a memory, a hard disk or a storage medium such as a CD-R, a CD-RW, a DVD-RAM, a DVD-R, or the like built in or externally installed in a computer.

(2) Operation of Color Component Interpolation Device 100 Next, the operation of the color component interpolation device 100 according to the present embodiment will be described using FIG. 1 and FIG. FIG. 2 is a flowchart showing the operation of the color component interpolation apparatus 100 according to this embodiment.

  In the present embodiment, a method for demosaicing the Bayer array (see FIG. 7) will be described. A method for coping with the change of the color filter will be described later.

(2-1) Definition of variables Variables are defined for the following explanation.

  First, in order to represent a point on a single-plate image sensor, a plane composed of an x-axis and a y-axis is defined.

On this plane, the point of the target position for which the color component (that is, color) is to be estimated is expressed as p _i * = (x _i *, y _i *) (where i = 0,..., M). .

The pixel used for estimating the color component is p _i = (x _i , y _i ) (where i = 0,..., N), and the signal amount obtained from p _i is s _i .

(2-2) Imaging data input unit 101
First, the imaging data input unit 101 acquires imaging data composed of combinations of positions, colors, and signal amounts of a plurality of pixels constituting a single-plate imaging element (step 1).

  For example, when performing demosaicing processing for estimating R (red) and G (green) components of pixels having a B (blue) color filter in the Bayer array of FIG. Image data of the pixels 1100 to 1108 in FIG. In the drawing, since RGB colors cannot be expressed, the R range is represented by right-down hatching, the G range is not hatched, and the B range is represented by left-down hatching.

When the process of estimating the missing color component of the block center p ₀ * = p ₄ is performed while shifting the 3 × 3 pixel block as shown in FIG. 11, a color image is obtained.

  In the following, the processing after the block and its internal data are obtained will be described.

(2-3) Parameter estimation unit 102
Next, the parameter estimation unit 102 estimates the parameters of the color component functions f _r , f _g , and f _b that describe the RGB components near the central pixel of the block, that is, the pixel 1104 (step 2).

First, the color component functions f _r , f _g , and f _b are modeled by a quadratic polynomial based on the position coordinates x and y as shown in Expression (1).

In this modeling, the derivatives obtained by differentiating the color component functions f _r , f _g , and f _b with x and y are all equal. With this modeling, the RGB components change with correlation in the vicinity of the pixel 1104, and color shift can be suppressed.

Next, an unknown parameter p = (β _r , β _g , β _b , β ₁ , β ₂ , β ₃ , β ₄ , β ₅ ) _t is estimated using imaging data of neighboring pixels 1100 to 1108. To do. Note that t represents transposition of a vector.

If each color component changes according to the model of Expression (1), for example, in the pixel 1100, it is considered that s ₀ = f _r (x ₀ , y ₀ ) holds. Considering the other points in the same way, if the color component functions f _r , f _g , and f _b accurately represent the neighboring color components, the color component model error E ₀ (p) shown in Expression (2) Is considered to be smaller.

  The left three columns of the matrix W in Equation (2) correspond to the color of the color filter.

For example, since the first row of W corresponds to pixel p ₀ in the upper left in FIG. 11, the elements of the head corresponding to a R component beta _r has a 1, the second corresponding to the G component The third element corresponding to the element and the B component is 0.

  W in Equation (2) is for a 3 × 3 pattern centered on B as shown in FIG. 11, but in the case of patterns such as 1201 to 1203 in FIG. This can be done by changing the position of “1”.

For example, in the case of the pattern 1201 in FIG. 12, a pixel having an R color filter (hereinafter referred to as “R pixel”) and a B pixel are interchanged as compared with the case of FIG. The matrix W in this case is Equation (3) in which the first and third columns of W in Equation (2) are interchanged.

(2-3-1) Parameter estimation method 1
The parameter estimation method 1 will be described.

As the parameter estimation method 1, the parameter p ′ of the function that minimizes E ₀ can be obtained as p ′ = W ⁺ s using the pseudo inverse matrix W ⁺ of the matrix W.

(2-3-2) Parameter estimation method 2
The parameter estimation method 2 will be described.

In the parameter estimation method 1, calculating W ⁺ at each point requires enormous calculation cost. However, in the case of the Bayer array, the arrangement of the color filters in the block has only the four patterns shown in FIGS. There are only 4 patterns. Therefore, if W ⁺ for all patterns is calculated in advance and stored in the memory, the calculation cost can be greatly reduced.

(2-3-3) Parameter estimation method 3
The parameter estimation method 3 will be described.

When the method that requires the calculation cost of the pseudo inverse matrix such as the parameter estimation unit 1 cannot be used, and when the memory that holds a plurality of W ⁺ cannot be used as in the parameter estimation method 2, the steepest descent method, the conjugate gradient method, etc. An approximate solution of p ′ is obtained using a method in which the initial value of is sequentially updated to approach the true solution.

  When this method is used, the calculation amount of the entire color component interpolation apparatus 1000 can be adjusted by changing the number of updates.

(2-3) Output image generation unit 103
Finally, the output image generation unit 103 estimates the signal amount of the color component at each point of the output image using the estimated color component functions f _r , f _g , and f _b (step 3). The signal amount to be estimated may be a signal amount in an output image different from the signal amount expressed by the voltage output from the image sensor (that is, the signal amount of the imaging data).

In the case of simple demosaicing processing, the RGB component at the point 1104 may be known. The signal amount of the RGB component at the point 1104 is obtained by substituting x = x ₄ , y = y ₄ into the equation (1), and f _r (x ₄ , y ₄ ) = β _r , f _g (x ₄ , y ₄ ). = Β _g , f _b (x ₄ , y ₄ ) = β _b

(2-4) Digital Zoom When digital zoom is performed simultaneously, each point of the image after zooming is associated with the image sensor, and the color component is estimated using Expression (1).

For example, when performing double digital zoom using FIG. 11, pixels of the image sensor are arranged as indicated by points 1100 to 1108 represented by circles in FIG. 14, and each pixel of the image after zooming is imaged. When correlated on the element, it is represented by a white square mark. At this time, in order to output a color image, four pixel points 1400 to 1403 (p _i * = (x _i *, y _i *)) near p ₄ (where i = 0,..., 4). )) RGB components need to be obtained.

In this case, for example, each RGB component of the point 1400 (p ₀ * = (x ₀ *, y ₀ *) = (x ₄ −0.25, y ₄ −0.25)) uses the expression (1). F _r (x ₀ *, y ₀ *), f _g (x ₀ *, y ₀ *), and f _b (x ₀ *, y ₀ *).

(3) Effect As described above, according to the color component interpolation device 100 according to the present embodiment, the color component of the gap portion of the pixels constituting the image sensor is obtained by using the color component function estimated using the image data. It is possible to estimate.

(4) Modification 1
In the above description, the color component functions f _r (x, y), f _g (x, y), and f _b (x, y) are modeled by a quadratic polynomial of x and y. However, functions other than quadratic polynomials can be used as the function form. For example, the order of the function can be arbitrarily increased, and the method is shown in Non-Patent Document 1.

  However, the order is not limited to an unlimited number. If the order is increased, it becomes easy to express a sharp rise of the edge, but it may be weak against noise. Therefore, an appropriate one is selected in the range of about 2 to 5 while viewing the image quality.

(5) Modification 2
The block shape used for demosaicing can be arbitrarily changed.

  In the above description, 3 × 3 blocks are used for demosaicing, but in the present embodiment, any block shape can be used.

For example, consider the case where the G pixel 1109 existing under the pixel 1106 is added in the demosaicing process of FIG. In this case, the signal _{s 9} obtained from the element to the pixel 1109 of the vector s of formula (2) added as shown in Equation (4), the element in row vectors of the matrix W (0,1,0, _{(x 9 -x 4), (y 9 -y} 4), may be added to ^{_{(x 9 -x 4) 2,}} (x 9 -x 4) (y 9 -y 4), (y 9 -y 4) 2) .

  However, it is desirable that the number of pixels used in the demosaicing process is larger than the number of function parameters.

The pixel value of a point distant from the center of the block p ₄ color component function f _r, f _g, since more likely to deviate from f _b, the shape of the block is circular or square, shapes such as a rhombus is desirable.

(Second Embodiment)
Hereinafter, a color component interpolation apparatus 100 according to the second embodiment of the present invention will be described with reference to FIGS. 3, 4, 11, 12, 17, and 18.

(1) Configuration of Color Component Interpolation Device 100 FIG. 3 is a block diagram showing the color component interpolation device 100 according to this embodiment.

  The color component interpolation apparatus 100 according to the present embodiment includes an imaging data input unit 101 that acquires imaging data including a combination of positions, colors, and signal amounts of a plurality of pixels that constitute an imaging element, pixels that constitute the imaging element, and Color component estimation weight determination unit 104 for assigning a positive real color component estimation weight to the imaging data, imaging data obtained from imaging data input unit 101 and color component estimation weight determination unit 104 A parameter estimation unit 105 that estimates the parameters of two or more color component functions with the same derivatives from the color component estimation weights, and a color component estimation unit that estimates color components at each point on the plane using the color component function 103.

(2) Operation of Color Component Interpolation Device 100 Next, the operation of the color component interpolation device 100 according to the present embodiment will be described using FIG. 3 and FIG.

  FIG. 4 is a flowchart showing the operation of the color component interpolation apparatus 100 according to this embodiment. In the following description, an example of Bayer array demosaicing using a 3 × 3 pixel block is used as in the first embodiment.

(2-1) Imaging data input unit 101
First, the imaging data input unit 101 acquires imaging data including a combination of positions, colors, and signal amounts of a plurality of pixels constituting the imaging element (step 1).

  The operation of this part is the same as that of the first embodiment.

(2-2) Color Component Estimation Weight Determination Unit 104
Next, as shown in FIG. 11, the color component estimation weight determination unit 104 assigns color component estimation weights K _{0 to} K ₈ that are positive real numbers to the pixels 1100 to 1108 (step 2).

  This weight is used for function parameter estimation, which is a process at a later stage, and a pixel given a large weight greatly contributes to estimation of a color component. Since there are three ways to assign weights, they will be explained in order.

(2-2-1) Color Component Estimation Weight Determination Method 1
First, it can set so that it may become so small that it leaves | separates from a center like Formula (5).

  α is a value of about 0.1 to 1.0, and is adjusted manually by looking at the image quality of the output image. This makes it possible to perform parameter estimation that places more importance on the block center.

(2-2-2) Color Component Estimation Weight Determination Method 2
However, for example, when a 45-degree oblique texture such as texture 1501 as shown in FIG. 15 is input, the edge is likely to deviate from the model of equation (1), so the weight is as in equation (5), etc. An ellipse 1502 is preferable to a square weight. Such a color component estimation weight K _i can be determined using the imaging data. Non-Patent Document 1 discloses a method for obtaining a two-dimensional Gaussian distribution weight from the amount of change in pixel value at each point of a grayscale image (hereinafter referred to as “conventional weight determination method”). The color component estimation weight determination method 2 is a further advancement of the conventional weight determination method.

In the conventional weight determination method, the amount of change dx _{i in} the x-axis direction of the pixel value at each point p _i = (x _i , y _i ) (i = 0,..., N) in the block w and the y-axis direction Using the variation dy _i and the positive real number H, a weight as shown in Equation (6) is given.

In the color component estimation weight determination method 2, for example, at the point p ₀ in FIG. 11, the same color in the vertical direction or the horizontal direction such as dx ₀ = s ₂ −s ₀ and dy ₀ = s ₆ −s _0. The amount of change can be defined using the difference in the signal amount of the pixel having the filter, and the weight can be determined using Equation (6).

  H is a positive real number arbitrarily set by the user and is adjusted while viewing the image quality. For calculation of C, for example, only pixels having a G color filter are used, so that the amount of calculation can be reduced.

Non-Patent Document 1 discloses that C in equation (6) can be expanded as in equation (7) using singular value decomposition.

  In Expression (7), θ represents the edge direction, and σ becomes a larger value as the edge strength increases.

(2-2-3) Color component estimation weight determination method 3
In the color component estimation weight determination method 2, one or more pixels having the same color filter must exist in the vertical direction and the horizontal direction in order to obtain a spatial differential amount in a certain pixel. Therefore, the amount of change cannot be taken from the G pixels 1101, 1103, 1105, and 1107 in FIG.

  Further, even when the block size is increased and the amount of change is taken from the G pixel, only the pixel in the vertical direction or the horizontal direction is used, and information on the pixel at the closest distance in the oblique direction is not used.

  Therefore, the color component estimation weight determination method 3 shows a method of obtaining the differential amount for each point using three pixels having the same color filter that do not exist on the same straight line.

Image data as shown in FIG. 11 is obtained, three pixels 1101, 1103, and 1107 having the same color filter as shown in FIG. 17 are selected, and the vector of the amount of change in f _g at the point 1103 is d ₃ = (dx ₃ , A case where dy ₃ ) ^t is obtained will be described as an example.

Consider a three-dimensional space (see FIG. 18) defined by using the s-axis corresponding to the signal amount in addition to the x and y axes representing the pixel positions used in FIG. When the pixel 1101 has the signal amount s ₁ , it can be considered that a point 1801 (x ₁ , y ₁ , s ₁ ) exists at the position of the height s ₁ above the pixel 1101 in the space of FIG. Considering similarly, a triangle 1800 passing through a point 1803 (x ₃ , y ₃ , s ₃ ), a point 1807 (x ₇ , y ₇ , s ₇ ), and points 1801 to 1803 can be considered. The inclination of the x-axis and y-axis direction of this plane is d _3.

Of the three selected pixels, a vector connecting 1103 and 1101 is v _31, and a vector connecting 1103 and 1107 is v ₃₇ . If the length of the vector v _ij is l _ij and the angle formed by the vector v _ij with respect to the x axis is θ _ij , then v _ij = (l _ij cos θ _ij , l _ij sin θ _ij ) ^t . Using these, d ₃ is

  It becomes. By using Expression (8), if there are at least three pixels having the same color filter in the block, the differential amount of the color component in each pixel can be obtained.

  There are many possible methods for selecting three pixels, but there are methods that use all possible combinations, such as using a triangle obtained by creating a Delaunay diagram that connects the centers of pixels with the same color filter inside the block. It is done.

(3) Parameter estimation unit 105
Next, the parameter estimation unit 105 minimizes the color component model error E ₁ (p) = | Ks−KWp | ² defined using the matrix K composed of the weights K _i in FIG. The parameter p to be converted is obtained.

When the color component estimation weight determination method 1 is used, the signal amount obtained at the block center and the estimated color component functions f _r (x, y), f _g (x, y), f _b (x, y) and if the large difference in the case where the color component that is estimated to amount of signal obtained in the peripheral block are greatly different, the former is the energy E ₁ becomes larger.

  As a result, the color component estimated using the color component estimation weight determination method 1 is more suitable for the signal measured at the block center than that obtained in the first embodiment.

  When the color component estimation weight determination method 2 is used, a large weight is given in the direction along the edge as shown in FIG. As a result, pixels in the region beyond the edge are less likely to contribute to the demosaicing result, which is useful for sharpening.

(3-1) Parameter estimation method 4
As the parameter estimation method 4, the parameter p ′ of the function that minimizes E ₁ can be obtained as shown in Equation (10) using a pseudo inverse matrix (KW) ⁺ of the matrix KW.

(3-2) Parameter estimation method 5
Since the matrix K is fixed when the color component estimation weight determination method 1 is used, the number of patterns in the matrix KW is four, which is the same as the number of W patterns. Therefore, if (KW) ₊ for all patterns is calculated in advance and stored in the memory, the calculation cost can be greatly reduced.

However, when the color component estimation weight determination method 2 or the color component estimation weight determination method 3 is used, since the matrix K changes point by point and the pattern is enormous, (KW) ₊ is all stored in the memory. It is not possible. A method for dealing with this problem will be described in the section of the third embodiment.

(3-3) Parameter estimation method 6
Similar to the parameter estimation method 3, it is possible to use any of the color component estimation weight determination methods 1 to 3 to obtain an approximate solution using a method such as the steepest descent method or the conjugate gradient method.

(4) Color component estimation unit 103
Finally, the color component estimation unit 103 estimates the color component of each point of the output image using the estimated model parameter p. The color component estimation unit 103 performs the same operation as in the first embodiment.

(5) Effect As described above, according to the color component interpolation apparatus 100 according to the present embodiment, the image quality is improved by, for example, sharpening the output image by introducing the color component model error using the weight K _i for each point. It becomes possible to raise.

(6) Example of change It should be noted that the weighting function can be developed as in Expression (11) from Expression (6) and Expression (7). G is a positive real number in which the user adjusts the value while looking at the image quality.

Here, when λ _l is too large, the ellipse 1203 in FIG. 15 is almost a straight line. At this time, K _i taking an extremely small value increases, and many of the row vectors of the matrix KW are close to 0, so that (KW) ⁺ may not be obtained correctly. In order to avoid this problem, the following may be performed.

First, a real number λ _max of about 0.5 is set as the upper limit of λ _l and λ _s .

Next, every time C is obtained, it expands into the form of equation (11). In the case of λ _l> λ _max replace the λ _l to λ _max.

Finally, when λ _s > λ ₁ , C is obtained again after replacing λ _s with λ ₁ .

By doing in this way, K _i taking an extremely small value is eliminated, and (KW) ⁺ can be obtained stably.

(Third embodiment)
Next, a color component interpolation apparatus 100 according to the third embodiment will be described with reference to FIGS.

(1) Purpose of this embodiment As described above, after the weight matrix K is obtained for each point using the color component estimation weight determination method 2 or the color component estimation weight determination method 3 in the second embodiment, When demosaicing is performed according to Equation (10), it is expected that the image quality is improved.

However, calculation of the matrix (KW) ⁺ K (hereinafter this matrix is referred to as “parameter estimation coefficient”) requires a large cost. If the weight matrix K has only a finite pattern, the calculation cost can be reduced by calculating all parameter estimation coefficients in advance and storing them in the memory.

  However, when the weight matrix K is obtained using the color component estimation weight determination method 2 or the color component estimation weight determination method 3, since the number of patterns is enormous, all the parameter estimation coefficients are held in the memory. Is difficult.

  Therefore, the present embodiment aims to address this problem.

(2) Configuration of Color Component Interpolation Device 100 FIG. 5 is a block diagram showing the color component interpolation device 100 according to the present embodiment.

  A color component interpolation apparatus 100 according to this embodiment includes an imaging data input unit 101 that inputs imaging data, a parameter estimation coefficient storage unit 106 that stores parameter estimation coefficients used for demosaicing, and a parameter estimation coefficient storage unit. The parameter estimation coefficient selection unit 107 that selects an appropriate one from the data obtained from the imaging data input unit 101 among the parameter estimation coefficients stored in 106 and the parameter estimation coefficient selection unit 107 are selected. A parameter estimation unit 108 that estimates parameters of a function that describes RGB components using parameter estimation coefficients, and an output image generation unit 103 that obtains color components at each point from the estimated parameters are provided.

  When the parameter estimation coefficient storage unit 106, the parameter estimation coefficient selection unit 107, and the parameter estimation unit 108 are combined, an operation similar to the parameter estimation unit 102 of the present embodiment can be realized.

(3) Operation of Color Component Interpolation Device 100 Next, the operation of the color component interpolation device 100 according to the present embodiment will be described using FIG. 5 and FIG. FIG. 6 is a flowchart showing the processing procedure of this embodiment.

(3-1) Parameter estimation coefficient storage unit 106
First, the parameter estimation coefficient accumulating unit 106 selects in advance a finite number from an infinite number of K patterns that can appear, calculates a parameter estimation coefficient corresponding to the selected K, and stores it in the memory (step S1). 1).

The weight function K (x, y) determined by the color component estimation weight determination method 2 or the color component estimation weight determination method 3 is a two-dimensional Gaussian function as shown in FIG. The parameters of the weighting function K (x, y) are represented by the equation (11). The major axis length λ ₊ , the minor axis length λ ₋ (λ ₋ <λ ₊ ), the major axis and x There are three angles θ (0 <= θ <π) formed by the axes. Therefore, for example, _a total of 60 patterns obtained by swinging λa in the range of 0 to 2.0 in increments of 0.5 and θ in the range of 0 degree <= θ <180 degrees in increments of 45 degrees are supported. The parameter estimation coefficient is obtained in advance and stored in the memory. Note that λ _a and θ corresponding to each parameter estimation coefficient are also recorded in the memory.

  As a result, all edge directions can be dealt with and various edge strengths can be dealt with.

(3-2) Imaging data input unit 101
Next, the imaging data input unit 101 acquires the brightness of pixels around the point where the color component is to be estimated from the imaging element and the color of the color filter (step 2).

  The imaging data input unit 101 performs the same operation as in the first embodiment.

(3-3) Parameter estimation coefficient selection unit 107
Next, the parameter estimation coefficient selection unit 107 selects an appropriate parameter estimation coefficient from the parameter estimation coefficient storage unit 106 using the data obtained from the imaging data input unit 101 (step 3).

First, an autocorrelation matrix C of the amount of change in RGB components in the vicinity of the area where demosaicing is performed is obtained from the imaging data. Each element of C relationship _{(C xx, C xy,} C _{xy, C yy)} and λ _A and θ is as Equation (12).

(3-4) Parameter estimation coefficient selection unit 107
Then, the parameter estimation coefficient selecting unit 107, a set of λ _A and θ are recorded in the parameter estimation coefficient storage unit 106, the formula (13) a set of λ _A and θ obtained from the real data using And the parameter estimation coefficient corresponding to the closest combination is output.

  For example, the closest combination may be selected as follows.

  First, the accumulated θ and the θ obtained from the actual data are compared, and those other than the closest are excluded from the candidates.

Next, λ ₊ corresponding to the remaining parameter estimation coefficient is compared with λ ₊ obtained from the actual data, and those other than the closest are excluded from the candidates.

Finally, λ ₋ corresponding to the remaining parameter estimation coefficient is compared with λ ₋ obtained from the actual data, and the closest one is output as the selection result.

  Since the selected parameter estimation coefficient is close to the parameter estimation coefficient calculated using C actually obtained from the imaging data input unit 101, the image quality of the output image is the color component in the second embodiment. This approach is closer to the case where the estimation weight determination method 2 or the color component estimation weight determination method 3 is used.

(3-5) Parameter estimation unit 102
Next, the parameter estimation unit 102 obtains a parameter p of a function that describes RGB components using Expression (10) (step 4).

(3-6) Color component estimation unit 103
Finally, the color component estimation unit 103 estimates the color component of each point of the output image using the estimated model parameter p. The color component estimation unit 103 performs the same operation as that in the first embodiment (step 5).

(4) Effect As described above, according to the color component interpolation apparatus 100 according to the present embodiment, an appropriate one is selected from the accumulated parameter estimation coefficients using the imaging data. It is possible to reduce pseudo-inverse matrix operations and reduce calculation costs.

(5) Example of change In this embodiment, it is not necessary to perform step 1 every time it images.

  For example, it is possible to execute Step 1 when manufacturing the color component interpolation apparatus 100 and always start from Step 2 at the time of subsequent imaging. In this way, color components can be interpolated at a higher speed. Is possible.

(Fourth embodiment)
Next, a fourth embodiment will be described. The Bayer array demosaicing has been described so far using the first to third embodiments, but the present invention can also be demosaiced even with a color filter pattern such as 1301 in FIG. In the fourth embodiment, a response to the change of the color filter will be described.

For example, in order to demosaicing a 3 × 3 pixel block in the upper left of FIG. 8, that is, a pattern such as 1301 in FIG. 13, the matrix W in Expression (2) is changed to Expression (13).

  The difference between W in Expression (2) and W in Expression (13) is the arrangement of 0 and 1 in the column vectors from the first column to the third column. Processing after changing W in accordance with the color filter is the same as in the first to third embodiments.

  Thus, in the case of an image sensor having a color filter made of RGB, it is possible to cope with a change in the color arrangement by changing the arrangement of 0 and 1 in the column vectors from the first column to the third column. However, in the pattern such as 1302 in Expression (12), the column vector in the first column is all 0, so that the block used for demosaicing always includes at least one pixel having an RGB color filter. Need to be.

  Although the color component weight estimation method 1 and the color component weight estimation method 3 can be applied, the color component weight estimation method 2 includes pixels having color filters of the same color in the vertical and horizontal directions in the block. It cannot be applied unless it is an image sensor.

(Fifth embodiment)
In the description of the first to fourth embodiments so far, the demosaicing of the image pickup element in which the pixels are arranged in a square lattice shape has been described. However, the present invention is, for example, a square lattice as shown in FIG. 9 or FIG. It is also possible to deal with an image sensor having a pixel array (PIA array, Pixel Interleaved Array) that is rotated 45 degrees. Further, the point p _i * at which the color component is to be calculated does not have to coincide with any of the pixel coordinates p _i . In the fifth embodiment, a response to a change in pixel arrangement will be described.

For example, in order to obtain a digital image in which pixels are arranged in a square grid pattern from an image sensor having an arrangement as shown in FIG. 9, for example, an RGB component at a point corresponding to a gap between the pixels, such as a point 1501 in FIG. 16, is estimated. There is a need to. Even in such a case, the matrix W of equation (2) can be obtained as in equation (14).

In this way, by using the pixel position p _i , it is possible to deal with a hexagonal lattice in which hexagonal pixels are spread without gaps in addition to the PIA arrangement.

  Although the color component weight estimation method 1 and the color component weight estimation method 3 can be applied, the color component weight estimation method 2 includes pixels having color filters of the same color in the vertical and horizontal directions in the block. It cannot be applied unless it is an image sensor.

(Sixth embodiment)
So far, in the description of the first to fifth embodiments, the demosaicing of the single-plate image sensor has been described, but the present invention can also be applied to an image sensor other than the single-plate image sensor.

  For example, Takashi Sakaguchi et al., “New 3CCD Camera System for Consumers”, Television Society Technical Report, Vol.19, No.20, pp47-52, 1995. There has been proposed a method (3CCD pixel shifting type imaging device) that can increase the sense of resolution by intentionally shifting.

In this imaging apparatus, each pixel of the three CCDs can be associated with a point on the output image. Therefore, the position on the output image pixel is associated With p _i, it is possible to obtain a matrix W of the formula (14). In the imaging apparatus because the position of the pixels of the imaging element corresponding to the R and B are completely overlapped, the value of two or more p _i match exactly. However, since the colors of the color filters are different (that is, the three elements on the left side of each row vector are different), the rank of W does not decrease, and the above embodiments can be applied.

(Seventh embodiment)
Non-Patent Document 1 also discloses a super-resolution technique in which a plurality of images obtained by capturing the same subject by moving a camera minutely are combined to obtain a higher resolution than each input image. In addition, the method of using the image data itself of the single-chip image sensor instead of the image is also described by Masatoshi Okutomi, Masao Shimizu, Tomoyuki Nakamura, Tomomasa Goto, Takahiro Yano, “Direct color super-resolution from single-plate CCD image data, "The 10th Symposium on Image Sensing Symposium (SSII2004), pp.507-512, 2004.) In super-resolution, the same subject is imaged multiple times by moving the camera. By associating each pixel of an image obtained each time an image is captured or each point on the output image with a single image, the number of pixels is increased in a pseudo manner to obtain a high-resolution image.

If the position where imaging data is associated with the p _i, it is possible to perform super-resolution by the proposed method.

  However, although the color component weight estimation methods 1 to 3 can be applied, since the regularity of the pixel arrangement is lost, it is impossible to increase the speed as in the third embodiment.

(Eighth embodiment)
In the eighth embodiment, an embodiment will be described in which the following three changes are made to the color component model used in the first to third embodiments.

(1) Correspondence to model for maintaining color mixture ratio In the first to third embodiments, modeling was performed in which the difference between color components was constant regardless of the position. However, in the present embodiment, it is also possible to model that the ratio between the color components is constant regardless of the position.

Therefore, s = (s ₀ , s ₁ ,...) ^T in equation (2) is a vector s ′ = (log (s ₀ + α), log (s ₁ + α), logarithm of each element,. ··)And it is sufficient.

α is a constant for preventing the value of s ′ from diverging infinitely even when s _i = 0, and is set to 1.0, for example. In this case, the RGB components as a result of demosaicing are exp (f _r (x, y)) − α, exp (f _g (x, y)) − α, exp (f _g (x, y)) − α. Become.

  Since the color is originally defined by the mixing ratio of the RGB components, the modeling in which the difference between the RGB components is constant changes the color in the local region, but if this change is made, the mixing ratio of the RGB components Can be kept constant. However, in a region where there is a large difference between the RGB components, a small variation in the color component having a small value may greatly affect the color component having a large value.

(2) Response to change of basis function In the first to third embodiments, the color component is modeled by a polynomial of x and y, but can be modeled by other expressions.

The modeling of the equation (1) is based on Taylor expansion of the color component functions f _r (x, y), f _g (x, y), and f _b (x, y) in the vicinity of p *, and the accuracy of approximation is a certain dimension. Is equivalent to censoring. However, in addition to Taylor expansion, the color component functions f _r (x, y), f _g (x, y), and f _b (x, y) are expanded using Fourier series and wavelets, and the method of each of the above embodiments. It is possible to apply this framework.

  A method of modeling a color component function with a weighted sum of a plurality of basis functions and estimating the weight from surrounding pixels is within the scope of the above embodiments.

(3) Response to splines The concept of spline interpolation can be incorporated.

All the methods described so far were measured as values at the position where the pixels of the functions f _r (x, y), f _g (x, y), f _b (x, y) describing the color components exist. This is a method of obtaining a function that can reduce the difference between the signals s _i .

On the other hand, when the concept of spline interpolation is introduced, the differential amounts of the color component functions f _r (x, y), f _g (x, y), f _b (x, y) of the RGB components and the points on the image Thus, it is possible to model the difference in the amount of change in the signal amount obtained from the pixel. A specific method will be described.

First, according to the model of Expression (1), the differential amount for each point of the R component is as shown in Expression (15). In the modeling method of the first embodiment, the differential amount of each GB component for each point is the same as that of the R component.

On the other hand, for example, in the case of 3 × 3 pixel demosaicing as shown in FIG. 11, since the spatial differential amounts of the RGB components are all the same, the differential amount of the B component in the center pixel 1104 in the x-axis direction is the differential amount of G. They are the same and can be approximated by 0.5 (s ₅ -s ₃ ) / (x ₅ -x ₃ ). With this it is possible to change the equation (2) to define a new color component model error E ₂ of Equation (16).

If you want a weighted as equation (10) in each row as the energy function E _1, Determination of weights attached to the bottom line of equation (16) is considered three.

The first determination method is a method in which the weight given to the point at which the color component estimation weight determination unit 102 obtains the differentiation is assigned as it is. For example, in the example of equation (16), the weight of the bottom line is the K _4.

The second determination method is a method of selecting one of the weights to be given to the points used by the color component estimation weight determination unit 102 to obtain the differentiation. For example, in the example of Expression (16), the weight of the bottom row is K ₃ or K ₅ .

  The third determination method is a method of multiplying the weight determined by the first and second determination methods by a positive real number α smaller than 1. The differential amount is the difference between the two signal amounts obtained from the pixel, and noise is added to each of the two signals, so that the value tends to deviate from the true value. Therefore, it is desirable that the weight corresponding to the differential amount is smaller than the weight for the other.

(Ninth embodiment)
In the methods described so far, the weight for color component estimation is determined using the difference in signal amount between two or more pixels having the same color filter, focusing on the edge. However, for example, when whiteout or blackout occurs, the signal amount of the pixel is likely to be out of the model of Expression (1). There are two possible countermeasures.

(1) First Method First, after giving the weight determined by the color component estimation weight determination methods 1 to 3 to each pixel, it is conceivable to correct the weight by referring to the signal amount of each pixel.

For example, if the difference between the maximum signal amount s _max and the signal amount s _{i that} can be output by the image sensor is equal to or less than a predetermined threshold Δs, it is considered that there is a high possibility of whiteout, and K _i is a real number α smaller than 1 (for example, 0 Similarly, if the signal amount s _i is equal to or less than the predetermined threshold value Δs, it is considered that there is a high possibility of blackout, and a measure of multiplying K _i by α can be taken.

(2) Second method Next, when pixels with overexposure or underexposure are concentrated in one color component, only that color component has a different derivative from other color components. It is possible to

For example, Figure 11, to remove the fifth line of W in formula (2), by excluding the beta _b from the elements of the vector p, can estimate R and G components. The B component may be supplemented by, for example, linear interpolation using neighboring pixels.

(Example of change)
The present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage.

  In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, some components may be deleted from all the components shown in the embodiment.

  Furthermore, constituent elements over different embodiments may be appropriately combined.

1 is a block diagram illustrating a configuration of a color component interpolation apparatus 100 according to a first embodiment of the present invention. It is a flowchart which shows operation | movement of 1st Embodiment. It is a block diagram which shows the structure of the color component interpolation apparatus 100 concerning 2nd Embodiment. It is a flowchart which shows operation | movement of 2nd Embodiment. It is a block diagram which shows the structure of the color component interpolation apparatus 100 concerning 3rd Embodiment. It is a flowchart which shows operation | movement of 3rd Embodiment. It is a figure of the color filter of a Bayer arrangement. It is a figure of the image pick-up element of the square lattice which mounted | wore with the color filter which is not a Bayer arrangement. It is an example of an image sensor that is not a square lattice. 10 is an example of an image sensor that is not a square lattice different from FIG. 9. It is a figure explaining the example of the method of cutting out the block of 3x3 pixels from a Bayer arrangement. FIG. 12 is a diagram illustrating an example different from FIG. 11 in a method of cutting out a 3 × 3 pixel block from a Bayer array. It is a figure explaining the example of the method of cutting out the block of 3x3 pixels from the image pick-up element of FIG. It is a figure which shows the relative positional relationship of the pixel of the image before an expansion, and the pixel of the image after an expansion. It is a figure showing the concept of the weight determination method 2 for color component estimation in connection with 2nd Embodiment. It is a figure showing the concept of the method of estimating the color component in the clearance gap between pixels in the image pick-up element of FIG. It is a figure showing the concept of the method of calculating | requiring the variation | change_quantity of the color component in each point in the weight determination method 3 for color component estimation in connection with 2nd Embodiment. It is a figure showing the concept of the three-dimensional space defined using the s axis | shaft corresponding to the signal amount in addition to the two axes of x and y showing the position of the pixel used in FIG.

Explanation of symbols

100 color component interpolation apparatus 101 imaging data input unit 102 parameter estimation unit 103 color component estimation unit

Claims (11)

An imaging data input unit for acquiring imaging data having a pixel position indicating a position of each pixel on the imaging device, a color of each pixel, and a signal amount of each pixel;
The parameters of the plurality of color component functions for estimating the signal amounts of the plurality of colors at arbitrary positions of interest on the image sensor are set so that the derivatives of the plurality of color component functions are similar to each other. A parameter estimation unit that estimates using imaging data of a plurality of pixels around the target position;
A color component estimation unit that estimates a signal amount of an arbitrary color at the position of interest using the color component function and the parameter;
A color component interpolation apparatus. A first weight determining unit for obtaining a first weight that decreases as the distance from the target position decreases;
The parameter estimation unit decreases the degree of similarity of the derivative for each color as the first weight is smaller.
The color component interpolation apparatus according to claim 1. A second weight determining unit that obtains a second weight that is larger in a region that extends in the edge direction obtained from the signal amount of each pixel centered on the target position;
The parameter estimation unit decreases the degree of similarity of the derivative for each color as the second weight is smaller.
The color component interpolation apparatus according to claim 1. The parameter estimation unit includes:
(1) A signal amount of imaging data of a plurality of pixels around the target position, and a value when a pixel position corresponding to each of the signal amounts is substituted into a color component function for each color A first error including the parameter calculated from the difference, or
(2) The parameter calculated from the difference between the change amount in the vicinity of the pixel position corresponding to each signal amount and the spatial differential value of the color component function corresponding to each pixel position, and the first error A second error including
Set one of the color component model errors as
Calculating the parameter so as to minimize the color component model error;
The color component interpolation apparatus according to claim 3. The parameter estimation unit calculates the parameter that minimizes the color component model error by performing a weighted sum of the signal amount using a third weight.
The color component interpolation apparatus according to claim 4. The parameter estimation unit includes:
A storage unit for storing the second weight and the third weight in association with each other;
The second weight is obtained using the signal amount, the third weight corresponding to the obtained second weight is called from the storage unit,
Calculating the parameter using the called third weight;
The color component interpolation apparatus according to claim 5. The color component function is a weighted sum of basis functions;
The parameter is a weight of the basis function;
The color component interpolation apparatus according to claim 1. The parameter of the color component function is a coefficient when the color component function is Taylor-expanded.
The color component interpolation apparatus according to claim 1. The second weight is estimated using a difference in the signal amount between pixel positions having the same color component.
The color component interpolation apparatus according to claim 3. An imaging data input step of acquiring imaging data having a pixel position indicating the position of each pixel on the imaging device, a color of each pixel, and a signal amount of each pixel;
The parameters of the plurality of color component functions for estimating the signal amounts of the plurality of colors at arbitrary positions of interest on the image sensor are set so that the derivatives of the plurality of color component functions are similar to each other. A parameter estimation step of estimating using imaging data of a plurality of pixels around the target position;
A color component estimation step for estimating a signal amount of an arbitrary color at the target position using the color component function and the parameter;
A color component interpolation method comprising: An imaging data input function for acquiring imaging data having a pixel position indicating a position of each pixel on the imaging device, a color of each pixel, and a signal amount of each pixel;
The parameters of the plurality of color component functions for estimating the signal amounts of the plurality of colors at arbitrary positions of interest on the image sensor are set so that the derivatives of the plurality of color component functions are similar to each other. A parameter estimation function for estimating using imaging data of a plurality of pixels around the target position;
A color component estimation function for estimating a signal amount of an arbitrary color at the target position using the color component function and the parameter;
This is a color component interpolation program that implements a computer.

Images