JP4364657B2 - Pixel interpolation filter generation device and pixel interpolation device - Google Patents

Description

  The present invention relates to a pixel interpolation technique.

  An input sensor such as a CCD inputs image information by photoelectrically converting light received through the color filter array. Here, when a single-plate CCD is used, the image information that can be acquired for each pixel is only information about any one color component. For example, in the case of a single-plate CCD using an RGB Bayer array color filter array, the information that can be acquired by the input sensor is the color component of any one of R, G, and B for each pixel. Information only.

  Therefore, pixel interpolation processing is performed on an image in which each pixel has only one color component. In the pixel interpolation process, image information of a target pixel is estimated from pixels around the target pixel. For example, when the target pixel has G component image information, the R component and B component image information of the target pixel is estimated from surrounding pixels.

  Here, conventionally, as a method of interpolating the image information of the target pixel, a method using a linear filter is generally performed. In addition, a non-linear filter may be used, but when a non-linear filter is used, each set value (coefficient) used for the filter cannot be generalized. Was cumbersome.

JP 2003-244715 A

  In the above-mentioned patent document 1 filed by the applicant of the present application, the problem of the interpolation method using a conventional linear filter or the like is solved by performing an interpolation process based on the local characteristics of the image. However, depending on the local characteristics, there are cases where appropriate interpolation is not performed.

  In view of the above problems, the present invention provides a technique that enables highly accurate interpolation processing by automatically generating a nonlinear filter.

According to the first aspect of the present invention, means for inputting a color image in which each pixel includes a plurality of color signals, means for extracting a block image including a pixel of interest from the color image, and each pixel of the block image Original image generation means for generating a pseudo original image in which each pixel has one color signal by selecting any one of the plurality of color signals according to a predetermined arrangement rule; Learning means that uses an original image as a learning signal, performs learning using a color signal that does not exist in the pseudo original image among all color signals included in the pixel of interest in the color image as a teacher signal, and generates a pixel interpolation filter; It is characterized by providing.

According to a second aspect of the present invention , in the pixel interpolation filter generation device according to the first aspect, the original image generation unit generates a plurality of patterns of pseudo original images corresponding to the plurality of color signals included in the target pixel. Means.

According to a third aspect of the present invention, in the pixel interpolation filter generation device according to the first or second aspect , the learning unit includes a neural network using an error back propagation method. To do.

According to a fourth aspect of the invention, in the pixel interpolation filter generator according to any one of claims 1 to 3, wherein the pixel of interest is characterized in that the a center pixel of the pseudo original image.

Invention according to claim 5, characterized in that in the pixel interpolating filter generator according to any one of claims 1 to 3, wherein the pixel of interest is a plurality of pixel groups contained in the pseudo original image And

The invention according to claim 6 is a pixel interpolating apparatus for interpolating a signal of a color component that does not exist in each pixel for an image in which each pixel has a signal of one color component, wherein each pixel has one color component. action means for inputting an original image with a signal, means for extracting the block original image from the original image, with respect to the block original image, a pixel interpolation filter according to any one of claims 1 to 5 Means for interpolating a color signal that does not exist in the target pixel of the original image.

  According to the present invention, it is possible to automatically generate a color interpolation filter by using a color image in which one pixel includes a plurality of color signals as a learning image and a teacher image. This makes it possible to automatically generate a color interpolation filter using various types of images, and to perform highly accurate color interpolation.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. First, terms used in this specification are defined. An original image (RAW image) indicates an image having only one color component signal for one pixel. An original image input from an input sensor such as a single-plate CCD is referred to as an input original image. On the other hand, a complete image indicates an image having a plurality of color component signals for one pixel. For example, in the case of an image in the RGB color space, an image having three color signals of R, G, and B for one pixel is a complete image. Conversely, the original image generated from the complete image is referred to as a pseudo original image.

{1. Pixel interpolation filter generation processing}
FIG. 1 is a block diagram of a pixel interpolation filter generation apparatus according to an embodiment of the present invention. The pixel interpolation filter device includes a blocking unit 1, a pseudo original image generation unit 2, a learning unit 3, and an error calculation unit 4. Each of these means 1 to 4 is configured with a hardware circuit in the present embodiment, but a part or all of these means may be configured with software. The blocking unit 1 includes a normalizing unit 10.

  First, the blocking unit 1 inputs an image. This image is a complete image having three color signals of R, G, and B for one pixel. This complete image has a role as a learning image for generating a nonlinear filter and a role as a teacher signal. Therefore, it is preferable to use a standard image for evaluation as a complete image.

  The complete image input to the blocking unit 1 is first normalized by the normalizing unit 10. If the complete image is an 8-bit image, for example, a 1-pixel image is given by an integer from 0 to 255, but the normalizing means 10 converts a 1-pixel image to a real number from 0 to 1 (for example, , A real number in the range of 0.1 to 0.9). Such a normalization process is performed in order to perform highly accurate learning in a later learning process, and depends on the characteristics of the neural network.

Next, the normalized complete image is blocked by the blocking unit 1 into, for example, a 5 × 5 region centered on the central pixel P ₂₂ as shown in FIG. In the figure, P _xy indicates a pixel at each coordinate position in a block area defined when x is a row number and y is a column number. Therefore, in the case of a block image extracted from a complete image, each pixel P _xy has three color signals of R, G, and B, respectively.

Next, the blocked complete image is input to the pseudo original image generating means 2 and converted into a pseudo original image (RAW image) composed of one color signal for each pixel. For example, as shown in FIG. 13, the pseudo original image has a fixed color arrangement of two G signals, one R signal, and one B signal for every four pixels (a region of 2 vertical pixels × 2 horizontal pixels). It is a Bayer array image having regularity. Here, the pseudo the original image, there are four array pattern D1~D4 as shown in FIG. 14 by the selection method of the color signal of the central pixel P _22. Each of the pseudo original images including the four arrangement patterns D1 to D4 is referred to as a learning image.

  The learning image generated by the pseudo original image generation unit 2 is then input to the learning unit 3. As shown in FIG. 2, the learning means 3 includes four neural networks NN1, NN2, NN3, and NN4 corresponding to the four learning image arrangement patterns D1 to D4 shown in FIG. Then, the switches 31 and 32 are switched according to the type of the learning image array patterns D1 to D4 input from the pseudo original image generation means 2, and the network to be used is selected. Specifically, the array pattern D1 shown in FIG. 14 corresponds to the neural network NN1, the array pattern D2 corresponds to the neural network NN2, and similarly, the array patterns D3 and D4 correspond to the neural networks NN3 and NN4. Yes.

  Each of the neural networks NN1 to NN4 includes a plurality of layers including a plurality of nodes 33, 33,... As shown in FIGS. Here, since the neural networks NN1 to NN4 have the same configuration, the neural network NNn will be described with reference to a common drawing. In the present invention, two types of patterns are assumed as the number of output signals of the neural network NNn. Therefore, FIG. 3 shows a neural network NNn having three output signals as a neural network NNnA. On the other hand, a neural network NNn having two output signals is shown as a neural network NNnB in FIG.

The number of layers of the neural networks NNnA and NNnB, the number of nodes in the intermediate layer, and the like are not particularly limited, and may be configured to balance the circuit scale and the calculation accuracy based on experience and the like. In the present embodiment, each of the neural networks NNnA and NNnB is a four-layer neural network. Specifically, an input layer to which pixels P _{00 to} P ₄₄ of the pseudo original image are input, and an output layer to which output signals S ₀ , S ₁ , S ₂ (only S ₀ , S _{1 in} NNnB) are output. And two intermediate layers hierarchized between the input layer and the output layer.

  Here, as the neural network NNn, either one shown in FIG. 3 or FIG. 4 is adopted. However, when the neural network NNnA is used, there are two cases where the output signal is three signals. It is possible to apply both to the case of making a signal. Hereinafter, the neural network NNnA and NNnB will be described as the neural network NNn without particularly distinguishing both.

  In the above configuration, when the switches 31 and 32 are switched and a network to be used is selected, a 5 × 5 learning image that is a pseudo original image is input to the neural network NNn. The input learning image is input to the input layer of the network.

Each node 33 of the neural network NNn includes a memory 34, a plurality of coefficient multipliers 35, 35,..., An adder 36, and a lookup table circuit 37, as shown in FIG. The memory 34 stores a weighting coefficient w that defines a pixel interpolation filter, a bias value b, and an output signal S _out of the node 33. In addition, when backpropagation (error back propagation method) described later is executed, the modified weighting coefficient w, bias value b, and output signal _Sout are sequentially updated and stored. The coefficients w, b and the output signal S _out are stored as unique values in the memories 34 of the nodes 33, 33,. In each node 33, the calculations shown in Equation 1 and Equation 2 are performed on the input signal.

First, the node 33, the input signal S _1, S _2, · · ·, the S _m is entered, the input signal S _1, S _2, · · ·, weighting factor S _m w _1, w _2, ..., w _m is multiplied. Here, the subscripts of the input signals S ₁ , S ₂ ,..., S _m and the weighting coefficients w ₁ , w ₂ ,..., W _m are the nodes 33, 33,. Where m is the number of nodes in the previous layer. The coefficients w ₁ , w ₂ ,..., W _m are stored in the memory 34 with initial values determined in advance at random. 3 and 4, two subscripts are appended to w of each node 33. Of these, the second half subscript is the number assigned to the node in the previous layer, as described above, and the first half subscript is the number assigned to the own node in the layer to which the own node belongs.

Next, the output of each multiplication result is added in the adder 36. The calculation results so far correspond to Σw _i · S _i in Equation (1). Further, the adder 36 adds the bias value b. An initial value of this bias value b is also randomly selected in advance and stored in the memory 34. The calculation results so far correspond to (b + Σw _i · S _i ) in Equation (1).

  Next, the LUT circuit 37 performs an operation based on the sigmoid function. The Sigmoid function is a function as shown in Formula 2, and is a function having an input / output relationship shown in FIG. Such a function operation may be directly executed, but here, in order to speed up the operation, information in which input / output relations are tabulated in advance is prepared, and an input signal input to the LUT circuit 37 is prepared. On the other hand, a corresponding output signal is output. This calculation result corresponds to Equation 1.

The output signal S _out of each node 33 calculated in this way becomes an input signal to the node 33 of the subsequent layer after the combination. Such an operation is similarly executed in the two intermediate layers and the output layer. The three (or two) output signals S _out output from the output layer are the output signals S ₀ , S ₁ , S ₂ (or S ₀ , S ₁ ) of the neural network NNn. That is, when the neural network NNnA is used, output signals S ₀ , S ₁ , S ₂ are output. When the neural network NNnB is used, output signals S ₀ and S ₁ are output. The output signal S _out output from each node 33 is stored in the memory 34 of each node 33.

In FIG. 1, output signals S ₀ , S ₁ , (S ₂ ) of the node 33 of the output layer of the neural network NNn are input to the error calculation means 4. The error calculation means 4 receives the teacher signals (R ₂₂ , G ₂₂ , B ₂₂ ) output from the blocking means 1. This teacher signal is image information of the target pixel (here, the central pixel P ₂₂ ) held in the complete image, and is three color signals of R, G, and B. Then, an error signal is calculated between the output signals S ₀ , S ₁ , S ₂ (or S ₀ , S ₁ ) output from the learning unit 3 and the teacher signal output from the blocking unit 1.

When the error calculation unit 4 performs error calculation, an error signal is output to the learning unit 3. Then, the learning means 3 performs backpropagation based on the input error signal, and corrects the weighting coefficient w and bias value b of each node 33. That is, when performing back pro vacation, coefficient weighting stored in the memory 34 the nodes 33 w, bias value b and the output signal S _out is referred to, the value after correction is updated. Then, after correcting the weighting coefficient w and the bias value b, the output signals S ₀ , S ₁ , (S ₂ ) are output again, and error calculation is performed. Such processing is repeated, and the processing is terminated when the error has converged, or when the error falls below a predetermined threshold.

  Thus, when the error converges, the node 33 in each neural network NNn stores the finally corrected weighting coefficient w and bias value b in the memory 34. Thereby, a pixel interpolation filter in which the weight w and the bias value b are set for each node 33 is automatically generated.

  Next, the above process will be described with a specific example.

<Learning method 1>
As an input signal, a pseudo original image having a Bayer array of a 5 × 5 block area centered on the center pixel P ₂₂ is used. As the teacher signal, all the R, G, and B color signals (R ₂₂ , G ₂₂ , B ₂₂ ) held in the complete image by the central pixel P ₂₂ are used. As the neural network NNn, a neural network NNnA that outputs three output signals S ₀ , S ₁ , S ₂ is used. That is, the number of nodes required for the input layer of each of the neural networks NN1A to NN4A is 25, and the number of nodes required for the output layer is three.

With the above settings, the weighting coefficient w and the bias value b are learned by backpropagation. An error is calculated between the output signals S ₀ , S ₁ , S ₂ from the learning means 3 and the teacher signals R ₂₂ , G ₂₂ , B ₂₂ , and each error converges or the error falls below a predetermined threshold value. Until the condition is satisfied, the learning is terminated.

  Such learning is repeated one or more times for a partial region or all regions of a plurality of complete images, and the learning process is terminated when the accumulated value of the error signal obtained by learning of one learning set converges. .

<Learning method 2>
As an input signal, a pseudo original image having a Bayer array of a 5 × 5 pixel block area centered on the center pixel P ₂₂ is used. The teacher signal, R center pixel P ₂₂ is retained in a fully image, G, the center pixel P ₂₂ of the pseudo original images of the three color signals of _{B (R 22, G 22,} B 22) is provided with Use signals other than color signals.

In other words,
When the arrangement pattern is D1, since the central pixel P ₂₂ of the pseudo original image is a G signal, the other R ₂₂ and B ₂₂ are used as teacher signals,
When the arrangement pattern is D2, the center pixel P ₂₂ of the pseudo original image since it is the R signal, the G _22, B ₂₂ other than this is a teacher signal,
When the arrangement pattern is D3, the central pixel P ₂₂ of the pseudo original image since it is the B signal, which G _22, R ₂₂ other than the teacher signal,
When the arrangement pattern is D4, the center pixel P ₂₂ of the pseudo original image because the G signal, and the teacher signal R _22, B ₂₂ other than this.

As the neural network NNn, a neural network NNnB that outputs two output signals S ₀ and S ₁ is used. That is, the number of nodes required for the input layer of each neural network NN1B to NN4B is 25, and the number of nodes required for the output layer is two.

With the above settings, the weighting coefficient w and the bias value b are learned by backpropagation. An error is calculated between any two of the output signals S ₀ , S ₁ and the teacher signals R ₂₂ , G ₂₂ , B ₂₂ from the learning means 3 until the error converges or falls below a predetermined threshold value. The process is repeated and learning is completed when the condition is satisfied.

  Such learning is repeated one or more times for a partial region or all regions of a plurality of complete images, and the learning process is terminated when the accumulated value of the error signal obtained by learning of one learning set converges. .

In the learning method 1 and the learning method 2 described above, the pseudo original image used for learning may use one type of arrangement pattern for one central pixel P ₂₂ , or all four types of arrangements shown in FIG. A pattern may be used. In this case, the for one of the center pixel P ₂₂ is four neural networks NN1~NN4 be learned once.

{2. Pixel interpolation processing}
Next, pixel interpolation processing using a pixel interpolation filter that is a nonlinear filter generated by the above processing will be described. FIG. 7 is a block diagram of the pixel interpolation device. The pixel interpolating apparatus includes a blocking unit 5 and a color interpolating unit 6. Each of these means 5 and 6 is constituted by a hardware circuit in the present embodiment, but a part or all of these means may be constituted by software. The blocking unit 5 includes a normalizing unit 50, and the color interpolation unit 6 includes a denormalizing unit 60.

  First, the blocking unit 5 inputs an image. This image is an original image of a Bayer array that includes any one of R, G, and B color signals for one pixel. This original image is an original image output from an input sensor such as a single-plate CCD, for example, and is referred to as an input original image as described above.

  The input original image input to the blocking unit 5 is first normalized by the normalizing unit 50. For example, if the input original image is an 8-bit image, one pixel image is given as an integer from 0 to 255, but the normalizing means 50 converts one pixel image to a real number from 0 to 1 ( For example, it is normalized to a real number in the range of 0.1 to 0.9. The reason why such normalization processing is performed is to match the image information of the learning image (normalized image) used in the learning unit 3.

Next, the normalized input original image is blocked by the blocking unit 5. This block area is a 5 × 5 pixel area centered on the central pixel P ₂₂ as in the case shown in FIG. However, since it is a block image extracted from the input original image, each pixel P _xy includes only one color signal of R, G, and B.

Next, the blocked input original image is input to the color interpolation means 6. Then, color interpolation processing is performed on the target pixel of the input original image that has been blocked. Here, a color interpolation processing as the target pixel centered image P ₂₂ of the block image.

  The processing contents of the color interpolation means 6 will be described. As shown below, four embodiments shown in FIGS. 8 to 11 will be described as the color interpolation means 6.

<Color interpolation means 6A>
The color interpolation unit 6A includes a learning unit 3 as shown in FIG. The learning means 3 has already been learned in the learning process described above, and is in a state in which an appropriate weighting coefficient w and bias value b are set for each node 33.

  The blocked input original image is input to the color interpolation means 6A as 25 pixel columns. Then, together with the 25 pixel columns, a signal designating the array pattern of the input original image is input to the learning means 3. The signal designating the arrangement pattern is a signal designating any of the arrangement patterns D1 to D4 shown in FIG. In other words, this is a signal that designates to which array pattern the input original image that has been blocked belongs.

  In the learning means 3 of the color interpolation means 6A, the switches 31 and 32 are switched according to the input array pattern signal, and the neural network NNn corresponding to the array pattern is selectively connected in the same manner as in the learning process.

On the other hand, the input original image is subjected to the calculation expressed by Equation 1 at each node 33 in each layer, and is interpolated as an output signal of the node 33 in the output layer of the neural network NNn, which does not exist in the input original image. Two color signals are output. That is, the neural network NNnB can be used as the network. In the embodiment shown in FIG. 8, the color component corresponding to the center pixel P ₂₂ of the input original image is not subjected to the color interpolation process, and the pixel value of the center pixel P _{22 is used} as it is.

In other words,
When the arrangement pattern is D1, G ₂₂ adopts the signal value of the center pixel P ₂₂ of the input original image, R ₂₂ and B ₂₂ adopt the interpolated signal value,
When the arrangement pattern is D2, R ₂₂ adopts the signal value of the center pixel P ₂₂ of the input original image, G ₂₂ and B ₂₂ adopt the interpolated signal value,
When the arrangement pattern is D3, B ₂₂ adopts the signal value of the center pixel P ₂₂ of the input original image, G ₂₂ and R ₂₂ adopt the interpolated signal value,
When the arrangement pattern is D4, G ₂₂ employs a signal value of the central pixel P ₂₂ of the input original image, R ₂₂ and B ₂₂ employs a signal value interpolated.

  In this way, R, G, and B color signals are generated for the target pixel, and color interpolation processing is performed. Then, such interpolation processing is executed for all pixels. That is, the color interpolation process is executed for each target pixel while moving the target pixel in the input original image. As a result, as shown in FIG. 15, for all the pixels of the input original image, all the R, G, B color signals are interpolated to generate a complete image.

<Color interpolation means 6B>
The color interpolation unit 6B includes a learning unit 3 as shown in FIG. As for the color interpolation means 6B, only the parts different from the color interpolation means 6A will be described.

The color interpolation means 6B outputs all three color signals of R, G and B as output signals of the node 33 in the output layer of the neural network NNn. That is, the neural network NNnA is adopted as the network. In this embodiment performs color interpolation processing for the color component corresponding to the central pixel P ₂₂ of the input original image, the center pixel P _22, is to employ the pixel value after all color interpolation. That is, G ₂₂ , R ₂₂ and B ₂₂ all adopt interpolated signal values regardless of the type of array pattern.

  Similarly, such interpolation processing is executed for all pixels. That is, the color interpolation process is executed for each target pixel while moving the target pixel in the input original image. As a result, as shown in FIG. 15, for all the pixels of the input original image, all the R, G, B color signals are interpolated to generate a complete image.

<Color interpolation means 6C>
The color interpolation unit 6C includes a learning unit 3 as shown in FIG. Regarding the color interpolation means 6C, only the parts different from the color interpolation means 6A will be described.

In the color interpolation means 6C, the three color signals R, G and B are output as output signals from the node 33 in the output layer of the neural network NNn. That is, the neural network NNnA is adopted as the network. Further, as shown in the figure, either one of the pixel value held by the center pixel P ₂₂ of the input original image and the pixel value of the same color pixel as the center pixel P ₂₂ output from the learning means 3 is obtained. A switch 51 for outputting is provided. In this embodiment, the color interpolation process is also performed for the color component held in advance by the center pixel P ₂₂ which is the target pixel of the input original image. For the color component signal that was not held by the center pixel P _22, the pixel value after color interpolation was adopted, and the color component signal that was previously held by the center pixel P ₂₂ was held. Either the signal value or the signal value after the interpolation processing is adopted.

  Similarly, such interpolation processing is executed for all pixels. That is, the color interpolation process is executed for each target pixel while moving the target pixel in the input original image. Thereby, for all the pixels of the input original image, all the color signals of R, G, B are interpolated to generate a complete image.

<Color interpolation means 6D>
The color interpolation unit 6D includes a learning unit 3 as shown in FIG. Regarding the color interpolation means 6D, only the parts different from the color interpolation means 6A will be described.

  In the color interpolation means 6D, the output layer node 33 of the neural network NNn outputs all three color signals of R, G and B as output signals. That is, the neural network NNnA is adopted as the network.

Further, as shown in the figure, the mixing means for mixing the pixel value held by the center pixel P ₂₂ of the input original image and the pixel value of the same color pixel as the center pixel P ₂₂ output from the learning means 3. 52 is provided. In this embodiment, the color interpolation processing is also performed on the color component held by the center pixel P ₂₂ which is the target pixel of the input original image. And, for the signal of the color component in which the center pixel P ₂₂ did not hold, to adopt a pixel value after color interpolation. For the color component signal held by the center pixel P _22, a color signal obtained by mixing the held signal value and the signal value after interpolation processing is employed. For example, the mixing ratio k is given, the pixel value generated by the interpolation process is multiplied by k, and the pixel value of the central pixel P _{22 included} in the input original image is multiplied by 1-k. Then, the result of adding these multiplication values is adopted.

  Similarly, such interpolation processing is executed for all pixels. That is, the color interpolation process is executed for each target pixel while moving the target pixel in the input original image. Thereby, for all the pixels of the input original image, all the color signals of R, G, B are interpolated to generate a complete image.

  As described above, when the color interpolation processing is executed in the color interpolation means 6A to 6D, the R, G, B signal values are converted into signals of a predetermined bit width in the inverse normalization means 60. For example, if an image of 8 bits per pixel is output, each pixel value is converted to an integer from 0 to 255.

{3. Modification}
In each of the above embodiments, the case where the target pixel is one pixel has been described as an example. That is, after the pseudo original image is generated from the complete image, the learning process is performed using the central pixel P ₂₂ of the pseudo original image as the target pixel. In the pixel interpolation process, interpolation processing is performed using the center pixel P ₂₂ as a target pixel.

  In the modification shown in FIG. 16, the target pixel is composed of a plurality of pixels. Here, nine pixels in the region S in the figure are the target pixels. Also in this case, it is possible to generate a color interpolation filter and execute a color interpolation process by the same process as in the above-described embodiment. The 7 × 7 pixel block shown in the figure is used in the following description as a pseudo original image or as an input original image.

  The neural network NNn shown in FIG. 3 and FIG. 4 can output three or two signals as output signals. On the other hand, in order to generate the color interpolation filter having the nine pixels shown in FIG. 16 as the target pixel, a neural network NNn capable of outputting 27 or 18 signals as output signals is used. Good. In other words, if a color interpolation filter is to be generated using all the color component signals including the pixel values held by the nine target pixels in the region S of the pseudo-original image as a teacher signal, per pixel. Three color signals, that is, 27 output signals are required. Similarly, in the case of performing color interpolation processing, for the nine target pixels in the area S of the input original image, when the signal values of all the color components are replaced with the pixel values after the interpolation processing, 27 outputs are performed. Requires a signal. Alternatively, as shown in FIGS. 10 and 11, even when both the signal value of the original image and the signal value after interpolation are required, 27 output signals are required.

  Then, as shown in the figure, in order to generate a pixel interpolation filter that uses a block area of 7 × 7 pixels as a learning image and outputs 27 output signals, the neural network has 49 nodes as input layers. Provided, and an output layer having 27 nodes may be used. The number of intermediate layers or the number of nodes in the intermediate layer may be selected optimally while balancing interpolation accuracy and circuit scale.

  On the other hand, the pixel values held by the nine target pixels in the area S of the pseudo original image are used as they are, and only the signals of the other color components are used as teacher signals. In the case of generating an interpolation filter, two color signals, that is, 18 output signals are required for each pixel. Similarly, when color interpolation processing is performed, when the signal values of two color components are replaced with pixel values after interpolation processing for nine target pixels in the area S of the input original image, 18 outputs are output. Requires a signal.

  In order to generate a pixel interpolation filter that uses a block area of 7 × 7 pixels as a learning image and outputs 18 output signals, the neural network includes 49 nodes as an input layer and 18 as an output layer. What has this node may be used. Then, the optimum number of intermediate layers or intermediate nodes may be selected.

{Others}
In the above embodiment, description has been made on an image (complete image, original image, etc.) composed of three color components of RGB. However, the pixel interpolation filter generation device and the pixel interpolation device of the present invention are not limited to this. For example, a color system composed of four color components of cyan, magenta, yellow, and green, or a color system composed of four color components of green, red, blue, and emerald, or other four or more than four A color system image having color components can be similarly processed.

  Further, the arrangement pattern is not limited to the Bayer arrangement method, and any original image can be applied as long as the arrangement pattern has a certain regularity such as an interline method, a stripe method, or a honeycomb arrangement.

  In the description of the above embodiment, the learning image is a 5 × 5 block pixel area or a 7 × 7 block pixel area. However, the learning image is not limited to this, and the learning image is arbitrarily large. It is also possible to use an image signal of an area having an arbitrary shape.

It is a block diagram of a pixel interpolation filter generation device. It is a block diagram of a learning means. It is a block diagram of a neural network. It is a block diagram of a neural network. It is a block diagram of each node with which a neural network is provided. It is a figure which shows the input / output relationship of LUT with which each node is provided. It is a block diagram of a color interpolation apparatus. It is a block diagram of a color interpolation means. It is a block diagram of a color interpolation means. It is a block diagram of a color interpolation means. It is a block diagram of a color interpolation means. It is a figure which shows the image made into the block. It is a figure which shows the pseudo | simulation original image produced | generated from the complete image and the complete image. It is a figure which shows the arrangement | sequence pattern of the pseudo | simulation original image produced | generated. It is a figure which shows the interpolation image produced | generated from the input original image. It is a figure which shows the Example which applies an interpolation filter by making a some pixel into a focused pixel.

Explanation of symbols

3 learning means 6 color interpolation means 33 nodes NNnA, NNnB neural network

Claims (6)

Means for inputting a color image in which each pixel comprises a plurality of color signals;
  Means for extracting a block image including a target pixel from the color image;
  For each pixel of the block image, by selecting any one color signal from the plurality of color signals according to a predetermined arrangement rule, an original that generates a pseudo original image in which each pixel has one color signal is generated. Image generating means;
  Learning means for generating a pixel interpolation filter by using the pseudo original image as a learning signal, learning as a teacher signal a color signal that does not exist in the pseudo original image among all color signals of the target pixel in the color image; ,
A pixel interpolation filter generation device comprising: The pixel interpolation filter generation device according to claim 1,
  The original image generation means includes
  Means for generating a plurality of patterns of pseudo original images corresponding to the plurality of color signals included in the target pixel;
A pixel interpolation filter generation device comprising: The pixel interpolation filter generation device according to claim 1 or 2,
  The learning means includes
  Neural network using back propagation method,
A pixel interpolation filter generation device comprising: The pixel interpolation filter generation device according to any one of claims 1 to 3,
  The pixel interpolation filter generation device, wherein the target pixel is a central pixel of the pseudo original image. The pixel interpolation filter generation device according to any one of claims 1 to 3,
  The pixel interpolation filter generation device, wherein the target pixel is a plurality of pixel groups included in the pseudo original image. A pixel interpolation device that interpolates a signal of a color component that does not exist in each pixel for an image in which each pixel has a signal of one color component,
  Means for inputting an original image in which each pixel comprises a signal of one color component;
  Means for extracting a block original image from the original image;
  Means for interpolating a color signal that does not exist in a target pixel of the original image by applying the pixel interpolation filter according to any one of claims 1 to 5 to the block original image;
A pixel interpolation apparatus comprising:

Images