JP6435560B1 - Image processing apparatus, image processing method, program, and imaging apparatus - Google Patents

Abstract

A high-quality image of each color component is generated from an image output from an image sensor in which a color filter array in a Bayer array is arranged.
A high-resolution G generation unit includes a target pixel selection unit that selects one target pixel from a first image output from an image sensor in which a color filter array having a Bayer array is arranged; A G code tap selection unit 61 that selects a code tap composed of a plurality of pixels that is a G component pixel of the image, a G code calculation unit 63 that calculates the G code of the selected code tap, and a plurality of codes for each code A G coefficient storage unit 64 that stores the G coefficient and outputs a plurality of G coefficients based on the calculated G code, and selects a filter tap that is a G component pixel of the first image and includes a plurality of pixels G product-tap calculation unit 62 that calculates the pixel value of the output pixel corresponding to each pixel of interest based on each pixel value of the selected filter tap and a plurality of output G coefficients 65 Equipped with a.
[Selection] Figure 11

Description

  The present invention relates to an image processing device, an image processing method, a program, and an imaging device.

  In order to reduce the size and weight of the camera, a single-plate camera using one image sensor is employed instead of a three-plate camera using three image sensors.

  A color coding filter including a color filter array assigned to each pixel is installed on the light incident surface of the image sensor of the single-plate camera. As a result, the image sensor generates a color component signal color-coded by the color coding filter for each pixel. As a color filter array constituting the color coding filter, for example, there are three primary color filter arrays of R (red), G (green), and B (blue).

  The single-plate camera obtains one color component signal for each pixel by an image sensor, and generates a color signal other than the color component signal possessed by each pixel by linear interpolation processing. Thereby, the single plate type camera obtains an image close to the image obtained by the three plate type camera.

  As the color filter array constituting the color coding filter, a Bayer array three primary color filter array is often used. In the Bayer array, the G color filters are arranged in a checkered pattern, and the R and B color filters are alternately arranged in a row in the remaining portion.

  Each pixel of the image sensor is provided with a filter of one of the three primary colors R, G, and B. Each pixel outputs only an image signal corresponding to the color of the arranged filter.

  That is, the R component image signal is output from the pixel in which the R color filter is arranged, but the G component and B component image signals are not output. Similarly, only the G component image signal is output from the G component pixel, and the R component and B component image signals are not output. From the B component pixel, only the B component image signal is output, and the R component and G component image signals are not output.

  Therefore, conventionally, a DLMMSE method, which is a technique for interpolating insufficient color components, has been disclosed (see Non-Patent Document 1). In the DLMMSE method, first, the interpolated color difference of each pixel position of the input image from the image sensor is obtained in two directions, horizontal and vertical, and the statistics of five pixels in the vicinity of the position of the pixel of interest are calculated. The

  By using the calculated statistics, the directionality (horizontal or vertical direction) of the pixel of interest is detected, and the horizontal color difference and the vertical color difference are mixed (prorated) according to the detection result. The color difference (RG, BG) is obtained.

  Based on the color differences (RG, BG) thus determined, an image signal for the G pixel, an image signal for the R pixel, and an image signal for the B pixel are respectively obtained and generated as an output image. .

  In addition, a technique is disclosed in which an image signal corresponding to a CCD output of a three-plate camera is obtained by a CCD image sensor of a single-plate camera (see Patent Document 1). The technique of Patent Document 1 predicts a pixel value of a pixel of interest in an output image by a product-sum operation using a coefficient corresponding to the pixel of interest in an input image and its surrounding pixel values as variables and obtained in advance through learning. To do.

  Further, a technique for obtaining an image signal of each color component from an output of an image sensor having a color filter array composed of a plurality of color components without degrading image quality is disclosed (see Patent Document 2).

  The technique of Patent Document 2 selects a designated area from a first image output from a single-plate pixel unit, calculates a representative value of each color component of the designated area, and is obtained from the pixel value of the designated area Based on the quantity, the coefficient stored in advance is read out. The technique of Patent Document 2 uses a pixel value related to a predetermined pixel in a designated area as a prediction tap, converts a pixel value of each color component of the prediction tap into a conversion value using a representative value, and uses the conversion value as a variable. Then, each pixel value of an image composed only of pixels of each color component is calculated by a product-sum operation using the read coefficient.

  In addition, a technique for reducing false color and noise of an image signal acquired by a single-plate camera is disclosed (Patent Document 3).

  The technique of Patent Document 3 sets a reference area from a first image configured by an image signal output from a single-plate pixel unit with a Bayer array, and changes the area of the reference area to change the reference area. By evaluating the statistic obtained from the pixel value of the pixel and detecting the direction of the target position in the first image, the direction of the target position is detected with higher accuracy. As a result, even when an input image including a high frequency component close to the Nyquist frequency is input, the reference area is widened to detect an accurate directionality and reliably reduce false colors and noise.

JP 2000-308079 A JP2014-200008 JP, 2014-200033, A

DLMMSE algorithm from L. Zhang and X. Wu, “Color demosaicking via directional linear minimum mean square-error estimation,” IEEE Trans. On Image Processing, vol. 14, no. 12, pp. 2167-2178, 2005.

  The technique of Non-Patent Document 1 generates an image signal of G pixels based on color differences (RG, BG), that is, using information on R pixels and B pixels. For this reason, since a prediction error occurs in the image signal of G pixel, there is a problem that zipper noise occurs and the image quality deteriorates.

  In the technique of Patent Document 1, pixel values corresponding to R, G, and B in the image sensor are used as taps that are variables for prediction calculation as they are.

  However, since the R, G, and B pixel values are originally low in correlation, for example, even if a plurality of pixel values around the target pixel are input as taps, a sufficient effect can be exhibited in the prediction calculation. There wasn't. For example, when a pixel value is generated by a prediction calculation in a region where there is almost no correlation between changes in R, G, and B pixel values, image quality degradation such as zipper noise, false color, color blur, and ringing is significant. There is a problem that occurs.

  Even in the technique of Patent Document 2, the prediction accuracy of the pixel value of the G pixel is insufficient, and there is a problem that zipper noise cannot be sufficiently suppressed.

  Further, even with the technique of Patent Document 3, the intermediate level cannot be processed well because the false color cannot be improved or the binary determination is performed with the horizontal and vertical thresholds. There were disadvantages such as large scale.

  The present invention solves the above-described conventional problems. That is, an object of the present invention is to obtain an image of each color component with high image quality from an image output from an imaging device in which pixels corresponding to each of a plurality of color components are configured in a predetermined arrangement without degrading the image quality. An image processing apparatus, an image processing method, a program, and an imaging apparatus that can be obtained are provided.

The image processing apparatus according to the present invention is configured so that one of the first images is output from a first image output from an imaging device in which pixels corresponding to each of a plurality of color components are configured in a predetermined arrangement. A pixel-of-interest selection unit that selects one pixel as a pixel of interest, and a pixel of a first color component that is a main component of the first image, and the pixel of interest selected by the pixel-of-interest selection unit A first code tap selection unit for selecting a code tap in which a plurality of pixels of the first color component are arranged in a first arrangement pattern; and a feature of the code tap selected by the first code tap selection unit A first code calculation unit for calculating a code indicating a quantity; and a plurality of first tap coefficients obtained in advance by a first learning process of a code classification type adaptive filter for each code; Performed by the code calculation unit A first coefficient storage unit that outputs a plurality of first tap coefficients based on the generated code, and a pixel of a first color component that is a main component of the first image, and the target pixel selection unit A first filter tap selection unit that selects a filter tap in which a plurality of pixels of the first color component are arranged in a second arrangement pattern for the selected pixel of interest; and the first filter tap selection. Based on each pixel value of the filter tap selected by the unit and a plurality of first tap coefficients output from the first coefficient storage unit, the pixel value of the output pixel corresponding to each pixel of interest is calculated. And a first image generation unit that generates an image of the first color component by calculation. The first tap coefficient is composed of only the first color component from an image in which the color component, the arrangement of the color components, and the size of the predetermined subject are the same as the first image. The first student image and the predetermined subject are composed of only the first color component having the same size as the first image, and half pixels in the horizontal direction and the vertical direction with respect to the first student image. The first learning process using the shifted first teacher image, and solving the mapped normal equation of each pixel of the first student image and each pixel of the first teacher image It is a tap coefficient calculated by the first learning process.

According to an image processing method of the present invention, one of the first images is output from a first image output from an imaging device in which pixels corresponding to a plurality of color components are configured in a predetermined arrangement. A pixel-of-interest selection step of selecting one pixel as a pixel of interest; and a pixel of a first color component that is a main component of the first image, and the pixel of interest selected in the pixel-of-interest selection step A first code tap selection step of selecting a code tap in which a plurality of pixels of the first color component are arranged in a first arrangement pattern; and a feature of the code tap selected in the first code tap selection step A first code calculation step for calculating a code indicating a quantity, and a coefficient storage unit for storing a plurality of first tap coefficients obtained in advance by learning processing of a code classification type adaptive filter for each code A first coefficient reading step of reading a plurality of first tap coefficients based on the code calculated in the first code calculation step; and a pixel of a first color component which is a main component of the first image A first filter tap selection step of selecting a filter tap in which a plurality of pixels of the first color component are arranged in a second arrangement pattern with respect to the attention pixel selected in the attention pixel selection step; , Corresponding to each pixel of interest based on each pixel value of the filter tap selected in the first filter tap selection step and a plurality of first tap coefficients read out in the coefficient reading step An image generation step of generating an image of the first color component by calculating a pixel value of the output pixel. The first tap coefficient is composed of only the first color component from an image in which the color component, the arrangement of the color components, and the size of the predetermined subject are the same as the first image. The first student image and the predetermined subject are composed of only the first color component having the same size as the first image, and half pixels in the horizontal direction and the vertical direction with respect to the first student image. The first learning process using the shifted first teacher image, wherein the first equation for solving the mapped normal equation of each pixel of the first student image and each pixel of the first teacher image 1 is a tap coefficient calculated by one learning process.

The program according to the present invention causes a computer to store a first image output from an imaging device in which pixels corresponding to each of a plurality of color components are configured in a predetermined arrangement, in the first image. A pixel-of-interest selection unit that selects one pixel as a pixel of interest, and a pixel of the first color component that is a main component of the first image, and the pixel of interest selected by the pixel-of-interest selection unit, A first code tap selection unit that selects a code tap in which a plurality of pixels of the first color component are arranged in a first arrangement pattern; and the code tap selected by the first code tap selection unit A first code calculation unit that calculates a code indicating a feature quantity; and a plurality of first tap coefficients previously determined by a learning process of a code classification type adaptive filter for each code; Part A first coefficient storage unit that outputs a plurality of first tap coefficients based on the calculated code; and a pixel of a first color component that is a main component of the first image, and the pixel of interest selection A first filter tap selection unit that selects a filter tap in which a plurality of pixels of the first color component are arranged in a second arrangement pattern for the target pixel selected by the unit; and the first filter Based on each pixel value of the filter tap selected by the tap selection unit and a plurality of first tap coefficients output from the first coefficient storage unit, a pixel of an output pixel corresponding to each target pixel This is a program for functioning as a first image generation unit that generates an image of the first color component by calculating a value. The first tap coefficient is composed of only the first color component from an image in which the color component, the arrangement of the color components, and the size of the predetermined subject are the same as the first image. The first student image and the predetermined subject are composed of only the first color component having the same size as the first image, and half pixels in the horizontal direction and the vertical direction with respect to the first student image. The first learning process using the shifted first teacher image, wherein the first equation for solving the mapped normal equation of each pixel of the first student image and each pixel of the first teacher image 1 is a tap coefficient calculated by one learning process.

The imaging apparatus according to the present invention, which performs an imaging device pixels corresponding to each of the plurality of color components is constituted in a predetermined arrangement, the image processing to the first image output from the image sensor And a processing device.

  An image processing apparatus, an image processing method, a program, and an imaging apparatus according to the present invention provide a high-quality image of each color component from an image output from an imaging device in which pixels corresponding to each of a plurality of color components are configured in a predetermined arrangement. Images can be obtained.

It is a block diagram which shows the structural example of one Embodiment of an imaging device. It is a block diagram which shows the structural example of a demosaic part. FIG. 4 is a block diagram illustrating a configuration example of a virtual color difference (RG) generation unit 33. It is a block diagram which shows the structural example of a temporary G production | generation part. It is a figure explaining the interpolation method of the G component value of the horizontal direction in a R pixel position. It is explanatory drawing of a horizontal interpolation color difference and a vertical interpolation color difference. It is a figure explaining how to obtain | require temporary Gr in case R44 pixel in an input image is an attention pixel. It is a figure explaining how to obtain | require temporary Gr in case B55 pixel in an input image is an attention pixel. It is a flowchart explaining the temporary G production | generation routine by the temporary G production | generation part in a virtual color difference (RG) production | generation part. It is a figure explaining the production | generation of provisional R in G pixel position. It is a block diagram which shows the structural example of a high-resolution G production | generation part. It is a figure which shows the structural example of a G code tap. It is a figure which shows the structural example of G filter tap. It is a block diagram which shows the structural example of a G code calculating part. It is a figure explaining the example of 1 bit DR quantization. It is a block diagram which shows the structural example of a high-resolution (RG) production | generation part. It is a figure which shows the structural example of a (RG) code tap. It is a figure which shows the structural example of a (RG) filter tap. (BG) It is a figure which shows the structural example of a code tap. It is a figure which shows the structural example of a (BG) filter tap. It is a flowchart which shows a demosaic processing routine. It is a figure which shows the high-resolution image produced | generated by the image process with respect to an input image. It is a block diagram which shows the structural example of a learning apparatus. It is a figure for demonstrating the student image and teacher image which are produced | generated from a high resolution image. It is a figure which shows the structural example of a G coefficient data generation part. It is a figure which shows the structural example of the (RG) coefficient data generation part. It is a flowchart explaining the learning process routine by a learning apparatus.

Hereinafter, embodiments of the present technology will be described with reference to the drawings.
<Configuration of Imaging Device 10>
FIG. 1 is a block diagram illustrating a configuration example of an embodiment of an imaging apparatus 10. The imaging device 10 can capture both still images and moving images.

  The imaging apparatus includes an optical system 11, an image sensor 12, an analog signal processing unit 13, an analog / digital (A / D) color separation unit 14, a defect correction unit 15, a demosaic unit 16, a color processing unit 17, a gamma correction unit 18, A Y color difference conversion unit 19, a codec unit 20, an output unit 21, and a control unit 22 are provided.

  The optical system 11 includes, for example, a zoom lens, a focus lens, a diaphragm, an optical low-pass filter, etc. (not shown). The optical system 11 causes subject light from the outside to enter the image sensor 12.

  The image sensor 12 is, for example, a CMOS (Complementary Metal Oxide Semiconductor) image sensor having a plurality of light receiving elements (pixels). On the light incident surface of the image sensor 12, a Bayer array color filter array composed of the three primary colors RGB is arranged.

  The image sensor 12 receives incident light from the optical system 11, performs photoelectric conversion for each light receiving element, and outputs an image signal as an electrical signal corresponding to the incident light from the optical system 11. Therefore, an R component image signal is obtained from a pixel (R pixel) in which an R color filter is arranged, but a G component and B component image signal cannot be obtained. Similarly, G component image signals can be obtained from the G pixels, but R component and B component image signals cannot be obtained. B pixel image signals are obtained from B pixels, but R component and G component image signals cannot be obtained.

The analog signal processing unit 13 performs predetermined analog signal processing such as correlated double sampling processing on the image signal output from the image sensor 12.
The A / D color separation unit 14 performs A / D conversion on the image signal output from the analog signal processing unit 13, further performs color separation processing, and each of red (R), green (G), and blue (B) Output image data.
The defect correction unit 15 performs defect correction processing on the RGB image data output from the A / D color separation unit 14.

  The demosaic unit 16 performs demosaic processing on the RGB image data output from the defect correction unit 15. The detailed configuration of the demosaic unit 16 will be described later.

The color processing unit 17 performs color correction processing on the RGB image data output from the demosaic unit 16.
The gamma correction unit 18 performs gamma correction processing on the RGB image data output from the color processing unit 17.
The Y color difference conversion unit 19 converts the RGB image data output from the gamma correction unit 18 into Y color difference image data.

The codec unit 20 compresses the Y color difference image data output from the Y color difference conversion unit 19 using an encoder such as MPEG.
The output unit 21 outputs the image data from the codec unit 20. For example, the output unit 21 includes a driver (not shown) that drives a recording medium such as a semiconductor memory, a magnetic disk, or an optical disk, and records the image data from the codec unit 20 on the recording medium. Further, the output unit 21 may output the image data from the codec unit 20 to a communication unit such as a network, various cables, or wireless.
The control unit 22 controls each block configuring the imaging device 10 according to a user operation or the like.

<Configuration example of demosaic unit 16>
FIG. 2 is a block diagram illustrating a configuration example of the demosaic unit 16. A demosaic process performed by the demosaic unit 16 will be described.

  The demosaic unit 16 detects the directionality of the pixel of interest (hereinafter referred to as the pixel of interest) in the input image, and generates a temporary G image based on the directionality. Thereafter, provisional R and provisional B obtained by interpolating R and B are generated, and further, a virtual color difference (RG) and a virtual color difference (B-G) are generated. The demosaic unit 16 determines each pixel position of the input image based on the virtual color difference (RG) and (BG) pixels and the coefficient obtained in advance by the learning process of the code classification type adaptive filter. To generate an image shifted by half a pixel.

  In addition, the demosaic unit 16 is a high-resolution G image that is shifted by a half pixel from each pixel position of the input image based on the pixel value of the G pixel and the coefficient obtained in advance by the learning process of the code classification type adaptive filter. Is generated. Finally, the demosaic unit 16 generates a high resolution R image or a high resolution B image based on the high resolution G image and the virtual color differences (RG) and (BG).

  The demosaic unit 16 performs demosaic processing on an image (input image) from the image sensor 12 in which the color filter array of the Bayer array is arranged, and thereby a high resolution R image, a high resolution G image, and a high resolution B. Output each image.

  In the following description, a signal value output from the R pixel of the image sensor 12 is referred to as a pixel value R. Similarly, signal values output from the G pixel and the B pixel are defined as a pixel value G and a pixel value B, respectively.

  The demosaic unit 16 includes a target pixel selection unit 31 that selects a target pixel from the input image, a delay unit 32 that performs a delay process on the input G image, and a virtual color difference (RG) generation unit that generates a virtual color difference (RG). 33. A virtual color difference (B−G) generation unit 34 that generates a virtual color difference (B−G) is provided.

  Furthermore, the demosaic unit 16 includes a high resolution G generation unit 35 that generates a high resolution G image, a high resolution (RG) generation unit 36 that generates a high resolution (RG), and a high resolution ( A high-resolution (BG) generation unit 37 that generates BG), an adder 38, and an adder 39 are provided.

  The target pixel selection unit 31 scans each pixel constituting the input image, and sequentially selects one pixel as the target pixel from the plurality of pixels. Thereby, in each block of the demosaic unit 16, image processing relating to the selected target pixel is executed.

  The delay unit 32 performs a delay process on the input image so that the image generation timings in the high-resolution G generation unit 35, the high-resolution (RG) generation unit 36, and the high-resolution (BG) generation unit 37 are aligned. And output to the high-resolution G generator 35.

  The virtual color difference (R−G) generation unit 33 uses the input images R and G, and the color difference (R−G) at the R pixel position and the G pixel position of the input image (hereinafter referred to as virtual color difference (R−G)). ) Is generated. Further, the virtual color difference (B−G) generation unit 34 uses the input images B and G to perform color difference (B−G) (hereinafter referred to as virtual color difference (B−G)) at the B pixel position and the G pixel position of the input image. Is generated).

  The high-resolution G generation unit 35 generates a high-resolution G image having twice the number of pixels with respect to the input image G using only the input image G output from the delay unit 32. The high resolution (RG) generation unit 36 uses the virtual color difference (RG) output from the virtual color difference (RG) generation unit 33 to increase the number of pixels four times that of the input image R. A high-resolution R image having the same is generated. The high resolution (BG) generation unit 37 uses the virtual color difference (BG) output from the virtual color difference (BG) generation unit 34 to increase the number of pixels four times that of the input image B. A high resolution B image is generated.

  The adder 38 adds the high resolution G image output from the high resolution G generation unit 35 and the high resolution (RG) output from the high resolution (RG) generation unit 36. By doing so, a high-resolution R image is generated. The adder 39 adds the high resolution G image output from the high resolution G generator 35 and the high resolution (BG) output from the high resolution (BG) generator 37. By doing so, a high-resolution B image is generated.

<Configuration Example of Virtual Color Difference (RG) Generation Unit 33>
FIG. 3 is a block diagram illustrating a configuration example of the virtual color difference (R−G) generation unit 33. The virtual color difference (R−G) generation unit 33 includes a temporary G generation unit 41, a temporary R generation unit 42, and a virtual color difference (R−G) calculation unit 43.

  The provisional G generation unit 41 uses the input image G to generate a provisional G that is a G component value interpolated at the R position of the input image. The provisional R generation unit 42 uses the input image R to generate a provisional R that is an R component value interpolated at the G position of the input image.

  The virtual color difference (RG) calculation unit 43 generates a color difference (RG) at each R position and G position of the input image (hereinafter referred to as virtual color difference (RG)). Specifically, the virtual color difference (RG) calculation unit 43 uses the input image R and the temporary G generated by the temporary G generation unit 41 to calculate the R-temporary G at the R position of the input image. By calculating, a virtual color difference (R−G) is generated. The virtual color difference (R−G) calculation unit 43 calculates a temporary RG using the input image G and the temporary R generated by the temporary R generation unit 42 at the G position of the input image, A virtual color difference (RG) is generated. Note that the virtual color difference (RG) calculation unit 43 does not generate a virtual color difference at the B position of the input image.

FIG. 4 is a block diagram illustrating a configuration example of the provisional G generation unit 41.
The provisional G generation unit 41 includes a horizontal interpolation unit 51, a horizontal interpolation color difference calculation unit 52, a horizontal interpolation color difference statistic calculation unit 53, a vertical interpolation unit 54, a vertical interpolation color difference calculation unit 55, a vertical interpolation color difference statistic calculation unit 56, and A provisional G calculation unit 57 is provided.
The horizontal interpolation unit 81 calculates an interpolated value Gh that is a G component value interpolated in the horizontal direction at the pixel position of the R pixel of the input image (hereinafter referred to as R pixel position).

FIG. 5 is a diagram for explaining a method of interpolating the horizontal G component value at the R pixel position.
In the Bayer array, when the R pixel is a central pixel, pixels G1 to G4, which are four G pixels, are arranged around the central pixel (up, down, left, and right). In this case, the horizontal interpolation unit 81 uses the pixel value G2 of the G2 pixel located on both the left and right sides of the center pixel and the pixel value G3 of the G3 pixel in the horizontal direction at the pixel position of the center pixel according to Equation (1). An interpolated value Gh, which is the G component value interpolated in (5), is calculated.
Gh = (G2 + G3) / 2 (1)

  The interpolation value Gh interpolated by the horizontal interpolation unit 51 is supplied to the horizontal interpolation color difference calculation unit 52 and the provisional G calculation unit 57. Thereby, the pixel value R and the interpolation value Gh are obtained in the R pixel arrangement of the input image.

The horizontal interpolation color difference calculation unit 52 performs horizontal interpolation that is a difference between the pixel value R at the R pixel position in the input image and the horizontal interpolation value Gh supplied from the horizontal interpolation unit 51 at each R pixel position. The color difference rh is calculated. Specifically, the horizontal interpolation color difference calculation unit 52 calculates the horizontal interpolation color difference rh according to Expression (2).
rh = R−Gh (2)

The horizontal interpolation color difference rh obtained by the horizontal interpolation color difference calculation unit 52 is supplied to the horizontal interpolation color difference statistic calculation unit 53 and the provisional G calculation unit 57.
Based on the horizontal interpolation color difference rh supplied from the horizontal interpolation color difference calculation unit 52, the horizontal interpolation color difference statistic calculation unit 53 calculates an average Argh that is an average of the horizontal interpolation color difference rh and a variance Vrgh that is a variance of the horizontal interpolation color difference rh. Are supplied to the provisional G calculation unit 57.
On the other hand, the vertical interpolation unit 54 calculates an interpolation value Gv that is a G component value interpolated in the vertical direction at the R pixel position of the input image.

As shown in FIG. 5, the vertical interpolation unit 54 uses the pixel value G1 of the G1 pixel and the pixel value G4 of the G4 pixel located on both the upper and lower sides of the central pixel and the pixel of the central pixel according to the equation (3). An interpolation value Gv that is a G component value interpolated in the vertical direction at the position is calculated.
Gv = (G1 + G4) / 2 (3)

  The interpolation value Gv interpolated by the vertical interpolation unit 54 is supplied to the vertical interpolation color difference calculation unit 55 and the provisional G calculation unit 57. Thereby, the pixel value R and the interpolation value Gv are obtained in the R pixel arrangement of the input image.

The vertical interpolation color difference calculation unit 55 performs vertical interpolation, which is a difference between the pixel value R at the R pixel position in the input image and the vertical interpolation value Gv supplied from the vertical interpolation unit 54 at each R pixel position. The color difference rv is calculated. Specifically, the vertical interpolation color difference calculation unit 55 calculates the vertical interpolation color difference rv according to Equation (4).
rv = R−Gv (4)

  The vertical interpolation color difference rv obtained by the vertical interpolation color difference calculation unit 55 is supplied to the vertical interpolation color difference statistic calculation unit 56 and the provisional G calculation unit 57.

  Based on the vertical interpolation color difference rv supplied from the horizontal interpolation color difference calculation unit 55, the vertical interpolation color difference statistic calculation unit 56 calculates the average Argv that is the average of the vertical interpolation color difference rv and the variance Vrgv that is the variance of the vertical interpolation color difference rv. Are supplied to the provisional G calculation unit 57.

  The provisional G calculation unit 57 is calculated by the horizontal interpolation color difference calculation unit 52, the horizontal interpolation color difference rh obtained by the horizontal interpolation color difference calculation unit 52, the average Argh and variance Vrgh obtained by the horizontal interpolation color difference statistic calculation unit 53, and the vertical interpolation color difference calculation unit 55. Using the vertical interpolation color difference rv and the average Argv and variance Vrgv obtained by the vertical interpolation color difference statistic calculation unit 56, a temporary G that is a G component value at the R pixel position is calculated.

  First, the provisional G calculation unit 57 determines the magnitude of the variance Vrgh of the horizontal interpolation color difference rh and the variance Vrgv of the vertical interpolation color difference rv, and sets the larger one as LVrg and the smaller one as SVrg. Then, the temporary G calculation unit 57 calculates the temporary G at the R pixel position according to the equation (5).

  On the other hand, the virtual color difference (B−G) generation unit 34 is configured in the same manner as the virtual color difference (R−G) generation unit 33, but the R pixel position and pixel used in the virtual color difference (R−G) generation unit 33. Instead of the value R, the B pixel position and the pixel value B are used.

  Therefore, similarly to the virtual color difference (RG) generation unit 33, the virtual color difference (B−G) generation unit 34 performs the interpolation value Gh that is the G component value interpolated in the horizontal direction at the B pixel position, and the vertical direction. An interpolated value Gv, which is a G component value interpolated in (5), is calculated.

  Further, the virtual color difference (B−G) generation unit 34 calculates the horizontal interpolation color difference bh (= B−Gh), and based on the horizontal interpolation color difference bh, the average Abgh that is the average of the horizontal interpolation color difference bh, and the horizontal A variance Vbgh which is a variance of the interpolated color difference bh is calculated.

  Similarly, the virtual color difference (B−G) generation unit 34 calculates the vertical interpolation color difference bv (= B−Gv), and based on the vertical interpolation color difference bv, an average Abgv that is an average of the vertical interpolation color difference bv, A variance Vbgv that is a variance of the vertical interpolation color difference bv is calculated.

  The virtual color difference (B−G) generation unit 34 calculates a temporary G that is a G component value at the B pixel position by using the horizontal interpolation color difference bh, the average Abgh and variance Vbgh, the vertical interpolation color difference bv, the average Abgv, and variance Vbgv. To do.

  At this time, the size of the variance Vbgh of the horizontal interpolation color difference bh and the variance Vbgv of the vertical interpolation color difference bv is determined, and the larger one is set to LVbg and the smaller one is set to SVbg. Then, a temporary G at the B pixel position is calculated according to the equation (6).

  FIG. 6 is an explanatory diagram of the horizontal interpolation color difference and the vertical interpolation color difference. Each circle in the figure represents one pixel. A color component value is given for each pixel.

  At the R pixel position in the input image, a horizontal interpolation value Gh is calculated. Then, the difference between the pixel value R and the interpolation value Gh is calculated for each R pixel position, and the horizontal interpolation color difference rh is obtained. That is, rh = R−Gh is calculated at the R pixel position of the input image.

  Further, an interpolation value Gv in the vertical direction is calculated at the R pixel position. Then, for each R pixel position, the difference between the pixel value R and the interpolation value Gv is calculated to obtain the vertical interpolation color difference rv. That is, rv = R−Gv is also calculated at the R pixel position of the input image.

  On the other hand, the horizontal interpolation value Gh is calculated at the B pixel position in the input image. Then, for each B pixel position, the difference between the pixel value B and the interpolation value Gh is calculated to obtain the horizontal interpolation color difference bh. That is, bh = B−Gh is calculated at the B pixel position of the input image.

  Further, the interpolation value Gv in the vertical direction is calculated at the B pixel position. Then, for each B pixel position, the difference between the pixel value B and the interpolation value Gv is calculated to obtain the vertical interpolation color difference bv. That is, bv = B−Gv is also calculated at the B pixel position of the input image.

  The provisional Gr at the R pixel position is obtained for each reference pixel centered on the target position in the vertical interpolation color difference rv or the horizontal interpolation color difference rh. The provisional Gb at the B pixel position is obtained for each reference pixel centered on the target position in the vertical interpolation color difference bv or the horizontal interpolation color difference bh.

  FIG. 7 is a diagram for explaining how to obtain the temporary Gr when the R44 pixel in the input image is the target pixel. Note that any of R, G, and B written in each upper circle in the figure represents the color component of each pixel. Numbers following any of R, G, and B are numbered with two-digit numbers from 0 to 9 indicating the position in the Y direction and 0 to 9 indicating the position in the X direction in order from the upper left.

  In addition, rh and bh written in each lower circle in the drawing represent the horizontal interpolation color difference at the position of the R pixel or the B pixel. rv and bv represent the vertical interpolation color difference at the position of the R pixel or the B pixel. Furthermore, the numbers following rh, bh, rv, and bv are the same numbers as the corresponding pixels of the input image.

The horizontal interpolation color difference rh is calculated as follows, for example.
rh00 = R00-Gh00
rh02 = R02-Gh02
rh04 = R04-Gh04

The horizontal interpolation color difference bh is calculated as follows, for example.
bh11 = B11−Gh11
bh13 = B13−Gh13
bh15 = B15−Gh15

The vertical interpolation color difference rv is calculated as follows, for example.
rv00 = R00-Gv00
rv02 = R02-Gv02
rv04 = R04-Gv04

The vertical interpolation color difference bv is calculated as follows, for example.
bv11 = B11−Gv11
bv13 = B13−Gv13
bv15 = B15−Gv15
The same calculation is performed at other pixel positions.

  When the R44 pixel is the target pixel, the horizontal interpolation color difference average Argh and variance Vrgh, and the vertical interpolation color difference average Argv and variance Vrgv are obtained based on the pixel values of a plurality of reference pixels centered on the pixel position of the R44 pixel. .

  The plurality of reference pixels corresponds to 9 pixels (thick line circle in the lower part of FIG. 7) arranged in a cross shape with the R44 pixel as the center. However, the reference pixels are not limited to nine pixels arranged in a cross shape. The reference pixels may be arranged in a rectangular shape with the target pixel as the center, but it is preferable that the reference pixels have a small number of pixels and can cover a wide area.

  Specifically, the horizontal interpolation color difference average Argh and variance Vrgh, and the vertical interpolation color difference average Argv and variance Vrgv are calculated according to the following equation (7).

Argh = A (rh04, rh24, rh40, rh42, rh44, rh46, rh48, rh64, rh84)
Argv = A (rv04, rv24, rv40, rv42, rv44, rv46, rv48, rv64, rv84)
Vrgh = V (rh04, rh24, rh40, rh42, rh44, rh46, rh48, rh64, rh84)
Vrgv = V (rv04, rv24, rv40, rv42, rv44, rv46, rv48, rv64, rv84)
... (7)

  The provisional Gr at the R pixel position is calculated according to the above-described equation (5) based on the horizontal interpolation color difference average Argh and variance Vrgh, the vertical interpolation color difference average Argv and variance Vrgv, and the like obtained as described above. .

FIG. 8 is a diagram for explaining how to obtain the temporary Gr when the B55 pixel in the input image is the target pixel.
When the B55 pixel is the target pixel, the horizontal interpolation color difference average Abgh and variance Vbgh, and the vertical interpolation color difference average Abgv and variance Vbgv are obtained based on the pixel values of a plurality of reference pixels centered on the pixel position of the B55 pixel. . The plurality of reference pixels correspond to nine pixels (thick lined circle in the lower part of FIG. 8) arranged in a cross shape with the B55 pixel as the center, but are not limited to this example.

  Specifically, the average Abgh and variance Vbgh of the horizontal interpolation color difference and the average Abgv and variance Vbgv of the vertical interpolation color difference are calculated according to the following equation (8).

Abgh = A (bh15, bh35, bh51, bh53, bh55, bh57, bh59, bh75, rh95)
Abgv = A (bv15, bv35, bv51, bv53, bv55, bv57, bv59, bv75, rv95)
Vbgh = V (bh15, bh35, bh51, bh53, bh55, bh57, bh59, bh75, rh95)
Vbgv = V (bv15, bv35, bv51, bv53, bv55, bv57, bv59, bv75, rv95)
... (8)

  The provisional Gb at the pixel position is calculated according to the above-described equation (6) based on the average Abgh and variance Vbgh of horizontal interpolation color differences, the average Abgv and variance Vbgv of vertical interpolation color differences obtained as described above.

<Provisional G generation routine>
FIG. 9 is a flowchart illustrating a temporary G generation routine by the temporary G generation unit 41 in the virtual color difference (RG) generation unit 33. Here, generation of the provisional G at the R pixel position will be described. The generation of the temporary G at the B pixel position, that is, the generation of the temporary G by the virtual color difference (B−G) generation unit 34 is performed in the same manner as the routine shown in FIG.

  The input image of the provisional G generation unit 41 is an image output from the image sensor 12 in which a Bayer array color filter array is arranged. Therefore, the input image is composed of a G image output from the G pixel of the image sensor 12, an R image output from the R pixel, and a B image output from the B pixel.

  In step S11, the horizontal interpolation unit 51 and the vertical interpolation unit 54 in the provisional G generation unit 41 calculate Expressions (1) and (3) at the pixel positions of the R pixel and the B pixel in the input image. As a result, an interpolation value Gh that is the value of the G component interpolated in the horizontal direction at the pixel position of the center pixel and an interpolation value Gv that is the value of the G component interpolated in the vertical direction are calculated. Then, the process proceeds to step S12.

  In step S12, the horizontal interpolation color difference calculation unit 52 and the vertical interpolation color difference calculation unit 55 calculate Expressions (2) and (4) at the respective pixel positions of the R pixel and the B pixel in the input image. Accordingly, the horizontal interpolation color difference rh and the vertical interpolation color difference rv are calculated, and the calculated values are held in a memory (not shown). Then, the process proceeds to step S13.

  In step S13, the horizontal interpolation color difference statistic calculation unit 53 and the vertical interpolation color difference statistic calculation unit 56 calculate the horizontal interpolation color difference average Argh and variance Vrgh, the vertical interpolation color difference average Argv, and the variance Vrgv.

  In step S14, the provisional G calculation unit 57 calculates the horizontal interpolation color difference average Argh and the variance Vrgh calculated by the horizontal interpolation color difference statistic calculation unit 53 and the vertical interpolation color difference statistic calculation unit 56, the average Argv of the vertical interpolation color difference, and the variance. Using Vrgv or the like, weighting is performed in the horizontal direction and the vertical direction in accordance with Expression (5), and a temporary Gr at the R pixel position is calculated. When the provisional Gr is generated, the provisional Gr generation routine ends.

  On the other hand, the temporary R generation unit 42 illustrated in FIG. 3 generates a temporary R that is an R component value using pixel values of a plurality of R pixels around the R pixel position at the G pixel position of the input image. .

FIG. 10 is a diagram for explaining the generation of the provisional R at the G pixel position.
For example, when the G pixel between the R00 pixel and the R02 pixel becomes the central pixel, the temporary R generation unit 42 uses the pixel values R00 and R02 of the R00 pixel and the R02 pixel adjacent to the left and right sides of the central pixel. Then, a temporary R (r01) that is an R component value interpolated in the horizontal direction is calculated according to the equation (9).
r01 = (R00 + R02) / 2 (9)

Further, for example, when the G pixel between the R00 pixel and the R20 pixel becomes the central pixel, the temporary R generation unit 42 calculates the pixel values R00 and R20 of the R00 pixel and the R20 pixel that are adjacent to the upper and lower sides of the central pixel. The temporary R (r10), which is the R component value interpolated in the vertical direction, is calculated according to the equation (10).
r01 = (R00 + R20) / 2 (10)

  As described above, the provisional R generation unit 42 in the virtual color difference (RG) generation unit 33 interpolates the pixel values of R pixels adjacent in the horizontal direction or the vertical direction at the G pixel position of the input image. , Provisional R is generated. Similarly, the virtual color difference (B−G) generation unit 34 generates a temporary B by interpolating the pixel values of B pixels adjacent in the horizontal direction or the vertical direction at the G pixel position of the input image.

  The virtual color difference (R−G) calculation unit 43 illustrated in FIG. 3 has the pixel value R at the R pixel position at the R pixel position of the input image and the temporary G generated by the temporary G generation unit 41 at the R pixel position. And a virtual color difference (R-provisional G) is generated. In addition, the virtual color difference (R−G) calculation unit 43, at the G pixel position of the input image, the temporary R generated by the temporary R generation unit 42 at the G pixel position, the pixel value G at the G pixel position, To calculate a virtual color difference (provisional RG). The virtual color difference (R−G) is a general term for the virtual color difference (R−provisional G) and the virtual color difference (provisional RG). Note that a virtual color difference (RG) is not generated at the B pixel position. The virtual color difference (R−G) is supplied to the high resolution (R−G) generation unit 36.

  As described above, the virtual color difference (RG) calculation unit 43 in the virtual color difference (RG) generation unit 33 generates the virtual color difference (RG) at the R pixel position and the G pixel position of the input image. Is done. Similarly, the virtual color difference (B−G) generation unit 34 generates a virtual color difference (B−G) at the B pixel position and the G pixel position of the input image. Note that a virtual color difference (B−G) is not generated at the R pixel position. The virtual color difference (B−G) is supplied to the high resolution (B−G) generation unit 37.

<Generation of high-resolution G image>
The high-resolution G generation unit 35 illustrated in FIG. 2 generates a high-resolution G image having a resolution twice that of the input G image based on the input G image supplied from the delay unit 32. Specifically, the high-resolution G generation unit 35 uses a pixel corresponding to the target pixel in the input image and its surrounding pixel values as variables, and performs a product-sum operation using coefficients obtained by learning in advance. The pixel value of the target pixel of the output image is predicted.

FIG. 11 is a block diagram illustrating a configuration example of the high-resolution G generation unit 35.
The high resolution G generation unit 35 includes a G code tap selection unit 61, a G filter tap selection unit 62, a G code calculation unit 63, a G coefficient storage unit 64, and a G product sum calculation unit 65.

  The G code tap selection unit 61 generates a code tap (hereinafter referred to as a G code tap) composed of a G component having a predetermined pattern necessary for generating a high resolution G image from the input image supplied from the delay unit 32. select. The G code tap is composed of, for example, a predetermined number of pixels centered on the pixel of the input image at a position corresponding to the target pixel of the output image. The G code tap selected by the G code tap selection unit 61 is supplied to the G code calculation unit 63.

<Configuration example of code tap>
FIG. 12 is a diagram illustrating a configuration example of the G code tap selected by the G code tap selection unit 61. A thin line circle represents an input pixel which is an input image. A dot-patterned circle represents an image of the pixel position phase with respect to the input pixel of the output image.

  As shown in FIG. 12, the output pixel exists at a position shifted in phase from the input pixel. Therefore, the input image is converted into an output image shifted by a half pixel phase. The center pixel of the cross-dotted line represents the input pixel that is the target pixel. A bold circle represents an input pixel to be a G code tap. Note that the G code tap is not limited to the configuration example shown in FIG. 12, and may have other configurations.

  For example, the G code tap selection unit 61 selects an input pixel to be a G code tap for the target pixel with reference to the input pixel closest to the position of the input pixel corresponding to the target pixel, and selects the selected G code tap as G. This is supplied to the code calculation unit 63.

  Based on the G code tap selected by the G code tap selection unit 69, the G code calculation unit 63 encodes, for example, quantization by DR quantization and other dynamic range widths to extract feature amounts and the like. The G code calculation unit 63 classifies the target pixel according to a predetermined rule using the extracted information such as the feature amount, and as a result, obtains a G code corresponding to the target pixel. The G code calculation unit 63 supplies the G code corresponding to the target pixel to the G coefficient storage unit 64. The detailed configuration of the G code calculation unit 63 will be described later.

  In the G coefficient storage unit 64, tap coefficients for each G code obtained by learning to be described later are stored in advance in association with the G code. And the G coefficient memory | storage part 64 reads the tap coefficient memorize | stored in the address corresponding to G code supplied from the G code calculating part 63 from the tap coefficients already memorize | stored, and reads the read tap coefficient. The G product-sum operation unit 65 is supplied.

  The G filter tap selection unit 62 selects and acquires from the input image a filter tap (hereinafter referred to as a G filter tap) made up of a predetermined pattern of G components necessary for generating a high resolution G image. The G filter tap is composed of a predetermined number of pixels, for example, with the pixel of the input image at the position corresponding to the target pixel of the output image as the central pixel. The G filter tap selected by the G filter tap selection unit 62 is supplied to the G product-sum operation unit 65.

<Configuration example of G filter tap>
FIG. 13 is a diagram illustrating a configuration example of filter taps selected by the G filter tap selection unit 62. A thin line circle represents an input pixel which is an input image. A dot-patterned circle represents an image of the pixel position phase with respect to the input pixel of the output image.

  As shown in FIG. 13, the output pixel exists at a position shifted in phase from the input pixel. Therefore, the input image is converted into an output image shifted by a half pixel phase. The center pixel of the cross-dotted line represents the input pixel that is the target pixel. Furthermore, a bold circle represents an input pixel that becomes a G filter tap. Note that the G filter tap is not limited to the configuration example illustrated in FIG. 13, and may have another configuration. Further, the G filter tap shown in FIG. 13 may have a different configuration from the G code tap shown in FIG.

  The G filter tap selection unit 62 selects, for example, an input pixel that serves as a filter tap for the target pixel with reference to the input pixel closest to the position of the input pixel corresponding to the target pixel.

  The G product-sum operation unit 65 substitutes the G filter tap output from the G filter tap selection unit 62 as a variable in a linear linear expression set in advance, and uses the coefficient supplied from the G coefficient storage unit 64 to perform filtering. Perform the operation. That is, the G product-sum operation unit 65 performs a filter operation on the pixel value of the output pixel corresponding to the target pixel in the input G image using the G filter tap according to the equation (11).

Here, N is the number of pixels of the G filter tap. _Xn is the pixel value of each G pixel that constitutes the G filter tap selected by the G filter tap selection unit 62. W _n is a tap coefficient read from the G coefficient storage unit 64.

  The G product-sum operation unit 65 predicts (calculates) an output pixel corresponding to the target pixel by performing a filter operation represented by Expression (11). The G product-sum operation unit 65 predicts (calculates) an output pixel corresponding to the target pixel every time one target pixel is selected from all the G pixels of the input image. As a result, a high-resolution G image having a resolution twice that of the input G image is generated.

  The high resolution G generator 35 shown in FIG. 2 outputs the generated high resolution G image to the outside and supplies it to the adder 38.

<Configuration Example of G Code Calculation Unit 63>
FIG. 14 is a block diagram illustrating a configuration example of the G code calculation unit 63 described above. The G code calculation unit 63 includes a quantization calculation unit 91 and a conversion table storage unit 92.
The quantization calculation unit 91 quantizes each pixel value of the input pixel constituting the G code tap supplied from the G code tap selection unit 61 using, for example, 1-bit DR quantization, and 1 bit of each input pixel The codes arranged in a predetermined order are supplied to the conversion table storage unit 92.

Here, as one method for performing code classification, for example, there is DR (Dynamic Range) quantization that quantizes pixel values as code taps.
In the DR quantization, the pixel value of each pixel constituting the code tap of the pixel of interest is quantized, and the code of the pixel of interest is determined according to the DR quantization code obtained by the quantization.

For example, in the case of N-bit DR quantization, the maximum value MAX and the minimum value MIN of the pixel values of each pixel constituting the code tap are detected. Then, a local dynamic range DR (= MAX−MIN) of a set of pixels constituting the code tap is calculated. Based on the dynamic range DR, the pixel value of each pixel constituting the code tap is quantized to N bits. More specifically, the minimum value MIN is subtracted from the pixel value of each pixel constituting the code tap, and the subtracted value is divided (quantized) by DR / ^2N . The pixel values of the N-bit pixels constituting the code tap obtained by the above processing are arranged in a predetermined order, and the arranged bit string is output as a DR quantization code.

  Therefore, when the code tap is subjected to, for example, 1-bit DR quantization processing, the pixel value of each pixel constituting the code tap is divided by the average of the maximum value MAX and the minimum value MIN (integer operation). Thereby, the pixel value of each pixel is set to 1 bit (binarization). Then, a bit string in which the 1-bit pixel values are arranged in a predetermined order is output as a DR quantization code. When code classification is performed only by DR quantization, for example, the DR quantization code is a code generated by the G code calculation unit 63.

For example, the G code calculation unit 63 can output the level distribution pattern of the pixel values of the pixels constituting the code tap as a class code as it is. In this case, if the code tap is composed of pixel values of M pixels, and the A bit is assigned to the pixel value of each pixel, the number of codes output from the code calculation unit 34 is (2 ^M ) ^A, and a huge number that is exponentially proportional to the number of bits A of the pixel value of the pixel.

  Therefore, the G code calculation unit 63 preferably performs code classification by compressing the information amount of the code tap by the above-described DR quantization or vector quantization.

  FIG. 15 is a diagram for explaining an example of 1-bit DR quantization. The horizontal axis represents the order (or position) of the input pixels that constitute the code tap. The vertical axis represents the pixel value of the input pixel constituting the code tap.

  In 1-bit DR quantization, the simple dynamic range DR is obtained by subtracting the minimum pixel value Min from the maximum pixel value Max among the pixel values of the input pixels constituting the code tap. Then, the pixel value of each input pixel constituting the code tap is binarized using a level that bisects the simple dynamic range DR as a threshold value, and converted into a 1-bit code.

  In the case of the code tap shown in FIG. 13, for example, when 1-bit DR quantization is performed on the code taps of all 8 pixels, an 8-bit DR quantization code is obtained. The code obtained by the quantization operation unit 91 is supplied to the conversion table storage unit 92.

  The conversion table storage unit 92 stores a conversion table. This conversion table optimizes the code (input code) obtained by the quantization operation unit 91 and converts it into a code (output code) that can correspond to the G coefficient storage unit 64. Therefore, the code obtained by the quantization operation unit 91 is converted into a code for the G coefficient storage unit 64 by the conversion table stored in advance in the conversion table storage unit 92. Then, the converted code is supplied to the G coefficient storage unit 64.

  When the storage capacity of the G coefficient storage unit 64 is sufficiently large, the conversion table storage unit 92 can be omitted. In this case, the code obtained from the quantization operation unit 91 is supplied to the G coefficient storage unit 64 as it is.

  Further, the conversion table stored in the conversion table storage unit 92 can be updated as appropriate so as to optimize the output code. Thereby, in the conversion table, for example, when the frequency of occurrence of a certain input code is low, the output code corresponding to the input code may be replaced with a representative code. In the conversion table, even if different output codes are associated with approximate coefficients, different output codes may be the same code.

<Generation of high-resolution (RG) image>
The high resolution (RG) generation unit 36 illustrated in FIG. 2 has the same resolution as the high resolution G image based on the virtual color difference (RG) generated by the virtual color difference (RG) generation unit 33. A high-resolution (RG) image is generated.

FIG. 16 is a block diagram illustrating a configuration example of the high resolution (RG) generation unit 36.
The high-resolution G generation unit 35 includes an (RG) code tap selection unit 71, an (RG) filter tap selection unit 72, an (RG) code calculation unit 73, and an (RG) coefficient storage unit. 74 and (R−G) product-sum operation unit 75.

  The (R−G) code tap selection unit 71 generates a high resolution (R−G) image from the virtual color difference (R−G) image generated by the virtual color difference (R−G) generation unit 33. For this purpose, a code tap (hereinafter referred to as (R-G) code tap) composed of color difference (RG) components of a predetermined pattern required for the purpose is selected. The (R−G) code tap selected by the (R−G) code tap selection unit 71 is supplied to the (R−G) code calculation unit 73.

<Configuration example of (R-G) code tap>
FIG. 17 is a diagram illustrating a configuration example of the (R−G) code tap selected by the (R−G) code tap selection unit 71. A thin line circle represents an input pixel which is an input image. A dot-patterned circle represents an image of the pixel position phase with respect to the input pixel of the output image.

  As shown in FIG. 17, the output pixel is present at a position shifted in phase from the input pixel. Therefore, the input image is converted into an output image shifted by a half pixel phase. The center pixel of the cross-dotted line represents the input pixel that is the target pixel. A bold circle represents an input pixel to be an (R−G) code tap. Note that the (R-G) code tap is not limited to the configuration example shown in FIG. 12, and may have other configurations.

  The (R−G) code tap selection unit 71 selects, for example, an input pixel that becomes a (R−G) code tap for the target pixel with reference to the input pixel closest to the position of the input pixel corresponding to the target pixel. .

  Based on the (R−G) code tap selected by the (R−G) code tap selection unit 71, the (R−G) code calculation unit 73, for example, performs quantization by DR quantization or other dynamic range widths. Encode and extract features. The (R−G) code calculation unit 73 classifies the pixel of interest according to a predetermined rule using information such as the extracted feature quantity, and as a result, obtains an (R−G) code corresponding to the pixel of interest. The G code calculation unit 63 supplies the (R−G) code corresponding to the target pixel to the (R−G) coefficient storage unit 74.

  In the (R−G) coefficient storage unit 74, tap coefficients for each (R−G) code obtained by learning described later are stored in advance in association with the (R−G) code. Then, the (R−G) coefficient storage unit 74 is stored at an address corresponding to the (R−G) code supplied from the (R−G) code calculation unit 73 among the already stored tap coefficients. The tap coefficient is read, and the read tap coefficient is supplied to the (R−G) product-sum operation unit 75.

  The (R−G) filter tap selection unit 72 generates a high resolution (R−G) image from the virtual color difference (R−G) image generated by the virtual color difference (R−G) generation unit 33. For this purpose, a filter tap (hereinafter referred to as (RG) filter tap) composed of (RG) components of a predetermined pattern required for this purpose is selected and acquired. The (R−G) filter tap selection unit 72 supplies the selected (R−G) filter tap to the (R−G) product-sum operation unit 75.

<Configuration example of (R-G) filter tap>
FIG. 18 is a diagram illustrating a configuration example of the (R−G) filter tap selected by the (R−G) filter tap selection unit 72. A thin line circle represents an input pixel which is an input image. A dot-patterned circle represents an image of the pixel position phase with respect to the input pixel of the output image.

  As shown in FIG. 18, the output pixel is present at a position shifted in phase from the input pixel. Therefore, the input image is converted into an output image shifted by a half pixel phase. The center pixel of the cross-dotted line represents the input pixel that is the target pixel. Further, a bold circle represents an input pixel that becomes an (R−G) filter tap. Note that the (R−G) filter tap is not limited to the configuration example illustrated in FIG. 18, and may have another configuration. Further, the G filter tap shown in FIG. 18 may have a different configuration from the G code tap shown in FIG.

  The (R−G) filter tap selection unit 72 selects, for example, an input pixel that becomes a (R−G) filter tap for the target pixel with reference to the input pixel closest to the position of the input pixel corresponding to the target pixel. .

  The (R−G) product-sum operation unit 75 substitutes the (R−G) filter tap output from the (R−G) filter tap selection unit 72 as a variable in a linear linear expression set in advance, and (R -G) The filter operation is performed using the coefficient supplied from the coefficient storage unit 74. That is, the (R−G) product-sum operation unit 75 sets the (R−G) filter tap to the pixel value of the output pixel corresponding to the target pixel in the input (R−G) image according to the above equation (11). Use to filter.

In Expression (11), N is the number of pixels of the (R−G) filter tap. _Xn is the (R−G) component value of each pixel constituting the (R−G) filter tap selected by the (R−G) filter tap selection unit 72. W _n is a tap coefficient read from the (R−G) coefficient storage unit 74.

  The (R−G) product-sum operation unit 75 predicts (calculates) an output pixel corresponding to the target pixel by performing a filter operation represented by Expression (11). The (R−G) product-sum operation unit 75 predicts an output pixel corresponding to a target pixel every time one target pixel is selected from pixels having all (R−G) components of the input image. (Calculate). As a result, a high resolution (RG) image having the same resolution as the high resolution G image is generated.

Then, the high resolution (RG) generator 36 shown in FIG. 2 supplies the generated high resolution (RG) image to the adder 38.
The high resolution (BG) generation unit 37 is configured in the same manner as the high resolution (RG) generation unit 36, and handles B pixels instead of R pixels.

<Configuration example of (B-G) code tap>
FIG. 19 is a diagram illustrating a configuration example of a (BG) code tap. A thin line circle represents an input pixel which is an input image. A dot-patterned circle represents an image of the pixel position phase with respect to the input pixel of the output image.

  As shown in FIG. 19, the output pixel is present at a position shifted in phase from the input pixel. Therefore, the input image is converted into an output image shifted by a half pixel phase. The center pixel of the cross-dotted line represents the input pixel that is the target pixel. Further, a bold circle represents an input pixel that becomes a (BG) filter tap. The (B-G) code tap is not limited to the configuration example shown in FIG. 19, and may have another configuration.

<(B-G) Configuration Example of Filter Tap>
FIG. 20 is a diagram illustrating a configuration example of the (BG) filter tap. A thin line circle represents an input pixel which is an input image. A dot-patterned circle represents an image of the pixel position phase with respect to the input pixel of the output image.

  As shown in FIG. 20, the output pixel is present at a position shifted in phase from the input pixel. Therefore, the input image is converted into an output image shifted by a half pixel phase. The center pixel of the cross-dotted line represents the input pixel that is the target pixel. Further, a bold circle represents an input pixel that becomes a (BG) filter tap. Note that the (B-G) filter tap is not limited to the configuration example illustrated in FIG. 20 and may have other configurations. Moreover, the (BG) filter tap shown in FIG. 20 may have a configuration different from the G code tap shown in FIG.

  The high resolution (BG) generation unit 37 performs the same processing as that of the high resolution (RG) generation unit 36, and thus a high resolution (BG) image having the same resolution as the high resolution G image. Is generated.

The adder 38 combines the high-resolution G image generated by the high-resolution G generation unit 35 and the high-resolution (RG) image generated by the high-resolution (RG) generation unit 36. By adding, a high-resolution R image having the same resolution as the high-resolution G image is generated.
The adder 39 combines the high resolution G image generated by the high resolution G generation unit 35 and the high resolution (BG) image generated by the high resolution (BG) generation unit 37. By adding, a high-resolution B image having the same resolution as the high-resolution G image is generated.

<Demosaic processing routine>
FIG. 21 is a flowchart showing a demosaic processing routine by the demosaic unit 16.

  Each block shown in FIG. 2 and FIG. 3 in the demosaic unit 16 executes the processing after step S21 on the input image formed by the output value of the image sensor 12 using the color filter array of the Bayer array. .

  In step S21, the provisional G generation unit 41 illustrated in FIG. 3 generates a provisional G that is interpolated from the pixel values G of adjacent G pixels at the R pixel position. Similarly, the virtual color difference (B−G) generation unit 34 generates a temporary G that is interpolated from the pixel values G of adjacent G pixels at the B pixel position.

  In step S22, the temporary R generation unit 42 generates a temporary R interpolated from the pixel value R of the adjacent R pixel at the G pixel position. Similarly, the virtual color difference (B−G) generation unit 34 generates a temporary B interpolated from the pixel value B of the adjacent B pixel at the G pixel position.

  In step S <b> 23, the virtual color difference (R−G) calculation unit 43 generates a virtual color difference (R−temporary G) using the temporary G generated by the temporary G generation unit 41 and the R pixel value of the input image. . Further, the virtual color difference (R−G) calculation unit 43 generates a virtual color difference (temporary RG) using the temporary R generated by the temporary R generation unit 42 and the pixel value G of the input image.

  Similarly, the virtual color difference (B−G) generation unit 34 generates a virtual color difference (B−temporary G) using the temporary G and the B pixel value of the input image, and the temporary B and the pixel value G of the input image Is used to generate a virtual color difference (temporary BG).

  In step S24, the pixel-of-interest selecting unit 31 selects one pixel that has not been selected as the pixel of interest among the pixels constituting the output image for the input image as the pixel of interest. That is, for example, the pixel-of-interest selection unit 31 selects, as the pixel of interest, a pixel that has not yet been selected as the pixel of interest in the raster scan order among the pixels that constitute the output image.

  In step S <b> 25, the G code tap selection unit 61 selects a G code tap for the pixel of interest for the input image. The (R−G) code tap selection unit 71 selects the (R−G) code tap for the pixel of interest for the input image. The high resolution (B-G) generation unit 37 selects a (B-G) code tap for the pixel of interest for the input image.

  In step S26, the G code calculation unit 63 encodes the supplied G code tap to generate an encoded code. The (R−G) code calculation unit 73 encodes the supplied (R−G) code tap to generate an encoded code. The high-resolution (BG) generation unit 37 encodes the (B-G) code tap to generate an encoded code.

  In step S27, the G filter tap selection unit 62 selects a G filter tap for the target pixel for the input image. The (R−G) filter tap selection unit 72 selects the (R−G) filter tap for the pixel of interest for the input image. The high resolution (B−G) generation unit 37 selects a (B−G) filter tap for the pixel of interest for the input image.

  In step S28, the G coefficient that is a tap coefficient associated with the encoded code calculated in step S26 is read from the G coefficient storage unit 64. The (R−G) coefficient associated with the encoded code calculated in step S <b> 26 is read from the (R−G) coefficient storage unit 74. Further, in the high resolution (BG) generation unit 37, the (B−G) coefficient associated with the encoded code calculated in step S26 is read from the internal storage unit.

In step S29, the G product-sum operation unit 65 performs product-sum operation on the target pixel. Specifically, G product-sum operation unit 65 substitutes _{X 1,} X 2, · · ·, each pixel value of the G filter tap selected at step S27 with respect to _{X N} of formula (11), substituting G coefficients read in step S28 with respect to the tap coefficient _{w n,} performing the calculation of equation (11). Similarly, the (R−G) product-sum operation unit 75 and the high-resolution (B−G) generation unit 37 also perform the filter operation on the target pixel according to Expression (11).

  In step S30, it is determined whether the process is a G pixel generation process. If the process is a G pixel generation process, the process proceeds to step S32. If the process is not a G pixel generation process, the process proceeds to step S31.

  In step S31, the adder 38 adds the pixel value G subjected to the filter operation in the process of step S29 and the color difference component (RG) similarly subjected to the filter operation for the target pixel, thereby obtaining a high resolution. A pixel value R constituting the image B image is generated. Further, the adder 39 adds the pixel value G subjected to the filter operation in the process of step S29 and the color difference component (BG) similarly subjected to the filter operation for the target pixel, thereby obtaining a high resolution B. A pixel value R constituting the image is generated.

  In step S <b> 32, the target pixel selection unit 31 determines whether there is a pixel that has not yet been selected as the target pixel. If there is a pixel that is not selected as the pixel of interest, the process returns to step S24, and if there is no pixel, this routine ends.

  FIG. 22 is a diagram illustrating generated pixels for input pixels of the demosaic unit 16. As shown in the figure, the resolution of the G generation pixel is doubled with respect to the G input pixel. The resolution of the R generation pixel is four times that of the R input pixel. The resolution of the B generation pixel is four times that of the B input pixel.

  As described above, the demosaic unit 16 uses the learning result of the code classification type adaptive filter by using only the pixel value of the G pixel from the input image of the image sensor 12 in which the color filters of the Bayer array are arranged. Thus, a high resolution G image can be generated.

  Further, the demosaic unit 16 generates a high resolution (RG) using the learning result of the code classification type adaptive filter using the pixel values of the R pixel and the G pixel from the input image. Then, the demosaic unit 16 generates a high resolution R image based on the high resolution (RG) and the high resolution G image. That is, the demosaic unit 16 can generate a high resolution R image without using the pixel value of the B pixel. Similarly, the demosaic unit 16 can generate a high-resolution B image without using the pixel value of the R pixel. As a result, a high-resolution image in which zipper noise, false color, color bleeding, ringing, and the like are significantly reduced is generated.

<Learning Principle of Code Classification Adaptive Filter>
The tap coefficient used for the demosaic process is obtained by the following procedure by the learning process of the code classification type adaptive filter.

  Here, the low-quality image (first image) refers to an image in which the image quality (resolution) has been reduced by filtering a high-quality image (second image) with an LPF (low pass filter).

  In the learning process of the code classification type adaptive filter, a filter tap is selected from the low-quality image, and the pixel value of the pixel of the high-quality image (high-quality pixel) is obtained by a predetermined calculation using the filter tap and the tap coefficient. . The predetermined calculation is, for example, a linear primary prediction calculation.

  In this case, the pixel value y of the high-quality pixel is obtained by a linear linear expression shown in Expression (12).

Filter tap, the pixel of the N low-quality image (low image quality _{pixel) x 1, x 2, ···} , x n, ···, composed of _{x N. xn} represents the pixel value of the nth low image quality pixel among the N low image quality pixels constituting the filter tap for the high image quality pixel y. w _n represents the n th tap coefficient to be multiplied by the n th low image quality pixel (its pixel value) x _n .

  The pixel value y of the high-quality pixel is not limited to the linear primary expression shown in Expression (12), and can be calculated by, for example, a higher-order expression of the second or higher order.

The true value of the pixel value of the high-quality pixel of the k-th sample (k-th) is expressed as y _k, and the predicted value of the true value y _k obtained by Expression (12) is expressed as y _k ′. In this case, the prediction error _{e k} with respect to the true value _{y k} of the prediction value _{y k} 'is expressed by equation (13).

The predicted value y _k ′ of Expression (13) is obtained according to Expression (12). Therefore, when y _k ′ in equation (13) is replaced according to equation (12), the following equation (14) is obtained.

x _{n, k} represents the n-th low-quality pixel that constitutes the filter tap for the high-quality pixel y _k of the k-th sample.

Equation (14) (or formula (13)) tap coefficients _{w n} to zero prediction error _{e k} of, the optimum tap coefficient for predicting the high-quality pixel _{y k.} However, for all of the high definition pixel y _k, to determine such tap coefficients w _n is generally difficult.

Therefore, as the standard for indicating that the tap coefficient w _n is the optimal value, for example, employs a method of least squares. In this case, the optimal tap coefficient w _n is a coefficient when the sum E of square errors expressed by the following equation (15) is minimized.

K is the number of samples in the set of the high image quality pixel y _k and the low image quality pixels x _{1, k} , x _{2, k} ,..., X _{n, k} that constitute the filter tap for the high image quality pixel y _k. (Number of learning samples).

The minimum value of the sum E of square errors of Equation (15) (minimum value), as shown in equation (16), given the results of the sum E were knitted differentiated by the tap coefficient w _n by w _n to 0.

Therefore, when knitting differentiated by the tap coefficient _{w n} of Equation (14) described above, the following equation (17) is obtained.

  Expression (18) is obtained from Expression (16) and Expression (17).

Substituting equation (14) to the _{e k} of the formula (18), equation (18) is represented by the normal equation shown in the following equation (19).

Normal equation of Equation (19), for example, by using a like sweeping-out method (Gauss-Jordan elimination method) can be solved for the tap coefficient _{w n.}

By solving the normal equations in equation (19) in each code, the optimum tap coefficient (a tap coefficient that minimizes the sum E of square errors) w _n is obtained for each code. Thus the tap coefficient w _n which is determined is stored in the G coefficient storage unit 64, for example, as a G factor. The tap coefficient is obtained, for example, by the learning device 100 shown in FIG.

<Configuration of Learning Device 100>
FIG. 23 is a block diagram illustrating a configuration example of the learning device 100. The learning apparatus 100 is used to generate a high-resolution G image, a high-resolution (RG) image, and a high-resolution (BG) image by learning using a teacher image and a student image, respectively. G coefficient, (RG) coefficient, and (BG) coefficient are generated.

  The learning apparatus 100 includes a student image generation unit 101, a teacher image generation unit 102, a pixel-of-interest selection unit 103, a delay unit 104, a virtual color difference (RG) generation unit 105, a virtual color difference (B-G) generation unit 106, G A coefficient data generation unit 107, an (RG) coefficient data generation unit 108, and a (B−G) coefficient data generation unit 109 are provided.

  The high-resolution image input to the learning device 100 is, for example, a high-resolution image RGB of each RGB component generated from each of the three image sensors corresponding to the R component, the G component, and the B component.

FIG. 24 is a diagram for explaining a student image and a teacher image generated from a high-resolution image.
The student image generation unit 101 down-converts the high-resolution images RGB input to the learning device 100, respectively. Here, the down-conversion filter characteristics may be, for example, 4-pixel averaging processing, LPF processing, or the like. Thereafter, the student image generation unit 102 generates a student image in which RGB is arranged according to the Bayer arrangement from the downconverted image RGB. That is, the student image corresponds to an image output as it is from the image sensor 12 in which the Bayer color filter array is arranged.

  The teacher image generation unit 102 down-converts the high-resolution image RGB input to the learning device 100 and shifts the down-converted image RGB by half a pixel, thereby generating a teacher image RGB for each component of RGB. The down conversion filter characteristics are the same as those of the student image generation unit 101.

  The target pixel selection unit 103 selects an arbitrary pixel in the teacher image generated by the teacher image generation unit 102 as the target pixel. The pixel-of-interest selection unit 103 provides information such as the coordinate value of the pixel selected as the pixel of interest, the virtual color difference (RG) generation unit 105, the G coefficient data generation unit 107, the (RG) coefficient data generation unit 108, etc. To each block.

  The virtual color difference (R−G) generation unit 105 is configured similarly to the virtual color difference (R−G) generation unit 33 illustrated in FIG. 2 and performs the same processing. That is, the virtual color difference (R−G) generation unit 105 uses the R component and the component G of the student image generated by the student image generation unit 101 to calculate the color difference (R−G) at the R pixel position and the G pixel position. A certain virtual color difference (R−G) is generated.

  The virtual color difference (B−G) generation unit 106 is configured similarly to the virtual color difference (B−G) generation unit 34 illustrated in FIG. 2 and performs the same processing. That is, the virtual color difference (B−G) generation unit 106 uses the component B and the component G of the student image generated by the student image generation unit 101 to calculate the color difference (B−G) at the B pixel position and the G pixel position. A certain virtual color difference (B−G) is generated.

  The delay unit 104 delays the student image generated by the student image generation unit 101 so as to match the output timings of the virtual color difference (RG) generation unit 105 and the virtual color difference (BG) generation unit 106. This is supplied to the G coefficient data generation unit 107.

<Configuration Example of G Coefficient Data Generation Unit 107>
FIG. 25 is a diagram illustrating a configuration example of the G coefficient data generation unit 107.
The G coefficient data generation unit 107 includes a G code tap selection unit 111, a G filter tap selection unit 112, a G code calculation unit 113, a G normal equation addition unit 114, and a G coefficient data calculation unit 115.

  The G code tap selection unit 111, the G filter tap selection unit 112, and the G code calculation unit 113 are configured similarly to the G code tap selection unit 61, the G filter tap selection unit 62, and the G code calculation unit 63 shown in FIG. The same processing is performed.

For each code supplied from the G code calculation unit 113, the G normal equation addition unit 114 generates a pixel of interest (teacher data) y _{k of} the teacher image G generated by the teacher image generation unit 102 and a G filter tap selection unit 112. The G filter taps (student data (pixels)) _{xn, k} for the target pixel selected in (1) are added.

Then, the G normal equation adding unit 114 uses the filter tap (student data) x _{n, k} for each code supplied from the G code calculating unit 113 and multiplies the student data in the matrix on the left side of Expression (19). An operation corresponding to (x _{n, k} x _{n ′, k} ) and summation (Σ) is performed.

Further, the G normal equation adding unit 114 uses the G filter tap (student data) x _{n, k} and the teacher data y _k for each code supplied from the G code calculating unit 113 _, and the right side of Expression (19) Multiplication (x _{n, k} y _k ) of student data x _{n, k} and teacher data y _{k in} the vector and calculation corresponding to summation (Σ) are performed.

That is, the G normal equation adding unit 114 performs the left-side matrix component (Σx _{n, k} x _{n ′, k} ) and the right-side vector component in the equation (19) obtained for the teacher data that is the previous pixel of interest. (Σx _{n, k} y _k ) is stored in a storage unit (not shown). Then, the G normal equation adding unit 114 applies the teacher data to the matrix data (Σx _{n, k} x _{n ′, k} ) or the vector component (Σx _{n, k} y _k ) for the teacher data newly set as the target pixel. The corresponding components x _{n, k + 1} x _{n ′, k + 1} or x _{n, k + 1} y _{k + 1} calculated by using the data y _{k + 1} and the student data x _{n, k + 1} are respectively added (represented by the summation of Expression (19)). Addition).

  Then, the G normal equation adding unit 114 performs the above-described addition using all the input high-resolution image teacher data as the pixel of interest, thereby forming the normal equation shown in Expression (19) for each code. . The G normal equation adding unit 114 supplies the normal equation of each code to the G coefficient data calculating unit 115.

G coefficient data calculation unit 115 solves the normal equations for each code supplied from the G normal equation adding section 114, for each code, calculates the optimal tap coefficient w _n. Then, G coefficient data calculation unit 115, depending on the type of the teacher image the pixel of interest is set (G component of the image), the tap coefficient w _n obtained, G coefficient necessary for performing the filter operation Output as. In this way, the obtained G coefficient for each code is stored in the G coefficient storage unit 64 shown in FIG.

<Configuration Example of G Coefficient Data Generation Unit 107>
FIG. 26 is a diagram illustrating a configuration example of the (R−G) coefficient data generation unit 108.
The (R−G) coefficient data generation unit 108 includes a (R−G) code tap selection unit 121, a (R−G) filter tap selection unit 122, a (R−G) generation unit 123, and a (R−G) code calculation. Unit 124, (R−G) normal equation addition unit 125, and (R−G) coefficient data calculation unit 126.

  The (R−G) code tap selection unit 121, the (R−G) filter tap selection unit 122, and the (R−G) code calculation unit 124 are the (R−G) code tap selection unit 71, ( (RG) It is comprised similarly to the filter tap selection part 72 and the (RG) code calculating part 33, and performs the same process.

  The (R−G) generation unit 123 calculates the difference between the R component and G component teacher images RG generated by the teacher image generation unit 102 at the R pixel position and the G pixel position of the RGB color filter array in the Bayer array. Thus, an (RG) component that is not a virtual color difference but an actual color difference is generated. The (R−G) generation unit 123 supplies the generated (R−G) component to the (R−G) normal equation addition unit 125 as teacher data.

The (R−G) normal equation adding unit 125 performs, for each code supplied from the (R−G) code calculation unit 114, the color difference component (R−G) of the target pixel generated by the (R−G) generation unit 123. ) (Teacher data) y _k and (RG) filter tap (student data (pixel)) x _{n, k} for the pixel of interest selected by the (RG) filter tap selection unit 122 Add.

  That is, the (RG) normal equation adding unit 125 performs the same process as the G normal equation adding unit 114 except that the (RG) component is used instead of the G component of the teacher image as the teacher data, For each (R−G) code, the normal equation shown in Equation (19) is established. Then, the (R−G) normal equation addition unit 125 supplies the normal equation of each (R−G) code to the (R−G) coefficient data calculation unit 126.

  The (RG) coefficient data calculation unit 126 performs the same processing as that of the G coefficient data calculation unit 115 to generate (RG) coefficient data.

<Learning processing routine by learning device>
FIG. 27 is a flowchart for explaining a learning processing routine by the learning apparatus 100 of FIG. When the high-resolution image RGB of each RGB component generated from each of the three image sensors is input, the learning apparatus 100 executes the processes after step S31.

  In step S31, the student image generation unit 102 shown in FIG. 23 downconverts the high-resolution image from the input high-resolution image RGB using, for example, an LPF simulation model, and RGB is converted according to the Bayer array. The arranged student image (student data) is generated.

  In step S32, the teacher image generation unit 103 down-converts the input high-resolution image RGB in the same manner as the student image, and generates a teacher image (teacher data) in which the half-pixel phase is shifted with respect to the student image. .

  In step S33, the virtual color difference (R−G) generation unit 105 illustrated in FIG. 23 generates a provisional G interpolated at the R position using the G component of the student image generated by the student image generation unit 101. The virtual color difference (R−G) generation unit 105 generates a provisional R interpolated at the G position using the R component of the student image. The virtual color difference (R−G) generation unit 105 uses the R component and the G component of the student image, and the provisional G and the provisional R, to calculate the color difference (RG) at the R pixel position and the G pixel position. A virtual color difference (RG) is generated.

  Further, the virtual color difference (B−G) generation unit 106 generates a temporary G interpolated at the B position using the G component of the student image generated by the student image generation unit 101. The virtual color difference (B−G) generation unit 106 generates a provisional B interpolated at the G position using the B component of the student image. Then, the virtual color difference (B−G) generation unit 106 uses the B component and the G component G of the student image and the provisional G and the provisional B to calculate the color difference (B−G) at the B pixel position and the G pixel position. A certain virtual color difference (B−G) is generated.

  In step S <b> 34, the target pixel selection unit 103 selects an unselected arbitrary pixel among the pixels constituting the teacher image generated by the teacher image generation unit 102 as the target pixel. That is, the pixel-of-interest selection unit 101 selects, as a pixel of interest, a pixel that has not yet been selected as a pixel of interest in the raster scan order among the pixels constituting the teacher image.

  In step S35, the G code tap selection unit 111 and the (R−G) code tap selection unit 116 illustrated in FIG. 25 perform the G code tap and the (R−G) code tap for the pixel of interest from the supplied student image. Select each one. Further, the (R−G) data generation unit 109 illustrated in FIG. 23 selects the (B−G) code tap for the target pixel from the student image.

  In step S36, the G code calculation unit 113 encodes the supplied G code tap to generate an encoded code. The (R−G) code calculation unit 124 encodes the supplied (R−G) code tap to generate an encoded code. The (BG) coefficient data generation unit 109 illustrated in FIG. 23 encodes the (BG) code tap to generate an encoded code.

  In step S37, the G filter tap selection unit 112 and the (RG) filter tap selection unit 122 select the G filter tap and the (RG) filter tap for the target pixel from each student image. Further, the (B−G) coefficient data generation unit 109 selects the (B−G) filter tap for the target pixel from the student image.

In step S38, the G normal equation adding unit 114 generates a linear linear expression represented by the above-described equation (12), and the G code tap selected by the G code tap selecting unit 111 is used as the pixel X _{1 of the} equation (12). , _2,..., used as _{X N.} The G normal equation adding unit 114 adds the linear linear expression generated in this way for each encoded code generated in step S36, and generates a normal equation of Expression (19). The G normalization equation addition unit 114 supplies the generated normalization equation to the G coefficient data calculation unit 115. The (R−G) normal equation adding unit 125 and the (B−G) coefficient data generating unit 109 similarly generate a normalized equation.

  In step S39, the target pixel selection unit 103 determines whether there is a next target pixel. If the determination is affirmative, that is, if there is a next pixel of interest, the process returns to step S34, and the processes after step S34 are repeatedly executed. If a negative determination is made, the process proceeds to step S40.

In step S40, the G coefficient data calculation unit 115 generates G coefficient data. Specifically, G coefficient data calculation unit 115, to the normal equation of Equation (19), calculates the tap coefficient _{w n} by using, for example, sweeping-out method (Gauss-Jordan elimination method).

G coefficient data calculation unit 115, depending on the type of the teacher image the pixel of interest is set (G component of the image), the tap coefficient w _n obtained, necessary to perform the prediction calculation of the G output image Output as G coefficient.
Similarly, the (R−G) coefficient data calculation unit 126 calculates (R−G) coefficient data, and the (B−G) coefficient data generation unit 109 calculates (B−G) coefficient data.

  In this way, the G coefficient, (RG) coefficient, and (BG) coefficient for each obtained encoded code are stored in the high resolution G generator 35 and the high resolution (RG) generator. 36 and the high-resolution (BG) generation unit 37 respectively.

  The present invention is not limited to the above-described embodiment, and can be applied to a design modified within the scope of the matters described in the claims.

  For example, the demosaic unit 16 illustrated in FIG. 1 or 2 can be realized by a semiconductor integrated circuit, a computer in which a program for executing each step described above is installed, or the like. The program may be installed in a computer from a recording medium such as an optical disk or a hard disk, or may be transmitted from the outside to the computer via a wired or wireless communication unit.

DESCRIPTION OF SYMBOLS 10 imaging device, 16 demosaic part, 33 virtual color difference (RG) production | generation part, 34 virtual color difference (BG) production | generation part, 35 high resolution G production | generation part, 36 high resolution (RG) production | generation part, 37 high-resolution (BG) generation unit, 38 adder, 39 adder, 41 provisional G generation unit, 42 provisional R generation unit, 43 virtual color difference (RG) calculation unit, 61 G code tap selection unit, 62 G filter tap selection unit, 63 G code calculation unit, 64 G coefficient storage unit, 65 G product-sum calculation unit, 71 (RG) code tap selection unit, 72 (RG) filter tap selection unit, 73 ( RG) code operation unit, 74 (RG) coefficient storage unit, 75 (RG) product-sum operation unit, 100 learning device

Claims (9)

Attention for selecting one pixel in the first image as a target pixel for a first image output from an imaging device in which pixels corresponding to each of a plurality of color components are arranged in a predetermined arrangement A pixel selector;
A plurality of pixels of the first color component are arranged in a first arrangement with respect to a pixel of first color component which is a main component of the first image and selected by the pixel of interest selection unit. A first code tap selection unit for selecting code taps arranged in a pattern;
A first code calculation unit that calculates a code indicating a feature amount of the code tap selected by the first code tap selection unit;
For each code, a plurality of first tap coefficients obtained in advance by the first learning process of the code classification type adaptive filter are stored, and a plurality of first tap coefficients are calculated based on the code calculated by the first code calculation unit. A first coefficient storage unit for outputting a tap coefficient of
A plurality of pixels of the first color component are arranged in a second array with respect to a pixel of interest of the first color component that is a main component of the first image and selected by the pixel of interest selection unit. A first filter tap selection unit for selecting filter taps arranged in a pattern;
Corresponding to each pixel of interest based on each pixel value of the filter tap selected by the first filter tap selection unit and a plurality of first tap coefficients output from the first coefficient storage unit A first image generation unit that generates an image of the first color component by calculating a pixel value of an output pixel to be output;
The first tap coefficient is
A first student image composed of only the first color component from an image in which the color component, the arrangement of the color components, and the size of the predetermined subject are the same as the first image;
For the predetermined subject, a first teacher image composed of only the first color component having the same size as the first image and shifted by half a pixel in the horizontal and vertical directions with respect to the first student image When,
Calculated by the first learning process for solving the mapped normal equation of each pixel of the first student image and each pixel of the first teacher image. An image processing device that is a tap coefficient. At the pixel position of each pixel of the first color component and at the pixel position of each pixel of the second color component which is one color component different from the first color component, the pixel of the first color component A color difference component generation unit that generates a color difference component that is a difference between a pixel value and a pixel value of a pixel of the second color component;
A plurality of pixels of the color difference component are arranged in a third arrangement pattern with respect to the pixel of interest that is the pixel of the first color component or the second color component and is selected by the pixel of interest selection unit. A second code tap selection unit for selecting a code tap;
A second code calculation unit that calculates a code indicating the feature quantity of the code tap selected by the second code tap selection unit;
For each code, a plurality of second tap coefficients obtained in advance by the second learning process of the code classification type adaptive filter are stored, and a plurality of second tap coefficients are stored based on the code calculated by the second code calculation unit. A second coefficient storage unit for outputting the tap coefficient of
A plurality of pixels of the color difference component are arranged in a fourth arrangement pattern with respect to the pixel of interest that is the pixel of the first color component or the second color component and is selected by the pixel of interest selection unit. A second filter tap selection unit for selecting a filter tap;
Corresponding to each pixel of interest based on each pixel value of the filter tap selected by the second filter tap selection unit and a plurality of second tap coefficients output from the second coefficient storage unit A second image generation unit that generates an image of the color difference component by calculating a pixel value of an output pixel to be output;
Based on the image of the first color component generated by the first image generation unit and the image of the color difference component generated by the second image generation unit, the second color component An arithmetic unit for calculating an image;
The image processing apparatus according to claim 1, further comprising: The second tap coefficient is
A color-difference image obtained as a difference between the first and second color components from an image in which the color component, the arrangement of the color components, and the size of the predetermined subject are the same as those of the first image. A second student image that is
A color difference image obtained as a difference between the image of the first color component having the same size as the first image and the image of the second color component having the same size for the predetermined subject, A second teacher image which is the color difference image shifted by half a pixel in the horizontal direction and the vertical direction with respect to the first student image;
Calculated by the second learning process that solves the mapped normal equation of each pixel of the second student image and each pixel of the second teacher image. The image processing apparatus according to claim 2, wherein the image processing apparatus is a tap coefficient. The color difference component generation unit, the second code tap selection unit, the second code calculation unit, the second coefficient storage unit, the second filter tap selection unit, the second image generation unit, and The calculation unit is provided for each second color component when there are a plurality of second color components different from the first color component in the first image. The image processing apparatus described. The color difference component generation unit
The interpolated pixel value of the first color component is calculated by interpolating from the pixel value of each pixel of the first color component adjacent to the pixel position at the pixel position of each pixel of the second color component. A first interpolation pixel value calculation unit;
The interpolation pixel value of the second color component is calculated by interpolating from the pixel value of each pixel of the second color component adjacent to the pixel position at the pixel position of each pixel of the first color component. A second interpolation pixel value calculation unit;
In the pixel position of each pixel of the second color component, the difference between the interpolated pixel value of the first color component at the pixel position and the pixel value of the second color component at the pixel position is calculated. In the pixel position of each pixel of the first color component, the pixel value of the first color component at the pixel position and the interpolated pixel value of the second color component at the pixel position are generated. A difference calculation unit that generates the color difference component by calculating a difference between;
The image processing apparatus according to claim 2, further comprising: When the first interpolation pixel value calculation unit includes a plurality of pixels of the first color component that are close to each other in the horizontal direction and the vertical direction of the pixel position at the pixel position of each pixel of the second color component. Calculating a horizontal interpolation value from the pixel value of each pixel of the first color component in the horizontal direction, calculating a vertical interpolation value from the pixel value of each pixel of the first color component in the vertical direction, and The image according to claim 5, wherein an interpolation value of the second color component is calculated by calculating a weighted average value of the horizontal interpolation value and the vertical interpolation value based on respective variances of the vertical interpolation value. Processing equipment. Attention for selecting one pixel in the first image as a target pixel for a first image output from an imaging device in which pixels corresponding to each of a plurality of color components are arranged in a predetermined array A pixel selection step;
A plurality of pixels of the first color component are in a first arrangement with respect to a pixel of the first color component that is a main component of the first image and selected in the pixel of interest selection step. A first code tap selection step of selecting code taps arranged in a pattern;
A first code calculation step of calculating a code indicating the feature quantity of the code tap selected in the first code tap selection step;
For each code, a plurality of first tap coefficients are stored on the basis of the code calculated in the first code calculation step from a coefficient storage unit that stores a plurality of first tap coefficients obtained in advance by the learning process of the code classification type adaptive filter. A first coefficient reading step for reading one tap coefficient;
A plurality of pixels of the first color component are arranged in a second arrangement with respect to the pixel of interest of the first color component that is the main component of the first image and selected in the pixel of interest selection step. A first filter tap selection step for selecting filter taps arranged in a pattern;
Output corresponding to each pixel of interest based on each pixel value of the filter tap selected in the first filter tap selection step and the plurality of first tap coefficients read out in the coefficient reading step An image generation step of generating an image of the first color component by calculating a pixel value of the pixel,
The first tap coefficient is
A first student image composed of only the first color component from an image in which the color component, the arrangement of the color components, and the size of the predetermined subject are the same as the first image;
For the predetermined subject, a first teacher image composed of only the first color component having the same size as the first image and shifted by half a pixel in the horizontal and vertical directions with respect to the first student image When,
A first learning processing using the first tap which is calculated by the first learning processing solving the mapped normal equation for each pixel of each pixel of the student image said first teacher image Image processing method that is a coefficient. Computer
Attention for selecting one pixel in the first image as a target pixel for a first image output from an imaging device in which pixels corresponding to each of a plurality of color components are arranged in a predetermined array A pixel selector;
A plurality of pixels of the first color component are arranged in a first arrangement with respect to a pixel of first color component which is a main component of the first image and selected by the pixel of interest selection unit. A first code tap selection unit for selecting code taps arranged in a pattern;
A first code calculation unit that calculates a code indicating a feature amount of the code tap selected by the first code tap selection unit;
For each code, a plurality of first tap coefficients obtained in advance by a learning process of a code classification type adaptive filter are stored, and a plurality of first tap coefficients are calculated based on the code calculated by the first code calculation unit. A first coefficient storage unit for outputting
A plurality of pixels of the first color component are arranged in a second array with respect to a pixel of interest of the first color component that is a main component of the first image and selected by the pixel of interest selection unit. A first filter tap selection unit for selecting filter taps arranged in a pattern;
Corresponding to each pixel of interest based on each pixel value of the filter tap selected by the first filter tap selection unit and a plurality of first tap coefficients output from the first coefficient storage unit Calculating a pixel value of an output pixel to function as a first image generation unit that generates an image of the first color component;
The first tap coefficient is
A first student image composed of only the first color component from an image in which the color component, the arrangement of the color components, and the size of the predetermined subject are the same as the first image;
For the predetermined subject, a first teacher image composed of only the first color component having the same size as the first image and shifted by half a pixel in the horizontal and vertical directions with respect to the first student image When,
A first learning processing using the first tap which is calculated by the first learning processing solving the mapped normal equation for each pixel of each pixel of the student image said first teacher image A program that is a coefficient. An image sensor in which pixels corresponding to each of a plurality of color components are configured in a predetermined arrangement;
The image processing apparatus according to any one of claims 1 to 6, wherein image processing is performed on the first image output from the imaging device.
An imaging apparatus comprising: