JP2013150123A - Image processor, control method, program, and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate an image improved in feeling of resolution, by using plural image signals obtained from an imaging device in which respective color components are arranged with a predetermined rule.SOLUTION: An image processor acquires plural image signals arranged in a Bayer array, generates one luminance signal and a color difference signal of an R component and a B component, and generates a development image. In generation of the luminance signal at this time, the image processor sets one image signal among the image signals as a reference image, and extracts a signal of a G component with respect to the reference image. With respect to each of missing pixels included in the signal of the G component, a pixel value calculated by adaptive interpolation from the peripheral pixels of the missing pixel and a pixel value of the nearest pixel which minimizes an image deviation amount from the missing pixel, the pixel value having been extracted from the image signals other than the image signal of the reference signal among the image signals undergo weighted addition to calculate the pixel value of the missing pixel.

Description

  The present invention relates to an image processing apparatus, a control method therefor, a program, and a storage medium, and more particularly to a technique for generating a developed image with a high resolution using a plurality of image signals output from an image sensor.

  In recent years, image pickup elements used in image pickup apparatuses such as digital cameras and digital video cameras have a tendency to increase the number of pixels. In order to display a still image or a moving image (hereinafter simply referred to as an image) picked up by an image pickup device having such an image pickup device having a large number of pixels, a display device such as a television receiver has a display area for displaying an image. The number of pixels is also increasing.

  However, when displaying an image with a small number of pixels in accordance with a display device having an increased number of pixels in the display area, the image is roughened by stretching the image according to the number of pixels in the display area. End up. For this reason, various image processing methods have been proposed in which an image with a small number of pixels is converted into an image with a large number of pixels without reducing the resolution.

  One of the proposed image processing methods is a method called “super-resolution processing”. This method can generate an image with improved resolution by using a plurality of highly related images such as continuous frames in a moving image or images continuously shot. Specifically, as shown in FIG. 16, in this method, a pixel (new pixel) indicated by a circle is replaced with another image ( Interpolation processing is performed with reference to the reference image. In general, when interpolating a new pixel, a linear interpolation method such as a bi-linear method using pixel values of a plurality of peripheral pixels existing around a position corresponding to the new pixel among the pixels of the reference image. A new pixel is generated.

  However, the method of generating new pixels by interpolation using a plurality of pixel values at the time of interpolation involves pixel value averaging processing. That is, since the image is smoothed by averaging the pixel values, the generated image may be an image with reduced edge strength.

  On the other hand, there is also a method of reducing the decrease in edge strength due to the averaging process by assigning the reference image pixel itself to the new pixel. According to Patent Document 1, the closest pixel is assigned to a new pixel according to the distance between the position corresponding to the new pixel in the reference image and the pixel closest to the position, or a new pixel from another pixel of the reference image A super-resolution process for switching whether to perform adaptive interpolation for generating the image is disclosed.

JP 2000-188680 A

  However, the super-resolution processing of Patent Document 1 is a method of generating one image with improved resolution from a plurality of already developed images, and does not improve the resolution in the development processing. There wasn't. That is, no technology has been disclosed so far that uses an image signal output from an imaging device by imaging a plurality of times and generates an image with improved resolution in development processing. Further, there has been no disclosure regarding a super-resolution technique in an image sensor in which each color component such as a Bayer array is arranged with a predetermined rule.

  The present invention has been made in view of the above-mentioned problems, and an image with improved resolution is obtained by using a plurality of image signals obtained from an image sensor in which each color component is arranged with a predetermined rule. The purpose is to generate.

In order to achieve the above object, an image processing apparatus of the present invention comprises the following arrangement.
An acquisition unit that acquires a plurality of image signals having a Bayer array, and a luminance signal generation unit that generates one luminance signal from the plurality of image signals acquired by the acquisition unit. Setting means for setting one image signal as a reference image, an extracting means for extracting a G component signal for the reference image set by the setting means, and no signal for the G component signal extracted by the extracting means Interpolating means for interpolating missing pixels, and luminance signal generating means, and color difference signal generating means for generating R component and B component color difference signals from a plurality of image signals acquired by the acquiring means, The interpolation means excludes the pixel value calculated by adaptive interpolation from the peripheral pixels of the missing pixel and the reference image among the plurality of image signals for each of the missing pixels in the G component signal. The pixel value of the missing pixel is calculated by weighted addition with the pixel value of a neighboring pixel that is the pixel included in the image signal and has the smallest image shift amount from the missing pixel. A luminance signal is generated from the G component signal in which the missing pixel is interpolated and the R component and B component signals of the reference image in which the missing pixel is interpolated by linear interpolation.

  With such a configuration, according to the present invention, it is possible to generate an image with improved resolution using a plurality of image signals obtained from an image sensor in which each color component is arranged with a predetermined rule. Become.

1 is a block diagram showing a functional configuration of a digital camera 100 according to an embodiment of the present invention. The block diagram which showed the function structure of the image process part 107 which concerns on embodiment of this invention. The figure for demonstrating the adaptive interpolation which concerns on embodiment of this invention. The figure for demonstrating the neighboring pixel which concerns on embodiment of this invention. The figure for demonstrating the weighted addition coefficient which concerns on Embodiment 1 of this invention. The figure for demonstrating the linear interpolation which concerns on embodiment of this invention. The figure for demonstrating the exposure time of the several image signal which concerns on embodiment of this invention. The figure which showed the Bayer arrangement | sequence which concerns on embodiment of this invention. The flowchart which illustrated the luminance signal generation processing concerning the embodiment of the present invention. 5 is a flowchart illustrating color difference signal generation processing according to the embodiment of the invention. The flowchart which illustrated the production | generation method determination process which concerns on Embodiment 2 of this invention. The flowchart which illustrated the luminance signal generation process for noise reduction which concerns on Embodiment 2 of this invention. The figure for demonstrating the weighted addition coefficient which concerns on Embodiment 2 of this invention. The figure for demonstrating the tilt phenomenon which concerns on the modification of this invention. The flowchart which illustrated the production | generation method determination process which concerns on the modification of this invention. The figure for demonstrating the conventional super-resolution process.

[Embodiment 1]
Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. In addition, one embodiment described below includes an image pickup device having a Bayer array as an example of an image processing apparatus, and can generate a single developed image using a plurality of image signals at the time of development processing. An example in which the present invention is applied to a camera will be described. However, the present invention is applicable to any apparatus that can generate a single developed image using a plurality of image signals having a Bayer array. Further, in this specification, the “image signal” is a signal group in which each pixel has a pixel value of any of R, G1, G2, and B in the Bayer array before the development processing is applied. And

<Configuration of Digital Camera 100>
FIG. 1 is a block diagram showing a functional configuration of a digital camera 100 according to an embodiment of the present invention.

  The control unit 101 is a CPU, for example, and controls the operation of each block included in the digital camera 100. Specifically, the control unit 101 controls the operation of each block by reading out an operation program for luminance signal generation processing or color difference signal generation processing, which will be described later, from the ROM 102, for example, and expanding and executing it in the RAM 103.

  The ROM 102 is, for example, a rewritable nonvolatile memory, and stores operation programs for other processes performed by the digital camera 100 in addition to operation programs for luminance signal generation processing and color difference signal generation processing. The ROM 102 stores information such as parameters necessary for the operation of each block of the digital camera 100 in addition to the operation program executed by the digital camera 100.

  The RAM 103 is a volatile memory and is used not only as a development area for the operation program of each block of the digital camera 100 but also as a storage area for storing intermediate data output in the operation of each block.

  The imaging unit 105 is an imaging element having a Bayer array, such as a CCD or a CMOS sensor. In this embodiment, an example of an image sensor using a Bayer array is shown. However, the present invention is not limited to this, and the present invention can be applied to an image sensor in which pixels of each color component are arranged with a predetermined rule. The imaging unit 105 photoelectrically converts an optical image formed on the light receiving unit via the imaging optical system 104 and outputs an analog image signal to the A / D conversion unit 106.

  The A / D conversion unit 106 applies A / D conversion processing to the input analog image signal, and outputs a digital image signal (hereinafter simply referred to as an image signal) to the image processing unit 107. The image signal is information indicating pixel values for the same number of pixels as the image sensor, and each pixel has any one of R, G1, G2, and B as described above.

  The display unit 108 is a display device included in the digital camera 100 such as a small LCD. The display unit 108 is used to display an image captured by the imaging unit 105 and an image recorded on the recording medium 109.

  The recording medium 109 is, for example, a recording device such as a built-in memory of the digital camera 100 or a memory card or HDD that is detachably connected to the digital camera 100. The recording medium 109 is imaged by the imaging unit 105 and records an image for recording output by a development process by an image processing unit 107 described later.

  The operation input unit 110 is a user input interface of the digital camera 100 such as a power button or a release button. When the operation input unit 110 detects that the user has operated the interface, the operation input unit 110 transmits a control signal corresponding to the operation to the control unit 101.

  The image processing unit 107 generates a final developed image luminance signal and color difference signal with improved resolution from a plurality of input image signals having the Bayer array. Here, the configuration of the image processing unit 107 of the present embodiment will be described in detail with reference to FIG.

(Configuration of the image processing unit 107)
As illustrated, the image processing unit 107 includes an image memory 120, a luminance signal generation unit 130, and a color difference signal generation unit 140. That is, the image processing unit 107 of the present embodiment separately generates and outputs a luminance signal and a color difference signal of one developed image from a plurality of input image signals.

  The image memory 120 is a volatile memory for image processing that the image processing unit 107 has. A plurality of image signals input to the image processing unit 107 are stored in the image memory 120 and then read out for processing executed in each block of the image processing unit 107 described later. Note that the information stored in the image memory 120 is not limited to the input image signal, and an image signal or the like to which processing is applied in each block of the image processing unit 107 may be stored.

◇ Luminance signal generator 130 ◇
The luminance signal generation unit 130 includes an adaptive interpolation unit 131, a neighboring pixel extraction unit 132, a weighted addition coefficient determination unit 133, a weighted addition unit 134, and a luminance signal conversion unit 135.

The adaptive interpolation unit 131 extracts a pixel value corresponding to a missing pixel from an image signal obtained by extracting only the G component (G1 and G2) of the image signal selected as the reference image from among the plurality of input image signals. G _adapt (interpolation signal) is generated by adaptive interpolation (color component signal generation).

Note that the adaptive interpolation performed by the adaptive interpolation unit 131 is performed by determining the edge direction. For example, as shown in FIG. 3, when a missing pixel and a G component pixel (peripheral pixel) adjacent to the missing pixel are arranged, the edge direction of the missing pixel is a gradient in the horizontal direction and the vertical direction of the adjacent pixel ( ΔH and ΔV). Specifically, the gradient in each direction is ΔH = | G10−G12 |
ΔV = | G01−G21 |
The edge direction is determined based on the gradient difference, and the pixel value of the missing pixel is G _adapt as follows.

i) If ΔH−ΔV> threshold, it is determined that the edge direction is the horizontal direction,
G _adapt = (G01 + G21) / 2
ii) If ΔV−ΔH> threshold, determine that the edge direction is the vertical direction;
G _adapt = (G10 + G12) / 2
iii) In other cases, determine that it is not an edge,
G _adapt = (G01 + G10 + G12 + G21) / 4
As described above, the adaptive interpolation unit 131 generates pixel values of missing pixels. Note that the adaptive interpolation method is not limited to this, and any method may be used as long as the pixel value generation method varies depending on the edge direction.

  The neighboring pixel extraction unit 132 extracts, for each pixel that is a missing pixel of the G component of the reference image, that is, each adaptive interpolation pixel generated by adaptive interpolation in the adaptive interpolation unit 131, an image other than the reference image located in the vicinity thereof. G component pixels are extracted. In the present embodiment, the G component pixel (nearest neighbor pixel) having the smallest relative positional deviation amount is extracted from the G component pixels included in the image signal (reference image) other than the standard image. However, the present invention is not limited to this, and it is also possible to select a plurality of images having a small positional deviation amount and to extract G component pixels in the vicinity of each adaptive interpolation pixel.

  Specifically, the neighboring pixel extraction unit 132 moves the G component of one reference image in a direction that eliminates the image shift from the G component of the reference image, and aligns the image signals of the G component of the reference image and the reference image. To do. That is, the neighboring pixel extraction unit 132 grasps an image shift amount between signals based on, for example, a pattern matching process or an acceleration change amount at the time of image signal shooting, and aligns two image signals according to the image shift amount. Do. At this time, it is difficult to perform accurate alignment in units of subpixels by any image shift amount detection method and alignment method. That is, it is conceivable that a deviation in sub-pixel units also occurs in the reference image after alignment and each reference image other than the reference image. In this embodiment using this, the neighboring pixel extraction unit 132 specifies the pixel closest to the position of each adaptive interpolation pixel of the G component of the reference image in the reference image as shown in FIG. 4 after alignment. The relative positional deviation amount d between the pixel and the adaptive interpolation pixel is acquired in units of subpixels. The neighboring pixel extraction unit 132 performs such processing on all the reference images, and finally corresponds to the information indicating the nearest pixel having the smallest relative displacement amount with respect to the adaptive interpolation pixel and the pixel. A distance d from the adaptive interpolation pixel is acquired.

For each adaptive interpolation pixel of the G component of the reference image, the weighted addition coefficient determination unit 133 weights the pixel value G _adapt of the adaptive interpolation pixel and the pixel value G _{ref of the} nearest pixel extracted by the neighboring pixel extraction unit 132. A weighted addition coefficient r (ratio) for weighted addition by the adding unit 134 is calculated. In this embodiment, the weighted addition coefficient r is defined as a value at which the behavior changes by providing two types of threshold values (th1 and th2) as shown in FIG. As shown, the weighted addition coefficient r (<0) is
i) 0 ≦ d ≦ th1
r = 1
ii) th1 <d <th2)
r = 1- (d-th1) / (th2-th1)
iii) th2 ≦ d
r = 0
The weighted addition unit 134 uses the calculated weighted addition coefficient r to calculate the pixel value G _mix after the weighted addition,
G _mix = G _adapt × (1-r) + G _ref × r
Calculate as This G _mix becomes a signal of a new G color component and is used for generating a luminance signal.

  Note that the method of calculating the weighted addition coefficient r is not limited to this, and any coefficient that changes in accordance with the relative positional deviation amount d between the two pixels to be weighted and added may be used. Specifically, the weighted addition coefficient determination unit 133 sets the pixel value of the adaptive interpolation pixel to the neighboring pixel when the positional deviation amount d is equal to or less than a threshold value that can be regarded as the corresponding pixel of the missing pixel. What is necessary is just to determine the weighted addition coefficient to be replaced with the pixel value.

  The luminance signal conversion unit 135 has the same number of pixels as the reference image from the G component image signal of the reference image after the weighted addition processing in the weighted addition unit 134 and the R component and B component image signals of the reference image. A luminance signal is generated.

Specifically, the luminance signal conversion unit 135 interpolates information on missing pixels of each image signal by performing linear interpolation on each of the R component and B component image signals of the reference image. Specifically, in the linear interpolation for the R component signal, R _linear (01), (10), (11), (12) that are missing pixels for the four R component pixels as shown in FIG. ) And (21) are generated as follows.

1. First, the pixel value 0 is assigned to the missing pixel. 2. Apply a filter represented by (1 2 1) / 2 in the vertical direction to generate pixel values of R _linear (10) and (12). Apply a filter represented by (1 2 1) / 2 in the horizontal direction

Then, the luminance signal conversion unit 135 generates and outputs a luminance signal Y by adding the pixel values of the G component, the R component, and the B component according to the following formula for each pixel of the reference image.
Y = (3R + 6G + B) / 10 (Formula 1)

  Note that, in the above formula 1, all the image signals input to the image processing unit 107 are formulas related to generation of a luminance signal in the case where the image is captured with appropriate exposure. However, when the appropriate exposure time determined by the control unit 101 is divided so that, for example, one imaging time as shown in FIG. The above equation 1 may be multiplied by n.

  In addition, the luminance signal generation unit 130 of the present embodiment performs neighboring pixel extraction processing and adaptive interpolation processing on the G component image signal of the reference image to interpolate missing pixels, and thereby perform R component and B For the component image signal, linear interpolation is performed to perform missing pixel interpolation. This is due to the following reason.

1. As for the R component and B component signals of the color distribution reference image of the Bayer array , as in the case of the G component signal, it is theoretically possible to execute the neighboring pixel extraction processing and adaptive interpolation processing for all missing pixels. Is possible. However, in the Bayer array, as described above with reference to FIG. 8, half of the 2 × 2 pixels are occupied by the G component and 1/4 by the R component or B component. That is, when the number of missing pixels in the G component image signal of the reference image is used as a reference, the number of missing pixels in the R component and B component image signals is tripled. In other words, the processing related to the extraction of neighboring pixels has a processing load three times that, which leads to an increase in the cost of the image processing unit 107 and is not realistic. In addition, adaptive interpolation processing can avoid edge strength reduction by determining the edge direction of surrounding pixels and dividing the processing, but if only one pixel can be referred to in 2 × 2 pixels, The judgment may be inaccurate, and there is a possibility that suitable interpolation cannot be performed.

2. In general, it is known that the resolution of an image is improved when the luminance signal reproduction rate is high. That is, in the generation of a luminance signal, a luminance signal having a higher reproducibility using a plurality of image signals as described in this specification than when a luminance signal is generated by interpolation processing from one image signal in a Bayer array. In the case of generating the image, the resolution of the image can be improved. As for the luminance signal, the contribution ratio of the G component is the highest as shown in the above-described equation 1, and the contribution ratio of the R component and the B component is less than half of that. In other words, with respect to the interpolation of missing pixels of the G component, which has a high contribution rate to the luminance component, the reproducibility is improved by executing the neighboring pixel extraction and adaptive interpolation processing, so that the luminance signal finally obtained is also reproducible. Can be high.

  For the reason described above, in the present embodiment, as a more preferable form, for the G component image signal, the neighboring pixel extraction processing or adaptive interpolation processing is executed to perform the missing pixel interpolation, and the R component and B component images are processed. For the signal, linear interpolation is performed to interpolate missing pixels. However, the present invention is not limited to this, and for the R component and B component as well as the G component, the neighboring pixel extraction processing, adaptive interpolation processing, and weighted addition processing in this embodiment are executed to perform missing pixel interpolation. The luminance signal may be generated from these signals.

◇ Color difference signal generator 140 ◇
On the other hand, the color difference signal generation unit 140 includes a linear interpolation unit 141, a color difference neighboring pixel extraction unit 142, a color difference weighted addition coefficient determination unit 143, a color difference weighted addition unit 144, a false color determination unit 145, and a color difference signal conversion unit 146. Consists of.

  The linear interpolation unit 141 performs linear interpolation processing on each of the R component, G1 component, G2 component, and B component signals of the reference image, and outputs a signal of each component obtained by interpolating the defective pixel (synchronization processing). ). In the generation of the color difference signal, unlike the generation of the luminance signal, G1 and G2 are separated and interpolation processing is performed separately. This is to suppress the generation of false colors due to the Bayer arrangement of the image sensor.

  The color difference neighboring pixel extraction unit 142, the color difference weighted addition coefficient determination unit 143, and the color difference weighted addition coefficient unit 144, for each of the signals of the G1 component and G2 component of the reference image subjected to linear interpolation in the linear interpolation unit 141, Since the same processing as the neighboring pixel extraction unit 132, the weighted addition coefficient determination unit 133, and the weighted addition unit 134 is performed, description of each block is omitted. Needless to say, the processing performed by the color difference neighboring pixel extraction unit 142, the color difference weighted addition coefficient determination unit 143, and the color difference weighted addition unit 144 is performed not on the adaptive interpolation pixel but on the linear interpolation pixel. .

  When generating the color difference signal, the false color determination unit 145 determines a pixel value calculation method for suppressing generation of a false color in each pixel of the color difference signal using a conventional method. A false color generation suppression method is a method of calculating a pixel value of a color difference signal by determining an edge direction from a correlation degree between G components in a horizontal direction and a vertical direction as disclosed in, for example, Japanese Patent Application Laid-Open No. 2002-300590. It may be. In addition, a false color generation suppression method is a method of calculating by performing weighted addition using each of G1 and G2 and a weighting coefficient corresponding to the difference between R and B pixel values as disclosed in JP-A-08-023541 It may be.

  The color difference signal conversion unit 146 generates two types of color difference signals (RG and BG) for the R component and the B component from the four types of image signals generated by the linear interpolation unit 141. At this time, the color difference signal conversion unit 146 calculates the pixel value of each pixel of the color difference signal with a calculation formula that suppresses the generation of the false color determined by the false color determination unit 145.

  In the present embodiment, the image processing unit 107 is described as having the above-described configuration, but it is easy to include blocks that perform other image processing such as development processing and display processing. Will be understood.

  In the present embodiment, description will be made assuming that processing is realized in each block included in the digital camera 100 as hardware. However, the present invention is not limited to this, and processing of each block is the same processing as that of each block. It may be realized by a program that performs.

<Luminance signal generation processing>
Specific processing of the luminance signal generation processing of the digital camera 100 of the present embodiment having such a configuration will be described with reference to the flowchart of FIG. The processing corresponding to the flowchart can be realized by the control unit 101 reading out a corresponding processing program stored in, for example, the ROM 102, developing it in the RAM 103, and executing it. In this luminance signal generation process, for example, the imaging unit 105 performs imaging a plurality of times in response to a shooting instruction from the user, and n image signals having the obtained Bayer array are output from the A / D conversion unit 106. It will be described as starting at the time.

  In step S <b> 901, the control unit 101 inputs n image signals having a Bayer array output from the A / D conversion unit 106 to the image processing unit 107 and stores them in the image memory 120.

  In step S <b> 902, the image processing unit 107 sets one image signal of n image signals as a reference image under the control of the control unit 101. Then, the image processing unit 107 extracts a G component signal of the image signal set as the reference image and inputs the signal to the adaptive interpolation unit 131.

  In step S <b> 903, under the control of the control unit 101, the image processing unit 107 causes the adaptive interpolation unit 131 to perform adaptive interpolation processing for missing pixels on the G component signal of the reference image, thereby generating adaptive interpolation pixels. The adaptive interpolation unit 131 outputs the G component signal after the adaptive interpolation processing to the neighboring pixel extraction unit 132.

  In step S904, under the control of the control unit 101, the image processing unit 107 extracts the nearest pixel of each adaptive interpolation pixel from the reference image after the adaptive interpolation process, and compares the pixel with the corresponding adaptive interpolation pixel. The neighboring pixel extraction unit 132 is made to acquire the positional deviation amount (distance) d.

  Specifically, the neighboring pixel extraction unit 132 selects n−1 image signals (reference images) other than the standard image stored in the image memory 120 one by one, and performs a predetermined alignment process. Then, the neighboring pixel extraction unit 132 specifies, for all the adaptive interpolation pixels of the base image, pixels that are present at the position with the smallest positional deviation amount among the G component pixels of the selected reference image. The neighboring pixel extraction unit 132 calculates the positional deviation amount of the pixel specified for each adaptive interpolation pixel and exists at the position with the smallest positional deviation amount, and stores it in the image memory 120 in association with the information indicating the reference image. To do. The neighborhood pixel extraction unit 132 repeats this for all reference images, and extracts the nearest neighborhood pixel having the minimum positional deviation amount for each adaptive interpolation pixel.

  As described above, by calculating the positional deviation amount of the pixel existing in the position where the positional deviation amount is the smallest with respect to each adaptive interpolation pixel for all the reference images, the nearest pixel can be extracted for each adaptive interpolation pixel. . However, for example, when the positional deviation amount d is equal to or less than a threshold value that can be regarded as a corresponding pixel of the adaptive interpolation pixel, the positional deviation amount d is calculated in the reference image after that for the adaptive interpolation pixel. It may not be performed. Alternatively, a plurality of corresponding pixels with a small amount of positional deviation may be extracted and added.

  In step S <b> 905, the image processing unit 107 transmits, to the weighted addition coefficient determination unit 133, information on the positional deviation amount d of the nearest neighboring pixel for each adaptive interpolation pixel of the G component of the reference image under the control of the control unit 101. Thus, the weighted addition coefficient r is determined.

  In step S906, the image processing unit 107 controls each adaptive interpolation pixel and the nearest neighboring pixel of the adaptive interpolation pixel according to the weighted addition coefficient determined by the weighted addition coefficient determination unit 133 under the control of the control unit 101. The weighted addition unit 134 performs weighted addition. The weighted addition unit 134 outputs a G component signal of the reference image for which interpolation has been completed.

  In step S907, under the control of the control unit 101, the image processing unit 107 transmits the G component signal of the reference image for which interpolation has been completed and the R component and B component signals of the reference image to the luminance signal conversion unit 135, A luminance signal is generated. Specifically, the luminance signal converter 135 first generates missing pixels by linear interpolation for each of the R component and B component signals of the input reference image. Then, using the G component signal of the reference image that has been interpolated and the R and B component signals after the linear interpolation processing, the pixel value of each pixel of the luminance signal is calculated according to Equation 1, and a luminance image is generated. And output.

  By doing in this way, the luminance signal generation part 130 of this embodiment can generate | occur | produce a luminance signal with high subject reproducibility.

<Color difference signal generation processing>
Next, specific processing of the color difference signal generation processing of the digital camera 100 of the present embodiment will be described using the flowchart of FIG. The processing corresponding to the flowchart can be realized by the control unit 101 reading out a corresponding processing program stored in, for example, the ROM 102, developing it in the RAM 103, and executing it. In this color difference signal generation process, for example, the imaging unit 105 performs imaging a plurality of times in response to a shooting instruction from the user, and n image signals having the obtained Bayer array are output from the A / D conversion unit 106. It will be described as starting at the time.

  In step S <b> 1001, the control unit 101 inputs n image signals having a Bayer array output from the A / D conversion unit 106 to the image processing unit 107 and stores them in the image memory 120. Note that the processing in this step is common to the processing in S901 of the luminance signal generation processing described above, and may be omitted.

  In step S <b> 1002, the image processing unit 107 sets one image signal of n image signals as a reference image under the control of the control unit 101. Then, the image processing unit 107 extracts the G1 component signal and the G2 component signal of the image signal set as the reference image (extraction for color difference) and inputs them to the linear interpolation unit 141. Note that the reference image set in this step is the same as the reference image set in the luminance signal generation process.

  In step S <b> 1003, under the control of the control unit 101, the image processing unit 107 causes the linear interpolation unit 141 to perform linear interpolation processing of missing pixels on the G1 component and G2 component signals of the reference image, thereby generating linear interpolation pixels. The linear interpolation unit 141 outputs the G1 component and G2 component signals after the linear interpolation processing to the color difference neighboring pixel extraction unit 142.

  In step S1004, under the control of the control unit 101, the image processing unit 107 extracts the nearest pixel of each linear interpolation pixel and corresponds to the pixel with respect to the G1 component and G2 component signals after the linear interpolation processing. The positional deviation amount (distance) d from the linear interpolation pixel is acquired by the color difference neighboring pixel extraction unit 142.

  Specifically, the color-difference neighboring pixel extraction unit 142 selects n−1 image signals (reference images) other than the standard image stored in the image memory 120 one by one, and performs a predetermined alignment process. Then, the color difference neighboring pixel extraction unit 142 is present at the position where the amount of positional deviation is the smallest among the corresponding component pixels of the selected reference image with respect to all the linear interpolation pixels of the G1 component and G2 component signals. Each pixel is specified. The color difference neighboring pixel extraction unit 142 calculates the positional deviation amount of the pixel specified for each linear interpolation pixel and exists at the position with the smallest positional deviation amount, and associates it with the information indicating the reference image, and the image memory 120. To remember. The color difference neighboring pixel extraction unit 142 repeats this for all reference images, and extracts the nearest neighbor pixel having the smallest amount of positional deviation for each linear interpolation pixel.

  As described above, by calculating the positional deviation amount of the pixel existing at the position with the smallest positional deviation amount with respect to each linear interpolation pixel for all the reference images, the nearest pixel can be extracted for each linear interpolation pixel. . However, for example, when the positional deviation amount d is equal to or less than a threshold value that can be regarded as a corresponding pixel of the linear interpolation pixel, the positional deviation amount d is calculated in the subsequent reference image for the linear interpolation pixel. It may not be performed. Alternatively, a plurality of corresponding pixels with a small amount of positional deviation may be extracted and added.

  In step S <b> 1005, under the control of the control unit 101, the image processing unit 107 uses the color difference weighted addition coefficient for information on the positional deviation amount d of the nearest neighboring pixel for each of the linear interpolation pixels of the G1 component and G2 component of the reference image. It transmits to the determination part 143, and makes each weighted addition coefficient r determine.

  In step S <b> 1006, the image processing unit 107 controls each linear interpolation pixel and the nearest neighbor pixel of the linear interpolation pixel according to the weighted addition coefficient determined by the color difference weighted addition coefficient determination unit 143 under the control of the control unit 101. Are weighted and added to the color difference weighted addition unit 144 (color difference interpolation). Then, the color difference weighted addition unit 144 outputs a signal of the G1 component or G2 component of the reference image for which interpolation has been completed.

  In step S <b> 1007, under the control of the control unit 101, the image processing unit 107 sends the G1 and G2 component signals of the reference image that has been interpolated and the R and B component signals of the reference image to the false color determination unit 145. The color difference signal calculation method that suppresses false color generation is determined. Then, the image processing unit 107 generates and outputs color difference signals for the R component and the B component according to the determined calculation method.

  By doing in this way, the luminance signal generation unit 130 of the present embodiment can generate a color difference signal for the G1 component and G2 component that have a high contribution rate to the luminance signal while increasing the subject reproducibility.

  Then, from the luminance signal and the color difference signal generated by the luminance signal generation process and the color difference signal generation process described above, the digital camera 100 of this embodiment can generate a developed image with improved resolution.

  As described above, the image processing apparatus according to the present embodiment generates an image with improved resolution using a plurality of image signals obtained from an image sensor in which each color component is arranged with a predetermined rule. be able to. Specifically, the image processing apparatus acquires a plurality of image signals having a Bayer array, generates one luminance signal, R component and B component color difference signals, and generates a developed image. At this time, in the generation of the luminance signal, the image processing apparatus sets one image signal among the plurality of image signals as a reference image, and extracts a G component signal for the reference image. For each missing pixel included in the G component signal, a pixel value calculated by adaptive interpolation from the neighboring pixels of the missing pixel and an image of the missing pixel extracted from the image signal excluding the reference image among the plurality of image signals The pixel value of the missing pixel is calculated by weighted addition with the pixel value of the nearest pixel with the smallest deviation amount.

  In the present embodiment, a neighboring pixel extraction process and a weighted addition process with an interpolation pixel that are characteristic of the present embodiment are performed for both the generation of the luminance signal and the generation of the color difference signal. However, it is not always necessary to use the processing characteristic of this embodiment for generating both luminance and color difference signals. The processing of this embodiment is performed on either the luminance signal or the color difference signal, and the other signal is converted into one image. Alternatively, a signal may be generated using a known interpolation process or the like. Even in that case, an image with improved resolution can be obtained in either the luminance signal or the color difference signal subjected to the processing in the present embodiment.

[Embodiment 2]
In the first embodiment described above, by performing weighted addition of the adaptive interpolation pixel generated for the G component signal of the base image and one nearest neighbor pixel having the smallest positional deviation amount from the pixel in the reference image. A method for generating a luminance signal has been described. That is, in the first embodiment described above, the method of calculating pixel values from at most two types of image signals for one missing pixel in the G component signal of the reference image has been described.

  On the other hand, noise may occur in the image signal output from the image sensor due to the sensitivity setting, the influence of the ambient temperature of the image sensor, or the like. Specifically, as the sensitivity setting is higher, the amount of noise included in the image signal output from the image sensor increases. Further, as the ambient temperature of the image sensor increases, the amount of noise included in the image signal output from the image sensor increases. In this way, when noise is included in the imaging signal, a pixel value with reduced noise is calculated by adding and synthesizing three or more images to one missing pixel included in the G component signal of the reference image. There is a technique called a composite method.

  In the present embodiment, by determining whether noise can be generated under the conditions when the imaging unit 105 performs imaging, the image signal used for calculating the pixel value of the missing pixel in the G component signal of the reference image is determined. A method of changing the number will be described.

<Configuration of Digital Camera 100>
The digital camera 100 according to the present embodiment includes a noise amount determination unit 111 in addition to the configuration of the digital camera 100 according to the first embodiment.

  The noise amount determination unit 111 acquires information on sensitivity settings of the image sensor at the time of imaging and environmental temperature around the image sensor, and calculates an evaluation value based on a predetermined calculation formula. The evaluation value is a value for quantitatively evaluating the amount of noise that can be included in the image signal output from the imaging unit 105 by imaging.

<Generation method determination process>
A specific process of the generation method determination process for determining the generation method of the luminance signal of the digital camera 100 according to the present embodiment will be described with reference to the flowchart of FIG. The processing corresponding to the flowchart can be realized by the control unit 101 reading out a corresponding processing program stored in, for example, the ROM 102, developing it in the RAM 103, and executing it. Note that this generation method determination processing is described as being started when the imaging unit 105 performs imaging a plurality of times, for example, in response to an imaging instruction from the user.

  In step S1101, the control unit 101 causes the noise amount determination unit 111 to calculate an evaluation value of the noise amount during imaging. Specifically, the control unit 101 transmits the sensitivity setting of the image sensor read from the ROM 102 and the temperature information around the image sensor measured by a temperature sensor (not shown) to the noise amount determination unit 111 to calculate the evaluation value. Let it be done.

  In step S1102, the control unit 101 determines that the noise amount evaluation value at the time of imaging calculated by the noise amount determination unit 111 is equal to or greater than a threshold value set as an evaluation criterion for performing noise reduction by adding and synthesizing three or more image signals. Judge whether there is. If the control unit 101 determines that the evaluation value of the noise amount at the time of imaging is equal to or greater than the threshold value, the control unit 101 moves the process to S1104.

  In step S1103, the control unit 101 executes the luminance signal generation process described in the first embodiment.

  In step S1104, the control unit 101 executes a noise reduction luminance signal generation process described later.

  By doing in this way, the digital camera 100 of this embodiment can determine and execute a method for generating a luminance signal by evaluating a noise generation condition during imaging.

<Noise reduction luminance signal generation processing>
Here, specific processing of the noise reduction luminance signal generation processing of the digital camera 100 of the present embodiment will be described with reference to the flowchart of FIG. In the following description of the noise reduction luminance signal generation process, the steps for performing the same process as the luminance signal generation process of the first embodiment are denoted by the same reference numerals, and the description thereof is omitted. I will only explain the typical steps.

  After the adaptive interpolation unit 131 generates the G component signal of the reference image after the adaptive interpolation process in S903, the control unit 101 moves the process to S1201.

  In step S <b> 1201, the image processing unit 107 extracts a plurality of neighboring pixels used for weighted addition to the neighboring pixel extraction unit 132 for each adaptive interpolation pixel of the G component of the reference image after adaptive interpolation processing under the control of the control unit 101. Let Further, the image processing unit 107 causes the neighboring pixel extraction unit 132 to acquire a positional deviation amount d from the corresponding adaptive interpolation pixel for each of the extracted plurality of neighboring pixels.

  Note that the number of neighboring pixels extracted for each adaptive interpolation pixel in this step may be the number of reference images (n−1) or a number determined by the noise amount evaluation value. Alternatively, all neighboring pixels whose positional deviation amount d is equal to or less than a threshold value determined to be regarded as a corresponding pixel of the adaptive interpolation pixel may be extracted as pixels used for weighted addition.

  In step S1202, the image processing unit 107 selects one extracted neighboring pixel for each adaptive interpolation pixel of the G component of the reference image under the control of the control unit 101, and information on the positional deviation amount d of the pixel. Is transmitted to the weighted addition coefficient determination unit 133 to determine the weighted addition coefficient r. In the noise reduction luminance signal generation processing of the present embodiment, the weighted addition coefficient is selected because the extracted plurality of neighboring pixels are selected one by one and weighted addition processing is performed for noise suppression with the adaptive interpolation pixel. As shown in FIG. 13, it is assumed that it is constant regardless of the amount of displacement.

  In step S1203, the image processing unit 107 controls each adaptive interpolation pixel and a plurality of neighbors extracted for the adaptive interpolation pixel according to the weighted addition coefficient determined by the weighted addition coefficient determination unit 133 under the control of the control unit 101. Pixels are weighted and added by the weighted addition unit 134. The weighted addition unit 134 outputs a G component signal of the reference image for which interpolation has been completed.

  As described above, the digital camera 100 according to the present embodiment determines the amount of noise generated by determining the conditions at the time of imaging. If it is determined that the amount of noise generated is equal to or greater than a predetermined amount, a plurality of image signals are output. By performing interpolation for addition and synthesis, it is possible to generate a luminance signal in which noise is preferably suppressed.

[Modification]
In the first and second embodiments described above, the description has been made on the assumption that the subject image is translated in the image signal obtained by imaging a plurality of times. That is, it has been described that the reference image and the reference image can be aligned simply by a method such as pattern matching in extracting the nearest pixel or the neighboring pixel. In the present modification, an aspect in which a luminance signal generation method is controlled in consideration of a situation in which a subject image is not translated in an image signal obtained by imaging a plurality of times will be described.

  For two image signals obtained by imaging the same subject, a case is considered in which the attitude of the digital camera 100 when imaging one image signal and the attitude of the digital camera 100 when imaging the other image signal are different. At this time, when rotation about the horizontal axis (pitch direction) of the image sensor or rotation about the vertical axis (yaw direction) of the image sensor occurs between the two postures, the subject image is deformed into a trapezoidal shape. A so-called tilt phenomenon occurs.

  For example, the attitude of the digital camera 100 when imaging one image signal is as shown in FIG. 14A, and the attitude when imaging the other image signal is a downward rotation in the pitch direction (referred to as the tilt angle θ). Consider the case where the resulting posture is as shown in FIG. At this time, the rectangular object 1401 facing the digital camera 100 in FIG. 14A is captured in a trapezoidal shape as indicated by 1402 in FIG. 14B.

  In order to perform image alignment with respect to such two image signals, after performing projective transformation according to the tilt angle θ on any of the image signals, an image shift amount such as pattern matching is determined. It is necessary to perform processing. However, since the projective transformation includes pixel interpolation, depending on the tilt angle, a zero drop occurs in the vicinity of Nyquist in the image signal obtained after the transformation. That is, there is a possibility that a suitable luminance signal cannot be obtained even if weighted addition is performed using pixels extracted from an image signal including zero drop.

  In the present modification, a method for generating a luminance signal is determined by determining whether or not an image signal including zero drop is output by projective transformation based on a change in posture of the digital camera 100 during imaging for a plurality of captured image signals. To control. Specifically, since the reproducibility of the luminance is reduced for the image signal including zero drop in the projective transformation, similarly to the noise reduction luminance signal generation processing of the second embodiment described above, for one missing pixel, Pixel values are calculated using weighted addition of pixel values extracted from three or more image signals.

<Generation method determination process>
Hereinafter, specific processing of the generation method determination processing for determining the generation method of the luminance signal of the digital camera 100 according to the present embodiment will be described with reference to the flowchart of FIG. The processing corresponding to the flowchart can be realized by the control unit 101 reading out a corresponding processing program stored in, for example, the ROM 102, developing it in the RAM 103, and executing it. Note that this generation method determination processing will be described as being started when the imaging unit 105 starts imaging a plurality of times in response to, for example, an imaging instruction from the user.

  In step S <b> 1501, the control unit 101 causes an angular velocity sensor (not illustrated) to detect angular velocity displacement during imaging of a plurality of image signals, and causes the RAM 103 to store posture information indicating the passage of time for the angular velocity displacement.

  In step S1502, the control unit 101 sets the posture at the time of shooting of the image selected as the reference image for generating the luminance signal in the image processing unit 107 as a reference posture, and integrates the angular velocity change from the posture to the time of shooting the reference image. Thus, the rotation angle from the standard posture for each of the reference images is calculated. The control unit 101 extracts a rotation angle (tilt angle) in a direction (pitch and yaw direction) in which tilt can occur from the calculated rotation angles.

In step S1503, the control unit 101 calculates an average value of the tilt angle θ calculated for each of the reference images, and outputs an image signal in which the calculated average tilt angle θ _a includes zero drop by a predetermined projective transformation. It is determined whether or not it is equal to or greater than a threshold value (posture determination). Control unit 101, when the average tilt angle theta _a is equal to or greater than the threshold moves the processing to S1505, if it is determined to be less than the threshold value shifts the processing to S1504.

  In step S1504, the control unit 101 executes the luminance signal generation process described in the first embodiment.

  In step S1505, the control unit 101 executes the noise reduction luminance signal generation process described in the second embodiment.

  By doing in this way, even if it is a case where the image signal imaged with the attitude | position which can produce a tilt is contained in several image signals, the luminance signal of suitable image quality can be produced | generated.

[Other Embodiments]
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via various storage media that can be read by a network or a computer, and the computer of the system or apparatus (or CPU, MPU, or the like). Is a process of reading and executing the program.

Claims (18)

Obtaining means for obtaining a plurality of image signals having a Bayer array;
Luminance signal generation means for generating one luminance signal from the plurality of image signals acquired by the acquisition means,
Setting means for setting one image signal as a reference image among the plurality of image signals;
Extracting means for extracting a G component signal for the reference image set by the setting means;
Interpolating means for interpolating missing pixels having no pixel value for the G component signal extracted by the extracting means, and a luminance signal generating means,
Color difference signal generation means for generating R component and B component color difference signals from the plurality of image signals acquired by the acquisition means;
The interpolation means includes, for each missing pixel in the G component signal, a pixel value calculated by adaptive interpolation from neighboring pixels of the missing pixel, and pixels included in the image signal excluding the reference image among the plurality of image signals The pixel value of the missing pixel is calculated by weighted addition of the pixel value of the nearest pixel with the smallest image shift amount with the missing pixel,
The luminance signal generation unit generates a luminance signal from the G component signal obtained by interpolating the missing pixel by the interpolation unit and the R component and B component signals of the reference image obtained by interpolating the missing pixel by linear interpolation. An image processing apparatus.   2. The image processing according to claim 1, wherein the interpolation unit changes a weighted addition ratio of the pixel value of the missing pixel in accordance with an image shift amount between the missing pixel and a neighboring pixel of the missing pixel. apparatus. The color difference signal generating means includes
Color difference extraction means for extracting a G1 component signal and a G2 component signal for the reference image set by the setting means;
Color difference interpolation means for interpolating missing pixels for the G1 component signal and the G2 component signal extracted by the color difference extraction means;
Have
The color-difference interpolating means, for each missing pixel included in each of the G1 component signal and the G2 component signal, a pixel value calculated by linear interpolation from a peripheral pixel of the missing pixel, and among the plurality of image signals Calculating the pixel value of the missing pixel by weighted addition of the pixel value of the pixel that is included in the image signal excluding the reference image and has the smallest image deviation amount from the missing pixel;
The chrominance signal generation unit changes a chrominance signal calculation method according to pixel values of the G1 component signal and the G2 component signal in which missing pixels are interpolated by the chrominance interpolation unit. The image processing apparatus according to claim 1. It further includes a determination unit that acquires sensitivity setting of the image sensor or an environmental temperature, and determines whether to suppress noise included in the image signal,
When the determination unit determines that the noise included in the image signal is to be suppressed by the determination unit, the interpolation unit calculates a pixel value calculated by adaptive interpolation from the peripheral pixels of the missing pixel and the reference image among the plurality of image signals. The image processing apparatus according to claim 1, wherein a pixel value of a missing pixel is calculated by weighted addition of pixel values of neighboring pixels included in a plurality of image signals other than the image signal. The attitude information of the imaging device when each of the plurality of image signals is imaged is obtained, and the attitude of the imaging device when each of the image signals excluding the reference image among the plurality of image signals is imaged is calculated. In addition, there is further provided posture determination means for determining whether or not an average rotation angle from the posture of the imaging device when the reference image is captured is equal to or greater than a predetermined threshold value.
The interpolation means, when the posture determination means determines that the average value of the rotation angle is equal to or greater than the predetermined threshold, the pixel value calculated by adaptive interpolation from the peripheral pixels of the missing pixel, and the plurality of 5. The pixel value of a missing pixel is calculated by performing weighted addition of pixel values of neighboring pixels included in a plurality of image signals excluding the reference image among the image signals of 5. The image processing apparatus according to item 1. An acquisition step in which an acquisition unit of the image processing apparatus acquires a plurality of image signals having a Bayer array;
The luminance signal generation means of the image processing device is a luminance signal generation step of generating one luminance signal from the plurality of image signals acquired in the acquisition step,
A setting step of setting one image signal as a reference image among the plurality of image signals;
An extraction step of extracting a G component signal for the reference image set in the setting step;
An interpolation step of interpolating missing pixels without pixel values for the G component signal extracted in the extraction step;
A color difference signal generation step of generating a color difference signal of an R component and a B component from the plurality of image signals acquired in the acquisition step, the color difference signal generation means of the image processing apparatus,
In the interpolation step, the luminance signal generation unit is configured to calculate, for each missing pixel included in the G component signal, a pixel value calculated from a neighboring pixel of the missing pixel by adaptive interpolation, and the reference image among the plurality of image signals. The pixel value of the missing pixel is calculated by weighted addition of the pixel value of the neighboring pixel with the smallest image deviation amount from the missing pixel, which is a pixel included in the image signal except
In the luminance signal generation step, the luminance signal generation means includes the G component signal obtained by interpolating the missing pixel in the interpolation step, and the R component and B component signals of the reference image obtained by interpolating the missing pixel by linear interpolation. Generating a luminance signal from the image processing apparatus.   The program for functioning a computer as each means of the image processing apparatus of any one of Claims 1 thru | or 5.   A computer-readable storage medium storing a program for causing a computer to function as each unit of the image processing apparatus according to any one of claims 1 to 5. An acquisition means for acquiring an image signal obtained from an image sensor in which each color component is arranged with a predetermined rule;
One image signal of the image signals acquired by the acquisition unit is used as a reference image, and one color component signal of the reference image is a pixel position having no signal of the one color component in the image signal. Extraction means for extracting a signal of the one color component of the plurality of image signals other than the reference image located in the vicinity;
Color component signal generation means for generating a new signal of the one color component using the signal of the one color component of the reference image and the signal of the one color component extracted by the extraction means;
An image processing apparatus comprising: generating means for generating one developed image using a luminance signal or a color difference signal generated using a signal generated by the color component signal generating means. The color component signal generation means calculates an interpolation signal corresponding to a pixel position having no signal of the one color component in the image signal by interpolation processing for the signal of the one color component,
The interpolation signal and the signal extracted by the extracting unit are weighted and added by a coefficient corresponding to the distance between the pixel position and the pixel position of the signal extracted by the extracting unit, thereby obtaining a new one color component. The image processing apparatus according to claim 9, wherein the signal is generated. The interpolation process is an adaptive interpolation process,
The image processing apparatus according to claim 10, wherein the generation unit generates the luminance signal from the new signal of the one color component. The interpolation process is a linear interpolation process,
The image processing apparatus according to claim 10, wherein the generation unit generates the color difference signal from the signal of the new one color component.   The image processing apparatus according to claim 9, wherein the one color component is a signal having a highest contribution rate to the luminance signal. The image sensor has a Bayer array having R, G, and B as color components,
The image processing apparatus according to claim 9, wherein the one color component is a G component.   The image processing apparatus according to claim 9, wherein the color component signal generation unit generates a signal of the new one color component for each color component. An acquisition step of the image processing apparatus for acquiring a plurality of image signals obtained from an image sensor in which each color component is arranged with a predetermined rule;
The extraction means of the image processing device uses one image signal among the plurality of image signals acquired in the acquisition step as a reference image, and the signal of one color component of the reference image An extraction step of extracting the signal of the one color component of the plurality of image signals other than the reference image located in the vicinity of a pixel position having no signal of one color component;
The color component signal generation means of the image processing device uses the signal of the one color component of the reference image and the signal of the one color component extracted in the extraction step to generate a new one color component. A color component signal generation step for generating a signal;
The generation unit of the image processing apparatus includes a generation step of generating one developed image using a luminance signal or a color difference signal generated using the signal generated in the color component signal generation step. A control method of the image processing apparatus.   The program for functioning a computer as each means of the image processing apparatus of any one of Claim 9 thru | or 15.   A computer-readable storage medium storing a program for causing a computer to function as each unit of the image processing apparatus according to any one of claims 9 to 15.

Images