JP2013009172A - Image pickup device and image formation method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image pickup device and an image formation method which can perform S/N ratio deterioration restricted removal of compound colors on high resolution images.SOLUTION: An image pickup device 100 includes a pixel array part 111, a readout control part 112, an estimation arithmetic part 200 and a compound color removal processing part 210. The basic block of color filters in the pixel array part 111 is M×Mpixels. The readout control part 112 sets an addition unit of N×Npixels differing from the M×Mpixels. The readout control part 112 performs weighted additions by setting addition units which are shifted while being superposed to acquire a plurality of low resolution images. The estimation arithmetic part 200 estimates high resolution images on the basis of the low resolution images thus acquired. The compound color removal processing part 210 performs removal processing on compound colors arising from the estimation of pixel values of high resolution images from the plurality of low resolution images and those arising from the addition of pixel values of plural colors by weighted additions.

Description

  The present invention relates to an imaging apparatus, an image generation method, and the like.

  Conventionally, many imaging devices represented by digital still cameras, video cameras, and the like perform imaging using a single-plate solid-state imaging device. In order to perform color photographing, most of the images are picked up by a solid-state imaging device having a plurality of color filters. In these imaging apparatuses, movie scanning is performed in moving image shooting to acquire an image by adding a plurality of pixels.

JP 2009-124621 A JP 2008-243037 A

  When it is desired to obtain a high resolution image from such a low resolution moving image, it is necessary to perform high resolution processing. However, in the conventional methods (for example, Patent Documents 1 and 2), the processing is complicated and the load on the hardware is large, so that it is difficult to mount on a small camera.

  Therefore, as a technique for simplifying the high-resolution processing, a technique for acquiring a plurality of low-resolution images by superposition shift addition and restoring the high-resolution images from the plurality of low-resolution images can be considered. In the superposition shift addition, an addition unit composed of pixels of a plurality of colors is shifted while being superimposed, a predetermined weight is given to each pixel in the addition unit, and addition in which a plurality of colors are mixed is performed.

  However, this method has a problem that mixed colors remain in the image restored by the high resolution processing. Therefore, it is necessary to perform color conversion when generating a high-resolution image and restore the original color. However, there is a problem that the S / N of the high resolution image is deteriorated by performing the color conversion.

  According to some aspects of the present invention, it is possible to provide an imaging apparatus, an image generation method, and the like that can perform mixed color removal with reduced S / N degradation on a high-resolution image.

One aspect of the present invention, a basic block is repeatedly arranged in the color filter, M _x × M _y pixels (M _x, M _y is an integer of 2 or more) and the pixel array unit is in the range of pixels to be added A readout control unit that sets a certain addition unit and obtains a low resolution image by weighted addition of pixel values included in the addition unit, and corresponds to the number of pixels in the pixel array unit based on the low resolution image An estimation calculation unit that estimates a high-resolution image with resolution, and a color mixture removal processing unit that performs a process of removing the color mixture in the estimated high-resolution image, and the readout control unit includes the pixel array unit A range of N _x × N _y pixels (N _x , N _y is an integer of 2 or more) different from the M _x × M _y pixels is set as the addition unit, and the addition unit shifted while being superimposed is set The low solution It relates to an imaging apparatus for obtaining a plurality of low-resolution image as an image.

According to one aspect of the present invention, an addition unit shifted while being superimposed is set, and pixel values in the addition unit are weighted and added to obtain a plurality of low-resolution images. The number of pixels in the addition unit N _x × N _y is different from the number of pixels M _x × M _{y in} the basic block of the color filter. A high-resolution image is estimated based on the acquired low-resolution image, and a process of removing the color mixture in the high-resolution image is performed. As a result, it is possible to remove the color mixture while suppressing the S / N degradation on the high resolution image.

  In one aspect of the present invention, the color mixture removal processing unit adds a color mixture obtained by estimating a pixel value of the high resolution image from the plurality of low resolution images and a plurality of color pixel values by the weighted addition. A process of removing the color mixture due to this may be performed.

  In this way, it is possible to perform the process of removing the color mixture by the restoration estimation process and the weighted addition process.

  In the aspect of the invention, the estimation calculation unit rearranges the pixel values of the plurality of low-resolution images into one image according to a setting position of the addition unit in the superposition shift, and performs the rearrangement. The estimated image is processed by a predetermined estimation filter, and the color mixture removal processing unit is defined by a first matrix representing a color mixture ratio defined by the estimation filter and a weighting coefficient in the weighted addition. The color mixture may be removed by processing the high-resolution image with a color mixture removal filter configured based on the second matrix representing the color mixture ratio.

  In this way, a color mixture removal filter can be configured based on the color mixture ratio defined by the estimation filter and the color mixture ratio defined by the weighting coefficient. Then, the color arrangement of the original color filter can be restored by calculating the color mixture removal filter on the high resolution image.

In one aspect of the present invention, the color filter, the a color filter of Bayer array to 2 × 2 pixels M _x × M _y pixels, the adding unit, 3 × the N _x × N _y pixels An addition unit of three pixels may be used.

  In this way, a higher S / N can be realized after mixed color removal and a higher resolution image with higher image quality can be obtained compared to the case where an addition unit other than 3 × 3 pixels is set.

  In one aspect of the present invention, when the 2 × 2 pixels are configured by one R pixel, two G pixels, and one B pixel, the pixel values of the rearranged image are R, G and B pixels are classified into first to third pixel values having different correspondences with weighting coefficients, and the first matrix is a mixture of the first to third pixel values in the pixel values of the high resolution image. The ratio is a matrix represented by the components of the estimation filter, and the second matrix is based on the weighting coefficient based on the mixture ratio of the R, G, and B pixels in the first to third pixel values. The color mixing removal filter is a filter configured by arranging the product of the inverse matrix of the first and second matrices in accordance with the pixel arrangement of the high resolution image. There may be.

  In this way, the color mixing removal filter can be configured by the first matrix based on the components of the estimation filter and the second matrix based on the weighting coefficient. Thereby, it is possible to configure a color mixture removal filter based on the color mixture ratio in the restoration estimation process and the weighted addition process.

In one aspect of the present invention, the addition unit set by the superposition shift is an N _{x obtained} by shifting the addition unit of the N _x × N _y pixels by N _x times for each column and N _y times for each row. _x × N _y addition units, where the weighted addition value of the pixel values of each column of the N _x × N _y addition units is an intermediate pixel value, and the intermediate pixel value of the first column is an unknown number The intermediate pixel value is represented by a relational expression of the unknown and the weighted addition value by the addition unit, and the estimation filter is represented by the intermediate pixel value represented by the relational expression and the addition unit. The filter may be set so that the square error with the weighted addition value is minimized.

  In this way, an estimation filter for restoring a high resolution image from a plurality of low resolution images can be configured. In addition, since a high-resolution image can be obtained simply by performing a convolution operation on the estimation filter, the restoration estimation process can be simplified.

In another aspect of the present invention, an addition unit that is a range of pixels to be added is set, a pixel value included in the addition unit is weighted and added to obtain a low resolution image, and the low resolution image is added to the low resolution image. Based on the estimation, a high resolution image having a resolution corresponding to the number of pixels in the pixel array unit is estimated, and in the color filter of the pixel array unit, when processing for removing color mixture in the estimated high resolution image is performed. is a basic block is repeatedly arranged _{M x} × _{M y} pixels _(M x, M _y is an integer of 2 or more) different _{N x} × _{N y} pixels and a range of _(N x, N _y is an integer of 2 or more) The present invention relates to an image generation method for acquiring a plurality of low resolution images as the low resolution image by setting the addition unit that is set while being superimposed and shifted while being superimposed.

2 is a configuration example of an imaging apparatus according to the present embodiment. An example of a pixel array of a pixel array unit. Explanatory drawing about the weighting in superposition shift addition. Explanatory drawing about a superposition shift. Explanatory drawing about a superposition shift. Explanatory drawing about a superposition shift. The flowchart of the process which this embodiment performs. Explanatory drawing about the rearrangement of a low-resolution image. Explanatory drawing about the weighting in the case of 2x2 pixel. Explanatory drawing about a color mixture removal process. The example of the S / N characteristic after a color mixture removal process. The example of the resolution characteristic after a mixed color removal process. FIGS. 13A and 13B are explanatory diagrams of the added pixel value and the intermediate pixel value.

  Hereinafter, preferred embodiments of the present invention will be described in detail. The present embodiment described below does not unduly limit the contents of the present invention described in the claims, and all the configurations described in the present embodiment are indispensable as means for solving the present invention. Not necessarily.

1. First, an outline of the present embodiment will be described. In this embodiment, as will be described later with reference to FIGS. 4 to 6, a super-shifted addition unit is set, and a pixel value included in the addition unit is weighted and added to obtain a low-resolution moving image. Then, as will be described later with reference to FIG. 7, an estimation filter is calculated for the low-resolution moving image, and a high-resolution image corresponding to the resolution of the image sensor is restored.

  In this way, since a high-resolution image can be obtained after the fact, the user can designate a desired timing and an image at a decisive moment can be easily obtained. Further, since the addition unit is shifted in a superimposed manner, it is possible to perform restoration by a simple process.

  In the present embodiment, the image sensor is a color single-plate image sensor such as a Bayer array, and the addition unit includes pixels of a plurality of colors. In the present embodiment, restoration is performed from the low-resolution image obtained by adding the pixel values of the plurality of colors. However, as will be described later with reference to FIG. 10, the color is not completely restored in the restored image. The original color is mixed.

  Therefore, in the present embodiment, a process for removing mixed colors is performed on the restored image to restore the original color. However, as will be described later with reference to FIG. 11, if the number of pixels in the addition unit is the same as the number of pixels in the basic block of the color filter (for example, 2 × 2 in the Bayer array), the noise is increased when the color mixture is removed and restored. The S / N of the image is deteriorated.

  Therefore, in this embodiment, the number of pixels in the addition unit is set to a number of pixels different from the number of pixels in the basic block of the color filter. As a result, as will be described later with reference to FIG. 11, S / N deterioration due to color mixture removal can be suppressed, and the image quality of the restored image can be improved.

2. Imaging Device FIG. 1 shows a configuration example of an imaging device according to the present embodiment that performs the restoration process and the color mixture removal process. An imaging apparatus 100 in FIG. 1 includes an imaging optical system 101 (lens), a control unit 102, a memory 103, a monitor display unit 104, a solid-state imaging device 110 (imaging device), and an image processing unit 120.

  The imaging device 100 is a device that converts optical image information into an electrical signal and records it electronically on a recording medium. As such an imaging apparatus 100, a digital still camera, a digital movie camera, etc. are assumed, for example.

  The imaging optical system 101 forms an image of a subject. The solid-state image sensor 110 is a color single-plate image sensor, and captures a formed subject image. Specifically, the solid-state imaging device 110 includes a pixel array unit 111 (light receiving element) and a readout control unit 112 (addition sampling unit).

  The read control unit 112 reads a low-resolution image in which a plurality of pixels of the pixel array unit 111 are weighted and added, and acquires a plurality of low-resolution images by sequentially shifting the range of the added pixels. The acquired image data is transmitted by the control unit 102 to the high resolution image generation unit 121 or the low resolution image generation unit 122 and the access control unit 124.

  The image processing unit 120 performs various processes on the captured low resolution image. Specifically, the image processing unit 120 includes a high-resolution image generation unit 121, a low-resolution image generation unit 122, a display processing unit 123, and an access control unit 124 (recording processing unit).

  The high-resolution image generation unit 121 receives low-resolution image data transmitted from the solid-state imaging device 110 or low-resolution image data read from the recording medium 105 (recording unit) and developed by the access control unit 124. Is done. The high resolution image generation unit 121 performs processing on the input low resolution image data, and generates image data having a higher resolution than the low resolution image data. Specifically, the high resolution image generation unit 121 includes an estimation calculation unit 200 and a color mixture removal processing unit 210 (color conversion processing unit).

  The estimation calculation unit 200 performs restoration estimation processing, which will be described later, on the low-resolution image data obtained by the superposition shift addition to obtain high-resolution image data. The color mixture removal processing unit 210 performs a color mixture removal process (color conversion process), which will be described later, on the restored high resolution image data, and removes the color mixture due to the superposition shift addition and the restoration estimation process.

  The low resolution image generation unit 122 receives the low resolution image data transmitted from the solid-state image sensor 110 or the low resolution image data read from the recording medium 105 and developed by the access control unit 124. The low resolution image generation unit 122 performs processing to be described later on the input low resolution image data, and generates image data having the same resolution as the low resolution image data.

  The display processing unit 123 performs display conversion processing on the image from the high-resolution image generation unit 121 or the low-resolution image generation unit 122 and performs processing for displaying the converted image on the monitor display unit 104. For example, as the conversion process for display, a resizing process for matching with the monitor resolution, a gamma correction process and a color correction process for making the image appear uncomfortable on the monitor are performed.

  The access control unit 124 performs a conversion process for recording on the image from the high-resolution image generation unit 121 or the low-resolution image generation unit 122 and performs control to record the converted image on the external recording medium 105. For example, as a conversion process for recording, a compression encoding process for reducing the data size or a conversion process to a specified data format is performed. Note that only one of the monitor display and the recording on the recording medium may be performed, or both of them may be performed.

3. Superposition Shift Addition Processing Next, the superposition shift addition processing performed by the present embodiment will be described in detail. First, FIG. 2 shows an example of a pixel array of the pixel array unit 111. In FIG. 2, x represents a column number of the pixel array, and y represents a row number of the pixel array (x and y are integers of zero or more). For example, the column number is a coordinate in the horizontal scanning direction, and the row number is a coordinate in the vertical scanning direction. In the following description, the direction in which the column number x decreases is referred to as “left”, the direction in which the column number x increases is referred to as “right”, the direction in which the row number y decreases is referred to as “up”, and the direction in which it increases

  As shown in FIG. 2, the color filters of the pixel array unit 111 are arranged in a so-called Bayer array. That is, 2 × 2 4 pixels are used as a basic block, and the basic block is repeatedly arranged. The basic block is the smallest structural unit in the repeated arrangement of color filters, and is a block having all colors in the color filter. The basic block of the Bayer array includes one R (red) pixel, two G (green) pixels, and one B (blue) pixel.

  Next, weighting in superposition shift addition will be described. As shown in FIG. 3, the readout control unit 112 sets 3 × 3 9 pixels as one unit (addition unit), weights and adds the pixel values in the unit, and outputs the added pixel value as one pixel signal. . In FIG. 3, the weighting coefficients for the pixels P1 to P9 in the unit are shown in parentheses. r is a weighting parameter and is a positive real number (for example, r = 2). FIG. 3 shows an example in which P1 is an R pixel, but the correspondence between P1 to P9 and the coefficients is the same for other colors.

  Next, the superposition shift will be described. As shown in FIGS. 4 to 6, in one read operation from the solid-state imaging device 110, the addition pixel signal is read from the entire region or a partial region of the pixel array unit 111. Then, an image in which the addition pixel signals are arranged is output from the solid-state imaging device 110 as one frame (field) low-resolution image data.

  Specifically, as shown in FIG. 4, in the 0th frame, 3 × 3 pixels at the upper left corner in the entire region or a partial region of the pixel array unit 111 are set as a unit. In the first frame, 3 × 3 pixels shifted to the right by one pixel with respect to the unit of the 0th frame are set as the unit. In the second frame, 3 × 3 pixels further shifted to the right by one pixel are set as a unit. As shown in FIG. 5, in the third frame, 3 × 3 pixels shifted by one pixel with respect to the unit of the 0th frame are set in the unit. In the fourth frame, 3 × 3 pixels shifted to the right by one pixel with respect to the unit of the third frame are set as the unit. In the fifth frame, 3 × 3 pixels shifted to the right by one pixel are set as a unit. As shown in FIG. 6, in the sixth frame, 3 × 3 pixels shifted by one pixel with respect to the unit of the third frame are set in the unit. In the seventh frame, 3 × 3 pixels shifted by one pixel to the right of the unit in the sixth frame are set as the unit. In the eighth frame, 3 × 3 pixels shifted further to the right by one pixel are set as a unit. In the ninth frame, the same 3 × 3 pixels as the unit in the 0th frame are set in the unit.

The addition pixel signal of 3 × 3 pixels acquired by the superposition shift is expressed by the following expression (1) when r = 2 is taken as an example.

Here, v _{x, y} is a pixel value at the position (x, y) of the pixel array unit 111, and a _{i, x, y} is at the position (x, y) of the low-resolution image of the i-th frame. It is a pixel value. In the following formula (1), normalization is performed to make the range of v _{x, y} and a _{i, x, y} the same.

When the above equation (1) is generalized to a unit of N _x × N _y pixels, the following equation (2) is obtained.

Here, t% s represents a remainder of t by s, and [t] represents a maximum integer not exceeding t. N _x and N _y are the numbers of added pixels in the horizontal direction and the vertical direction, respectively. For example, in this embodiment, N _x = N _y = 3. r _x and r _y are weighting parameters in the horizontal direction and the vertical direction, respectively. For example, r _x = r _y = 2 in this embodiment.

  Although an example in which the color filter is an RGB Bayer array has been described above, the present embodiment is not limited to this. For example, the color filter may be CMY or an array other than the Bayer array.

  Further, the example in which the unit is 3 × 3 pixels has been described above. However, the present embodiment is not limited to this, and the number of pixels of the unit is different from the number of pixels as long as the number of pixels is different from the basic block of the color filter. It may be set.

  Moreover, although the example which the solid-state image sensor 110 acquires and outputs a low-resolution image sequentially from the 0th frame was demonstrated above, this embodiment is not limited to this. For example, the frame may be output from an arbitrary frame, or the output order is not limited to the serial number, and may be output in an arbitrary order. A plurality of low resolution images may be acquired or output in one reading operation (one frame).

  Here, the frame is, for example, one shooting operation timing by the image sensor or a timing at which one image (for example, one low resolution image or an image X described later) is processed in the image processing. Alternatively, one low resolution image or high resolution image in the image data is also referred to as a frame as appropriate.

4). Restoration Estimation Process and Color Mixing Removal Process Next, the restoration estimation process and the color mixing removal process performed by this embodiment will be described in detail. FIG. 7 shows a flowchart of processing performed by the present embodiment.

  For example, steps S2 to S5 of the flowchart are executed by the high resolution image generation unit 121. Steps S7 to S9 are executed by the low-resolution image generation unit 122. The other steps S1 and S6 are executed by the control unit 102. In addition, each part of this embodiment may be comprised as hardware, and may be comprised by software. When configured by software, processing of each unit is realized by a program describing each unit of the present embodiment being executed by an information processing apparatus such as a CPU.

  As shown in FIG. 7, when this process is started, it is determined whether or not to output a high-resolution image (S1). For example, the determination is made based on a user instruction to the imaging apparatus 100. Note that the user's instruction may be given before imaging or may be requested from the user at the timing of processing step S1.

  When generating a high-resolution image (S1, YES), image data from the solid-state imaging device 110 or the recording medium 105 is read (S2). In this embodiment, since nine consecutive frames of images are required, nine consecutive frames of low-resolution image data are read. Next, the pixel value of the high resolution image is estimated from the read low resolution image data (S3).

A pixel value estimation method will be described in detail. First, as shown in FIG. 8, pixels a _{i, x, y} of 9 frames of low resolution image data are arranged to generate an image X of 1 frame. The following equation (3) shows the correspondence between the pixel A _{x, y} and the pixel a _{i, x, y} in the image X. As illustrated in FIG. 8, the addition pixel value of the addition unit GPxy at the same position (x, y) illustrated in FIGS. 4 to 6 is disposed at the position (x, y) of the image X.

When the above equation (3) is generalized to a unit of N _x × N _y pixels, the following equation (4) is obtained.

Next, as shown in the following formula (5), a convolution operation is performed between the pixel A _{x, y} of the image X and the estimation filter ρ, and the pixel value u _{x, y} of the high-resolution image is estimated. As shown in the following equation (6), the estimation filter ρ is represented by a matrix, and ρ _kj is a kj component of the matrix. The estimation filter ρ will be described in detail later.

When the estimation filter ρ is generalized to a unit of N _x × N _y pixels and weighting parameters of r _x and r _y , the following expression (7) is obtained. Here, M ^T represents the transpose of the matrix M.

Next, in the pixel value ux _{, y} obtained by the above estimation calculation, since the color is not correctly reproduced as will be described later, color conversion processing is performed on the image after the estimation calculation (S4). Specifically, as shown in the following equation (8), color conversion processing is performed using the color conversion matrix τ.

The matrices α and β are matrices represented by the following equation (9).

Here, σ _kj is the kj component of the matrix σ shown in the following equation (10), and ρ _kj is the kj component of the matrix ρ shown in the above equation (6).

When the above equation (10) is generalized to the unit of N _x × N _y pixels and the weighting parameters of r _x and r _y , the following equation (11) is obtained.

For example, taking FIG. 3 as an example, the components σ ₀₀ , σ ₁₀ , σ ₀₁ , and σ ₁₁ in the above equation (11) are the color mixture ratios corresponding to the colors of the pixels P1, P2, P4, and P5, respectively. It is a value normalized by adding a weighting coefficient. As shown in FIGS. 4 to 6, since the color corresponding to each pixel differs depending on the frame, the color mixture ratio of each color changes depending on the frame.

The matrices α, β, and τ in this embodiment are expressed by the following equation (12) from the above equations (6) and (8) to (10).

As shown in the following equation (13), a color mixture removal filter T is configured from the components of the matrix τ. As shown in the following formula (14), the pixel value u ′ _{x, y} is obtained by performing a convolution operation with the pixel value u _{x, y} and the color mixing removal filter T.

Next, development processing is performed on the pixel value u ′ _{x, y} (S5). For example, when the color filter has a Bayer array, the restored image has a Bayer array. In the development process, the Bayer array image is interpolated to obtain RGB pixel values of all pixels. Next, the developed image is output as final image data, and the process ends (S6).

  Next, when it is determined in step S1 that a low-resolution image is to be generated (S1, NO), low-resolution image data from the solid-state imaging device 110 or the recording medium 105 is read (S7). In the case of generating a high resolution image, a plurality of frames of low resolution image data are read. However, in step 5, one frame of low resolution image data may be used.

Next, since the read low-resolution image data does not reproduce colors correctly, color conversion processing is performed on the low-resolution image data (S8). The following equation (15) shows the color conversion matrix τ. The matrix α is the matrix α in the above equation (9).

  Similar to the above equations (13) and (14), a color mixture removal filter T is constructed from the matrix τ, and a convolution operation between the low resolution image and the color mixture removal filter T is performed. Next, development processing is performed on the image from which the color mixture has been removed (S9), the image after the development processing is output as final image data, and the processing ends (S6).

In the above description, the example in which the high-resolution image estimation process (S3) and the color conversion process (S4) are performed separately has been described. However, the present embodiment is not limited to this and may be performed in a single process. Good. In this case, a new matrix is generated by convolution of the matrix ρ in the above equation (7) and the matrix τ in the above equation (8). Then, a convolution operation between the matrix and each pixel A _{x, y} of the image X is performed to obtain a pixel value u ′ _{x, y} after color mixture removal.

  In the above description, nine low-resolution images are read and high-resolution image estimation processing is performed. However, the present embodiment is not limited to this. For example, an image having a smaller number of frames may be read by performing an interpolation process or the like, or an image having a larger number of frames may be read in order to improve estimation accuracy.

5. Next, the fact that the color mixture is actually removed by the color mixture removal filter T will be described in detail. In the following, for the sake of simplicity, the addition unit is 2 × 2 pixels and the weighting parameter r = 2 as shown in FIG.

First, a description will be given of the point where color mixture remains in a high-resolution image after restoration estimation. As shown in FIG. 10, an image before addition is assumed to be V, an image after weighted addition is assumed to be X, a high-resolution image after restoration processing is assumed to be U, and an image after color mixture removal is assumed to be U ′. Note that, in FIG. 10, in order to focus only on how the colors are mixed, the position xy is omitted and the same reference numerals are assigned to the same color pixels. In the following description, the same color pixels are collectively defined as G = Gr = Gb, a _1/2 = a ₁ = a ₂ , g = gr = gb, and G ′ = Gr ′ = Gb ′.

The matrix σ representing the color mixture by weighted addition is expressed by the following expression (16) from the above expression (11). Since the image X is a convolution of the image V and the matrix σ, the pixel values a ₀ , a _1/2 , and a ₃ of the image X are expressed by the following expression (17).

The estimation filter ρ is expressed by the following expression (18) from the above expression (7). Since the image U is a convolution of the image X and the filter ρ, the pixel values r, g, and b of the image U are expressed by the following equation (19).

If the 3 × 3 matrices of the above equations (17) and (19) are α and β, the following equation (20) is obtained. This matrix α is the same as α calculated from the above equation (9), and is a matrix representing the color mixture of (R, G, B) by the superposition shift addition process. The matrix β is the same as β calculated from the above equation (9), and is a matrix representing the color mixture of (a ₀ , a _1/2 , a ₃ ) by the restoration estimation process.

From the above expressions (17), (19), and (20), the following expression (21) is established. As shown in the following equation (21), the pixel values r, gr, gb, and b of the image U estimated to be restored are the sums of the original colors R, Gr, Gb, and B, respectively. It can be seen that it has not been completely restored.

Next, the point that the mixed color is removed by the mixed color removal filter T will be described. The color mixing removal filter T is expressed by the following expression (22) from the above expression (13). Since the image U ′ is a convolution of the image U and the filter T, the pixel values R ′, Gr ′, Gb ′, and B ′ of the image U ′ are expressed by the following equation (23).

When the 3 × 3 matrix of the above equation (23) is set to τ, the following equation (24) is obtained. This matrix τ is the same as τ calculated from the above equation (8), and is a matrix representing color conversion from (r, g, b) to (R ′, G ′, B ′) in the color mixture removal processing. is there.

From the above expressions (21), (23), and (24), the following expression (25) is established. From the following equation (25), it can be seen that RGB is separated in the image U ′, and the original Bayer array color is restored.

6). S / N Deterioration Suppression Next, the point that S / N deterioration due to color mixture removal is suppressed by setting an addition unit to 3 × 3 pixels different from the basic block 2 × 2 of the Bayer array will be described in detail.

  When the addition unit is 2 × 2 pixels, the matrices α, β, and τ are as described in the above equations (20) and (24). The filters σ, ρ, and T are as described in the above equations (16), (18), and (22).

  When the addition unit is 3 × 3 pixels, the matrices α, β, and τ are as described in the above equation (12). The filters σ, ρ, and T are as described in the above equations (10), (6), and (13).

  As described in the above equation (17) and the like, the matrices α, β, and τ represent coefficients that are multiplied by each color in each process. That is, the matrices α, β, and τ are matrices representing the gain increase amounts of the respective colors. In actual computation, the filters σ, ρ, T composed of the matrices α, β, τ are convolved with each pixel, so the gain increase amount is defined by the computation coefficients (components) of these filters. Will be.

Since the noise level of an image depends on the square sum of gains, the increase / decrease in the noise level in each computation is represented by the square sum of filter computation coefficients. Assuming that the noise level of the pre-addition image acquired by the image sensor is n, the noise level after each calculation is represented by the following equation (26) when the addition unit is 2 × 2 pixels, and the following equation (27) when the addition unit is 3 × 3 pixels: ) Here, n _α represents the noise level after weighted addition, n _β represents the noise level after high resolution, and n _τ represents the noise level after color conversion (final image).

  From the above equations (26) and (27), the noise increases 2.1 times and 1.2 times in the final image, respectively. That is, the increase in noise is smaller in the 3 × 3 pixel addition than in the 2 × 2 pixel addition, and the S / N deterioration due to the calculation is suppressed.

  FIG. 11 shows an example of the S / N characteristic after the color mixture removal process when the noise level of the image before addition is set as a reference (0 dB). As shown in A1 of FIG. 11, the S / N of 3 × 3 pixel addition at r = 2 corresponds to the above noise level and is about 4.5 dB than the S / N of 2 × 2 pixels shown in A2. good. In other r, the S / N of 3 × 3 pixel addition is higher than the S / N of 2 × 2 pixels. As indicated by A3, when the addition unit is 4 × 4 pixels, the S / N can be improved as compared with 2 × 2 pixels. From the above, S / N can be improved by performing addition with a different number of pixels from the basic block 2 × 2 pixels of the Bayer array, compared with the case of adding with the same number of pixels as the basic block.

  Further, the S / N is better for the 3 × 3 pixel addition shown in A1 than in the case of the 4 × 4 pixel addition shown in A3. That is, by setting the addition unit to 3 × 3 pixels, it is possible to increase the S / N more than when the number of pixels is set.

  FIG. 12 shows an example of the resolution characteristics after the color mixture removal process when the resolution of the image before the addition is set as a reference (100%). The resolution of the 3 × 3 pixel addition shown in B1 of FIG. 12 is slightly different from the resolution of the 2 × 2 pixel addition shown in B2 except when r is close to 1 (for example, 1.5 or less). The S / N and resolution of an image often have an inverse correlation, but there is almost no difference in visual resolution between 3 × 3 pixel addition and 2 × 2 pixel addition. S / N degradation can be suppressed without losing the power. As shown in B3, in the case of 4 × 4 pixel addition, S / N deterioration can be suppressed without losing a sense of resolution.

  In addition, the 3 × 3 pixel addition shown in B1 has a better resolution than the 4 × 4 pixel addition shown in B3. That is, by setting the addition unit to 3 × 3 pixels, it is possible to increase the S / N while maintaining a higher resolution than when the number of pixels is set.

According to the above embodiment, as illustrated in FIG. 1, the imaging apparatus 100 includes the pixel array unit 111, the read control unit 112, the estimation calculation unit 200, and the color mixture removal processing unit 210. As described in FIG. 2, the pixel array section 111, a basic block is repeatedly arranged in the color filter is a _{M x} × _{M y} pixels _(M x, _{M y} is an integer of 2 or more). As described with reference to FIGS. 3 to 6, the readout control unit 112 sets an addition unit that is a range of pixels to be added, and obtains a low-resolution image by weighted addition of pixel values included in the addition unit. To do. As described with reference to FIG. 7, the estimation calculation unit 200 estimates a high-resolution image having a resolution corresponding to the number of pixels of the pixel array unit 111 based on the low-resolution image. The color mixture removal processing unit 210 performs processing to remove color mixture in the estimated high resolution image.

In this case, as described with reference to FIG. 3, the read control unit 112 has N _x × N _y pixels different from the M _x × M _y pixels of the pixel array unit 111 (N _x and N _y are integers of 2 or more). Is set as the increment unit. As described with reference to FIGS. 4 to 6, the readout control unit 112 sets a plurality of low resolution images (a _{0, x, y to} a _{8, x, y} ) is obtained.

  In this way, it is possible to remove the mixed color while suppressing the S / N deterioration for the high resolution image. That is, as described with reference to FIG. 11, by setting the addition unit of the number of pixels different from that of the basic block of the color filter, the gain increase amount at the time of color conversion can be made smaller than in the case of the same number of pixels as the basic block. it can. Thereby, the deterioration of S / N in a high resolution image can be suppressed. In addition, as described with reference to FIG. 12, the sense of resolution can be maintained.

  In addition, by shifting while superimposing the addition units, a plurality of low resolution images are acquired, so that a high resolution image at an arbitrary timing can be restored from the plurality of low resolution images. As will be described later with reference to FIG. 13A and the like, restoration estimation processing can be simplified by superimposing shift addition, and hardware load can be reduced.

Here, the addition unit is superimposed means that the two addition units have a common pixel. For example, in FIGS. 4 to 6, a plurality of addition units GP00, GP10,..., GP22 are set by shifting the addition unit horizontally or vertically one pixel at a time, and adjacent additions among the plurality of addition units. The unit has a common pixel. For example GP00 and GP10 share pixel _{v 10 ~v 12, v 20 ~v} 22.

The color mixture is a state in which a plurality of color components before processing are mixed in the processed pixel. For example, as described in the above equation (17), in the pixel a ₀ after the superposition shift addition, the plurality of colors R, G, B before the addition have a color mixture ratio 4: 4: 1 defined by the matrix α. It has been added. The color mixture removal is to restore the color component before the superposition shift addition. In this restoration, it is only necessary that the colors are separated and the original color arrangement is restored, and the original pixel values themselves do not need to be restored.

  In this embodiment, as described with reference to FIG. 10, the color mixture removal processing unit 210 uses the color mixture (the above equation (19)) generated by estimating the pixel value of the high resolution image from a plurality of low resolution images. Then, a process of removing the color mixture (the above expression (17)) due to the addition of the pixel values of a plurality of colors by weighted addition is performed (the above expressions (23) to (25)).

More specifically, as described with reference to FIG. 8, the estimation calculation unit 200 uses the pixel values a _{i, x, y of} a plurality of low-resolution images as the setting position (x, y) of the addition unit GPxy in the superposition shift. ) Are rearranged into one image X in accordance with () (formulas (3) and (4)). As described with reference to FIG. 7, the estimation calculation unit 200 performs estimation by processing the rearranged image X (pixel value A _{x, y} ) with a predetermined estimation filter ρ (the above equation (6), (7)). The color mixture removal processing unit 210 is configured based on a first matrix β representing a color mixture ratio defined by the estimation filter ρ and a second matrix α representing a color mixture ratio defined by a weighting coefficient in weighted addition. A process for removing the color mixture is performed by processing the high resolution image (pixel value u _{x, y} ) with the color mixture removal filter T (the above formulas (8) to (14)).

  In this way, it is possible to perform the process of removing the color mixture by the restoration estimation process and the weighted addition process. Specifically, based on the color mixture ratio defined by the estimation filter ρ and the color mixture ratio defined by the weighting coefficient, the color mixture removal filter T for removing these color mixture can be configured. Then, by calculating T on the restored image, it is possible to restore the original color filter color array.

In this embodiment also, as described in FIG. 2, the color filters are color filters of the Bayer array to 2 × 2 pixels M _x × M _y pixels. As described in FIG. 3, the addition unit is an addition unit in which N _x × N _y pixels are 3 × 3 pixels.

  In this way, as described with reference to FIG. 11, a higher S / N can be realized after the color mixture removal compared to the case where an addition unit other than 3 × 3 pixels (for example, 4 × 4 pixels) is set. Higher resolution and higher resolution images can be obtained. In addition, as described with reference to FIG. 12, a higher resolution can be realized.

In the present embodiment, as described with reference to FIG. 2, the 2 × 2 pixel includes one R pixel, two G pixels, and one B pixel. As shown in FIG. 10, the pixel value _{A x} of rearranged image _{X, y} is, R, G, the first to third pixel value _a corresponding to the B pixel and the weighting coefficients are different 0, _{a 1 / 2} (= a ₁ = a ₂ ) and a ₃ . In the first matrix β, the mixing ratio of the first to third pixel values a ₀ , a _1/2 , and a ₃ in the pixel values ux _{, y} of the high-resolution image U is expressed by components of the estimation filter ρ. (The above formulas (9), (18) to (20)). The second matrix α is a matrix in which the mixing ratio of R, G, and B pixels in the first to third pixel values a ₀ , a _1/2 , and a ₃ is expressed based on a weighting coefficient (matrix σ). (The above formulas (9), (16), (17), (20)). The color mixture removal filter T is a filter configured by arranging components of the inverse matrix product τ = α ⁻¹ β ⁻¹ of the first and second matrices in accordance with the pixel arrangement of the high resolution image ( (8), (13), (22) to (24)).

In this way, the first matrix β can be configured based on the components of the estimation filter ρ, the second matrix α can be configured based on the weighting coefficient, and the color mixing removal filter T can be configured using these matrices. Further, by configuring the color mixture removal filter T based on the product τ = α ⁻¹ β ⁻¹ of the inverse matrix of the first and second matrices, as described in the above equation (25), τ is the color conversion βα. And can restore RGB in color conversion.

Here, the classification into the first to third pixel values a ₀ , a _1/2 , and a ₃ means that the correspondence relationship between each color and the weighting coefficient is changed by the shift of the addition unit, and the correspondence relationship is classified into three types. It is to be classified. For example, in FIGS. 4 to 6, the addition unit is classified into three groups (GP00, GP20, GP02, GP22), (GP10, GP01, GP21, GP12), and (GP11). Are the same.

  The arrangement according to the pixel arrangement of the high resolution image means that the components of the matrix τ represent the coefficients (color mixture ratio) for the pixels r, g, and b as shown in the above equations (23) and (24). The coefficients are rearranged as shown in the above equation (13) in correspondence with the arrangement of the pixels r, g, and b of the image U shown in FIG.

7). Estimation Filter The above estimation filter ρ will be described in detail. In the following, for the sake of simplicity of explanation, a case where the addition unit is 2 × 2 pixels and the weighting parameter r = 2 will be described as an example.

As shown in FIG. 13A, the addition pixel value a _xy is obtained by the superposition shift of the addition unit of 2 × 2 pixels. A ₀₀ and a ₁₀ are shown in the following formula (28). Here, v _xy is a pixel value before addition.
a ₀₀ = v ₀₀ + (1/2) v ₀₁ + (1/2) v ₁₀ + (1/4) v ₁₁ ,
_{a 10 = v 10 + (1/2} ) v 11 + (1/2) v 20 + (1/4) v 21 (28)

As shown in FIG. 13B, an intermediate pixel value b _xy is defined. The intermediate pixel value b _xy is obtained by weighted addition of pixel values included in each column of the addition unit. For example, the first to third columns of pixels included in a ₀₀ and a ₁₀ are the intermediate pixel value b ₀₀ and b ₁₀ and b ₂₀ . B ₀₀ , b ₁₀ and b ₂₀ are shown in the following formula (29).
b ₀₀ = v ₀₀ + (1/2) v ₀₁ ,
b ₁₀ = v ₁₀ + (1/2) v ₁₁ ,
b ₂₀ = v ₂₀ + (1/2) v ₂₁ (29)

When the above equation (28) is transformed using the above equation (29), the following equation (30) is established.
a ₀₀ = b ₀₀ + (1/2) b ₁₀ ,
a ₁₀ = b ₁₀ + (1/2) b ₂₀ (30)

When the above equation (30) is solved for b ₁₀ and b ₂₀ , the following equation (31) is established.
_{b 10 = -2b 00 + 2a 00} ,
_{b 20 = 4b 00 +2 (a} 10 -2a 00) (31)

As shown in the following equation (32), the evaluation function Ey is obtained from the square error between the intermediate pixel value b _xy and the added pixel value a _xy .

Substituting the above equation (31) into the above equation (32), the unknown b ₀₀ that minimizes Ey is obtained, and b ₀₀ is determined. For b ₀₁ , b ₁₁ , and b ₂₁ , b ₀₁ is determined in the same manner as described above, with the unknown being b ₀₁ . The determined b ₀₀ and b ₀₁ are shown in the following formula (33). Here, in the following formula (33), normalization is performed so that the sum of components is 1 for the right-side matrix.

B ₀₀ and b _{01 in} the above formula (33) are represented by the following formula (34). When the above equation (30) is replaced with the following equation (34) and v ₀₀ is determined in the same manner as described above with the unknown as v ₀₀ , the following equation (35) is obtained.
b ₀₀ = v ₀₀ + (1/2) v ₀₁ ,
b ₀₁ = v ₀₁ + (1/2) v ₀₂ (34)

From the above equations (33) and (35), the following equation (36) is obtained. When the coefficients of a ₀₀ , a ₁₀ , a ₀₁ , and a ₁₁ are rewritten as a filter having components ρ ₀₀ , ρ ₁₀ , ρ ₀₁ , and ρ ₁₁ , the following expression (37) is obtained. In this way, the estimation filter ρ described in the above equations (7) and (18) is obtained.

According to the above embodiment, as shown in FIG. 13 (A), the addition unit set by superposition shifts, N _x × N _{y (e.g.} 2 × 2) adding a unit of pixel, one row N _x N _x × N _y addition units a _xy shifted N _y times each time and one row. As described with reference to FIG. 13B, the weighted addition values of the pixel values in each column of the N _x × N _y addition units are set to the intermediate pixel value b _xy, and the intermediate pixel value b ₀₀ in the first column is an unknown number. And As described in the above equation (30), the intermediate pixel values b ₁₀ and b ₂₀ are represented by relational expressions between the unknown number b ₀₀ and the weighted addition values a ₀₀ and a ₁₀ based on the addition unit. As described in the above equations (32) and (33), the estimation filter ρ includes the intermediate pixel values b ₀₀ , b ₁₀ , and b ₂₀ expressed by the relational expressions, and the weighted addition value a ₀₀ by the addition unit, square errors between a ₁₀ is a filter that is set so as to minimize.

In this way, the estimation filter ρ for restoring a high resolution image from a plurality of low resolution images can be configured. In addition, it is possible to simplify the process of restoring and estimating a high resolution image. That is, a high-resolution image can be obtained by simply performing the convolution operation on the estimation filter ρ with respect to the pixel value a _xy obtained by the superposition shift addition. This eliminates the need for complicated processing such as, for example, repetitive calculation of a two-dimensional filter (Japanese Patent Laid-Open No. 2009-124621) and searching for an appropriate part for setting an initial value (Japanese Patent Laid-Open No. 2008-243037). Become.

  Although the present embodiment has been described in detail as described above, it will be easily understood by those skilled in the art that many modifications can be made without departing from the novel matters and effects of the present invention. Accordingly, all such modifications are intended to be included in the scope of the present invention. For example, a term described with a different term having a broader meaning or the same meaning at least once in the specification or the drawings can be replaced with the different term in any part of the specification or the drawings. Further, the configurations and operations of the estimation calculation unit, the color mixture removal processing unit, the imaging device, and the like are not limited to those described in the present embodiment, and various modifications can be made.

DESCRIPTION OF SYMBOLS 100 Imaging device, 101 Imaging optical system, 102 Control part, 103 Memory,
104 monitor display unit, 105 recording medium,
110 solid-state imaging device, 111 pixel array unit, 112 readout control unit,
120 image processing units, 121 high-resolution image generation units,
122 low-resolution image generation unit, 123 display processing unit, 124 access control unit,
200 estimation calculation unit, 210 color mixture removal processing unit,
A _{x, y} pixel value, Ey evaluation function, GPxy addition unit,
M _x × M _y number of basic block pixels, N _x × N _y number of pixels of addition unit,
P1-P9 pixels, R, G, B colors, T color mixture removal filter,
U, U ′, V, X image, a _{i, x, y} pixel value, a _xy addition pixel value,
b ₀₀ unknown, b _xy intermediate pixel value, r weighting parameter,
u _{x, y} pixel values, v _{10 to} v ₁₂ pixels, x, y positions, α second matrix,
β first matrix, ρ estimation filter, σ matrix

Claims (7)

Basic blocks are repeatedly arranged in the color filter, a pixel array portion is a _{M x} × _{M y} pixels _(M x, _{M y} is an integer of 2 or more),
A readout control unit that sets an addition unit that is a range of pixels to be added, obtains a low resolution image by weighted addition of pixel values included in the addition unit, and
Based on the low-resolution image, an estimation calculation unit that estimates a high-resolution image having a resolution corresponding to the number of pixels of the pixel array unit;
A color mixture removal processing unit that performs a process of removing the color mixture in the estimated high resolution image;
Including
The read control unit
The set the M _x × M _y different from the pixel N _x × N _y pixels of the pixel array portion a range of (N _x, N _y is an integer of 2 or more) to said summation unit, which is shifted with superimposed the An imaging apparatus, wherein a plurality of low resolution images are acquired as the low resolution image by setting an addition unit. In claim 1,
The color mixture removal processing unit
A process of removing a color mixture obtained by estimating a pixel value of the high-resolution image from the plurality of low-resolution images and a color mixture obtained by adding pixel values of a plurality of colors by the weighted addition are performed. An imaging device. In claim 2,
The estimation calculation unit includes:
The pixel values of the plurality of low-resolution images are rearranged into one image according to the setting position of the addition unit in the superposition shift, and the rearranged image is processed by a predetermined estimation filter, thereby Make an estimate,
The color mixture removal processing unit
The high resolution is achieved by the color mixture removal filter configured based on the first matrix representing the color mixture ratio defined by the estimation filter and the second matrix representing the color mixture ratio defined by the weighting coefficient in the weighted addition. An image pickup apparatus that performs a process of removing the color mixture by processing an image. In claim 3,
The color filter is
A color filter of Bayer array to 2 × 2 pixels the M _x × M _y pixels,
The addition unit is
An imaging apparatus, wherein the imaging unit is an addition unit in which the N _x × N _y pixels are 3 × 3 pixels. In claim 4,
When the 2 × 2 pixel is composed of one R pixel, two G pixels, and one B pixel,
The pixel values of the rearranged image are:
The R, G, B pixels are classified into first to third pixel values having different correspondences with weighting coefficients,
The first matrix is
The mixing ratio of the first to third pixel values in the pixel values of the high-resolution image is a matrix represented by components of the estimation filter,
The second matrix is
A mixture ratio of the R, G, and B pixels in the first to third pixel values is expressed based on the weighting coefficient;
The color mixing removal filter is
An image pickup apparatus, comprising: a filter configured by arranging a component of an inverse matrix of the first and second matrices in accordance with a pixel arrangement of the high resolution image. In any of claims 3 to 5,
The addition unit set by the superposition shift is
The _{N x} × _{N y} the summation unit pixels, an _{N x} × _{N y} number of summing units which are _{N y} times shifted by _{N x} times and one row by one row,
When the weighted addition value of the pixel value of each column of the N _x × N _y addition units is an intermediate pixel value and the intermediate pixel value of the first column is an unknown number,
The intermediate pixel value is
It is represented by a relational expression between the unknown and the weighted addition value by the addition unit,
The estimation filter is
An image pickup apparatus, comprising: a filter set to minimize a square error between the intermediate pixel value represented by the relational expression and the weighted addition value based on the addition unit. Set an addition unit that is a range of pixels to be added, obtain a low-resolution image by weighted addition of pixel values included in the addition unit,
Based on the low-resolution image, estimating a high-resolution image having a resolution corresponding to the number of pixels of the pixel array unit,
When performing the process of removing the color mixture in the estimated high resolution image,
Wherein the color filter of the pixel array unit is a basic block is repeatedly arranged _{M x} × _{M y} pixels _(M x, M _y is an integer of 2 or more) different from the _{N x} × _{N y} pixels _(N x, N _y is A plurality of low-resolution images are acquired as the low-resolution image by setting the addition unit shifted while being superimposed. Image generation method.

Images