JP5269954B2 - Imaging device - Google Patents

Abstract

An imaging device includes a read control unit (131), mixed-color elimination process unit (120), color temperature detection unit (121), color temperature adjustment unit (122) and estimation computation unit. By setting a unit of addition that has been shifted while being superimposed, the read control unit (131) acquires multiple low-resolution images for which the correspondence between each of multiple colors and a weighting coefficient is different. The mixed-color elimination process unit (120) subjects these multiple low-resolution images to a process that eliminates mixed colors by means of weighted addition, and finds pixel values corresponding to each color. The color temperature detection unit detects color temperatures on the basis of the low-resolution images having mixed colors eliminated. The color temperature adjustment unit (122) adjusts the color temperatures of the low-resolution images on the basis of the detected color temperatures. The estimation computation unit estimates high-resolution images on the basis of the low-resolution images having color temperatures compensated.

Description

  The present invention relates to an imaging device and the like.

  Conventionally, many imaging devices represented by digital still cameras, video cameras, and the like perform imaging using a single-plate solid-state imaging device. In order to perform color photographing, most of the images are picked up by a solid-state imaging device having a plurality of color filters. In an image photographed using such an imaging device, the white balance is generally lost due to the characteristics of the solid-state imaging device, the conditions of the light source or the subject, and the like. Since the color balance and false colors are generated due to the collapse of the white balance, the color balance and false colors are suppressed by adjusting the white balance.

  For example, in Patent Document 1, a captured image is divided into a plurality of blocks, whether or not there is a possibility of gray in the block, and a color average value of the blocks that are determined to be gray is determined. Discloses a method of determining the gain of each color so as to be gray and adjusting the white balance of the image using the gain.

Special table 2009-506700 gazette

  In the conventional method, white balance adjustment is performed so that a subject that should originally be white is white on the image.

  However, the conventional method has a problem that it cannot be applied when an image captured by the solid-state imaging device is intentionally accompanied by a color shift. For example, when pixel values of a plurality of colors are subjected to addition reading with arbitrary weighting applied to each pixel, the pixel value read by addition is a pixel value in which a plurality of colors are mixed. In this case, even if the white balance is set, the white subject is not white in the image obtained by the addition reading. For this reason, when the conventional method is applied to adjust the white color, the white balance is lost.

  According to some aspects of the present invention, it is possible to provide an imaging device or the like that can adjust the white balance of a captured image obtained by weighted addition of pixel values of a plurality of colors.

  According to one embodiment of the present invention, when pixels of a plurality of colors are arranged on an image sensor, an addition unit having the pixels of the plurality of colors is set for each of the plurality of pixels of the image sensor, and the pixels included in the addition unit A read control unit that obtains a low-resolution image by weighted addition of values, and a color mixture that performs processing for removing the color mixture caused by weighted addition of the pixels of the plurality of colors with respect to the acquired low-resolution image A color processing unit that detects a color temperature based on the low-resolution image from which the color mixture has been removed, and a color temperature of the low-resolution image that is adjusted based on the detected color temperature. A color temperature adjustment unit; and an estimation calculation unit that estimates a high-resolution image having a resolution corresponding to the number of pixels of the image sensor based on the low-resolution image that has undergone color temperature adjustment, and the read control unit Is the addition unit shifted while being superimposed. To obtain a plurality of low-resolution images having different correspondences between the respective colors of the plurality of colors and weighting coefficients, and the color mixture removal processing unit removes the color mixture from the plurality of low-resolution images. The present invention relates to an image pickup apparatus that performs processing to obtain pixel values corresponding to the respective colors.

  According to one aspect of the present invention, an addition unit shifted while being superimposed is set, and a plurality of low-resolution images having different correspondences between colors and weighting coefficients are acquired. The acquired low-resolution image is subjected to processing for removing mixed colors, and pixel values corresponding to the respective colors are obtained. The color temperature is detected based on the pixel value after the color mixture removal, and the color temperature of the low resolution image is adjusted. Then, a high resolution image is estimated from the adjusted low resolution image. This makes it possible to adjust the white balance of the captured image obtained by weighted addition of pixel values of a plurality of colors.

  In one aspect of the present invention, the color mixture removal processing unit configures a color mixture removal filter that removes the color mixture based on an inverse matrix of a coefficient matrix that defines the correspondence between each color and a weighting coefficient, and configures the color mixture You may perform the process which removes the said color mixture with a removal filter.

  In one embodiment of the present invention, when the pixels of the plurality of colors of the image sensor have a Bayer arrangement, the readout control unit includes the first to fourth weighting factors for each color and the first to fourth weighting coefficients. The first to fourth low-resolution images corresponding to the combination of are acquired as the plurality of low-resolution images, and the pixel values of the first to fourth low-resolution images are added in the superposition shift. The color mixture removal processing unit is rearranged into one image in accordance with a set position of the unit, and the coefficient matrix is a matrix in which the first to fourth combinations are arranged in the first to fourth rows. The color mixture removal filter may be configured by arranging the inverse matrix components according to the rearrangement, and a convolution operation may be performed between the configured color mixture removal filter and the rearranged image.

  According to these aspects, the color mixture removal filter can be configured based on the correspondence between each color and the weighting coefficient. As a result, it is possible to obtain color information of an image before color mixing by weighted addition, and color temperature detection can be performed based on the color information.

  In one aspect of the present invention, the color temperature adjustment unit obtains an adjustment gain for each color based on the detected color temperature, a coefficient matrix that defines a correspondence between each color and a weighting coefficient, and the adjustment gain. Based on the above, an adjustment filter may be configured, and the color temperature may be adjusted by the configured adjustment filter.

  In one embodiment of the present invention, when the pixels of the plurality of colors of the image sensor have a Bayer arrangement, the readout control unit includes the first to fourth weighting factors for each color and the first to fourth weighting coefficients. The first to fourth low-resolution images corresponding to the combination of are acquired as the plurality of low-resolution images, and the pixel values of the first to fourth low-resolution images are added in the superposition shift. The color temperature adjusting unit is rearranged into one image according to the set position of the unit, and the color temperature adjusting unit uses the matrix in which the first to fourth combinations are arranged in the first to fourth rows as the coefficient matrix, and the coefficient matrix The adjustment filter is configured by arranging the components of the product of the matrix in which the adjustment gains of each color are arranged diagonally and the inverse matrix of the coefficient matrix according to the rearrangement, and the adjustment Convolution of filters and the rearranged images Deployment calculates the may be performed.

  According to these aspects, the adjustment filter can be configured based on the correspondence between each color and the weighting coefficient. As a result, it is possible to perform white balance adjustment processing on the low-resolution image mixed by weighted addition.

  In one aspect of the present invention, the first addition unit set at the first position overlaps with the second addition unit set at the second position where the first position is shifted. In this case, the estimation calculation unit includes a first addition pixel value obtained by weighted addition of the pixel value of the first addition unit and a second addition pixel obtained by weighted addition of the pixel value of the second addition unit. A first intermediate pixel value that is an added pixel value of the first area obtained by removing the overlap area from the first addition unit, and the overlap area is removed from the second addition unit. The relational expression with the second intermediate pixel value that is the added pixel value of the second region is expressed using the difference value, and the first and second intermediate pixel values are estimated using the relational expression. A pixel of each pixel included in the addition unit using the estimated first intermediate pixel value The may be obtained.

  In this way, it is possible to estimate the intermediate pixel value from the addition pixel value of the addition unit that is pixel-shifted while being superimposed, and obtain the final estimated pixel value from the estimated intermediate pixel value. Thereby, a captured image can be estimated by simple processing.

  In the aspect of the invention, the estimation calculation unit may include an intermediate included in the intermediate pixel value pattern when successive intermediate pixel values including the first and second intermediate pixel values are used as an intermediate pixel value pattern. A relational expression between pixel values is expressed using the first and second addition pixel values, and the intermediate pixel value pattern and the first and second addition pixel values expressed by a relational expression between the intermediate pixel values. And the similarity may be evaluated, and an intermediate pixel value included in the intermediate pixel value pattern may be determined based on the similarity evaluation result so that the similarity becomes the highest.

  In this way, an intermediate pixel value can be obtained by estimation based on a plurality of added pixel values acquired by an addition unit that is pixel-shifted while being superimposed.

2 is a configuration example of an imaging apparatus according to the present embodiment. The modification structural example of an imaging device. An example of a pixel array of a pixel array unit. Explanatory drawing about the weighting in superposition shift addition. Explanatory drawing about superposition shift addition. The example of the image which rearranged the 1st-4th low-resolution image. The flowchart of a white balance adjustment process. FIG. 5 is a detailed explanatory diagram of color mixture removal processing. FIG. 5 is a detailed explanatory diagram of color mixture removal processing. FIG. 5 is a detailed explanatory diagram of color mixture removal processing. 11A and 11B are explanatory diagrams of an estimated pixel value and an intermediate pixel value. Explanatory drawing about a restoration estimation process. Explanatory drawing about a restoration estimation process. Explanatory drawing about a restoration estimation process.

  Hereinafter, preferred embodiments of the present invention will be described in detail. The present embodiment described below does not unduly limit the contents of the present invention described in the claims, and all the configurations described in the present embodiment are indispensable as means for solving the present invention. Not necessarily.

1. First, an outline of the present embodiment will be described. In this embodiment, as will be described later with reference to FIG. 5 and the like, a low-resolution moving image is acquired by setting an addition unit that is shifted by superimposition and weighting and adding pixel values included in the addition unit. Then, as will be described later with reference to FIGS. 11A to 14, a high-resolution still image and a high-resolution moving image corresponding to the resolution of the image sensor are restored from the low-resolution moving image.

  In this way, since a high-resolution image can be obtained after the fact, the user can designate a desired timing and an image at a decisive moment can be easily obtained. Further, since the addition unit is shifted in a superimposed manner, it is possible to perform restoration by a simple process.

  In the present embodiment, the image sensor is a color single-plate image sensor such as a Bayer array, and the addition unit includes pixels of a plurality of colors. In the present embodiment, a high-resolution image is restored from the low-resolution image obtained by adding the pixel values of the plurality of colors. However, if the white balance of the low-resolution image is not appropriate, the restored image deteriorates (for example, false Color). Therefore, it is necessary to adjust the white balance of the low resolution image.

  However, the low resolution image has a pixel value in which a plurality of colors are mixed by weighted addition. Therefore, even if it is attempted to detect the color temperature as it is from the low-resolution image, the color information of the image before addition is not known, so that the correct color temperature cannot be detected, and the white balance cannot be adjusted.

  Therefore, in the present embodiment, as will be described later with reference to FIG. 7 and the like, the low-resolution image is processed by a color mixture removal filter to obtain an image reflecting color information before addition, and the color temperature is based on the image. Is detected. Based on the color temperature, white balance adjustment (color temperature adjustment in a narrow sense) of the low resolution image is performed. As a result, the white balance can be adjusted even for an image mixed by weighted addition.

2. Next, the present embodiment will be described in detail. First, FIG. 1 shows a configuration example of an imaging apparatus of the present embodiment that performs the above-described color mixture removal processing and white balance adjustment processing.

  1 includes an imaging optical system 101 (lens), a solid-state imaging device 102 (imaging device), a white balance adjustment unit 103, a high-resolution image generation unit 104 (estimation calculation unit), a development processing unit 105, and a control unit. 106, a memory 107, a display processing unit 108, a monitor display unit 109, and an access control unit 110 (recording processing unit).

  The imaging device 100 is a device that converts optical image information into an electrical signal and records it electronically on a recording medium. As such an imaging apparatus 100, a digital still camera, a digital movie camera, etc. are assumed, for example.

  The imaging optical system 101 forms an image of a subject. The solid-state image sensor 102 is a color single-plate image sensor, and captures a formed subject image. Specifically, the solid-state imaging element 102 includes a pixel array unit 130 (light receiving element) and a readout control unit 131. The read control unit 131 reads a low resolution image in which a plurality of pixels of the pixel array unit 130 are weighted and added, and acquires a plurality of low resolution images by sequentially shifting the range of the added pixels. The acquired image data is transmitted by the control unit 106 to the white balance adjustment unit 103, the high-resolution image generation unit 104, and the access control unit 110.

  Image data before white balance adjustment from the solid-state imaging device 102 or the recording medium 111 is input to the white balance adjustment unit 103. Then, white balance adjustment is performed on the input image data, and the image data after the white balance adjustment is output. Specifically, the white balance adjustment unit 103 analytically obtains the color temperature of the image data, and performs auto white balance adjustment using the color temperature. The white balance adjustment unit 103 includes a color mixture removal processing unit 120 (color mixture removal filter calculation unit), a color temperature detection unit 121, and a color temperature adjustment unit 122 (color temperature correction unit).

  The color mixture removal processing unit 120 performs a process of removing the color mixture generated by the weighting of the solid-state imaging element 102 from the image data before white balance adjustment. The color temperature detection unit 121 detects the color temperature from the image data from which the color mixture has been removed. The color temperature adjustment unit 122 performs a process of adjusting the detected color temperature to the target color temperature for the image data before white balance adjustment. These processes will be described in detail later.

  The high-resolution image generation unit 104 performs a restoration estimation process, which will be described later, on the image data that has been subjected to white balance adjustment, and generates image data having a higher resolution than the image data from the solid-state image sensor 102. As will be described later, the high resolution image generation unit 104 generates one frame of high resolution image data based on four consecutive frames of image data in order to acquire accurate resolution information and color information.

  The development processing unit 105 develops the high resolution image data and outputs final image data. For example, when the solid-state image sensor 102 has a Bayer array, the restored image has a Bayer array. The development processing unit 105 interpolates the Bayer array image to obtain RGB pixel values of all pixels.

  The display processing unit 108 performs a display conversion process on the developed image, and performs a process of displaying the converted image on the monitor display unit 109. For example, as the conversion process for display, a resizing process for matching with the monitor resolution, a gamma correction process and a color correction process for making the image appear uncomfortable on the monitor are performed.

  The access control unit 110 performs recording conversion processing on the developed image, and performs control to record the converted image on the external recording medium 111. For example, as a conversion process for recording, a compression encoding process for reducing the data size or a conversion process to a specified data format is performed. Note that only one of the monitor display and the recording on the recording medium may be performed, or both of them may be performed.

  Note that the present embodiment is not limited to the configuration shown in FIG. 1, and various modifications such as omitting some of the components or adding other components are possible. For example, although the white balance adjustment unit 103 is provided outside the solid-state image sensor 102 in the above, the white balance adjustment unit 103 may be formed on a silicon substrate of the solid-state image sensor 102 as shown in FIG. In this case, image data is acquired by the read control unit 131, white balance adjustment is performed on the image data, and the adjusted image data is output from the solid-state imaging device 102. When image data that has not undergone white balance adjustment is output from the solid-state imaging device 102, a circuit that bypasses the white balance adjustment unit 103 may be provided in the solid-state imaging device 102 so that the circuit can be switched.

  In addition, although an example in which auto white balance is performed has been described above, the present embodiment is not limited to this. For example, a user is given a color temperature of an image, and white balance adjustment is performed using a preset value corresponding to the color temperature. May be performed.

  In the above description, an example in which one frame of a high resolution image is restored from four frames of a low resolution image has been described. However, the present embodiment is not limited to this, and for example, a low number of frames with more or less than four frames. A high-resolution image of one frame may be restored based on the resolution image.

  Further, the example in which the recording medium 111 is provided outside has been described above, but the present embodiment is not limited to this, and the recording medium 111 may be provided inside the imaging apparatus 100 or both inside and outside. May be provided.

3. Superposition shift addition Next, the superposition shift addition performed by the present embodiment will be described in detail. First, FIG. 3 shows an example of the pixel array of the pixel array unit 130. In FIG. 3, x represents the column number of the pixel array, and y represents the row number of the pixel array (x and y are integers of zero or more). For example, the column number is a coordinate in the horizontal scanning direction, and the row number is a coordinate in the vertical scanning direction.

  As shown in FIG. 3, the pixels of the pixel array unit 130 are arranged in a so-called Bayer array. That is, 2 × 2 4 pixels are defined as one block, and the block is repeatedly arranged. Each block includes one R (red) pixel, two G (green) pixels, and one B (blue) pixel. The R pixel is provided with a color filter that transmits R color light, the Gr pixel and Gb pixel are provided with color filters that transmit G color light, and the B pixel transmits B color light. A color filter is provided.

  In the present embodiment, the color filter of the pixel array unit 130 is not limited to RGB, and may be CMY, for example. Further, the arrangement of the color filters is not limited to the Bayer arrangement, and may be another arrangement.

Next, weighting in superposition shift addition will be described with reference to FIG. As shown in FIG. 4, the readout control unit 131 sets 2 × 2 4 pixels as one unit (addition unit), weights and adds the pixel values in the unit, and outputs the added pixel value as one pixel signal. . The weight coefficient “1” for the upper left pixel P1, the weight coefficient “1/2” for the upper right pixel P2, the weight coefficient “1/2” for the lower left pixel P3, and the weight coefficient “1 /” for the lower right pixel P4. 4 "is weighted. The obtained addition pixel value GP is shown in the following formula (1).
GP = P1 + (1/2) * P2 + (1/2) * P3 + (1/4) * P4 (1)

  The unit is four pixels that are the same as or different from the blocks in the Bayer array described above. The readout control unit 131 reads out the added pixel values of the entire region or a partial region of the pixel array unit 130 in one (one frame) readout, and obtains one frame of image data in which the added pixel values are arranged. To do.

  Specifically, as shown in FIG. 5, the position of the unit changes for each frame. In FIG. 5, small squares represent pixels, and hatching applied to the small squares represents the color of the color filter. The correspondence between hatching and color is the same as in FIG. A square with rounded corners represents a unit for performing addition.

  In the f-th frame (f is a natural number), the same four pixels as the color filter block are set as a unit. Next, in the (f + 1) th frame, four pixels shifted to the right by one pixel with respect to the unit of the fth frame are defined as a unit. In the (f + 2) th frame, four pixels shifted by one pixel with respect to the unit in the fth frame are defined as a unit. In the f + 3 frame, 4 pixels shifted to the right by 1 pixel with respect to the unit of the f + 2 frame are defined as a unit. In this way, the first to fourth low-resolution images are acquired in the fth to f + 3th frames, respectively.

  FIG. 6 shows an image X obtained by rearranging the acquired first to fourth low-resolution images. The white balance adjustment unit 103 performs white balance adjustment processing using the rearranged image X. As shown in FIG. 6, the addition pixels of the f-th to (f + 3) -th frames are alternately arranged pixel by pixel so as to be in the upper left, upper right, lower left, and lower right, respectively, thereby generating an image X of one frame. For example, the addition pixels GP00, GP01, GP10, and GP11 shown in FIG. 5 are arranged at the coordinates xy = 00, 01, 10, and 11 of the image X, respectively, as shown in FIG.

Although an example in which one unit is 2 × 2 four pixels has been described above, the present embodiment is not limited to this, and M × N pixels (M and N are integers of 2 or more) may be one unit. Good. In the above description, the weighting factors are set to “1”, “1/2”, and “1/4”. However, the present embodiment is not limited to this. For example, weighting may be performed as shown in the following equation (2). Good (r is a real number where r ≧ 1).
GP = P1 + (1 / r) * P2 + (1 / r) * P3 + (1 / r ² ) * P4 (2)

  In the above description, the weighting factor is fixed. However, the present embodiment is not limited to this, and the weighting factor may be changed in each frame, for example. In the above description, the first to fourth low-resolution images are read in the f-th to f + 3 frames, respectively. However, the present embodiment is not limited to this, and for example, the third and fourth low-resolution images are respectively read. You may read in the f + 3th and f + 2th frame. In the above description, an example in which one low-resolution image is read out in one reading operation (one frame) has been described. However, the present embodiment is not limited to this, and a plurality of low-resolution images are read out in one reading operation. May be read out. For example, the first and fourth low-resolution images may be read out in the f-th frame, and the second and third low-resolution images may be read out in the f + 1-th frame.

  Here, the frame is, for example, the timing at which one low-resolution image is captured by the image sensor, or the timing at which one image (for example, one low-resolution image or image X) is processed in image processing. Alternatively, one low resolution image or high resolution image in the image data is also referred to as a frame as appropriate.

4). White Balance Adjustment Process Next, the white balance adjustment process performed by the present embodiment will be described in detail. FIG. 7 shows a flowchart of white balance adjustment processing.

  For example, step S4 in the flowchart is executed by the color mixture removal processing unit 120. Step S5 is executed by the color temperature detection unit 121. Steps S6, S7, S12, and S13 are executed by the color temperature adjusting unit 122. The other steps S1 to S3 and S8 to S11 are executed by the control unit 106. In addition, each part of this embodiment may be comprised as hardware, and may be comprised by software. When configured by software, processing of each unit is realized by a program describing each unit of the present embodiment being executed by an information processing apparatus such as a CPU.

  As shown in FIG. 7, when the white balance adjustment process is started, image data from the solid-state imaging device 102 or the recording medium 111 is read (S1). Since the subsequent high-resolution image generation requires images of four consecutive frames, the image data of four consecutive frames is read, the pixels are arranged, and an image X of one frame is generated.

  Next, it is determined whether or not to perform auto white balance adjustment (S2). For example, the determination is made based on a user instruction to the imaging apparatus 100. Note that the user's instruction may be given before imaging, or may be requested from the user at the timing of processing step S2.

  When it is determined that the auto white balance adjustment is to be performed (S2, YES), the image X obtained in step S1 is temporarily stored in the memory 107 (S3). This is because an image obtained by changing the image X is used in the color temperature detection in step S5, and the original image is used in the white balance adjustment in step S12.

Next, a color mixture removal filter is calculated for the image X to obtain a new image X ′ from which the intentional color mixture is excluded from the image X (S4). This color mixture removal filter is obtained as follows. First, as shown in the following equation (3), a matrix A representing a mixed color of R, Gr, Gb, and B in the image X is obtained. In the following equation (3), two pixels G in one block are treated as different pixels Gr and Gb.

Here, the component A _ij in the i-th row and j-th column of the matrix A is the color mixture ratio of the j-th color in the i-th frame (i, j = 1, 2, 3, 4). That is, the first to fourth columns of the matrix A correspond to the first to fourth colors R, Gr, Gb, and B, respectively. In addition, the first to fourth rows of the matrix A correspond to the first to fourth frames, respectively. For example, when f = 1 in FIG. 5, R, Gr, Gb, and B pixels are mixed with weight coefficients 1, 1/2, 1/2, and 1/4, respectively, in the first frame. Corresponding to this weighting factor, the components of the first row of the matrix A are 1, 1/2, 1/2, 1/4.

Next, as shown in the following equation (4), an inverse matrix A ⁻¹ of the matrix A is obtained, and a color mixture removal filter α is obtained using the component (A ⁻¹ ) _nm of the inverse matrix A ⁻¹ . Since α does not change depending on the image, α is stored in advance in a heap memory, and the storage area is referred to for each calculation.

  Next, convolution of the color mixture removal filter α and each pixel of the image X is performed for the entire area of the image X, and an image X ′ from which the intentional color mixture has been removed from the image X is obtained. This image X ′ is an image of a Bayer array. As will be described later with reference to FIGS. 8 to 10, the pixel value before the addition is not restored in the image X ′, but since the white balance adjustment is performed on the entire region (or every predetermined region), the color Is separated, it is possible to detect the color temperature from the image X ′.

Next, a process of detecting a color temperature from the image X ′ from which the color mixture has been removed is performed (S5). Next, based on the detected color temperature, white balance adjustment gains g _R , g _Gr , g _Gb , and g _B are obtained for each color so that the white object becomes white (S6). In the image X ′, unlike the image X, intentional color mixture is excluded, so that a general color temperature detection method can be applied. For example, the color temperature may be detected and the white balance adjustment gain may be obtained by the method disclosed in Patent Document 1 described above.

Next, a white balance adjustment filter is generated based on the obtained white balance adjustment gain (S7). This white balance adjustment filter is obtained as follows. First, as shown in the following formula (5), a matrix W is obtained.

Next, as shown in the following equation (6), white balance adjustment filters ω _{1 to} ω ₄ are obtained using the components of the matrix W.

Next, the image X temporarily stored in step S3 is read (S8). Next, convolution of the read image X and the white balance adjustment filters ω _{1 to} ω ₄ is performed to generate an image Y with adjusted white balance (S9). Which of ω _{1 to} ω ₄ is used in the convolution depends on the pixel position of the image X. That is, for example, when f = 1 in FIG. 6, ω _i is used when the white balance adjustment filter is calculated on 3 × 3 pixels centered on the pixel of the i-th frame in the image X (i = 1, 2, 3, 4).

  Next, the image Y is output (S10), and the auto white balance adjustment process is terminated.

If it is determined in step S2 that automatic white balance adjustment is not performed (S2, NO), first, a color temperature designated in advance by the user is read (S11). Next, the white balance gains g _R , g _Gr , g _Gb , and g _B corresponding to the read color temperature are referred to and determined (S12). The white balance gain is set in advance corresponding to each color temperature (for example, a lookup table). Next, white balance adjustment filters ω _{1 to} ω ₄ are obtained by the same method as in step S 7 (S 13). Next, steps S8 to S10 are executed, and the white balance adjustment process is terminated.

In the above description, the matrix A is represented by the above equation (3), but the present embodiment is not limited to this. For example, as shown in the following equation (7), normalization may be performed so that the sum of the components in each row of the matrix A is 1. That is, since the component of the matrix A is the color mixture ratio of each color, it is only necessary that the ratio is maintained.

In the above equation (3), the matrix A is configured with the color filters of four colors R, Gr, Gb, and B, but the present embodiment is not limited to this. For example, as shown in the following equation (8), the matrix A may be configured with three color filters of R, G, and B.

In the above description, an example in which 2 × 2 pixels are one block of the pixel array and each block has four color filters is described. However, the present embodiment is not limited to this. For example, m × n
Pixels (m and n are natural numbers, m × n ≧ 2) may be one block, and one block may have k = m × n color filters. In this case, the matrix A is generally represented by the following equation (9).

Here, the component A _ij in the i-th row and j-th column of the matrix A is the color mixture ratio of the j-th color in the i-th frame (i and j are natural numbers less than or equal to k).

In the above equation (4), the color mixture removal filter α is configured symmetrically with respect to the central component ((A ⁻¹ ) ₁₁ ) in the convolution, but the present embodiment is not limited to this, and the inverse matrix A ^−1. It is sufficient that α is configured using the components for one line. For example, when the matrix A shown in the above equation (9) is used, the color mixture removal filter α _i (i is a natural number equal to or less than k) may be configured as an m × n matrix as shown in the following equation (10).

Here, which _one of α _{1 to} α _k is used in the convolution of the color mixing removal filter depends on the pixel position of the image X. That is, for example, when f = 1 in FIG. 6, α _i is used when the color mixture removal filter is calculated on m × n pixels centered on the pixel of the i-th frame in the image X (i is equal to or less than k). Natural number).

  In the above description, the color mixture removal filter α does not change, and α is stored in the heap memory in advance and referred to at the time of calculation. However, the present embodiment is not limited to this. For example, when α changes (for example, changes according to an image or temporally), α may be obtained from the matrix A as needed, and the α may be used for calculation.

In the above description, the color mixture removal is performed by the color mixture removal filter α, but the present embodiment is not limited to this. For example, for each pixel of the image X, a pixel value corresponding to each frame of the fth to f + 3th frames is obtained by interpolation, and a product of the obtained component of each pixel of the image X and the inverse matrix A ⁻¹ is taken. Alternatively, color mixture removal may be performed.

In the above equation (5), the white balance adjustment gain is set as an absolute value for each color, but the present embodiment is not limited to this. For example, it may be set as a relative gain for a predetermined color. For example, as shown in the following formula (11), the relative gain of R, Gb, and B with respect to Gr may be set as γ _R , γ _Gb , and γ _B with reference to the pixel value of Gr.

In the above equation (5), the matrix W is configured with four color filters, but this embodiment is not limited to this. For example, when the color filter has k colors, the following equation (12) is generally used. A matrix W is constructed.

In the above equation (6), the white balance adjustment filter ω _i is configured symmetrically with respect to the central component (W _ii ) in the convolution. However, the present embodiment is not limited to this, and one row of the matrix W It is sufficient that ω _i is configured using components. For example, when the matrix W shown in the above equation (12) is used, ω _i (i is a natural number of k or less) may be configured as an m × n matrix as shown in the following equation (13).

5. Next, a detailed derivation example of the above-described color mixture removal filter α will be described. In addition, a description will be given of that the image X ′ whose color temperature can be detected is restored by convolving the α.

As shown in FIG. 8, first in f~ the f + 3 frame, it performs pixel addition while changing the weighting, obtains each addition pixel value C _f ~C _f + _3. Focusing on color, this operation is equivalent to the operation shown in the following equation (14). As shown in the following equation (15), the 4 × 4 matrix on the right side of the following equation (14) corresponds to the matrix A.

The added pixel values C _{f to} C _{f + 3} for the obtained four frames are rearranged to generate an image X. In order to return the color of the image X to the state before addition, the inverse matrix A- ¹ shown in the following equation (16) may be applied to the image X. That is, the operation shown in the following formula (17) may be performed. The following equation (17) naturally holds from the above equations (14) to (16) and the definition of the inverse matrix.

As shown in FIG. 9, the operation of the above equation (17) is actually realized by performing convolution of the image X and the color mixture removal filter α on all pixels of the image X. As shown in the following equation (18), α is obtained by rearranging the components of the inverse matrix A ⁻¹ corresponding to the color arrangement of the color filter. This α is a filter having the upper left (1 row, 1 column) component as a reference position.

As shown in the following equation (19), there are four patterns in the convolution depending on the correspondence between the reference position of α and the pixel of the image X. The right equal sign of the following expression (19) is established from the above expression (17).

  As shown in FIG. 9, it can be seen that the image X ′ generated by this convolution has a Bayer array, and the color has surely returned.

Since the same calculation as in the above equation (19) may be performed in the convolution, the color mixture removal filter α may be expanded as shown in the following equation (20). This expanded α is a detailed example corresponding to α in the above equation (4). The reference position is a central component (2 rows and 2 columns). For example, as shown in the following equation (21), when the reference position corresponds to the pixel _Cf , the color of R is returned in the same manner as in the above equation (19).

Next, the RGB color components restored as described above will be described in more detail. Hereinafter, the R component r ₁₁ of the image X ′ illustrated in FIG. 10 will be described as an example. In FIG. 10, the pixel at the position (x, y) is represented as R _xy , C _xy , r _{xy, and the} like.

As shown in the following equation (22), the pixel value of the image X is obtained by pixel addition. Further, as shown in the following equation (23), the R component r ₁₁ of the image X ′ is obtained by the color mixture removal process. In the following formula (23), α in the above formula (18) is used.

If the above equations (22) and (23) are used, the following equation (24) is established. From the following equation (24), it can be seen that the R component r ₁₁ is the result of calculating the same color pixels in a certain range of the image before addition. That is, the R component r ₁₁ reflects the R pixel value before addition, although the original pixel value R ₁₁ itself is not restored.

  As described above, in the image X ′, the color components R, Gr, Gb, and B reflecting the same color pixel values before the addition are restored, and it is understood that the color temperature can be detected from the image X ′.

  According to the above embodiment, as illustrated in FIG. 1, the imaging apparatus 100 includes a read control unit 131, a color mixture removal processing unit 120, a color temperature detection unit 121, a color temperature adjustment unit 122, and an estimation calculation unit (high resolution image). Generator 104).

  As shown in FIG. 3, pixels of a plurality of colors are arranged on the image sensor (solid-state image sensor 102). As shown in FIG. 5, the readout control unit 131 sets the addition unit (unit GP00 or the like) having the pixels of the plurality of colors for each of the plurality of pixels of the image sensor, and weights and adds the pixel values included in the addition unit. To obtain a low resolution image. As described with reference to FIGS. 1, 7, and the like, the color mixture removal processing unit 120 performs a process of removing color mixture due to weighted addition of a plurality of pixels on the acquired low-resolution image. The color temperature detection unit 121 detects the color temperature based on the low resolution image from which the mixed colors are removed. The color temperature adjustment unit 122 adjusts the color temperature of the low resolution image based on the detected color temperature. As will be described later with reference to FIGS. 11A to 14, the estimation calculation unit estimates a high-resolution image having a resolution corresponding to the number of pixels of the image sensor based on the low-resolution image whose color temperature has been corrected.

  In this case, as shown in FIG. 5, the readout control unit 131 sets a plurality of low resolution images (first images) in which the correspondence between each color of the plurality of colors and the weighting coefficient is set by setting the addition unit shifted while being superimposed. 1 to 4 low-resolution images). As described with reference to FIGS. 1, 7, etc., the color mixture removal processing unit 120 performs a process of removing the color mixture on the plurality of low-resolution images, and pixel values (color components; for example, FIG. 10) corresponding to each color. The pixel value of the image X ′ shown in FIG.

  In this way, the color temperature can be detected by removing the mixed color, and the white balance can be adjusted using the color temperature. Thereby, the white balance of an image in which a plurality of color signals are mixed at an arbitrary ratio can be adjusted. In addition, according to the present embodiment, a low resolution image is acquired by superposition shift addition, so that a high resolution image can be restored from the low resolution image. Thereby, a high-resolution image at an arbitrary timing can be acquired by a simple restoration process.

  Here, adjusting the white balance (color temperature) of the low-resolution image means obtaining a low-resolution image in which the white balance is adjusted in the image before addition. That is, a low-resolution image equivalent to that obtained by weighted addition of the pre-addition image with adjusted white balance is obtained by processing the low-resolution image.

  In the present embodiment, as described with reference to FIG. 7, the color mixture removal processing unit 120 is based on the inverse matrix of the coefficient matrix A (the above formula (3)) that defines the correspondence (color mixture ratio) between each color and the weighting coefficient. Then, the color mixture removal filter α (the above formula (4)) for removing the color mixture is configured, and the color mixture is removed by the configured color mixture removal filter α.

More specifically, as shown in FIG. 3, the pixels of the plurality of colors (R, Gr, Gb, B) of the image sensor are in a Bayer array. As shown in FIG. 8, the read control unit 131 corresponds to each color and the first to fourth weighting coefficients (1, 1/2, 1/2, 1/4) corresponding to the first to fourth combinations. First to fourth low-resolution images (pixel values C _{f to} C _{f + 3} ) are acquired. The pixel values C _{f to} C _{f + 3} of the first to fourth low-resolution images are rearranged into one image X according to the set position of the addition unit in the superposition shift. As described in the above equations (3) and (14), the color mixture removal processing unit 120 uses the matrix in which the first to fourth combinations are arranged in the first to fourth rows as the coefficient matrix A. As described in the above equations (17) to (19), the color mixture removal processing unit 120 arranges the components of the inverse matrix A ⁻¹ of the coefficient matrix A in accordance with the rearrangement (pixel arrangement of the image X). Thus, the color mixture removal filter α is configured. As described with reference to FIG. 9, the color mixture removal processing unit 120 performs a convolution operation between the configured color mixture removal filter α and the rearranged image X.

  In the present embodiment, the process of rearranging the first to fourth low-resolution images into the image X is performed by, for example, the control unit 106 in FIG. Alternatively, the reading control unit 131 or the color mixture removal processing unit 120 in FIG. Here, the rearrangement according to the set value of the addition unit in the superposition shift means that, for example, the added pixel value GPxy at the position (x, y) in FIG. 5 is the same position (x, y) in the image X in FIG. ) Is arranged.

  In this way, the color mixture removal filter α can be configured based on the correspondence between each color and the weighting coefficient. As a result, it is possible to obtain color information of an image before color mixing by weighted addition, and color temperature detection can be performed based on the color information.

In this embodiment, as described with reference to FIG. 7, the color temperature adjustment unit 122 obtains an adjustment gain (such as g _{R in the} above equation (5)) for each color based on the detected color temperature, and the coefficient matrix A The adjustment filters ω _{1 to} ω ₄ (formula (6)) are configured based on the adjustment gain and the color temperature is adjusted by the configured adjustment filters ω _{1 to} ω ₄ .

More specifically, as described in the above equation (5), the color temperature adjustment unit 122 arranges the coefficient matrix A and the adjustment gains g _R , g _Gr , g _Gb , and g _B for each color diagonally. A component of the product W of the matrix and the inverse matrix A- ¹ of the coefficient matrix is obtained. As described in the above equation (6), the color temperature adjustment unit 122 configures the adjustment filters ω _{1 to} ω ₄ by arranging them according to the rearrangement (pixel arrangement of the image X). The color temperature adjustment unit 122 performs a convolution operation between the configured adjustment filters ω _{1 to} ω ₄ and the rearranged image X.

In this way, the adjustment filters ω _{1 to} ω ₄ can be configured based on the correspondence between each color and the weighting coefficient. As a result, it is possible to perform white balance adjustment processing on the low-resolution image mixed by weighted addition. That is, the color information before the addition is reproduced by the inverse matrix A- ¹ , the white balance adjustment is performed on the color information by the adjustment gain, and the image after the addition is restored by the coefficient matrix A.

6). Restoration Estimation Process Next, a process for restoring a high-resolution image from the added pixel value acquired by superposition shift addition will be described in detail. In the following, the addition pixel values {a ₀₀ , a ₁₀ , a ₁₁ , a ₀₁ } will be described as an example, but the same applies to other addition pixel values. Although the case where the addition unit is 2 × 2 pixels will be described as an example, the present invention is not limited to this, and may be 3 × 3 pixels, for example.

FIGS. 11A and 11B are explanatory diagrams of the estimated pixel value and the intermediate pixel value. In the estimation processing, the intermediate pixel values b _{00 to} b ₂₁ are estimated from the added pixel values a _{00 to} a _11, and the final estimated pixel values v _{00 to} v ₂₂ are estimated from the intermediate pixel values b _{00 to} b ₂₁ . The suffix such as a _xy corresponds to the pixel position (x, y), and x and y are integers of zero or more. The addition pixel values {a ₀₀ , a ₁₀ , a ₁₁ , a ₀₁ } correspond to the addition units GP 00, GP 10, GP 11, GP 01 described in FIG.

First, the process of estimating the intermediate pixel value will be described using the intermediate pixel values b _{00 to} b ₂₀ in the first row in the horizontal direction as an example. As illustrated in FIG. 12, the intermediate pixel values b _{00 to} b ₂₀ are estimated based on the added pixel values a ₀₀ and a ₁₀ . For the sake of simplicity, for example, assuming that the weighting factor r = 2, the addition pixel values a ₀₀ and a ₁₀ are expressed by the following expression (25).
a ₀₀ = v ₀₀ + (1/2) v ₀₁ + (1/2) v ₁₀ + (1/4) v ₁₁ ,
_{a 10 = v 10 + (1/2} ) v 11 + (1/2) v 20 + (1/4) v 21 (25)

B ₀₀ , b ₁₀ , and b ₂₀ are defined as shown in the following formula (26).
b ₀₀ = v ₀₀ + (1 / r) v ₀₁ = v ₀₀ + (1/2) v ₀₁ ,
b ₁₀ = v ₁₀ + (1 / r) v ₁₁ = v ₁₀ + (1/2) v ₁₁ ,
b ₂₀ = v ₂₀ + (1 / r) v ₂₁ = v ₂₀ + (1/2) v ₂₁ (26)

Next, when the above equation (25) is transformed using the above equation (26), the following equation (27) is established.
a ₀₀ = b ₀₀ + (1/2) b ₁₀ ,
a ₁₀ = b ₁₀ + (1/2) b ₂₀ (27)

In the above equation (27), a difference δi ₀ is obtained by multiplying a ₀₀ and a ₁₀ by a predetermined weighting factor, and rearranged, the following equation (28) is established.
δi ₀ = a ₁₀ -2a ₀₀
= (1/2) b ₂₀ -2b ₀₀ (28)

If b ₀₀ is an unknown (initial variable), intermediate pixel values b ₁₀ and b ₂₀ are obtained as a function of b ₀₀ as shown in the following equation (29).
b ₀₀ = (unknown number),
b ₁₀ = 2 (a ₀₀ −b ₀₀ ),
b ₂₀ = 4b ₀₀ + 2δi ₀ = 4b ₀₀ +2 (a ₁₀ −2a ₀₀ ) (29)

Next, as shown in FIG. 13, the added pixel value pattern {a ₀₀ , a ₁₀ } is compared with the intermediate pixel value pattern {b ₀₀ , b ₁₀ , b ₂₀ }, and an unknown b ₀₀ that minimizes the error is obtained. To derive. Then, the derived value is set as the intermediate pixel value b ₀₀ .

Specifically, the relationship of the following expression (30) is established between the added pixel value {a _xy } and the intermediate pixel value {b _xy , b _{(x + 1) y} }. Considering the weighting by the following equation (30), the evaluation function Ey shown by the following equation (31) is obtained.
a _xy = b _xy + (1/2) b _{(x + 1) y} (30)

As shown in FIG. 14, an unknown b ₀₀ (= α) that minimizes Ey is obtained, and the value of b ₀₀ is determined. Then, the estimated value of b ₀₀ is substituted into the above equation (29) to obtain b ₁₀ and b ₂₀ .

The final estimated pixel value v _xy is obtained in the same manner as the method for _obtaining the intermediate pixel value b _xy . Taking the first column in the vertical direction (x = 0 column) as an example, if the above equation (27) is replaced with the following equation (32), the subsequent processing is the same.
b ₀₀ = v ₀₀ + (1/2) v ₀₁ ,
b ₀₁ = v ₀₁ + (1/2) v ₀₂ (32)

According to the above embodiment, as illustrated in FIG. 1, the imaging apparatus 100 includes the estimation calculation unit (high-resolution image generation unit 104). As shown in FIG. 11A, the first addition unit (for example, a ₀₀ ) set to the first position and the second addition set to the second position where the first position is shifted. Units (eg, a ₁₀ ) overlap.

In this case, as shown in the above equation (28), the estimation calculation unit performs the first and second addition in which the pixel values of the first and second addition units (for example, a ₀₀ and a ₁₀ ) are weighted and added. A difference value δi ₀ between the pixel values a ₀₀ and a ₁₀ is obtained. As shown in FIG. 11B, the first intermediate pixel value b ₀₀ is an addition of the first area (v ₀₀ , v ₀₁ ) obtained by removing the overlapping area (v ₁₀ , v ₁₁ ) from the addition unit a _00. It is a pixel value. The second intermediate pixel value b ₂₀ is an addition pixel value of the second region (v ₂₀ , v ₂₁ ) obtained by removing the overlap region (v ₁₀ , v ₁₁ ) from the addition unit a ₁₀ . As shown in the above equation (29), a relational expression between the first and second intermediate pixel values b ₀₀ and b ₂₀ is expressed using a difference value δi ₀ . As shown in FIG. 13 and the like, the first and second intermediate pixel values b ₀₀ and b ₂₀ are estimated using the relational expression. Using the estimated first intermediate pixel value b ₀₀ , pixel values {v ₀₀ , v ₁₀ , v ₁₁ , v ₀₁ } of each pixel included in the addition unit are obtained.

  In this way, it is possible to simplify the estimation process of the high-resolution image by once estimating the intermediate pixel value from the addition pixel value subjected to the superposition shift and obtaining the estimation pixel value from the intermediate pixel value subjected to the superposition shift. . For example, complicated processing such as a repetitive calculation of a two-dimensional filter (Japanese Patent Laid-Open No. 2009-124621) and a search for a part suitable for setting an initial value (Japanese Patent Laid-Open No. 2008-243037) are not required.

Here, superimposing means having an area where the addition unit overlaps the addition unit. For example, as shown in FIG. 11A, the addition unit a ₀₀ and the addition unit a ₁₀ are two estimated pixels v _10. is to share the _{v 11.}

  The position of the addition unit is the position and coordinates of the addition unit in the pixel array of the image sensor. Alternatively, it is the position or coordinates of the addition unit on the image data in the estimation process. The shifted position is a position shifted from the original position by a pixel, and is a position where the position and coordinates do not coincide with the original position.

In this embodiment, continuous intermediate pixel values including the first and second intermediate pixel values (for example, b ₀₀ , b ₂₀ ) are set as intermediate pixel value patterns {b ₀₀ , b ₁₀ , b ₂₀ }. As shown in the above equation (29), the estimation calculation unit represents the relational expression between the intermediate pixel values included in the intermediate pixel value pattern using the first and second addition pixel values a ₀₀ and a ₁₀ . As shown in FIG. 13, the similarity is evaluated by comparing the intermediate pixel value pattern expressed by the relational expression between the intermediate pixel values with the first and second addition pixel values. Based on the similarity evaluation result, intermediate pixel values {b ₀₀ , b ₁₀ , b ₂₀ } included in the intermediate pixel value pattern are determined so that the similarity is the highest.

  In this way, the intermediate pixel value can be estimated based on a plurality of added pixel values acquired by the addition unit that is pixel-shifted while being superimposed.

  Here, the intermediate pixel value pattern is a data string (a set of data) of intermediate pixel values in a range used for estimation processing. The addition pixel value pattern is a data string of addition pixel values in a range used for the estimation process.

Further, in the present embodiment, as shown in the above equation (31), the estimation calculation unit includes the intermediate pixel value pattern {b ₀₀ , b ₁₀ , b ₂₀ } represented by the relational expression between the intermediate pixel values and the addition pixel. An evaluation function Ey representing an error from the values {a ₀₀ , a ₁₀ } is obtained. Intermediate pixel values {b ₀₀ , b ₁₀ , b ₂₀ } included in the intermediate pixel value pattern are determined so that the value of the evaluation function Ey is minimized.

  In this way, the value of the intermediate pixel value can be estimated by expressing the error by the evaluation function and obtaining the intermediate pixel value corresponding to the minimum value of the evaluation function. For example, as described above, the initial value of the intermediate pixel estimation can be set with a simple process by obtaining the unknown using the least square method. For example, searching for an image portion suitable for initial value setting (Japanese Patent Laid-Open No. 2008-243037) is unnecessary.

  Although the present embodiment has been described in detail as described above, it will be easily understood by those skilled in the art that many modifications can be made without departing from the novel matters and effects of the present invention. Accordingly, all such modifications are intended to be included in the scope of the present invention. For example, a term described with a different term having a broader meaning or the same meaning at least once in the specification or the drawings can be replaced with the different term in any part of the specification or the drawings. In addition, the configuration and operation of the solid-state imaging device, the imaging device, and the like are not limited to those described in the present embodiment, and various modifications can be made.

100 imaging device, 101 imaging optical system, 102 solid-state imaging device,
103 white balance adjustment unit, 104 high-resolution image generation unit,
105 development processing unit, 106 control unit, 107 memory,
108 display processing unit, 109 monitor display unit, 110 access control unit,
111 recording medium, 120 color mixture removal processing unit, 121 color temperature detection unit,
122 color temperature adjustment unit, 130 pixel array unit, 131 readout control unit,
A matrix, B _{blue, C f ~C} f _{+ 3} addition pixel value, Ey evaluation function,
GP00, GP01, GP10, GP11 addition unit, Gr, Gb green,
P1-P4 pixels, R red, W matrix, X, X 'image,
a _{00 to} a ₁₁ addition pixel value, b ₀₀ unknown, b _{00 to} b ₂₁ intermediate pixel value,
b ₁₀ intermediate pixel value, g _R , g _Gr , g _Gb , g _B white balance adjustment gain,
v _{00 to} v ₂₂ estimated pixel value, α color mixture removal filter, δi ₀ difference value,
ω _{1 to} ω ₄ white balance adjustment filter

Claims (7)

When pixels of a plurality of colors are arranged on the image sensor, an addition unit having the pixels of the plurality of colors is set for each of the plurality of pixels of the image sensor, and pixel values included in the addition unit are weighted and added to reduce the value. A readout control unit for acquiring a resolution image;
A color mixture removal processing unit that performs a process of removing a color mixture due to the weighted addition of the plurality of color pixels with respect to the acquired low-resolution image;
A color temperature detection unit that detects a color temperature based on the low-resolution image from which mixed colors have been removed;
A color temperature adjusting unit that adjusts the color temperature of the low-resolution image based on the detected color temperature;
Based on the low-resolution image whose color temperature has been adjusted, an estimation calculation unit that estimates a high-resolution image having a resolution corresponding to the number of pixels of the imaging element;
Including
The read control unit
By setting the addition unit shifted while being superimposed, a plurality of low resolution images having different correspondences between the colors of the plurality of colors and weighting coefficients are obtained,
The color mixture removal processing unit
An image pickup apparatus that performs a process of removing the mixed color on the plurality of low resolution images and obtains pixel values corresponding to the colors. In claim 1,
The color mixture removal processing unit
A color mixing removal filter that removes the color mixing is configured based on an inverse matrix of a coefficient matrix that defines the correspondence between each color and a weighting coefficient, and the color mixing is removed by the configured color mixing removing filter. An imaging device. In claim 2,
When the pixels of the plurality of colors of the image sensor are in a Bayer array,
The read control unit
Obtaining the first to fourth low-resolution images corresponding to the respective colors and the first to fourth weighting coefficients in the first to fourth combinations as the plurality of low-resolution images;
The pixel values of the first to fourth low-resolution images are:
Rearranged into one image according to the set position of the addition unit in the superposition shift,
The color mixture removal processing unit
The matrix in which the first to fourth combinations are arranged in the first to fourth rows is used as the coefficient matrix, and the color mixing removal filter is configured by arranging the inverse matrix components of the coefficient matrix according to the rearrangement. An imaging apparatus that performs a convolution operation between the configured color mixing removal filter and the rearranged image. In any one of Claims 1 thru | or 3,
The color temperature adjusting unit is
The adjustment gain of each color is obtained based on the detected color temperature, an adjustment filter is configured based on the coefficient matrix that defines the correspondence between each color and a weighting coefficient, and the adjustment gain, and the adjustment is configured. An image pickup apparatus, wherein the color temperature is adjusted by a filter. In claim 4,
When the pixels of the plurality of colors of the image sensor are in a Bayer array,
The read control unit
Obtaining the first to fourth low-resolution images corresponding to the respective colors and the first to fourth weighting coefficients in the first to fourth combinations as the plurality of low-resolution images;
The pixel values of the first to fourth low-resolution images are:
Rearranged into one image according to the set position of the addition unit in the superposition shift,
The color temperature adjusting unit is
A matrix in which the first to fourth combinations are arranged in the first to fourth rows is the coefficient matrix, the coefficient matrix, a matrix in which the adjustment gains of the respective colors are diagonally arranged, and an inverse matrix of the coefficient matrix The adjustment filter is configured by arranging the components of the product in accordance with the rearrangement, and a convolution operation between the configured adjustment filter and the rearranged image is performed. . In any one of Claims 1 thru | or 5,
When the first addition unit set in the first position and the second addition unit set in the second position where the first position is shifted overlap,
The estimation calculation unit includes:
Obtaining a difference value between a first addition pixel value obtained by weighted addition of the pixel value of the first addition unit and a second addition pixel value obtained by weighted addition of the pixel value of the second addition unit;
A first intermediate pixel value that is an addition pixel value of the first area obtained by removing the overlap area from the first addition unit, and an addition pixel of the second area obtained by removing the overlap area from the second addition unit. A relational expression with the second intermediate pixel value that is a value is expressed using the difference value,
The first and second intermediate pixel values are estimated using the relational expression, and the pixel value of each pixel included in the addition unit is obtained using the estimated first intermediate pixel value. Imaging device. In claim 6,
The estimation calculation unit includes:
When successive intermediate pixel values including the first and second intermediate pixel values are used as an intermediate pixel value pattern, a relational expression between intermediate pixel values included in the intermediate pixel value pattern is expressed by the first and second Expressed using the sum pixel value,
Comparing the intermediate pixel value pattern represented by the relational expression between the intermediate pixel values and the first and second addition pixel values to evaluate similarity;
An image pickup apparatus, comprising: determining an intermediate pixel value included in the intermediate pixel value pattern based on the similarity evaluation result so that the similarity becomes the highest.

Images