JP2018056848A - Imaging apparatus and imaging apparatus control method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique for acquiring a sheet of HDR image by a smaller number of times of imaging than before.SOLUTION: An imaging apparatus comprises an imaging sensor in which first color filters and second color filters are two-dimensionally arranged according to Bayer arrangement. The first color filters have the same peak wavelength in a spectral transmission factor and mutually different distributions of the spectral transmission factor, and the second color filters have peak wavelength in the spectral transmission factor different from any one of the first color filters.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an HDR image generation technique.

  In an image processing apparatus capable of shooting a high dynamic range (HDR) video, if multiple images with different shutter speeds are HDR combined, it is possible to create a composite image at a frame rate that is higher than the shutter speed in terms of calculation. Can not. Furthermore, when combining from a plurality of shots, there is a problem that when a moving object is photographed and HDR combined, it cannot be combined well.

  However, according to the following configuration, an HDR image can be acquired in one shot (Patent Document 1). That is, this configuration includes a lens optical system having a first optical region and a second optical region, and at least a plurality of first pixels and a plurality of second pixels to which light that has passed through the lens optical system is incident. And an image pickup device. Further, this configuration includes a divided dimming element including a first dimming unit and a second dimming unit located in the first optical region and the second optical region, respectively, and the first dimming device A control unit that changes at least one of the transmittance of the light control unit and the transmittance of the second light control unit. Further, this configuration is an array-like optical element disposed between the lens optical system and the image pickup element, in which light that has passed through the first optical region is incident on a plurality of first pixels, and the second optical element An array-like optical element that causes light having passed through the region to enter the plurality of second pixels is provided.

  The color filter of the CCD image pickup device uses a multiple Bayer array in which 2 × 2 4 pixel blocks have the same color and the array when the block is a unit is a Bayer array. In this case, the same color filters of two levels of high and low are arranged in a checkered pattern (diagonal layout) in each multi-pixel composed of four pixel blocks. For example, the R double pixel is composed of 2 × 2 4 pixels having the spectral sensitivity characteristic of R. Of these, 2 pixels arranged diagonally are high sensitivity pixels (LR) having a low filter density. The two diagonally arranged pixels are low-sensitivity pixels (dR) having a high filter density. The G and B double pixels have the same structure. If the difference between the colors of the double pixels is ignored, two types of high and low sensitivity pixels (L, d) are arranged in a checkered pattern throughout the pixel array. (Patent Document 2).

International Publication No. 2013/114889 JP 2002-165226 A

  However, in the invention described in Patent Document 1, a complicated optical system is required, and the transmittance of the filter is changed using a dimming unit such as an ND filter, so that the apparatus becomes expensive. In the invention described in Patent Document 2, since all the RGB filters are prepared for each of the shades, and the shades are controlled as a set of 4 pixels of RG1G2B, there is a high risk that irregular moire occurs when demosaiced. .

  The present invention has been made in view of such a problem, and provides a technique for acquiring a single HDR image with a smaller number of times of shooting than before.

  According to one embodiment of the present invention, the first color filter group having the same peak wavelength in the spectral transmittance and the distribution of the spectral transmittance is different from each other, and the peak wavelength in the spectral transmittance is different from each of the first color filter group. The image sensor includes two color filter groups that are two-dimensionally arranged according to the Bayer arrangement.

  According to the configuration of the present invention, it is possible to acquire one HDR image with a smaller number of shooting times than in the past.

1 is a block diagram illustrating a configuration example of an imaging device. The flowchart of a HDR synthetic | combination process. The flowchart which shows the detail of the process in step S3. The figure for demonstrating the effect of 1st Embodiment. The figure which shows a color filter and a plane. The figure for demonstrating the modification 2. FIG. The flowchart which shows the detail of the process in step S3. The flowchart which shows the detail of the process in step S3. The figure which shows a color filter and a plane. The figure for demonstrating the effect of 2nd Embodiment.

  Embodiments of the present invention will be described below with reference to the accompanying drawings. The embodiment described below shows an example when the present invention is specifically implemented, and is one of the specific examples of the configurations described in the claims.

[First Embodiment]
An imaging apparatus according to each of the following embodiments and modifications will be described as an example of an imaging apparatus having the following configuration. That is, the first color filter group having the same peak wavelength in spectral transmittance and the distribution of spectral transmittance being different from each other, and the second color filter group having a peak wavelength different in spectral transmittance from both of the first color filter groups. Will be described as an example of an imaging apparatus having imaging sensors arranged two-dimensionally according to a Bayer arrangement.

  Furthermore, an imaging apparatus according to each of the following embodiments and modifications will be described as an example of an imaging apparatus having the following configuration. Here, the first color filter group has a peak wavelength corresponding to green as a peak wavelength in spectral transmittance and has a first spectral transmittance, and corresponds to green as a peak wavelength in spectral transmittance. And a color filter having a second spectral transmittance lower than the first spectral transmittance. Then, the imaging apparatus includes a first color image that is an output of the imaging element on the imaging sensor corresponding to the color filter having the first spectral transmittance in the first color filter group, and the first color filter group. A second color image that is an output of the image sensor on the image sensor corresponding to the color filter having the second spectral transmittance is acquired. Then, the luminance value of the third color image is determined by selectively using the luminance value of the first color image and the luminance value of the second color image for each pixel. Further, the color difference component of the third color image is determined based on the luminance value of the color image that is the output of the image sensor on the image sensor corresponding to the second color filter group.

  First, a configuration example of the imaging apparatus according to the present embodiment will be described with reference to the block diagram of FIG. FIG. 1 shows a main configuration related to the following description, and does not show all the configurations of the imaging apparatus according to the present embodiment. Further, one or more of the functional units shown in FIG. 1 may be combined into one functional unit.

  The setting unit 1 sets any one of shutter speed / aperture / exposure sensitivity to an appropriate shooting condition so that a proper exposure state is obtained. The imaging unit 2 includes a shutter, an imaging lens, an aperture, an optical LPF (Low Pass Filter), a color filter, an imaging sensor, an A / D conversion unit, and the like. The shutter speed / aperture / exposure is set according to the setting by the setting unit 1. Imaging is performed while changing one of the sensitivities.

  A light beam from the outside, such as a subject, enters the optical LPF through an imaging lens having a diaphragm / focus function, and is subjected to low-pass filter processing by the optical LPF and enters a color filter. As is well known, the color filter has a spectral transmittance characteristic corresponding to each pixel (each image sensor included in the image sensor). The image sensor is a two-dimensional arrangement of image sensor groups such as CMOS and CCD, and light that has passed through a color filter enters the image sensor. Each image sensor generates and outputs a signal (pixel signal) having a charge amount corresponding to the amount of light received by itself. The A / D converter performs A / D conversion on the pixel signal output from each image sensor, that is, the image signal output from the image sensor, thereby converting the image signal as an analog signal into a digital signal. To the image signal. The image signal as a digital signal obtained by this conversion is a mosaic color image.

  The synthesizing unit 3 performs HDR synthesis on each image (mosaic color image) obtained by the imaging unit 2 performing imaging while changing any one of the shutter speed / aperture / exposure sensitivity. An HDR image corresponding to the image is generated. The output unit 4 outputs each HDR image generated by the synthesis unit 3.

  The control unit 9 performs operation control using the computer program and data stored in a memory (not shown) in the imaging device, thereby controlling the operation of the entire imaging device including the above-described functional units.

  Next, HDR synthesis processing by the imaging apparatus having the configuration shown in FIG. 1 will be described with reference to the flowchart of FIG. In step S <b> 1, the setting unit 1 sets shooting conditions for the imaging unit 2. There are three types of shooting conditions: shutter speed, aperture, and exposure sensitivity. The setting unit 1 sets one or more of the shutter speed, aperture, and exposure sensitivity so that a proper exposure state is obtained. At this time, the setting unit 1 may fix one or two shooting conditions according to the subject. For example, in the case of a moving subject, the shutter speed is fixed at a high speed, the exposure sensitivity is increased indoors, the exposure sensitivity is decreased outdoors, and the remaining aperture is set to have proper exposure. In addition, when it is desired to obtain an HDR image of a landscape outdoors, the exposure speed is set low, the aperture is fixed high, and the shutter speed is set so as to achieve proper exposure.

  In step S2, the imaging unit 2 performs imaging while changing any one of the shutter speed, aperture, and exposure sensitivity according to the imaging conditions set by the setting unit 1, and synthesizes each color image obtained by the imaging in the subsequent stage. Output to part 3.

  In step S <b> 3, the synthesizing unit 3 performs an HDR synthesizing process on each color image output from the imaging unit 2 to generate an HDR image. Details of the processing in step S3 will be described later.

  In step S4, the output unit 4 outputs the HDR image generated by the synthesizing unit 3. The output unit 4 may output the generated HDR image itself, or may output an image processed image obtained by performing image processing including demosaicing processing on the HDR image. The output destination of the HDR image / image processed image by the output unit 4 is not limited to a specific output destination, a memory (not shown) in the imaging device, a compact flash (registered trademark) inserted in the imaging device, an SD card, or the like This recording medium may be the output destination of the HDR image / image processed image. The output unit 4 may output the HDR image / image processed image to an external device via a USB cable or wirelessly. The output unit 4 may display the image-processed image on a display screen (not shown) included in the imaging apparatus.

  Next, details of the processing in step S3 will be described with reference to the flowchart of FIG. The process according to the flowchart of FIG. 3 is a process for generating one HDR image from one color image. However, a plurality of HDR images can be generated by performing processing according to the flowchart of FIG. 3 for each color image output from the imaging unit 2.

  In step S31, the composition unit 3 decomposes the mosaic color image output from the imaging unit 2 into planes for each color. In the present embodiment, the color filter included in the imaging unit 2 is assumed to have the configuration shown in FIG. That is, when two pixels (one square represents one pixel) × 2 pixels are taken as one unit, the R (red) pixel and B (blue) are located at the upper left and lower right pixel positions in the unit, respectively. Pixels are arranged. In the unit, a G (green) pixel and a G2 (dark green) pixel are arranged at the lower left and upper right pixel positions, respectively. Here, the R pixel is a pixel whose peak wavelength is a red wavelength in the spectral transmittance, and the B pixel is a pixel whose peak wavelength is a blue wavelength in the spectral transmittance. , G pixels are pixels whose peak wavelength is a green wavelength in spectral transmittance. The G2 pixel has a green wavelength with a peak wavelength in the spectral transmittance, and (1 / n) times the spectral transmittance corresponding to each wavelength in the G spectral transmittance distribution. is there. n is 2-4, for example. As for the peak wavelength, the peak wavelength of the B pixel <the peak wavelength of the pixel of G (G2) <the peak wavelength of the pixel of R <R.

  As is well known, the color filter is for transmitting light of a specific color to each image sensor on the image sensor. However, in FIG. 5 (a), for example, a portion marked with “R” is a color filter for transmitting light of R (red) color, and for receiving light transmitted through the color filter. An image sensor is arranged. Hereinafter, an image sensor for receiving light transmitted through a color filter (R pixel) for transmitting light of R color may be referred to as “image sensor R”. The same applies to other colors (B, G, G2 in the case of FIG. 5A). Therefore, what is shown in FIG. 5A is also an arrangement distribution of the image sensor R, the image sensor B, the image sensor G, and the image sensor G2 on the image sensor.

  In step S31, the synthesizing unit 3 generates an image (G plane: upper left in FIG. 5B) in which the luminance values of pixels other than the corresponding pixel (G pixel) corresponding to the image sensor G in the color image are 0. Furthermore, the synthesis unit 3 generates an image (G2 plane: lower left in FIG. 5B) in which the luminance values of pixels other than the corresponding pixel (G2 pixel) corresponding to the imaging element G2 in the color image are 0. Furthermore, the synthesis unit 3 generates an image (R plane: upper right in FIG. 5B) in which the luminance values of pixels other than the corresponding pixel (R pixel) corresponding to the image sensor R in the color image are 0. Further, the synthesis unit 3 generates an image (B plane: lower right in FIG. 5B) in which the luminance values of pixels other than the corresponding pixel (B pixel) corresponding to the image sensor B in the color image are 0.

  In step S32, the synthesis unit 3 performs a blurring process (demosaic process) using a blur filter on each plane, and interpolates the luminance value of the non-corresponding pixel with the luminance value of the corresponding pixel (FIG. 5). (C)). For example, a filter having the following configuration may be used as the blur filter.

| 1 2 1 |
1/4 × | 2 4 2 |
| 1 2 1 |
By performing a blurring process on a plane using such a blur filter, there is a corresponding pixel at any one of the vertical position, the horizontal position, and the four diagonal positions adjacent to the non-corresponding pixel. In this case, the average value of the luminance values of the corresponding pixels becomes the luminance value of the non-corresponding pixel.

  In step S33, the combining unit 3 combines the G plane and the G2 plane that have been subjected to the blurring process. The luminance value of the pixel of interest in the G plane is G, and the luminance value of the pixel at the same position as the pixel of interest in the G2 plane is G2. At this time, the luminance value Ga of the pixel at the same position as the target pixel in the combined plane (plane corresponding to the luminance component of the HDR image: luminance plane) obtained by combining the G plane and the G2 plane is obtained by the following equation.

Ga = max (G, G2 × n)
Here, max (a, b) is a function that returns the larger of a and b. By determining the luminance value of each pixel of the luminance plane based on such an expression, it is possible to reproduce on the luminance plane the gradation of the portion that has been whitened with only the G plane. The process in step S33 corresponds to a process for determining the luminance component of each pixel in the HDR image.

  In step S34, the synthesizing unit 3 out of the pixels of the synthesis plane generated in step S33, the pixel whose luminance value exceeds the luminance value of the pixel at the same position in the G plane (luminance value G2 × n) is specified as a target pixel. Then, the combining unit 3 calculates the luminance value of the pixel having the maximum luminance value (the maximum luminance value is 255 when the luminance value is 0 to 255) among the pixels at the same position as the target pixel in the R plane. The brightness value of the corresponding pixel is updated. Similarly, the synthesizing unit 3 has the luminance value of the pixel having the maximum luminance value (the luminance value is 255 when the luminance value is 0 to 255) among the pixels at the same position as the target pixel in the B plane. Is updated to the luminance value of the corresponding pixel. By such processing, it is possible to determine the R luminance value and the B luminance value corresponding to each pixel of the HDR image. In this way, the gradation that has been skipped at the proper exposure can be accurately reproduced by setting the same value as G2 × n, and the color that has not been skipped can be accurately reproduced. . The process in step S34 corresponds to a process for determining the color difference component in the HDR image.

  As described above, according to the present embodiment, the luminance component and the color difference component of the HDR image can be determined by one imaging (that is, the HDR image can be completed). In addition, since a high frame rate can be taken for one image pickup, a moving subject is not blurred and moiré does not occur during demosaicing. This point will be described with reference to FIG.

  Conventionally, when one HDR image is created with one camera, it is necessary to photograph at least two images with different exposures. Therefore, as shown in FIG. Thus, four HDR images (composite images) are created. On the other hand, if one HDR image can be created by one shooting as in the present embodiment, as shown in FIG. 4B, six HDR images (synthesized images) by six shootings. Can be created, and a moving image can be created with 1.5 times the number of frames compared to FIG.

<Modification 1>
In the following embodiments and modifications, including this modification example, differences from the first embodiment will be mainly described, and unless otherwise noted, the same as the first embodiment. . In the first embodiment, the G2 pixel in the color filter is obtained by multiplying the spectral transmittance corresponding to each wavelength by (1 / n) in the distribution of the spectral transmittance of G, whereby a region having a high luminance value (strong) It was possible to improve the tone reproduction of the part that was whiteout due to reflection or the like. However, the G2 pixel in the color filter may be obtained by multiplying the spectral transmittance corresponding to each wavelength by n times in the distribution of the spectral transmittance of G. n is 2-4, for example. As a result, tone reproduction in a region with a low luminance value (such as a portion shaded by light) can be improved.

  In the present embodiment, the combining unit 3 combines the G plane and the G2 plane that have been subjected to the fraud process in step S33 among the steps of the flowchart of FIG. The luminance value of the pixel of interest in the G plane is G, and the luminance value of the pixel at the same position as the pixel of interest in the G2 plane is G2. At this time, the luminance value Ga of the pixel at the same position as the target pixel in the combined plane (plane corresponding to the luminance component of the HDR image: luminance plane) obtained by combining the G plane and the G2 plane is obtained by the following equation.

Ga = min (G, G2 × 1 / n)
Here, min (a, b) is a function that returns the smaller one of a and b. By determining the luminance value of each pixel of the luminance plane based on such an expression, it is possible to reproduce the gradation of the portion that has been crushed black with only the G plane.

  Further, in step S34, the combining unit 3 does not update the luminance value for the R plane and the G plane, unlike the first embodiment. As described above, according to the present modification, the gray level can be accurately reproduced by the G2 plane in the portion that was originally blacked out with the proper exposure.

<Modification 2>
In the method of sharing the pixel value of the color difference component, there is a possibility that noise of the color difference component is emphasized. Therefore, as shown in FIG. 6, in order to reduce the noise of the color difference component, a composite plane (composite color difference image) obtained by combining the R planes and the B planes obtained for each color image obtained by two imaging operations. Is combined with the luminance plane. Thereby, the noise of the color difference component in an HDR image can be reduced.

  In this modification, the synthesis unit 3 performs processing according to the flowchart of FIG. 7 in step S3. In the flowchart of FIG. 7, the same processing steps as those shown in FIG. 3 are denoted by the same step numbers, and description thereof will be omitted.

  In step S38, the synthesizing unit 3 generates an R plane (R1 plane) generated from one color image of two color images obtained by two imaging operations, and an R plane (R2 plane) generated from the other color image. ) And. If the luminance value of the target pixel in the R1 plane is R1, and the luminance value of the pixel at the same position as the target pixel in the R2 plane is R2, the pixel at the same position as the target pixel in the composite plane obtained by combining the R1 plane and the R2 plane. Is obtained by the following equation.

R = (R1 + R2) / 2
Similarly, the synthesis unit 3 includes a B plane (B1 plane) generated from one color image of two color images obtained by two imaging operations, and a B plane (B2 plane) generated from the other color image. , Is synthesized. If the luminance value of the target pixel in the B1 plane is B1, and the luminance value of the pixel at the same position as the target pixel in the B2 plane is B2, the pixel at the same position as the target pixel in the composite plane obtained by combining the B1 plane and the B2 plane. Is obtained by the following equation.

B = (B1 + B2) / 2
In step S39, the synthesis unit 3 performs the processing in step S34 described above, assuming that the synthesis plane of the R1 plane and the R2 plane is the R plane, and the synthesis plane of the B1 plane and the B2 plane is the B plane.

<Modification 3>
In step S33 in the first embodiment and the second modification, the luminance value Ga of the pixel in the combined plane obtained by combining the G plane and the G2 plane is set, and the luminance value of the corresponding pixel in the G plane and the corresponding pixel in the G2 plane are set. The larger one of the luminance values × n was adopted. However, other methods may be adopted when determining the luminance value Ga in the first embodiment and the second modification. For example, a threshold value Th (= about 90% of the maximum luminance value in the G plane) in consideration of noise may be set, and the luminance value Ga may be determined according to the following equation.

Ga = G (when G <Th)
G2 × n (when G ≧ Th)
By determining the luminance value Ga according to such an expression, it becomes possible to reproduce the gradation of the portion where the noise is less noticeable and has been whitened with only the G plane. Similarly, in Step S33 in Modification 1, the luminance value Ga of the pixel in the combined plane obtained by combining the G plane and the G2 plane is set, and the luminance value of the corresponding pixel in the G plane and the luminance value of the corresponding pixel in the G2 plane are used. The smaller one of × 1 / n was adopted. However, another method may be adopted when determining the luminance value Ga in the first modification. For example, a threshold value Th2 (= maximum luminance value in the G2 plane × about 90% of 1 / n) may be set in consideration of noise, and the luminance value Ga may be determined according to the following equation.

Ga = G2 × 1 / n (when G2 × 1 / n <Th)
G (when G2 × 1 / n ≧ Th)
By determining the luminance value Ga according to such an expression, it becomes possible to reproduce the gradation of the portion that is less noticeable and is blackened only by the G plane.

[Second Embodiment]
In the first embodiment and its modification example, the luminance component and the color difference component of the HDR image are created using four planes of the G plane, the G2 plane, the R plane, and the B plane. It is not limited to the combination of planes. For example, four types of planes may be created for the G plane, and two types of planes may be created for the R plane and the B plane. In such a case, the color filter included in the imaging unit 2 has a configuration as shown in FIG. That is, when 4 pixels (one square represents 1 pixel) × 4 pixels are defined as 1 unit, R2 pixel, R4 pixel, G1 pixel, G2 pixel, G3 pixel, G4 Eight pixels of the pixel, the B2 pixel, and the B4 pixel are arranged two by two in the arrangement pattern shown in FIG.

  Here, the R2 pixel is a pixel whose peak wavelength is a red wavelength in the spectral transmittance, and the B2 pixel is a pixel whose peak wavelength is a blue wavelength in the spectral transmittance. . The pixel of R4 has a peak wavelength of red in the spectral transmittance, and the spectral transmittance corresponding to each wavelength in the distribution of the spectral transmittance of R2 is (1 / m) times. The B4 pixel has a peak wavelength of blue in the spectral transmittance, and the spectral transmittance corresponding to each wavelength in the distribution of the spectral transmittance of B2 is (1 / m) times. Here, m = n × n. The pixel of G1 is a pixel whose peak wavelength is a green wavelength in the spectral transmittance. The G2 pixel has a peak wavelength of green in the spectral transmittance, and the spectral transmittance corresponding to each wavelength in the spectral transmittance distribution of G1 is (1 / n) times. The G3 pixel has a peak wavelength of green in the spectral transmittance, and the spectral transmittance corresponding to each wavelength in the spectral transmittance distribution of G2 is (1 / n) times. The G4 pixel has a peak wavelength of green in the spectral transmittance, and the spectral transmittance corresponding to each wavelength in the spectral transmittance distribution of G3 is (1 / n) times. n is 1.4-2, for example.

  In FIG. 9 (a), for example, a color filter for transmitting light of the color R2 (red) and an image sensor for receiving the light transmitted through the color filter are provided at locations marked with “R2”. Will be arranged. Hereinafter, an image sensor for receiving light transmitted through a color filter (R2 pixel) for transmitting R2 color light may be referred to as “image sensor R2”. The same applies to the other colors (R4, B2, B4, G1, G2, G3, and G4 in the case of FIG. 9A). Therefore, what was shown to Fig.9 (a) is also arrangement | positioning distribution of image pick-up element R2, R4, image pick-up element B2, B4, and image pick-up element G1-G4 on an image sensor.

  In the present embodiment, the synthesis unit 3 performs processing according to the flowchart of FIG. 8 in step S3. In step S301, the synthesizing unit 3 sets the luminance value of pixels other than the corresponding pixel (Gi pixel) corresponding to the imaging element Gi (i = 1, 2, 3, 4) in the color image to 0 (Gi plane: FIG. 9 (b) left) is generated. Further, the synthesizing unit 3 is an image (Rj plane: right of FIG. 9B) in which the luminance values of pixels other than the corresponding pixel (Rj pixel) corresponding to the imaging element Rj (j = 2, 4) in the color image are set to 0. ) Is generated. Further, the synthesizing unit 3 generates an image (Bj plane: right in FIG. 9B) in which the luminance values of pixels other than the corresponding pixel (Bj pixel) corresponding to the image sensor Bj in the color image are 0.

  In step S302, the synthesizer 3 performs a blurring process (demosaic process) using a blurring filter on each plane, and interpolates the luminance value of the non-corresponding pixel with the luminance value of the corresponding pixel (FIG. 9). (C)). For example, a filter having the following configuration may be used as the blur filter.

| 0 3 6 3 0 |
| 3 12 18 12 3 |
1/24 × | 6 18 24 18 6 |
| 3 12 18 12 3 |
| 0 3 6 3 0 |
In step S303, the synthesizer 3 synthesizes the G1 to G4 planes that have been subjected to the blurring process (FIG. 9D). The luminance value of the pixel of interest in the G1 plane is G1, the luminance value of the pixel at the same position as the pixel of interest in the G2 plane is G2, the luminance value of the pixel at the same position as the pixel of interest in the G3 plane is the attention value in the G3, G4 plane Let G4 be the luminance value of the pixel at the same position as the pixel. At this time, the luminance value Ga of the pixel at the same position as the pixel of interest in the combined plane (plane corresponding to the luminance component of the HDR image: luminance plane) obtained by combining the G1 to G4 planes is obtained by the following equation.

Ga = max (max (G, G2 × n), max (G3 × n × n, G4 × n × n × n))
By determining the luminance value of each pixel of the luminance plane based on such an expression, it is possible to reproduce the gradation of the portion that has been whitened with only the G1 plane on the synthesis plane.

  In step S304, the synthesizer 3 synthesizes the B2 plane and the B4 plane that have been subjected to the blurring process, and synthesizes the R2 plane and the R4 plane that have been subjected to the blurring process. If the luminance value of the target pixel in the B2 plane is B2, and the luminance value of the pixel at the same position as the target pixel in the B4 plane is B4, the pixel at the same position as the target pixel in the combined plane obtained by combining the B2 plane and the B4 plane Is obtained by the following equation.

Ba = max (B2, B4 × m)
By determining the luminance value in the combined plane of the B2 plane and the B4 plane in this way, it is possible to reproduce the gradation of the portion that cannot be reproduced only by the B2 plane. Similarly, the luminance value of the pixel of interest in the R2 plane is R2, and the luminance value of the pixel at the same position as the pixel of interest in the R4 plane is R4. At this time, the luminance value Bb of the pixel at the same position as the target pixel in the composite plane obtained by combining the R2 plane and the R4 plane is obtained by the following expression.

Bb = max (R2, R4 × m)
In step S305, the combining unit 3 selects a pixel (G2 × n) in which the luminance value of the pixel exceeds the luminance value of the pixel at the same position in the G1 plane among the pixels of the combined planes G1 to G4. , G3 × n × n or G4 × n × n × n) is specified as the target pixel. For each of the R2 and R4 planes, the synthesizing unit 3 determines that the luminance value of the pixels at the same position as the target pixel in the plane is the maximum luminance value (when the luminance value takes a value of 0 to 255, the maximum luminance value is The brightness value of the pixel 255) is updated to the brightness value of the corresponding pixel. Similarly, for each of the B2 and B4 planes, the synthesizing unit 3 determines that the luminance value is the maximum luminance value among the pixels at the same position as the target pixel in the plane (if the luminance value takes a value of 0 to 255, the maximum luminance value). Updates the luminance value of the pixel 255) to the luminance value of the corresponding pixel. By such processing, it is possible to determine the R luminance value and the B luminance value corresponding to each pixel of the HDR image. In this way, the portion that was originally flying at the appropriate exposure can be accurately reproduced by setting the same value as any of G2 × n, G3 × n × n, and G4 × n × n × n, On the other hand, the color that is not flying can be accurately reproduced.

  Here, conventionally, when one HDR image is created with one camera, it is necessary to take a plurality of images with different exposures. Therefore, as shown in FIG. Three HDR images (composite images) are created by photographing (12 times in FIG. 10A). On the other hand, if one HDR image can be created by one shooting as in the present embodiment, as shown in FIG. 10B, six HDR images (composite images) by six shootings. Can be created, and a moving image can be created with three times the number of frames as compared with FIG. Note that a part or all of the configurations described in the first embodiment, its modified examples, and the second embodiment may be appropriately combined or selectively used.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  1: Setting unit 2: Imaging unit 3: Combining unit 4: Output unit 9: Control unit

Claims (8)

  The first color filter group having the same peak wavelength in the spectral transmittance and the distribution of the spectral transmittance being different from each other and the second color filter group having the different peak wavelength in the spectral transmittance are the Bayer. An imaging apparatus having an imaging sensor arranged two-dimensionally according to the arrangement.   The first color filter group includes a color filter having a peak wavelength corresponding to green as a peak wavelength in spectral transmittance and having a first spectral transmittance, and a peak corresponding to green as a peak wavelength in spectral transmittance. The image pickup apparatus according to claim 1, further comprising: a color filter having a wavelength and a second spectral transmittance lower than the first spectral transmittance.   The second color filter group includes a color filter having a peak wavelength in spectral transmittance that is longer than a peak wavelength corresponding to green, and a color filter having a peak wavelength in spectral transmittance that is shorter than a peak wavelength corresponding to green. The imaging apparatus according to claim 1, further comprising: The imaging device further includes:
A first color image that is an output of an image sensor on the image sensor corresponding to a color filter having the first spectral transmittance in the first color filter group, and the first color filter group in the first color filter group Means for obtaining a second color image that is an output of the image sensor on the image sensor corresponding to a color filter having a second spectral transmittance;
Means for selectively using a luminance value of the first color image and a luminance value of the second color image for each pixel to determine a luminance component of the third color image;
Determining means for determining a color difference component of the third color image based on a luminance value of a color image that is an output of an image sensor on the image sensor corresponding to the second color filter group. The imaging apparatus according to claim 2.   The determining means determines a color difference component of the third color image based on a luminance value of an image obtained by synthesizing a color image that is an output of an image sensor on the image sensor corresponding to the second color filter group. The imaging apparatus according to claim 4, wherein:   The determining means selects, for each pixel, a luminance value of a color image that is an output of an image sensor on the image sensor corresponding to the color filter for each color filter having the same peak wavelength in the second color filter group. The imaging apparatus according to claim 4, wherein the imaging device is used to generate a fourth color image and determine a color difference component of the third color image based on the fourth color image. The imaging device further includes:
7. The image processing apparatus according to claim 4, further comprising means for performing image processing including demosaicing processing on the third color image to generate an image processed image and outputting the image processed image. The imaging apparatus of Claim 1. The first color filter group having the same peak wavelength in the spectral transmittance and the distribution of the spectral transmittance being different from each other and the second color filter group having the different peak wavelength in the spectral transmittance are the Bayer. An imaging apparatus having an imaging sensor arranged two-dimensionally according to the arrangement,
It is the output of the image sensor on the image sensor corresponding to the color filter having the peak wavelength corresponding to green as the peak wavelength in the spectral transmittance of the first color filter group and having the first spectral transmittance. The first color image has a peak wavelength corresponding to green as a peak wavelength in the spectral transmittance of the first color filter group, and has a second spectral transmittance lower than the first spectral transmittance. Obtaining a second color image that is an output of the image sensor on the image sensor corresponding to a color filter;
Determining the luminance component of the third color image by selectively using the luminance value of the first color image and the luminance value of the second color image for each pixel;
The color difference component of the third color image is determined based on a luminance value of a color image that is an output of an image sensor on the image sensor corresponding to the second color filter group. Control method.