WO2016167140A1

WO2016167140A1 - Image-capturing device, image-capturing method, and program

Info

Publication number: WO2016167140A1
Application number: PCT/JP2016/060897
Authority: WO
Inventors: 亮介古川
Original assignee: ソニーセミコンダクタソリューションズ株式会社
Priority date: 2015-04-13
Filing date: 2016-04-01
Publication date: 2016-10-20
Also published as: JP2016201733A

Abstract

The present technology pertains to an image-capturing method, a program, an image-capturing device configured to make it possible to improve image quality. The present invention is provided with a processor such that, when a 2×2 array of pixels having the same spectral sensitivity is defined as one block, two pixels within the block are long-exposure pixels, two pixels are short-exposure pixels, pixels having the same exposure time are disposed diagonally, and signals from pixels disposed on the image-capturing surface are processed in block units. The processor: adds signals from the long-exposure pixels in the block to generate a long-exposure image; adds signals from the short-exposure pixels to generate a short-exposure image; synthesizes the resulting long-exposure image and short-exposure image at a prescribed synthesis ratio; detects a moving body based on the difference between the long-exposure image and the short-exposure image; and detects a folding component based on the long-exposure image and the short-exposure image. The synthesis ratio is set based on the result of detecting the moving body and the result of detecting the folding component.

Description

Imaging apparatus, imaging method, and program

The present technology relates to an imaging device, an imaging method, and a program. Specifically, the present invention relates to an imaging apparatus, an imaging method, and a program that can perform imaging with an expanded dynamic range.

In recent years, CCD (Charge Coupled Device) image sensors and amplification-type image sensors, known as solid-state imaging devices suitable for applications such as video cameras and digital still cameras, are due to an increase in the number of pixels with high sensitivity and a reduction in image size. The pixel size is becoming finer. On the other hand, in general, solid-state imaging devices such as CCD image sensors and CMOS (Complementary Metal Oxide Semiconductor) image sensors tend to be used in various environments such as indoors, outdoors, daytime, and nighttime. Accordingly, it is often necessary to perform an electronic shutter operation or the like that adjusts the exposure time by controlling the charge accumulation period in the photoelectric conversion element and optimizes the sensitivity.

By the way, in the CMOS image sensor, as a method of expanding the dynamic range, a method of adjusting the exposure time by turning the electronic shutter at a high speed, a method of photographing a plurality of frames at a high speed, and a photoelectric conversion of the light receiving unit A method of making the characteristic logarithmic response is known.

When processing one piece of image data having different exposure times or analog gains or different sensitivities, pixels having different exposure times or different sensitivities in a local area of a predetermined size are replaced with pixels having the other exposure time or sensitivity. It has been proposed to increase the dynamic range by combining processing (see, for example, Patent Document 1). In order to expand the dynamic range, the gain multiplied by the set exposure time ratio or the sensitivity ratio calculated in advance is applied to the pixel signal with a low signal amount, and this value and the pixel signal with a high signal amount are fixed. It has been proposed to synthesize in proportions.

Usually, an HDR image can be obtained by synthesizing a plurality of differently exposed images obtained through multiple exposures and shutters by the above method. However, in this method, there is a possibility that the image is collapsed in the area of the moving object.

Using the method proposed in Patent Document 1, the pixels having different exposure times and different sensitivities are arranged two-dimensionally and periodically, and the readout timing is aligned between the differently exposed pixels. Patent Document 2 proposes to suppress blurring of a moving object by selecting an optimum combination ratio along the region at each pixel position.

JP 2013-66145 A JP 2013-66142 A

With the above-described technology, an image with an extended dynamic range can be obtained, which can contribute to higher image quality.

However, there is a possibility that false colors due to aliasing components may occur when an image including a high-frequency signal is taken, and it is desired to suppress the occurrence of such false colors.

The present technology has been made in view of such a situation, and suppresses the generation of false colors and enables photographing with a wide dynamic range.

In the imaging device according to one aspect of the present technology, when 2 × 2 pixels having the same spectral sensitivity are defined as one block, two pixels out of 2 × 2 pixels in the one block are exposed for a long time. 2 pixels are short-time exposure pixels, and pixels having the same exposure time are arranged in an oblique direction, and include a processing unit that processes signals from pixels arranged on the imaging surface in units of the blocks, The processing unit generates a long-time exposure image by adding signals from the long-time exposure pixels in the one block, and adds a signal from the short-time exposure pixels to generate a short-time exposure image. A generating unit, a combining unit that combines the long-time exposure image and the short-time exposure image generated by the generation unit at a predetermined combining ratio, and a difference between the long-time exposure image and the short-time exposure image From the moving object detection unit for detecting the moving object, A folding component detection unit that detects a folding component from the long-time exposure image and the short-time exposure image, and the composition ratio is the detection result of the moving object in the moving object detection unit, and the folding It is set from the detection result of the folded component in the component detection unit.

The aliasing component detection unit determines whether or not a difference between the long-time exposure image and the short-time exposure image and a saturation of each of the long-time exposure image and the short-time exposure image satisfy a predetermined condition. By doing so, the folded component can be detected.

The folding component detection unit can detect the folding component by determining whether or not the following first to fourth conditions are satisfied.
First condition: There is a difference between the long-time exposure image and the short-time exposure image Second condition: There is a difference in saturation between the long-time exposure image and the short-time exposure image Third condition: Saturation A large signal has a green or magenta color. Fourth condition: When a generated signal is subtracted from a signal having no aliasing component, the difference is determined as G pixel and R pixel or G pixel and B. It occurs in the pixel with the same amplitude in the reverse direction.

The short-time exposure image can be an exposure-corrected image.

The long exposure image and the short exposure image can be transferred to a predetermined color space to obtain the saturation.

The composite ratio is the long-time exposure image or the short-time exposure in which it is determined that no aliasing component has occurred in a pixel where the aliasing component is detected by the aliasing component detection unit. The ratio of using a large amount of images can be set.

The composition ratio is a pixel in which the moving object is detected by the moving object detection unit, but it is determined that no folding component is generated in the pixel in which the folding component is detected by the folding component detection unit. The long exposure image or the short exposure image that is frequently used can be used.

In the imaging method according to one aspect of the present technology, when 2 × 2 pixels having the same spectral sensitivity are defined as one block, two pixels out of 2 × 2 pixels in the one block are exposed for a long time. An image is provided with a processing unit that processes signals from pixels arranged on the imaging surface in units of blocks, the pixels having the same exposure time are arranged in an oblique direction. In the imaging method of the apparatus, the processing unit generates a long-time exposure image by adding signals from the long-time exposure pixels in the one block, and adds the signals from the short-time exposure pixels. A short-exposure image, and the generated long-exposure image and the short-exposure image are synthesized at a predetermined composition ratio, and from the difference between the long-exposure image and the short-exposure image, the moving object Detecting the long exposure image and A step of detecting a folding component from the short-time exposure image, wherein the composition ratio is a detection result of the moving body in the moving body detection unit and a detection result of the folding component in the folding component detection unit Set from

According to a program of one aspect of the present technology, when 2 × 2 pixels having the same spectral sensitivity are defined as one block, 2 pixels out of 2 × 2 pixels in the one block are long-time exposure pixels. And 2 pixels are short-time exposure pixels, pixels having the same exposure time are arranged in an oblique direction, and an image pickup apparatus including a processing unit that processes signals from the pixels arranged on the image pickup surface in units of blocks In addition, the processing unit generates a long-time exposure image by adding signals from the long-time exposure pixels in the one block, and adds short-time exposure by adding signals from the short-time exposure pixels. Generating an image, combining the generated long-time exposure image and the short-time exposure image at a predetermined composition ratio, detecting a moving object from a difference between the long-time exposure image and the short-time exposure image; The long exposure image and the short exposure A process including a step of detecting a folded component from an image is executed, and the composition ratio is calculated based on the detection result of the moving object in the moving object detection unit and the detection result of the folded component in the folded component detection unit. Causes the computer to execute the process set by

In the imaging device, the imaging method, and the program according to one aspect of the present technology, when 2 × 2 pixels having the same spectral sensitivity are defined as one block, 2 × 2 pixels out of 2 × 2 pixels in one block are used. The pixels are long-time exposure pixels, the two pixels are short-time exposure pixels, the pixels with the same exposure time are arranged in an oblique direction, and signals from the pixels arranged on the imaging surface are processed in block units. The The process generates a long exposure image by adding signals from long exposure pixels in one block, and generates a short exposure image by adding signals from short exposure pixels. The long-time exposure image and the short-time exposure image are combined at a predetermined composition ratio, and the moving object is detected from the difference between the long-time exposure image and the short-time exposure image. The synthesis ratio is set based on the detection result of the moving body in the moving body detection unit and the detection result of the folding component in the folding component detection unit.

According to one aspect of the present technology, it is possible to suppress the occurrence of false colors. In addition, shooting with a wide dynamic range can be performed.

It should be noted that the effects described here are not necessarily limited, and may be any of the effects described in the present disclosure.

It is a figure showing the composition of the 1 embodiment of the imaging device to which this art is applied. It is a figure for demonstrating pixel arrangement | positioning. It is a figure for demonstrating control of exposure time. It is a figure for demonstrating the production | generation of a HDR image. It is a figure for demonstrating suppression of blur. It is an image which shows an example of a folding signal. It is a figure for demonstrating 4th conditions. It is a figure for demonstrating the structure of a HDR image generation part. It is a figure for demonstrating the structure of a folding | turning component detection part. It is a figure for demonstrating the usage example of an imaging device. It is a figure for demonstrating a recording medium.

Hereinafter, modes for carrying out the present technology (hereinafter referred to as embodiments) will be described. The description will be given in the following order.
1. Configuration of imaging apparatus 2. Pixel arrangement 3. Processing of image processing unit 4. Generation of blur 5. Generation of false color due to aliasing components 6. Configuration of HDR image generation unit Configuration of aliasing component detection unit 8. 8. Usage example of imaging device About recording media

<Configuration of imaging device>
FIG. 1 is a diagram illustrating a configuration of an embodiment of an imaging apparatus to which the present technology is applied. In the imaging apparatus 100 illustrated in FIG. 1, light incident through the optical lens 101 enters an imaging element 102 configured by an imaging unit, for example, a CMOS image sensor, and outputs image data by photoelectric conversion. The output image data is input to the image processing unit 103.

The output image of the image sensor 102 is a so-called mosaic image in which any pixel value of RGB is set for each pixel. The image processing unit 103 generates a high dynamic range (HDR) image based on demosaic processing that sets all RGB pixel values for each pixel and synthesis processing of a long-time exposure image and a short-time exposure image, which will be described later. Processing, blur correction processing, etc.

The output of the image processing unit 103 is input to the signal processing unit 104. The signal processing unit 104 performs signal processing in a general camera, such as white balance (WB) adjustment and gamma correction, and generates an output image 120. The output image 120 is stored in a storage unit (not shown). Or it outputs to a display part (not shown).

The control unit 105 outputs a control signal to each unit according to a program stored in a memory (not shown), for example, and controls various processes.

<Pixel arrangement>
Hereinafter, the arrangement of pixels in an image sensor to which the present technology is applied will be described with reference to FIG. In FIG. 2, each rectangle schematically represents a pixel. Each rectangle has a symbol indicating the type of color filter (color light output from each pixel). For example, “R” is assigned to R (Red) pixels, “G” is assigned to G (Green) pixels, and “B” is assigned to B (Blue) pixels. The same applies to the following description.

In the pixel arrangement shown in FIG. 2, the arrangement of the R pixel, the G pixel, and the B pixel is repeated in units of 4 × 4 in the vertical and horizontal directions. In the example shown in FIG. 4, the 2 × 2 four pixels at the upper left are all R pixels. These 2 × 2 R pixels are defined as R blocks. All 4 pixels of 2 × 2 adjacent to the right side of the 4-pixel R block are G pixels (referred to as G blocks).

The lower 2 × 2 4 pixels of the R block are all G pixels (G block). All 2 × 2 4 pixels on the lower right side of the R block are B pixels (referred to as B blocks). In this way, all the 2 × 2 4 pixels have the same color, and the R block, G block, G block, and B block in units of 4 pixels are arranged in the 4 × 4 pixel region.

As described above, in the pixel array shown in FIG. 2, the R pixel, the G pixel, the G pixel, and the B pixel are configured in 4 × 4 units each including four pixels. Hereinafter, such an arrangement of pixels is appropriately described as a four-divided Bayer RGB arrangement.

Here, the description will be continued with an example in which RGB pixels are arranged, but a configuration including W (White) pixels is also possible. Further, the present technology can be applied to a combination of cyan, magenta, and yellow instead of RGB.

When the W pixel is included, the W pixel functions as a spectral sensitivity pixel having total color matching, and the R pixel, the G pixel, and the B pixel are spectral sensitivity pixels having characteristics in their respective colors. Function. The present technology can also be applied to an image sensor (image sensor) in which pixels of four types of spectral sensitivities including spectral sensitivities having total color matching are arranged on an imaging surface.

In the pixel arrangement to which the present technology is applied, four blocks are included in one unit composed of 4 × 4, and two of them are G blocks. One of the two G blocks may be a W block in which W pixels are arranged.

In the image sensor arranged based on the pixel arrangement shown in FIG. 2, the four pixels included in one block have the same color, but two types of exposure times are set. Four pixels included in one block are set as long exposure pixels L or short exposure pixels S, respectively. The relationship of exposure time is as shown below.
Long exposure L> Short exposure S

Referring to FIG. 2 again, the pixel arrangement focusing on the exposure time will be described. Note the four pixels in the R block located in the upper left. The R pixels located at the upper left and lower right in the R block are long-time exposure pixels L. Hereinafter, the R pixel set as the long-time exposure pixel L is described as an RL pixel. Similarly, for the G pixel and the B pixel, the pixels set as the long-time exposure pixel L are described as a GL pixel and a BL pixel, respectively.

The R pixels located at the upper right and lower left in the R block are the short-time exposure pixels S. Hereinafter, the R pixel set as the short-time exposure pixel S is described as an RS pixel. Similarly, for the G pixel and the B pixel, the pixels set as the short-time exposure pixel S are described as a GS pixel and a BS pixel, respectively.

This arrangement is the same for other color blocks. For example, G pixels located at the upper left and lower right in the G block are GL pixels set as long exposure pixels L, and G pixels located at the upper right and lower left are set as short exposure pixels S. GS pixel.

Similarly, B pixels located at the upper left and lower right in the B block are BL pixels set as long exposure pixels L, and B pixels located at the upper right and lower left are set as short exposure pixels S. This is the BS pixel.

As described above, the arrangement of the pixels to which the present technology is applied is that the same color pixels are arranged as four blocks of 2 × 2 pixels, and the four pixels of the same color in one block are pixels that are photographed with long exposure. It is set for each pixel for which shooting is performed with short-time exposure.

Here, the description will be continued assuming 2 × 2 as one block, but it is not a description indicating that the number of pixels in one block is limited to four. is there. For example, 3 × 3 may be one block, and nine pixels may be included in one block.

In addition, here, one block is described as an example in which the vertical and horizontal are M × M and the number of pixels in the vertical and horizontal directions is the same, but the vertical and horizontal are M × N, The present technology can be applied even when the number of pixels in the vertical direction and the horizontal direction is different. The long exposure pixel L and the short exposure pixel S are arranged according to the number of pixels in one block.

Further, the arrangement of pixels with different exposure times shown in FIG. 2 is an example, and other arrangements may be used. For example, in FIG. 2, the long-time exposure pixels L are arranged at the upper left and lower right, but other arrangements such as the long-time exposure pixels L at the upper right and lower left may be used.

In addition, although the arrangement of the pixels having different exposure times in the R block, the G block, and the B block has been described as being the same, the arrangement may be different for each color. For example, different exposures in the R block, the G block, and the B block are arranged such that the long exposure pixel L in the R block and the long exposure pixel L in the G block located on the right are adjacent to each other. The arrangement of the temporal pixels may be the same or different.

As described above, in the imaging apparatus 100 to which the present technology is applied, the long exposure pixel and the short exposure pixel are set for each pixel included in one photographed image, and the synthesis process (blend) between these pixels is performed. Thus, an HDR image is generated. This exposure time control is performed under the control of the control unit 105.

Fig. 3 shows an example of the timing of exposure time for each pixel. The long exposure pixel L is subjected to a long exposure process. The short exposure pixel S is subjected to a short exposure process. As shown in FIG. 3, the exposure start timings of the short-time exposure pixels S and the long-time exposure pixels L do not match, but the exposure times are controlled so that the exposure end timings match.

<About the processing of the image processing unit>
A process performed by the image processing unit 103 that processes a signal from the image sensor 102 in which the short-time exposure pixels S and the long-time exposure pixels L are arranged as described above will be described. Here, an outline of processing performed by the image processing unit 103 will be described, and details will be described later. As an explanation to be described later, there is a process for suppressing the occurrence of false color due to the aliasing component.

FIG. 4 shows a processing example when the exposure time is changed in an oblique direction in a four-divided Bayer RGB array. FIG. 4 shows the following three data.
(4a) Imaging data (4b) Intermediate data (4c) Output data

(4a) The imaging data is imaging data of the imaging device, and shows an image taken when the exposure time is changed for each column in the Bayer array. In FIG. 4, the white portion indicates the long-time exposure pixel L, and the dark gray portion indicates the short-time exposure pixel S. For example, RL00 is the long-time exposure pixel L of the R pixel at the coordinate position (0, 0). GL20 is the long-time exposure pixel L of the G pixel at the coordinate position (2, 0). The coordinates are shown in a format such as GSxy, GLxy, etc. by applying coordinates (x, y) where x is the vertical downward direction and y is the horizontal right direction.

As described with reference to FIG. 2, in the present embodiment, the long exposure pixels L and the short exposure pixels S are alternately set in an oblique direction.

(4a) The imaging data indicates a 4 × 6 pixel area.
(4b) Intermediate data indicates intermediate data generated based on 4 × 6 (4a) imaging data.

In such a case, in step S1 (STEP 1), 12 pieces of intermediate pixel data are calculated based on 4 × 6 (4a) imaging data.

(4c) Output data indicates output data generated based on 12 (4b) intermediate pixel data. This output data is output data generated as a wide dynamic range image.

(Step 1)
The process of generating (4b) intermediate data from (4a) imaging data in step S1 is performed by the following diagonal addition process of a plurality of pixel values. For example, the pixel value of RLA00 (RLA00) and the pixel value of RSA00 (RSA00) calculated from the R block shown in FIG. 4 (4b) are the values obtained by applying pixel values of a plurality of pixels included in (4a) imaging data. It is calculated by the diagonal addition process according to the equation.
RLA00 = (RL00 + RL11) / 2
RSA00 = (RS01 + RS10) / 2 (1)

Thus, the average value of the RL pixels arranged in the diagonal direction in the R block is calculated, and the average value of the RS pixels is calculated, so that intermediate data is generated. Instead of obtaining an average value, a value obtained by adding pixel values may be used as it is.

Similarly, for the G block and B block, intermediate data is generated by calculating the average value of the pixel values of the long-time exposure pixels L and the average value of the pixel values of the short-time exposure pixels S arranged in an oblique direction. Is done.

An expression expressing Expression (1) as an expression common to the R pixel, the G pixel, and the B pixel is defined as the following Expression (2).
DLA = (DL + Dl) / 2
DSA = (DS + Ds) / 2 (2)

In Expression (2), DLA represents the average value of the pixel values of the long-time exposure pixels L in one block, and DSA represents the average value of the pixel values of the short-time exposure pixels S in one block. DL represents one pixel value of the two long-time exposure pixels L in one block, and Dl represents the other pixel value. Similarly, DS represents one pixel value of the two short-time exposure pixels S in one block, and Ds represents the other pixel value.

In this way, by adding together pixels having the same exposure time arranged in an oblique direction within one block, it is possible to suppress color mixture components.

(Step 2)
(4b) Output data generation processing from (4b) intermediate data in step S2 is performed by (4b) blend processing of pixel values included in the intermediate data as follows. For example, the pixel value (R00) of the R00 pixel shown in FIG. 4 (4c) is calculated according to the following calculation formula (1) to which the pixel value of a plurality of pixels included in the intermediate data (4b) and the blend coefficient α are applied. Is done.
R00 = (1-α) × RSA00 × Gain + α × RLA00 (3)

However,
Gain: Gain multiplied by the pixel value of the short-time exposure pixel (exposure ratio between the long-time exposure pixel and the short-time exposure pixel)
α: The blend coefficient of the pixel value of the long exposure pixel and the pixel value of the short exposure pixel.

Similarly, output data is generated in the G pixel and B pixel in accordance with a calculation formula using a gain and a blend coefficient. Note that the R pixel, the G pixel, and the B pixel have different sensitivities, and therefore, for example, different values may be used for the R pixel, the G pixel, and the B pixel for the gain and the blend coefficient.

The following expression (4) is obtained by expressing the expression (3) as an expression common to the R pixel, the G pixel, and the B pixel.
DH = (1−α) × DS + α × DL (4)

In Expression (4), DH represents a pixel value of a predetermined pixel in the HDR image. DS corresponds to RSA00 × GAIN in equation (3). DL corresponds to DLA in equation (2). It is also possible to multiply DLA by a predetermined gain so that DL = DLA × Gain.

In this way, an HDR image is generated.

<About the occurrence of blur>
In the wide dynamic range (HDR) image generation process as described above, blurring may occur when a moving subject is photographed.

Generally, when a change occurs in the relative positional relationship between the subject and the imaging device during the exposure time of the image sensor, the subject is imaged over a plurality of pixels, so that the fineness of the obtained image is lost. In particular, long exposure images tend to lose fineness because exposure time is longer than short exposure images. For this reason, when blurring occurs on a moving subject or an imaging device, the corresponding pixel positions of the long-exposure image and the short-exposure image are corrected even if the exposure of the short-exposure image and the long-exposure image is corrected to make the brightness uniform. Deviation occurs in the pixel values.

In this way, the pixel value of the corresponding pixel position between the long-exposure image and the short-exposure image that have been corrected for exposure by changing the relative positional relationship between the subject and the imaging device during the exposure time. The phenomenon of deviation is defined as “blur”.

In order to suppress the occurrence of such blur, the following processing is performed. With reference to FIG. 5, an example of determination processing of the blend coefficient α between the long exposure image and the short exposure image will be described. FIG. 5A shows an example of the blend processing configuration.

In this embodiment, blur information is measured based on both the short-time exposure image and the long-time exposure image, and the blend coefficient is calculated based on the measured blur information and both the short-time exposure image and the long-time exposure image. An example will be described in which blending of a short-exposure image and a long-exposure image is executed by applying the determined blend coefficient.

“Blur” can be defined as a shift in the pixel value of the pixel at the corresponding pixel position between the long-exposure image and the short-exposure image that have been corrected based on the exposure ratio, and “blur information” It can be set as an index value indicating the degree of occurrence of blur corresponding to the shift amount of the pixel value.

“Blur information” for each pixel is acquired from the captured image, and the blending coefficient determined based on the acquired “blur information” is applied to execute the blending process between the short exposure image and the long exposure image. , An HDR image is generated.

In the configuration of the present disclosure, the blend coefficient α in consideration of blur is calculated according to the following equation (5), for example.

In equation (5),
M: Blur information (= blur degree index value indicating the size of blur)
max (a, b): a function for obtaining the maximum value of a and b. min (a, b): a function for obtaining the minimum value of a and b. K ₁ and k ₀ are parameters. Details will be described later.

Note that the value M indicating the magnitude of the blur of the long exposure image is ideally calculated by the following equation.
M = (μL−μS) ² (6)
Note that μL and μS have the following values.
μL: Ideal pixel value of exposure-corrected long-exposure image obtained when there is no influence of noise (corresponding to DL in equation (4))
μS: Exposure correction obtained when there is no influence of noise Ideal pixel value of short-exposure image (corresponding to DS in equation (4))

However, in the above equation (6), in reality, μL and μS cannot be directly obtained because of the influence of noise. Therefore, M is approximately obtained using, for example, the following equation (7) using a pixel group around the target pixel.

In equation (7),
M (x, y): Blur information at pixel position (x, y) DL (x, y): Pixel value at pixel position (x, y) of exposure-corrected long exposure image DS (x, y): Pixel value VL (x, y) at pixel position (x, y) of exposure-corrected short-exposure image VS: variance value VS (x, y) of noise at pixel position (x, y) of exposure-corrected long-exposure image ): Exposure correction Noise variance value at pixel position (x, y) of short-exposure image φ (dx, dy): Weight coefficient of low-pass filter min (a, b): Minimum value of value a and value b Function to calculate max (a, b): Function to calculate the maximum value of value a and value b p: Parameter for adjustment. A constant greater than or equal to zero.

Also, as a result of the calculation of equation (7), when M (x, y) becomes a negative value, the difference absolute value or the long / short difference is set to zero. Alternatively, the calculation of the value M indicating the blur size of the long exposure image may be simplified and calculated according to the following equation (8).

The blending process is performed by determining such a blending coefficient α. As a result of this processing, α is set to preferentially output a pixel value based on a short-exposure image with little blur because α approaches 0 at a portion where blur is large, and α is a value equivalent to the conventional method at a portion where blur is small. And a pixel value corresponding to a predetermined blend coefficient is generated.

Such processing is realized, and as a result, an HDR image with a good S / N ratio is obtained from a dark part to a bright part in the moving part with less blur in the moving subject part. Note that the amount of calculation of the blend coefficient is not so large and can be processed at high speed. For example, it can be applied to HDR image generation processing of moving images.

<Generation of false color due to aliasing components>
By the processing as described above, an HDR image in which the occurrence of blur is suppressed can be generated. However, when an image including a high-frequency component is captured, a false color due to the aliasing component may occur. The false color due to the aliasing component will be described.

Here, reference is again made to equation (6). Expression (6) is an expression for calculating a value M indicating the magnitude of blur of a long exposure image.
M = (μL−μS) ² (6)
μL: Pixel value of exposure-corrected long-exposure image μS: Pixel value of exposure-corrected short-exposure image

For example, when the exposure ratio is 16, the exposure correction long-time exposure image μL is calculated by multiplying the pixel value of the long-time exposure pixel L by 1, and the exposure correction short-time exposure image μS is calculated as the short-time exposure pixel. It is calculated by multiplying the pixel value of S by 16. The exposure correction long-time exposure image μL and the exposure correction short-time exposure image μS calculated in this way are images in which the brightness is matched.

The exposure-corrected long-time exposure image μL and the exposure-corrected short-time exposure image μS, which are adjusted in brightness in this way, have substantially the same signal value as long as there is no influence of noise. By using this characteristic, blur detection is performed as described above. That is, in Expression (6), if the exposure correction long-time exposure image μL and the exposure correction short-time exposure image μS are the same, the value is 0. However, by capturing a moving subject or the like, the exposure correction long-time exposure image μL is obtained. When a difference occurs between the exposure correction short-exposure image μS, some value is calculated.

Incidentally, in the case of the color arrangement of the color filter shown in FIG. 2, there is a possibility that a difference occurs between the exposure-corrected long-time exposure image μL and the exposure-corrected short-time exposure image μS due to the influence from adjacent pixels. Reference is now made to FIG. 4 again. In the (4a) imaging data shown in FIG. 4, attention is paid to the BL22 pixel. The BL22 pixel is adjacent to the GS21 pixel on the left, the RL11 pixel on the upper left, and the GS12 pixel on the upper left. The BL22 pixel is affected by these GS21 pixel, RL11 pixel, and GS12 pixel. That is, the BL22 pixel is easily affected by adjacent different color pixels.

Next, pay attention to the BL33 pixel. The BL33 pixel is adjacent to the BS32 pixel on the left, the BL22 pixel on the upper left, and the BS23 pixel on the upper left. The BL33 pixel is affected by these BS32 pixel, BL22 pixel, and BS23 pixel. However, the adjacent pixels affected by the BL33 pixel have the same color, and thus the influence is small.

As described above, the BL22 pixel and the BL33 pixel are the long-time exposure pixels L in the same B block, but there is a difference in whether adjacent pixels are the same color or different colors. As described above, there is a possibility that a difference occurs in the signal value. This is considered to be the influence of the color mixture due to the oblique light component.

Such an influence of the color mixture due to the oblique light component occurs not only in the long-time exposure pixels L in the B block exemplified in the above description. When the influence of the color mixture due to the oblique light component appears in the short-time exposure pixel S, as described above, when the exposure correction is converted into the short-time exposure image μS, an exposure ratio, for example, a value of 16 times is obtained. Since the multiplication is performed, the influence becomes large.

When there is an influence of such an oblique light component, the relationship that “the exposure corrected long exposure image and the exposure corrected short exposure image having the same brightness have substantially the same signal value” does not hold.

However, as described with reference to Expression (1) and Expression (2), it is possible to suppress mixed color components by adding pixels having the same exposure time arranged in an oblique direction within one block. it can. This is considered to be because the light component in the oblique direction, which is the main component of the color mixture, can be canceled by adding pixels of the same exposure time arranged in the oblique direction.

For example, as shown in FIG. 2 and FIG. 4, long exposure pixels L and short exposure pixels S are set in one block, the long exposure pixels L are arranged in an oblique direction, and the short exposure pixels S It was set as the structure arrange | positioned in the diagonal direction. In this way, by arranging pixels of the same exposure in pixels of the same color diagonal direction and adding them, a signal in which the color mixture component is suppressed can be obtained.

Thus, the influence of the color mixture can be suppressed by performing the calculation shown in the equation (2). Also, the HDR image is generated by performing the calculation shown in Expression (4) using the signal in which the influence of the color mixture is suppressed. The blend coefficient α in the equation (4) is set as a coefficient for suppressing the occurrence of blur as described above.

Therefore, it is possible to acquire an HDR image in which the influence of color mixture and blur is suppressed. However, such a process is performed when the difference between the pixel value of the long-time exposure pixel L and the pixel value of the short-time exposure pixel S whose exposure ratio has been corrected occurs only due to the influence of noise components or moving subjects. Although effective, there is a possibility that the influence of the aliasing component described below cannot be suppressed.

As described above, when pixels having the same exposure time arranged in an oblique direction are added to reduce the influence of the oblique light component, the signal from the long exposure pixel L and the short exposure pixel S are added. The difference in the frequency characteristics in the oblique direction is caused by the signal from the signal, and there is a difference in the pixel value of the long-time exposure pixel L and the pixel value of the short-time exposure pixel S whose exposure ratio is corrected even in the region including the oblique high frequency. End up.

An example will be described in which an image including a high-frequency signal is taken with the imaging apparatus 100 (FIG. 1) having the pixel arrangement shown in FIG. 2 (FIG. 4). Although not shown, for example, when a still image in which concentric circles are drawn in a narrow state is captured and the difference between the long-time exposure image and the short-time exposure image is acquired as an image, it is shown in FIG. Such an image may be acquired.

When a still image is captured and the difference between the long exposure image and the short exposure image is represented as an image, the difference between the long exposure image and the short exposure image is 0, so the image represents 0. An image of a color, for example, a black color in FIG. However, the image shown in FIG. 6 has a white portion (hereinafter referred to as a folding signal), which indicates that there is a portion where a difference occurs between the long-time exposure image and the short-time exposure image. .

The difference between the long-time exposure image and the short-time exposure image can be calculated by performing the calculation according to the above equation (6). As described above, it is possible to determine whether or not there is an influence (blurring) due to the moving subject based on the calculation result according to Expression (6). That is, if there is a difference between the long exposure image and the short exposure image, it can be determined that the pixel (region) is affected by the moving subject, and if there is no difference, the pixel (region) is not affected by the moving subject. It can be determined that there is.

However, as shown in FIG. 6, when a still image is captured, in other words, even when an image without a moving subject is captured, the difference between the long exposure image and the short exposure image is obtained. May end up. In FIG. 6, there is a aliasing signal because of the addition in the oblique direction for reducing the mixed color component, a difference occurs in the frequency characteristics in the oblique direction between the signal of the long exposure pixel L and the signal of the short exposure pixel S, This is presumably because a difference occurs between the signal of the long-time exposure pixel L and the signal of the short-time exposure pixel S whose exposure ratio has been corrected even in the region including the oblique high frequency.

As described above, when a still image is captured, if there is a region where a difference between the long-exposure image and the short-exposure image occurs, the region is influenced by the moving subject based on the above processing. Is treated as an area where blur is likely to occur.

Although the processing for suppressing the occurrence of blur has been described above, in brief, in a region where blurring may occur, the possibility of blurring increases when the signal of the long-time exposure pixel L is used. Therefore, the signal of the short-time exposure pixel S is used.

However, the signal of the short-time exposure pixel S is likely to contain a noise component, which is higher than the signal of the long-time exposure pixel L. By using the signal of the short-time exposure pixel S, the SN ratio is lowered. There is a possibility.

For this reason, when a still image is taken, a difference between the long-time exposure image and the short-time exposure image occurs, and if the processing for blur suppression is executed, the SN ratio may be reduced. There is sex.

For this reason, the processing for suppressing blur and the processing for suppressing the aliasing signal due to the high frequency signal are not the same processing but different processing. By using different processing, it is possible to appropriately suppress blurring and suppress a folding signal due to a high-frequency signal.

In order to perform the suppression of blurring and the influence of high-frequency components in the diagonal direction in different processes, identify whether the influence is due to moving subjects or the influence of high-frequency components in the diagonal direction, It is necessary to divide the processing.

Here, the high-frequency information in the oblique direction is detected, and the portion where the detection signal is high is that even if there is a difference between the exposure-corrected long-time exposure image μL and the exposure-corrected short-time exposure image μS whose exposure ratio is corrected, It is determined that it is due to high frequency, not caused by the influence of moving objects, and by reducing the strength of the long / short difference used to detect the moving object region, it suppresses erroneous determination of moving object region detection. .

In addition, in the portion where the high frequency information is detected, the signal having the reduced aliasing signal of the exposure correction long exposure image μL and the exposure correction short exposure image μS is combined so as to be selectively strongly combined. By controlling the ratio, the folding signal is reduced.

The Applicant has obtained an analysis result that the following four conditions are satisfied for a pixel (region) that is affected by a high-frequency component in an oblique direction. Hereinafter, “the influence of the oblique high-frequency component” is described as “the influence of the aliasing component”.

First condition: There is a difference between the exposure correction long-time exposure image μL and the exposure correction short-time exposure image μS.
Second condition: There is a large difference in saturation between the exposure-corrected long-time exposure image μL and the exposure-corrected short-time exposure image μS.
Third condition: a large saturated signal has a green or magenta color.
Fourth condition: When the generated signal is subtracted from the signal in which no aliasing occurs, the difference is generated with the same amplitude in the opposite direction between G and R or G and B.

The first condition is that there is a difference between the exposure-corrected long-time exposure image μL and the exposure-corrected short-time exposure image μS even when a still image is captured as described above. When the difference is imaged, for example, it means that an image as shown in FIG. 6 is obtained. First, in an image that is affected by the aliasing component, there is a condition that there is a difference between the exposure-corrected long-time exposure image μL and the exposure-corrected short-time exposure image μS.

The second condition is that the exposure correction long exposure image μL and the exposure correction short exposure image μS are acquired separately, and when the same area is compared, the area affected by the aliasing component is the exposure correction long exposure image. This means that there is a large difference in saturation between μL and the exposure-corrected short exposure image μS.

In the region where there is no influence by the aliasing component, when the same region of the exposure-corrected long-time exposure image μL and the exposure-corrected short-time exposure image μS are compared, the hue is the same and the saturation is the same. On the other hand, in the area affected by the aliasing component, when comparing the same area of the exposure-corrected long-time exposure image μL and the exposure-correction short-time exposure image μS, the hue is greatly different, and there is a large difference in saturation. Arise.

In the area affected by the aliasing component, when the exposure correction long exposure image μL and the exposure correction short exposure image μS are acquired separately and the same area is compared, the exposure correction long exposure image μL or exposure correction is obtained. A folding signal is generated in one of the short-time exposure images μS. Therefore, when the exposure correction long exposure image μL and the exposure correction short exposure image μS are compared, if a folding signal is generated in one of the images, the exposure correction long exposure is performed in an area where the folding signal is present. A difference is generated between the image μL and the exposure correction short-time exposure image μS.

Next, the third condition will be described. The third condition is that a large saturation signal has a green or magenta color. When there is an influence due to the aliasing component, when the image is acquired without suppressing the influence, the influence is an image having a color of green or magenta.

Even if there is an area (pixel) that satisfies the first condition and the second condition, if the color of the area is not green or magenta, the area is not affected by the aliasing component. It can be determined that there is no.

The fourth condition will be described with reference to FIG. In FIG. 7, capital letters R, G, and B represent the R pixel, G pixel, and B pixel of the long-time exposure pixel L, respectively. In FIG. 7, lowercase r, g, and b represent the R pixel, G pixel, and B pixel of the short-time exposure pixel S, respectively. FIG. 7 shows pixels (signals) in a region affected by the folding signal.

7A shows the pixel arrangement, which is basically the same as the pixel arrangement shown in FIG. 2, but is shown in an obliquely inclined state in order to illustrate pixels to be added diagonally in the horizontal direction. It is shown. B of FIG. 7 represents the result of the diagonal addition of the same color. C of FIG. 7 represents the addition result (long livestock addition result) of the long exposure pixel L. D of FIG. 7 represents the addition result (short livestock addition result) of the short-time exposure pixels S. E of FIG. 7 represents a result when the difference between the long stock addition result and the short stock addition result is calculated. F of FIG. 7 represents the result of correcting the difference result shown in E of FIG. 7 in consideration of the sensitivity ratio.

As shown in FIG. 7B, the pixel values of the long-time exposure pixels L in one block are calculated by adding the pixel values of the long-time exposure pixels L arranged in an oblique direction in one block. The pixel values of the short-time exposure pixels S are calculated by adding the pixel values of the short-time exposure pixels S to each other.

7C and FIG. 7D illustrate the G pixel and the R pixel. C in FIG. 7 represents the G pixel and R pixel of the long-time exposure pixel L diagonally added, and D in FIG. 7 represents the G pixel and R pixel of the short-time exposure pixel S diagonally added.

7E shows the difference (Gg) between the G pixel of the long stock addition result of FIG. 7C and the G pixel of the short stock addition result of FIG. 7D. Further, E in FIG. 7 represents the difference (R−r) between the R pixel of the long stock addition result of C in FIG. 7 and the R pixel of the short stock addition result of D in FIG.

Since the G pixel and the R pixel have different sensitivity ratios, and the G pixel has a higher sensitivity than the R pixel, in order to match the signal level of the G pixel and the signal level of the R pixel, a predetermined gain is applied to the signal of the R pixel. Need to be multiplied. Such processing is performed as white balance adjustment in a general imaging apparatus.

The signal intensity obtained by multiplying the R pixel difference (R−r) shown in E of FIG. 7 by a predetermined gain to match the sensitivity is shown in F of FIG. As shown in FIG. 7F, the adjusted R pixel difference (R−r) and the G pixel difference (G−g) have substantially the same signal intensity and are in the opposite directions.

Here, the R pixel has been described as an example, but the same applies to the B pixel. That is, within the region affected by the aliasing signal, the difference (B−b) between the B pixels after sensitivity adjustment and the difference (G−g) between the G pixels have substantially the same signal intensity and are in the opposite directions.

Since this occurs in the region affected by the aliasing component, the fourth condition is that if the generated signal is subtracted from the signal that does not cause aliasing, the difference However, G and R or G and B are generated with the same amplitude in the opposite directions. " Natural signals tend to satisfy the fourth condition.

It can be determined that a pixel (region) that satisfies all of the first to fourth conditions is a pixel that is affected by the aliasing component.

Thus, by determining whether or not the first to fourth conditions are satisfied, it is possible to determine whether or not the pixel is affected by the aliasing component. By making such a determination and using the determination result, it is possible to suppress the blur due to the influence of the moving subject and to suppress the generation of the false color due to the influence of the aliasing component.

<Configuration of HDR image generation unit>
About the HDR image generation part which produces | generates an HDR image by processing the signal from the image pick-up element 102 (FIG. 1) in which the long exposure pixel L and the short exposure pixel S are arrange | positioned as shown in FIG. explain. FIG. 8 is a diagram illustrating a configuration of the HDR image generation unit 200.

The HDR image generation unit 200 illustrated in FIG. 8 includes an RGB interpolation signal generation unit 211, an exposure correction unit 212, an SN maximization synthesis ratio calculation unit 213, a aliasing reduction synthesis ratio calculation unit 214, a blur reduction synthesis ratio calculation unit 215, The long-lived saturation-considering synthesis ratio calculation unit 216, the long / short synthesis processing unit 217, the aliasing component detection unit 218, the moving object detection unit 219, and the noise reduction processing unit 220 are included.

The HDR image generation unit 200 receives a signal from the long exposure pixel L and a signal from the short exposure pixel S from the image sensor 102. The input signal is a signal after the diagonal addition of the same color and is a signal that has been subjected to the color mixture reduction process. The RGB interpolation signal generation unit 211 interpolates signals of R pixels, G pixels, and B pixels that have been exposed for a long time at all pixel positions, and generates a long exposure image that includes R pixels and a long time that includes G pixels. An exposure image and a long exposure image composed of B pixels are generated.

In addition, the RGB interpolation signal generation unit 211 interpolates signals of R pixels, G pixels, and B pixels that have been exposed to a short time at all pixel positions, and includes a short exposure image that includes R pixels and a G pixel. A short-time exposure image and a short-time exposure image composed of B pixels are generated. Each of these generated images is supplied to the exposure correction unit 212.

The exposure correction unit 212 performs correction to absorb the difference in sensitivity between the R pixel, the G pixel, and the B pixel. As described above, since the G pixel has higher sensitivity than the R pixel and the B pixel, exposure correction is performed by multiplying the R pixel signal and the B pixel signal by respective predetermined gains. The exposure correction unit 212 generates a signal for the long exposure pixel L (hereinafter referred to as a long exposure signal) and a signal for the short exposure pixel S (hereinafter described as a short exposure signal). From the exposure correction unit 212, a long exposure signal for the R pixel, a long exposure signal for the G pixel, a long exposure signal for the B pixel, a short exposure signal for the R pixel, a short exposure signal for the G pixel, and a B pixel The short exposure signal is output.

The long-time exposure signals for R, G, and B from the exposure correction unit 212 and the short-time exposure signals for R, G, and B are respectively converted into an SN maximizing synthesis ratio calculation unit 213, an aliasing component detection unit 218, And supplied to the moving object detection unit 219. Further, the long-time exposure signal from the exposure correction unit 212 is also supplied to the long-lived saturation-considering synthesis ratio calculation unit 216 and the long / short synthesis processing unit 217. The short-time exposure signal from the exposure correction unit 212 is also supplied to the noise reduction processing unit 220.

The SN maximized combination ratio calculation unit 213 calculates a combination ratio that maximizes the SN ratio, and supplies the SN maximized combination ratio to the aliasing reduction combination ratio calculation unit 214.

The aliasing reduction synthesis ratio calculation unit 214 corrects the SN maximization synthesis ratio based on the aliasing component information from the aliasing component detection unit 218. The configuration and processing of the aliasing component detection unit 218 will be described later with reference to FIG. The aliasing component information is information obtained by determining whether or not the first to fourth conditions described above are satisfied, and is information indicating whether or not the pixel is influenced by the aliasing signal. is there.

As described above, if the folding signal is generated on the long exposure signal side, the folding signal is not generated on the short exposure signal side. Further, if the folding signal is generated on the short exposure signal side, the folding signal is not generated on the long exposure signal side.

For this reason, the ratio of using the signal on the side where the folding signal is generated is lowered, and the folding is performed so that the signal on the side where the folding signal is not generated is preferentially used. The reduced synthesis ratio calculation unit 214 calculates a synthesis ratio.

Specifically, for example, the calculation based on the following equation is performed in the aliasing reduction combination ratio calculation unit 214.
A = {SN maximization composite ratio × (1.0−short accumulation folding component)} + (1.0 × short accumulation folding component)
OUT = {A × (1.0-long animal folding component)} + (0.0 × long animal folding component)

In this equation, the short accumulation folding component and the long stock folding component are information supplied from the folding component detection unit 218 as folding component information. In this way, an operation for selectively bringing the SN maximal composite ratio close to 1.0 (uses 100% long exposure signal) or 0.0 (uses 100% short exposure signal) is performed using the aliasing component. .

The aliasing reduction composition ratio calculated in this way is supplied to the blur reduction composition ratio calculation unit 215. The blur reduction composition ratio calculation unit 215 calculates a composition ratio for suppressing blur as described above in <About occurrence of blur>. Specifically, the blend coefficient α is calculated by the calculation as described above.

The blur reduction synthesis ratio calculation unit 215 selectively selects the folding reduction synthesis ratio supplied from the folding reduction synthesis ratio calculation unit 214 using the moving object detection information supplied from the moving object detection unit 219. Calculation is performed so as to approach 100 (use 100% long exposure signal) or 0.0 (use 100% short exposure signal).

The blur reduction composition ratio calculation unit 215 is supplied with moving object detection information indicating whether a moving object is detected from the moving object detection unit 219, that is, whether a pixel is likely to cause blur. The The animal body detection unit 219 is supplied with the folding component information from the folding component detection unit 218.

When the folded component information is information that the folded component is generated, the moving object detection unit 219 assumes that the detected component is not a moving object but a folded component even if it is detected as a moving object. The information that the moving object is not detected is supplied to the blur reduction composition ratio calculation unit 215. Therefore, it is possible for the blur reduction composition ratio calculation unit 215 to perform control so as not to execute the process for reducing the blur for the pixel in which the aliasing component is generated.

By performing such processing, when the pixel is affected by the moving object, processing for suppressing blur is executed, and when the pixel is affected by the aliasing component, It is possible to execute processing for suppressing generation of false colors due to aliasing components.

Note that the folding component information output from the folding component detection unit 218 may be, for example, 0 or 1 information indicating whether or not the folding component is generated, or the folding component is generated. For example, it may be information having a value of 0 to 1 that represents the likelihood of being unclear.

Similarly, the moving object detection information output from the moving object detection unit 219 may be, for example, 0 or 1 information indicating whether or not blur has occurred due to the influence of the moving object. It may be information having a value of 0 to 1, for example, representing the certainty of the possibility of being.

The processing up to this point suppresses color mixing, suppresses false colors that may occur due to the effects of aliasing components, and suppresses blurs that may occur due to effects of moving objects.

The blur reduction composition ratio from the blur reduction composition ratio calculation unit 215 is supplied to the long livestock saturation consideration composition ratio calculation unit 216.

The long-lived saturation-considering composition ratio calculation unit 216 refers to the long-time exposure signal supplied from the exposure correction unit 212 and determines whether or not the long-time exposure pixel L is saturated. The pixel value (signal) of the saturated pixel is not used, and for the saturated pixel, the supplied blur reduction composition ratio is converted to a ratio that uses the pixel value (signal) of the short-time exposure pixel S. To do.

The long-lived saturation-considering composition ratio calculation unit 216 comprehensively synthesizes a ratio (0.0 (uses 100% short-time exposure signal)) that uses a short-time exposure signal instead of a long-time exposure signal for saturated pixels. The ratio is output to the subsequent long / short synthesis processing unit 217 as the ratio, and for the pixels that are not saturated, the input blur reduction synthesis ratio is output to the subsequent long / short synthesis processing unit 217 as the total synthesis ratio.

The total synthesis ratio is a synthesis ratio that maximizes the S / N ratio in the flat part, and is a synthesis ratio that reduces the strength if the area is the aliasing component. The composition ratio is reduced.

The long / short synthesis processing unit 217 is supplied with the long time exposure signal from the exposure correction unit 212 and the short time exposure signal via the noise reduction processing unit 220. The short exposure signal is supplied to the long / short combination processing unit 217 after noise is reduced by the noise reduction processing unit 220.

The long / short combination processing unit 217 combines the supplied long-time exposure signal and short-time exposure signal based on the total combination ratio from the long-lived saturation consideration component ratio calculation unit 216. The signal synthesized in this way is output as an HDR image signal.

<Configuration of folding component detection unit>
When an HDR image is generated as described above, SN optimization, suppression of color mixing, suppression of false colors that may occur due to the influence of aliasing components, and generation of effects due to the moving object may occur. It is possible to generate an HDR image in consideration of blur suppression and long-time exposure signal saturation.

The configuration of the aliasing component detection unit 218 of the HDR image generation unit 200 is shown in FIG. The aliasing component detection unit 218 detects the aliasing component by determining whether or not the first to fourth conditions are satisfied.

The aliasing component detection unit 218 includes a strong saturation generation region detection unit 251, a local color ratio calculation unit 252, each color length difference calculation unit 253, each color length difference difference normalization unit 254, and a normalized amplitude intensity similarity calculation unit 255. It is said that.

The folding component detection unit 218 is supplied with a long exposure signal and a short exposure signal from the exposure correction unit 212 (FIG. 8). Since the long exposure signal and the short exposure signal include an R signal, a G signal, and a B signal, respectively, six-color signals are supplied to the aliasing component detection unit 218.

The intense saturation generation area detection unit 251 is a part that mainly determines whether or not the area (pixel) satisfies the second condition and the third condition.

The strong saturation generation area detection unit 251 performs conversion to a color space, detection of a saturation difference, and detection of a specific color. The intense saturation generation area detection unit 251 converts the supplied long exposure signal and short exposure signal into color spaces, respectively. For example, the image is transferred to a color space such as a Lab color space, and saturation and color difference are obtained to determine whether the color is a specific color.

The second condition is that there is a large difference in saturation between the exposure correction long exposure image μL and the exposure correction short exposure image μS. It is determined whether or not.

Further, as a third condition, there is a condition that “a signal with a large saturation has a green or magenta color”, so that the color of an area (pixel) determined to be a signal with a large saturation Is determined to be green or magenta. This determination result (determination result E) is supplied to the local color ratio calculation unit 252 and the normalized amplitude intensity similarity calculation unit 255.

In order to determine whether or not the first condition and the fourth condition are satisfied, first, each color length difference calculating unit 253 determines the exposure correction long exposure image μL for each of the R signal, the G signal, and the B signal. The difference from the exposure corrected short exposure image μS is calculated.

Further, each color length short / difference calculating unit 253 divides the calculated difference using a signal with lower saturation in order to perform normalization. The signal having the lower saturation is a signal having a lower saturation when the saturation of the long-time exposure signal and the saturation of the short-time exposure signal are compared. Further, as described above, it can be determined that the signal with the lower saturation is the signal on the side where the aliasing component does not appear. That is, here, a long-time exposure signal or a short-time exposure signal with no aliasing component is selected, and normalization of the long-short difference is performed.

Each color length difference calculation unit 253 obtains a difference between the exposure-corrected long-time exposure image μL and the exposure-correction short-time exposure image μS for each of the R signal, the G signal, and the B signal, Each of the lower R signal, G signal, and B signal is divided to generate a standardized long / short difference signal, which is output to each subsequent color length / short difference normalization unit 254.

If the long / short difference signal has a predetermined value, the first condition, that is, the condition that “the difference occurs between the exposure-corrected long-time exposure image μL and the exposure-corrected short-time exposure image μS”. Is satisfied. On the other hand, if the long / short difference signal does not have a predetermined value, it can be determined that the first condition is not satisfied. In such a case, the long / short difference signal is processed as 0 in the subsequent processing. Therefore, as a result, the folding component information becomes a determination result that the folding component is not detected.

Note that the difference between the exposure-corrected long-exposure image μL and the exposure-corrected short-exposure image μS is affected by noise, and there is a possibility that the difference may be calculated even if there is no difference. Considering this, a predetermined threshold is provided, and it is determined whether or not the difference between the exposure-corrected long-time exposure image μL and the exposure-corrected short-time exposure image μS is equal to or larger than the predetermined threshold. In this case, it may be determined that the first condition is satisfied.

The local color ratio calculation unit 252 obtains a local color ratio using a signal with lower saturation. The local color ratio calculation unit 252 refers to the determination result E from the strong saturation generation region detection unit 251 to determine the signal with the lower saturation, and based on the determination, the supplied exposure correction long time The signal of the exposure image μL or the signal of the exposure corrected short-time exposure image μS is selected, and the local color ratio is obtained. The obtained local color ratios of R, G, and B are supplied to each color length difference difference normalizing unit 254.

Each color length difference difference normalizing unit 254 multiplies each of the R, G, and B difference signals from each color length difference calculating unit 253 by the local ratio of each of R, G, and B from the local color ratio calculating unit 252. Thus, the RGB color levels are aligned to generate a standardized R component long / short difference signal, a standardized G component long / short difference signal, and a standardized B component long / short difference signal, respectively.

Each color length difference standardization unit 254 unifies the RB value by selecting the larger normalized value of the normalized R component length difference signal or the B component length difference signal, The selected R component or B component long / short difference signal is supplied to the normalized amplitude intensity similarity calculation unit 255. Each color length difference normalization unit 254 supplies the normalized G component length difference signal to the normalized amplitude intensity similarity calculation unit 255.

The standardized amplitude intensity similarity calculation unit 255 calculates the similarity between the supplied standardized G component long / short difference signal, the standardized R (or B) component long / short difference signal, and the sign direction. Is evaluated, and it is determined whether or not both signals have the same amplitude by the inverse method. This determination is the determination described with reference to FIG. 7 and is a determination as to whether or not the fourth condition is satisfied.

In this way, finally, the normalized amplitude intensity similarity calculation unit 255 determines whether or not the fourth condition is satisfied. The standardized amplitude intensity similarity calculation unit 255 is also supplied with the determination result E from the strong saturation generation region detection unit 251. This determination result E is information including a determination result as to whether or not the second condition and the third condition are satisfied.

When the normalized amplitude intensity similarity calculation unit 255 determines that the fourth condition is satisfied, and the supplied determination result E also indicates a result determined to satisfy the second and third conditions, If the region (pixel) to be processed is a region where a aliasing component is generated, a final determination is made, and the determination result is used as aliasing component information to determine the aliasing reduction combination ratio calculation unit 214 and the animal. It supplies to the body detection part 219 (FIG. 8).

As described above, the folding component information output from the folding component detection unit 218 may be, for example, 0 or 1 information indicating whether or not the folding component is generated. For example, it may be information having a value of 0 to 1 that represents the probability of occurrence of a component.

Further, when it is determined that any one of the first to fourth conditions is not satisfied in the calculation result of each part in the aliasing component detection unit 218, the process for the region (pixel) to be processed is terminated. Alternatively, information indicating that there is no folding component may be output as the folding component information.

For example, when the strong saturation generation region detection unit 251 determines that there is no significant difference in saturation, or when no color such as green or magenta is detected, the local color ratio calculation unit 252, each color length difference calculation unit 253, The processing in each color length difference normalization unit 254 may not be performed, and the standardized amplitude intensity similarity calculation unit 255 may output information indicating that there is no aliasing component.

In this way, the folded component detection unit 218 can detect a region where the folded component is generated.

As described above, it is possible to detect a region where the aliasing component is generated, so that it is a region where a difference between the long-time exposure signal and the short-time exposure signal whose exposure ratio is corrected occurs. However, it is possible to perform a process of weakening the signal value of the difference at a place where the detection strength of the aliasing component is high. By enabling such processing, it is possible to prevent erroneous detection of the moving object.

As described above, a composite ratio that maximizes the S / N ratio is obtained, and if it is a folded area, a composite ratio that reduces the strength is adopted (in a four-part Bayer array, a right diagonal pattern and a left diagonal pattern are used. In the case of a moving object region, an optimum combining ratio can be calculated by adopting a combining ratio that reduces blur (moving object blur).

As described above, according to the present technology, the adjacent 2 × 2 pixels are set to the same color filter, and the 2 × 2 color filter is used with an imaging element using a four-divided Bayer type array arranged in a Bayer type. In an image taken by changing the exposure time in a spatial cycle, the effects of color mixing are alleviated, blurring of moving objects, suppression of aliasing signals due to high-frequency signals, optimization of SN, saturation of long-time exposure signals Can be obtained.

<Usage example of imaging device>
FIG. 10 is a diagram illustrating a usage example in which the above-described imaging device and an electronic apparatus including the imaging device are used.

The imaging device described above can be used in various cases for sensing light such as visible light, infrared light, ultraviolet light, and X-rays as follows.

・ Devices for taking images for viewing, such as digital cameras and mobile devices with camera functions ・ For safe driving such as automatic stop and recognition of the driver's condition, Devices used for traffic, such as in-vehicle sensors that capture the back, surroundings, and interiors of vehicles, surveillance cameras that monitor traveling vehicles and roads, and ranging sensors that measure distances between vehicles, etc. Equipment used for home appliances such as TVs, refrigerators, air conditioners, etc. to take pictures and operate the equipment according to the gestures ・ Endoscopes, equipment that performs blood vessel photography by receiving infrared light, etc. Equipment used for medical and health care ・ Security equipment such as security surveillance cameras and personal authentication cameras ・ Skin measuring instrument for photographing skin and scalp photography Such as a microscope to do beauty Equipment used for sports-Equipment used for sports such as action cameras and wearable cameras for sports applications-Used for agriculture such as cameras for monitoring the condition of fields and crops apparatus

<About recording media>
The series of processes described above can be executed by hardware or can be executed by software. When a series of processing is executed by software, a program constituting the software is installed in the computer. Here, the computer includes, for example, a general-purpose personal computer capable of executing various functions by installing various programs by installing a computer incorporated in dedicated hardware.

FIG. 11 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processing by a program. In the computer, a CPU (Central Processing Unit) 301, a ROM (Read Only Memory) 302, and a RAM (Random Access Memory) 303 are connected to each other by a bus 304. An input / output interface 305 is further connected to the bus 304. An input unit 306, an output unit 307, a storage unit 308, a communication unit 309, and a drive 310 are connected to the input / output interface 305.

The input unit 306 includes a keyboard, a mouse, a microphone, and the like. The output unit 307 includes a display, a speaker, and the like. The storage unit 308 includes a hard disk, a nonvolatile memory, and the like. The communication unit 309 includes a network interface and the like. The drive 310 drives a removable medium 311 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

In the computer configured as described above, the CPU 301 loads the program stored in the storage unit 308 to the RAM 303 via the input / output interface 305 and the bus 304 and executes the program, for example. Is performed.

The program executed by the computer (CPU 301) can be provided by being recorded in, for example, a removable medium 311 as a package medium or the like. The program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

In the computer, the program can be installed in the storage unit 308 via the input / output interface 305 by attaching the removable medium 311 to the drive 310. Further, the program can be received by the communication unit 309 via a wired or wireless transmission medium and installed in the storage unit 308. In addition, the program can be installed in the ROM 302 or the storage unit 308 in advance.

The program executed by the computer may be a program that is processed in time series in the order described in this specification, or in parallel or at a necessary timing such as when a call is made. It may be a program for processing.

In addition, in this specification, the system represents the entire apparatus composed of a plurality of apparatuses.

It should be noted that the effects described in the present specification are merely examples and are not limited, and other effects may be obtained.

Note that the embodiments of the present technology are not limited to the above-described embodiments, and various modifications can be made without departing from the gist of the present technology.

In addition, this technique can also take the following structures.
(1)
When 2 × 2 pixels having the same spectral sensitivity are made into one block, out of 2 × 2 pixels in the one block, two pixels are long-time exposure pixels, and two pixels are short-time exposure. A pixel having the same exposure time is disposed in an oblique direction, and includes a processing unit that processes a signal from the pixel disposed on the imaging surface in the block unit,
The processor is
A generating unit that generates a long-time exposure image by adding signals from the long-time exposure pixels in the one block and generates a short-time exposure image by adding signals from the short-time exposure pixels; ,
A combining unit that combines the long-time exposure image generated by the generation unit and the short-time exposure image at a predetermined combining ratio;
From the difference between the long-time exposure image and the short-time exposure image, an animal body detection unit that detects an animal body,
A folding component detection unit for detecting a folding component from the long exposure image and the short exposure image;
The imaging ratio is set from the detection result of the moving object in the moving object detection unit and the detection result of the folded component in the folded component detection unit.
(2)
The aliasing component detection unit determines whether or not a difference between the long-time exposure image and the short-time exposure image and a saturation of each of the long-time exposure image and the short-time exposure image satisfy a predetermined condition. The imaging device according to (1), wherein the folded component is detected.
(3)
The aliasing component detection unit detects the aliasing component by determining whether or not the following first to fourth conditions are satisfied. First condition: long-time exposure image and short-time exposure image Second condition: There is a difference in saturation between the long-time exposure image and the short-time exposure image Third condition: A large signal with saturation has a green or magenta color Condition 4: When the generated signal is subtracted from the signal having no aliasing component, the difference is generated with the same amplitude in the reverse direction between the G pixel and the R pixel or the G pixel and the B pixel. The imaging device according to (1).
(4)
The imaging device according to (3), wherein the short-time exposure image is an exposure-corrected image.
(5)
The imaging device according to (3), wherein the long-time exposure image and the short-time exposure image are transferred to a predetermined color space to obtain the saturation.
(6)
The composite ratio is the long-time exposure image or the short-time exposure in which it is determined that no aliasing component has occurred in a pixel where the aliasing component is detected by the aliasing component detection unit. The imaging apparatus according to any one of (1) to (5), wherein a ratio of using a large amount of images is used.
(7)
The composition ratio is a pixel in which the moving object is detected by the moving object detection unit, but it is determined that no folding component is generated in the pixel in which the folding component is detected by the folding component detection unit. The imaging device according to any one of (1) to (6), wherein the ratio is a ratio in which the long-time exposure image or the short-time exposure image is frequently used.
(8)
When 2 × 2 pixels having the same spectral sensitivity are made into one block, out of 2 × 2 pixels in the one block, two pixels are long-time exposure pixels, and two pixels are short-time exposure. In the imaging method of the imaging apparatus, the pixels having the same exposure time are arranged in an oblique direction and include a processing unit that processes signals from the pixels arranged on the imaging surface in units of the blocks.
The processor is
A long exposure image is generated by adding signals from the long exposure pixels in the block, and a short exposure image is generated by adding signals from the short exposure pixels.
The generated long exposure image and the short exposure image are synthesized at a predetermined synthesis ratio,
From the difference between the long exposure image and the short exposure image, the moving object is detected,
Detecting a folded component from the long exposure image and the short exposure image,
The imaging ratio is set from the detection result of the moving object in the moving object detection unit and the detection result of the folded component in the folded component detection unit.
(9)
When 2 × 2 pixels having the same spectral sensitivity are made into one block, out of 2 × 2 pixels in the one block, two pixels are long-time exposure pixels, and two pixels are short-time exposure. An image pickup apparatus including a processing unit that processes a signal from a pixel that is arranged in an oblique direction and is arranged on the image pickup surface in a block unit.
The processor is
A long exposure image is generated by adding signals from the long exposure pixels in the block, and a short exposure image is generated by adding signals from the short exposure pixels.
The generated long exposure image and the short exposure image are synthesized at a predetermined synthesis ratio,
From the difference between the long exposure image and the short exposure image, the moving object is detected,
From the long-time exposure image and the short-time exposure image, to execute a process including a step of detecting a folding component,
The composition ratio is a program for causing a computer to execute a process set based on the detection result of the moving object in the moving object detection unit and the detection result of the folding component in the folding component detection unit.

100 imaging device, 101 optical lens, 102, imaging device, 103 image processing unit, 104 signal processing unit, 105 control unit, 200 HDR image generation unit, 211 RGB interpolation signal generation unit, 212 exposure correction unit, 213 SN maximization synthesis Ratio calculation unit, 214 Folding reduction synthesis ratio calculation unit, 215 Blur reduction synthesis ratio calculation unit, 216 Long livestock saturation consideration synthesis ratio calculation unit, 217 Long and short synthesis processing unit, 218 Folding component detection unit, 219 Animal body detection unit, 220 noise reduction processing unit, 251 strong saturation generation region detection unit, 252 local color ratio calculation unit, 253 each color length short difference calculation unit, 254 each color length short difference normalization unit, 255 standardized amplitude intensity similarity calculation unit

Claims

When 2 × 2 pixels having the same spectral sensitivity are made into one block, out of 2 × 2 pixels in the one block, two pixels are long-time exposure pixels, and two pixels are short-time exposure. A pixel having the same exposure time is disposed in an oblique direction, and includes a processing unit that processes a signal from the pixel disposed on the imaging surface in the block unit,
The processor is
A generating unit that generates a long-time exposure image by adding signals from the long-time exposure pixels in the one block and generates a short-time exposure image by adding signals from the short-time exposure pixels; ,
A combining unit that combines the long-time exposure image generated by the generation unit and the short-time exposure image at a predetermined combining ratio;
From the difference between the long-time exposure image and the short-time exposure image, an animal body detection unit that detects an animal body,
A folding component detection unit for detecting a folding component from the long exposure image and the short exposure image;
The imaging ratio is set from the detection result of the moving object in the moving object detection unit and the detection result of the folded component in the folded component detection unit.
The aliasing component detection unit determines whether or not a difference between the long-time exposure image and the short-time exposure image and a saturation of each of the long-time exposure image and the short-time exposure image satisfy a predetermined condition. The imaging apparatus according to claim 1, wherein the folded component is detected.
The aliasing component detection unit detects the aliasing component by determining whether or not the following first to fourth conditions are satisfied. First condition: long-time exposure image and short-time exposure image Second condition: There is a difference in saturation between the long-time exposure image and the short-time exposure image Third condition: A large signal with saturation has a green or magenta color Condition 4: When the generated signal is subtracted from the signal without the aliasing component, the difference is generated with the same amplitude in the reverse direction in the G pixel and the R pixel or the G pixel and the B pixel. Item 2. The imaging device according to Item 1.
The imaging device according to claim 3, wherein the short-time exposure image is an exposure-corrected image.
The imaging device according to claim 3, wherein the long-time exposure image and the short-time exposure image are transferred to a predetermined color space to obtain the saturation.
The composite ratio is the long-time exposure image or the short-time exposure in which it is determined that no aliasing component has occurred in a pixel where the aliasing component is detected by the aliasing component detection unit. The imaging apparatus according to claim 1, wherein the imaging apparatus is configured to use a large amount of images.
The composition ratio is a pixel in which the moving object is detected by the moving object detection unit, but it is determined that no folding component is generated in the pixel in which the folding component is detected by the folding component detection unit. The imaging device according to claim 1, wherein the ratio is a ratio of using a lot of the long-time exposure image or the short-time exposure image.
When 2 × 2 pixels having the same spectral sensitivity are made into one block, out of 2 × 2 pixels in the one block, two pixels are long-time exposure pixels, and two pixels are short-time exposure. In the imaging method of the imaging apparatus, the pixels having the same exposure time are arranged in an oblique direction and include a processing unit that processes signals from the pixels arranged on the imaging surface in units of the blocks.
The processor is
A long exposure image is generated by adding signals from the long exposure pixels in the block, and a short exposure image is generated by adding signals from the short exposure pixels.
The generated long exposure image and the short exposure image are synthesized at a predetermined synthesis ratio,
From the difference between the long exposure image and the short exposure image, the moving object is detected,
Detecting a folded component from the long exposure image and the short exposure image,
The imaging ratio is set from the detection result of the moving object in the moving object detection unit and the detection result of the folded component in the folded component detection unit.
When 2 × 2 pixels having the same spectral sensitivity are made into one block, out of 2 × 2 pixels in the one block, two pixels are long-time exposure pixels, and two pixels are short-time exposure. An image pickup apparatus including a processing unit that processes a signal from a pixel that is arranged in an oblique direction and is arranged on the image pickup surface in a block unit.
The processor is
A long exposure image is generated by adding signals from the long exposure pixels in the block, and a short exposure image is generated by adding signals from the short exposure pixels.
The generated long exposure image and the short exposure image are synthesized at a predetermined synthesis ratio,
From the difference between the long exposure image and the short exposure image, the moving object is detected,
From the long-time exposure image and the short-time exposure image, to execute a process including a step of detecting a folding component,
The composition ratio is a program for causing a computer to execute a process set based on the detection result of the moving object in the moving object detection unit and the detection result of the folding component in the folding component detection unit.