JP2014039170A - Image processing device, image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a device and method for generating a high dynamic range image by pixel synthesis of an image composed of long exposure pixels and short exposure pixels.SOLUTION: The device comprises: an edge detection unit for executing edge detection of respective constituent pixels of an input image containing long exposure pixels and short exposure pixels; a centroid correction unit for executing centroid correction processing of the long exposure pixels and the short exposure pixels being synthesis targets; and a pixel synthesis unit for executing synthesis processing of the pixels generated by the centroid correction unit. The pixel synthesis unit selects an optimal synthesis result corresponding to the situation of the respective pixels from among a flat portion synthesis result having a blend rate (α) corresponding to the pixel value of the long exposure pixels applied thereto, an edge portion synthesis result having a fixed blend rate applied thereto, and a saturated portion synthesis result adapted to a saturated portion area, generates a high dynamic range (HDR) image on the basis of the selected synthesis result, and outputs the image.

Description

  The present disclosure relates to an image processing device, an image processing method, and a program. More specifically, the present invention relates to an image processing apparatus, an image processing method, and a program that generate a high dynamic range image by combining pixels set at different exposure times.

  Solid-state imaging devices such as CCD image sensors and CMOS (Complementary Metal Oxide Semiconductor) image sensors used in video cameras and digital still cameras store charges according to the amount of incident light and output electrical signals corresponding to the stored charges. Photoelectric conversion is performed to generate an image composed of pixel values corresponding to the accumulated charge amount.

  Recently, an imaging apparatus (camera) that generates a high dynamic range (HDR) image in which highly accurate pixel values are set from a low luminance region to a high luminance region has been proposed.

  A high dynamic range (HDR) image is, for example, a combination of a plurality of short exposure images and long exposure images, or long exposure pixels set in one captured image and short exposure pixels. Generated by pixel synthesis.

FIG. 1 is a diagram showing a high dynamic range (HDR) image generation processing configuration by combining a plurality of short exposure images and long exposure images.
The long exposure image 11 and the short exposure image 12 are continuously photographed by setting two different exposure times. In the synthesizing unit 20, the pixel values of the corresponding pixels of these two images are blended to determine the pixel value of the high dynamic range (HDR) image 21 as the output image.

For example, the pixel value D (HDR) of the high dynamic range (HDR) image 21 is
When the pixel values of the corresponding pixels of the long exposure image 11 and the short exposure image 12 are D (L) and D (S), respectively, they are calculated by the following equations.
D (HDR) = (1−α) × D (L) + α × D (S) × (GAIN)
However,
D (L) is the pixel value of the long exposure image,
D (S) is the pixel value of the short-time exposure image,
GAIN is the exposure time ratio (exposure ratio) between the long exposure image and the short exposure image,
α is the blend ratio,
It is.
The pixel value calculation process according to the above formula is called an α blend process.

  For example, in a pixel area where the pixel value of the long-exposure image is saturated, such as a high-luminance pixel area, the pixel value of the long-exposure image is not used in the above-described α blend process, or the pixel of the long-exposure image The pixel value D (HDR) of the high dynamic range (HDR) image 21 that is the output image is calculated by reducing the blend ratio of the values.

In the low luminance region, the pixel value of the short-time exposure image is not used or the blend ratio of the pixel value of the long-time exposure image is increased in the α blending process.
By executing such processing and generating a composite image, it is possible to generate a high dynamic range (HDR) image in which more accurate pixel values are set from the low luminance region to the high luminance region.

FIG. 2 is a diagram illustrating a configuration for generating a high dynamic range (HDR) image by combining a long-time exposure pixel set in one captured image and a short-time exposure pixel.
In the captured image 30, long exposure pixels and short exposure pixels are set in predetermined pixel area units.
The example shown in FIG. 2 is an example in which long exposure pixels and short exposure pixels are alternately set in units of two rows in an image sensor having a Bayer array. It should be noted that the long-exposure pixel and the short-exposure pixel can be set in various other settings such as every other row.

The synthesizer 40 calculates the pixel value of the output image by blending the pixel values of the long-time exposure pixels and the short-time exposure pixels in the vicinity area included in the captured image 30 to generate a high dynamic range (HDR) image 41. To do.
For example, one R pixel value of the high dynamic range (HDR) image 41 is determined using the two R pixels shown in the drawing, that is, the long exposure pixel R31 and the short exposure pixel R32.

The calculation process of the pixel value D (HDR) of the high dynamic range (HDR) image 41 using the single photographed image 30 is basically the same as the blend process described above with reference to FIG. Executed. That is,
The pixel value D (HDR) of the high dynamic range (HDR) image 41 is as follows when the pixel values of the long exposure pixel and the short exposure pixel to be blended are D (L) and D (S), respectively. Calculated by the formula.
D (HDR) = (1−α) × D (L) + α × D (S) × (GAIN)
However,
D (L) is the pixel value of the long exposure pixel,
D (S) is the pixel value of the short-time exposure pixel,
GAIN is the exposure time ratio (exposure ratio) between the long exposure pixel and the short exposure pixel,
α is the blend ratio,
It is.

  For example, Patent Document 1 (Japanese Patent Application Laid-Open No. 2011-244309), Japanese Patent Laid-Open No. 2011-244309 discloses a process for setting pixels with different exposure times in one image and combining these pixels to generate a high dynamic range image. Document 2 (Japanese Patent Laid-Open No. 2012-80297) and the like.

However, when generating a high dynamic range (HDR) image based on a single photographed image in this way, a combination process of long exposure pixels and short exposure pixels at different pixel positions in one image is performed. Necessary.
Strictly speaking, a long-time exposure pixel and a short-time exposure pixel in one image are shooting pixels of a subject at different positions, and pixel values of one output image are synthesized by combining pixel values obtained by shooting the different subject positions. Must be calculated.

When a high dynamic range image is generated by such processing, jaggies are likely to occur in an edge region having a large luminance change. Jaggy is a phenomenon in which a boundary such as an edge is output in a stepped jagged pattern.
Originally, even a smooth edge boundary is output as a step-like unnatural edge boundary. In many cases, “jaggy” is aliasing that occurs because the number of samplings is insufficient with respect to the band of the input signal, and is likely to appear in, for example, a subject boundary or a textured area with a pattern in an image.

JP 2011-244309 A JP 2012-80297 A

  The present disclosure has been made in view of, for example, such a situation, and an image processing apparatus that performs high-quality image generation in a configuration that generates a high dynamic range image using a single captured image, and image processing It is an object to provide a method and a program.

The first aspect of the present disclosure is:
A signal processing unit that generates an output image by executing pixel value synthesis processing of pixels included in an input image including the long-time exposure pixel and the short-time exposure pixel;
The signal processing unit
An edge detection unit that performs edge detection on each component pixel of the input image and outputs edge information corresponding to each pixel;
A centroid correction unit that executes centroid correction processing of the long-time exposure pixel and the short-time exposure pixel to be combined, and outputs the centroid correction unit output long-time exposure pixel and centroid correction unit output short-time exposure pixel;
The image processing apparatus includes a pixel synthesizing unit that performs a synthesizing process between the centroid correction unit output long-time exposure pixel and the centroid correction unit output short-time exposure pixel generated by the centroid correction unit.

  Furthermore, in an embodiment of the image processing apparatus according to the present disclosure, the centroid correction unit performs a process of matching the pixel positions of the long-time exposure pixel and the short-time exposure pixel that are to be combined as the centroid correction process. The pixel value of the long-time exposure pixel output from the centroid correction unit is calculated by weighted addition using pixel values of a plurality of long-exposure pixels of the same color at different pixel positions, and a plurality of identical pixels at different pixel positions are calculated. Processing for calculating the pixel value of the short-time exposure pixel output by the centroid correction unit is performed by weighted addition using the pixel value of the short-time exposure pixel of color.

  Furthermore, in an embodiment of the image processing apparatus of the present disclosure, the centroid correction unit includes a saturation determination unit that determines whether or not a pixel applied to the centroid correction process is saturated, and the centroid correction process includes When the applied pixel is saturated, the pixel value of the input image is set as the pixel value of the centroid correction unit output long exposure pixel and the centroid correction unit output short exposure pixel, and the pixel applied to the centroid correction process is If not saturated, the pixel value calculated by weighted addition using the pixel values of a plurality of long-time exposure pixels of the same color at different pixel positions is used as the pixel value of the centroid correction unit output long-time exposure pixel and is different A pixel value calculated by weighted addition using pixel values of a plurality of short-time exposure pixels of the same color at pixel positions is selectively output as a pixel value of the centroid correction unit output short-time exposure pixel.

  Furthermore, in an embodiment of the image processing apparatus of the present disclosure, the centroid correction unit includes a saturation determination unit that determines whether or not a pixel applied to the centroid correction process is saturated, and the centroid correction process includes When the applied pixel is saturated, the pixel value of the centroid correction unit output long-time exposure pixel is set as the pixel value of the long-time exposure pixel of the input image, and the pixel value of the centroid correction unit output short-time exposure pixel is The pixel value calculated by the smoothing process based on the plurality of pixel values of the short-time exposure pixels of the input image is set.

  Furthermore, in an embodiment of the image processing apparatus according to the present disclosure, the center-of-gravity correction unit applies the smoothing process to a smoothing filter in which a coefficient corresponding to an edge direction input from the edge detection unit is set. Run.

  Furthermore, in an embodiment of the image processing device of the present disclosure, the pixel synthesis unit is a flat as a synthesis unit that performs a synthesis process of the centroid correction unit output long-time exposure pixel and the centroid correction unit output short-time exposure pixel. A flat part synthesis unit that executes synthesis processing adapted to the sub-region, an edge synthesis unit that performs synthesis processing adapted to the edge region, a saturation unit synthesis unit that performs synthesis processing adapted to the saturation region, A selection unit configured to select a synthesis result of each of the synthesis units and generate a synthesis result image, and the selection unit determines whether the edge information and a pixel applied to the centroid correction processing are saturated pixels; When the saturation flag is input and the pixel applied to the centroid correction process is a saturated pixel, the combination result of the saturation unit synthesizer is selected, and the pixel applied to the centroid correction process is not a saturated pixel but an edge In the area If selects the synthesis results of the edge portion combining unit, the center of gravity correction processing pixel is not saturated pixel applied to, if not an edge region, selects the combined result of the flat portion synthesis unit.

  Furthermore, in an embodiment of the image processing apparatus of the present disclosure, the flat portion synthesis unit includes the centroid correction unit according to a blend rate (α) that changes according to a pixel value of the centroid correction unit output long-time exposure pixel. The pixel value of the output long time exposure pixel and the centroid correction unit output short time exposure pixel are blended to calculate a pixel value as a synthesis result, and the edge synthesis unit calculates the pixel value of the centroid correction unit output long time exposure pixel. The saturation value is calculated by blending pixel values of the centroid correction unit output long-time exposure pixel and the centroid correction unit output short-time exposure pixel according to a fixed blend rate (β) independent of The synthesis unit calculates the pixel value of the centroid correction unit output short-time exposure pixel as a pixel value of the synthesis result.

  Furthermore, in an embodiment of the image processing apparatus according to the present disclosure, the pixel composition unit further includes a moving subject unit composition unit that performs composition processing adapted to the moving subject region, and the selection unit includes the center-of-gravity correction. When a moving subject detection flag indicating whether or not a pixel applied to the process is a moving subject region is input, and a pixel applied to the centroid correction processing is a moving subject region, the composition result of the moving subject composition unit is displayed. select.

  Furthermore, in an embodiment of the image processing device according to the present disclosure, the input image is a Bayer array image composed of RGB pixels, and the edge detection unit performs an edge detection process using only G pixels.

Furthermore, the second aspect of the present disclosure is:
A signal processing unit that generates an output image by executing pixel value synthesis processing of pixels included in an input image including the long-time exposure pixel and the short-time exposure pixel;
The signal processing unit
An edge detection unit that performs edge detection on each component pixel of the input image and outputs edge information corresponding to each pixel;
An array conversion centroid correction unit that generates a specific color high-frequency signal and an all-color low-frequency signal of the input image;
A pixel synthesizer that inputs the specific color high-frequency signal and the all-color low-frequency signal, and generates a specific color high-frequency high-dynamic range (HDR) signal and an all-color low-frequency high dynamic range (HDR) signal;
Correlation processing for generating the high dynamic range (HDR) image having the same number of pixels as the input image by inputting the specific color high frequency range high dynamic range (HDR) signal and the all color low frequency range high dynamic range (HDR) signal. An image processing apparatus having a unit.

  Furthermore, in an embodiment of the image processing device of the present disclosure, the input image is a Bayer array image configured by RGB pixels, and the array conversion centroid correction unit includes a G color high-frequency signal of the input image and RGB Each color low-frequency signal is generated, and the pixel synthesizing unit inputs the G color high-frequency signal and the RGB low-frequency signal, the G color high-frequency high dynamic range (HDR) signal, and the RGB low-frequency color high-frequency signal. A dynamic range (HDR) signal is generated, and the correlation processing unit inputs the G color high frequency high dynamic range (HDR) signal and the RGB color low frequency high dynamic range (HDR) signal, and inputs the input A high dynamic range (HDR) image including a Bayer array having the same number of pixels as the image is generated.

Furthermore, the third aspect of the present disclosure is:
An image processing method executed in an image processing apparatus,
The signal processing unit executes a signal processing step of generating an output image by performing pixel value synthesis processing of pixels included in the input image including the long exposure pixels and the short exposure pixels,
In the signal processing step,
Edge detection processing for executing edge detection for each component pixel of the input image and outputting edge information corresponding to each pixel;
The center-of-gravity correction processing for executing the center-of-gravity correction processing of the long-time exposure pixel and the short-time exposure pixel to be combined and outputting the center-of-gravity correction unit output long-time exposure pixel and the center-of-gravity correction unit output
In the image processing method, the centroid correction unit output long exposure pixel and the centroid correction unit output short exposure pixel generated by the centroid correction unit are combined.

Furthermore, the fourth aspect of the present disclosure is:
A program for executing image processing in an image processing apparatus;
Causing the signal processing unit to execute a signal processing step of generating an output image by performing pixel value synthesis processing of pixels included in the input image including the long exposure pixels and the short exposure pixels;
In the signal processing step,
Edge detection processing for executing edge detection for each component pixel of the input image and outputting edge information corresponding to each pixel;
The center-of-gravity correction processing for executing the center-of-gravity correction processing of the long-time exposure pixel and the short-time exposure pixel to be combined and outputting the center-of-gravity correction unit output long-time exposure pixel and the center-of-gravity correction unit output
A program for executing a synthesis process of the center-of-gravity correction unit output long-time exposure pixel and the center-of-gravity correction unit output short-time exposure pixel generated by the barycentric correction unit.

  Note that the program of the present disclosure is a program that can be provided by, for example, a storage medium or a communication medium provided in a computer-readable format to an information processing apparatus or a computer system that can execute various program codes. By providing such a program in a computer-readable format, processing corresponding to the program is realized on the information processing apparatus or the computer system.

  Other objects, features, and advantages of the present disclosure will become apparent from a more detailed description based on embodiments of the present disclosure described below and the accompanying drawings. In this specification, the system is a logical set configuration of a plurality of devices, and is not limited to one in which the devices of each configuration are in the same casing.

According to the configuration of an embodiment of the present disclosure, an apparatus and a method for generating a high-quality, high dynamic range image by pixel synthesis of an image including long-time exposure pixels and short-time exposure pixels are realized.
Specifically, an edge detection unit that performs edge detection for each component pixel of the input image including the long exposure pixel and the short exposure pixel, and a centroid correction process of the long exposure pixel and the short exposure pixel to be combined And a pixel composition unit that performs composition processing of the generated pixels of the center of gravity correction unit. The pixel compositing unit is a flat part compositing result obtained by applying a blend ratio (α) corresponding to the pixel value of the long-time exposure pixel, an edge part compositing result applying a fixed blend ratio, and a saturated part compositing adapted to the saturated part region. From the result, an optimum synthesis result corresponding to the situation of each pixel is selected, and a high dynamic range (HDR) image is generated and output based on the selected synthesis result.
By applying the above method, it is possible to obtain an optimum synthesis result corresponding to the pixel in accordance with the situation of each constituent pixel, and to generate a high-quality, high dynamic range image.

It is a figure explaining the high dynamic range image generation process based on multiple images. It is a figure explaining the high dynamic range image generation process based on a single image. It is a figure explaining an example of 1 composition of an image processing device of this indication. It is a figure explaining the example of 1 composition of the signal processing part of the image processing device of this indication. It is a figure explaining the example of pixel arrangement composition of an image sensor. It is a figure explaining the exposure control structure of an image pick-up element. It is a figure explaining the structural example of the HDR image generation part of the image processing apparatus of this indication. It is a figure explaining one structural example of the edge detection part of the HDR image generation part of the image processing apparatus of this indication. It is a figure explaining an example of the process which an edge detection part performs. It is a figure explaining an example of the process which an edge detection part performs. It is a figure explaining one structural example of the gravity center correction | amendment part of the HDR image generation part of the image processing apparatus of this indication. It is a figure explaining an example of the process which a gravity center correction | amendment part performs. It is a figure explaining an example of the process which a gravity center correction | amendment part performs. It is a figure explaining an example of the process which a gravity center correction | amendment part performs. It is a figure explaining one structural example of the pixel synthetic | combination part of the HDR image generation part of the image processing apparatus of this indication. It is a figure explaining an example of the process which a pixel synthetic | combination part performs. It is a figure which shows the flowchart explaining the sequence of the process which a pixel synthetic | combination part performs. It is a figure explaining an example of the process which a pixel synthetic | combination part performs. It is a figure explaining one structural example of the gravity center correction | amendment part of the HDR image generation part of the image processing apparatus of this indication. It is a figure explaining the example of 1 process performed in a gravity center correction | amendment part. It is a figure explaining one structural example of the pixel synthetic | combination part of the HDR image generation part of the image processing apparatus of this indication. It is a figure which shows the flowchart explaining the sequence of the process which a pixel synthetic | combination part performs. It is a figure explaining the example of pixel arrangement composition of an image sensor. It is a figure explaining the exposure control structure of an image pick-up element. It is a figure explaining the structural example of the HDR image generation part of the image processing apparatus of this indication. It is a figure explaining an example of the process which an edge detection part performs. It is a figure explaining an example of the process which an edge detection part performs. It is a figure explaining the structural example of the arrangement | sequence conversion gravity center correction | amendment part of the HDR image generation part of the image processing apparatus of this indication. It is a figure explaining one process example of the process which the arrangement | sequence conversion gravity center correction | amendment part performs. It is a figure explaining one process example of the process which the arrangement | sequence conversion gravity center correction | amendment part performs. It is a figure explaining one process example of the process which the arrangement | sequence conversion gravity center correction | amendment part performs. It is a figure explaining one structural example of the pixel synthetic | combination part of the HDR image generation part of the image processing apparatus of this indication. It is a figure explaining one structural example of the correlation process part of the HDR image generation part of the image processing apparatus of this indication. It is a figure explaining an example of 1 composition of an image processing device of this indication.

The details of the image processing apparatus, the image processing method, and the program of the present disclosure will be described below with reference to the drawings. The description will be made according to the following items.
1. 1. Example of overall configuration of image processing apparatus 2. Configuration and processing of HDR image generation unit 2-1. Configuration and processing of edge detection unit 2-2. Regarding the configuration and processing of the gravity center correction unit 2-3. 2. Configuration and processing of pixel composition unit Example 2: Example in which the configuration of the gravity center correction unit is changed. Example 3 Example in which the configuration of the pixel synthesis unit is changed. Example 4: Example of change in pixel arrangement of image sensor and long / short exposure pixel setting Example 5: Example of generating an HDR image without reducing the number of pixels 6-1. Configuration and processing of edge detection unit 6-2. Configuration and processing of array conversion centroid correction unit 6-3. Configuration and processing of pixel composition unit 6-4. 6. Configuration and processing of correlation processing unit Example 6: Configuration example of imaging apparatus having gradation conversion unit and camera signal processing unit Summary of composition of this disclosure

[1. Example of overall configuration of image processing apparatus]
First, an overall configuration example of the image processing apparatus according to the first embodiment of the present disclosure will be described.
FIG. 3 illustrates a configuration example of the imaging apparatus 100 that is a configuration example of the image processing apparatus of the present disclosure.

  Light incident through the optical lens 101 is incident on an image sensor 102 constituted by an imaging unit, for example, a CMOS image sensor, and outputs image data by photoelectric conversion. The output image data is input to the signal processing unit 103. For example, the signal processing unit 103 performs pixel synthesis processing described below, and further performs signal processing in a general camera such as white balance (WB) adjustment and gamma correction to generate the output image 120. The output image 120 is stored in a storage unit (not shown). Or it outputs to a display part.

  The control unit 105 outputs a control signal to each unit according to a program stored in a memory (not shown), for example, and controls various processes.

Next, a configuration example of the signal processing unit 103 will be described with reference to FIG.
FIG. 4 is a diagram illustrating a configuration of the signal processing unit 103 according to an embodiment of the present disclosure.
As shown in FIG. 4, the signal processing unit 103 includes an HDR image generation unit 131, a subsequent signal processing unit 132, and an output unit 133.

The HDR image generation unit 131 executes a synthesis process of pixel signals input from the image sensor 102.
The post-stage signal processing unit 132 executes signal processing in a general camera such as white balance (WB) adjustment and gamma correction.
The output unit 133 outputs the processed image that has been subjected to the signal processing as an output image 120.

The image sensor 102 accumulates electric charges based on subject light in each of a large number of pixels and outputs image data. Note that the image sensor 102 has a configuration including a high-sensitivity pixel that performs long-time exposure and a low-sensitivity pixel that performs short-time exposure.
In the following explanation,
High-sensitivity pixels that are exposed for a long time are called “long exposure pixels”.
Low-sensitivity pixels that perform short-time exposure are called “short-exposure pixels”
Will be described.

The HDR image generation unit 131 executes a synthesis (blending) process of the pixel value of the long exposure pixel and the pixel value of the short exposure pixel. For example, a process of combining (blending) the pixel value of the long exposure pixel and the pixel value of the short exposure pixel in the vicinity of the vertical direction of the image sensor is performed, and after this combining, the combined pixels in the horizontal direction are recombined. . Such a pixel value synthesis process is executed to generate a high dynamic range (HDR) image with a reduced number of pixels.
There is also a method of generating a high dynamic range (HDR) image without reducing the number of pixels. The generation configuration of these high dynamic range (HDR) images will be described in detail later.

  Hereinafter, as an example, a processing example in the case where the image sensor 102 is a CMOS (Complementary Metal Oxide Semiconductor) image sensor having an RGB array, that is, a Bayer array as illustrated in FIG. 5 will be described.

As shown in FIG. 6, the image sensor 102 is configured to have pixels with different exposure times for every two rows, that is, short exposure pixels and long exposure pixels.
This exposure time control for each pixel is executed under the control of the control unit 105.

  The gray area shown in FIG. 6 is a short exposure pixel area with a short exposure time, and the white area is a long exposure pixel area with a long exposure time. Each rectangular area represents one pixel.

In the following embodiment, a processing example for an image photographed by setting a short exposure pixel and a long exposure pixel in units of two rows as shown in FIG. 6 will be described.
Note that such an exposure process that changes the exposure time for each pixel is referred to as an SVE (Spatially Variable Exposure Exposure) method, and an image obtained by this SVE method is referred to as an SVE image, an SVE array image, or an SVE signal.

  In this way, an image having a wide dynamic range can be obtained by combining the pixel signals of the long exposure pixel and the short exposure pixel of the SVE image captured by changing the exposure time for each pixel. An image generated by this combining process is called a high dynamic range (HDR) image.

[2. Regarding configuration and processing of HDR image generation unit]
Next, the configuration and processing of the HDR image generation unit 131 in the image processing apparatus according to the first embodiment of the present disclosure will be described.
FIG. 7 is a block diagram illustrating a detailed configuration of the HDR image generation unit 131.
The HDR image generation unit 131 inputs an output image of the image sensor 102, that is, an input image (SVE array image) 141 in which long exposure pixels and short exposure pixels are alternately set in units of two rows.
The HDR image generation unit 131 generates a high dynamic range (HDR) image 142 by combining the long exposure pixels and the short exposure pixel pixels that form the input image (SVE array image) 141 and outputs the high dynamic range (HDR) image 142.
For example, an N × M pixel input image (SVE array image) 141 is input, and a P × Q pixel HDR image 142 is generated and output.
However,
P ≦ N,
Q ≦ M
It is.

As illustrated in FIG. 7, the HDR image generation unit 131 includes an edge detection unit 151, a centroid correction unit 152, and a pixel synthesis unit 153.
The edge detection unit 151 analyzes the presence / absence of an edge and the edge direction in each pixel region of the input SVE array, and generates and outputs edge information including the analysis information.
The centroid correction unit 152 uses the edge information detected by the edge detection unit 151 to correct the centroid position of the long exposure pixel and the short exposure pixel to be combined.
The pixel synthesis unit 153 uses the centroid correction pixel value that is the output of the centroid correction unit 152 to synthesize the long exposure pixel and the short exposure pixel using the edge component information.
Hereinafter, the detailed configuration and processing of each component of the HDR image generation unit 131 will be sequentially described.

(2-1. Configuration and processing of edge detection unit)
First, the configuration and processing of the edge detection unit 151 will be described with reference to FIG.
FIG. 8 is a block diagram showing a detailed configuration of the edge detection unit 151.
The edge detection unit 151 uses the input image (SVE array image) 141 shown in FIG. 7, which is an output image from the image sensor 102, and 9 × 9 pixels in the vicinity of the pixel of interest with the pixel of interest as the edge detection target being substantially the central pixel. Edge information corresponding to the target pixel is generated using an image signal composed only of G pixels in the region, that is, the edge detection image 141G shown in FIG. 8, and the centroid correction unit 152 and the pixel synthesis unit 153 shown in FIG. Output to.

Each pixel of the HDR image 142 finally generated by the HDR image generation unit 131 shown in FIG. 7 is a plurality of pixels including at least one long exposure pixel and short exposure pixel in the input image (SVE array image) 141. Generated by pixel value synthesis.
Therefore, the HDR image generation unit 131 may be set to perform processing by sequentially selecting only the long exposure pixels in the input image (SVE array image) as processing target pixels (target pixels). The short exposure pixel is automatically selected as a pixel to be applied to synthesis with the long exposure pixel selected as the processing target pixel (target pixel).

  Note that the short exposure pixel to be combined with the long exposure pixel selected as the processing target pixel (target pixel) is, for example, a short exposure pixel of the same color in the vicinity of the selected long exposure pixel in the vertical direction. A combination of the long exposure pixel and the short exposure pixel to be combined is set in advance, and the HDR image generation unit 131 performs pixel combination based on the setting information.

The edge information generated by the edge detection unit 151 shown in FIG. 8 is the following signals as shown in FIG.
(A) Long exposure edge flag (LEedge_flg) 171,
(B) Long exposure edge direction flag (LDir_flg) 172,
(C) Short exposure edge flag (SEedge_flg) 173,
(D) Short exposure edge direction flag (SDir_flg) 174,
The edge detection unit 151 generates edge information composed of (a) to (d) and outputs the edge information to the centroid correction unit 152 and the pixel synthesis unit 153.

As illustrated in FIG. 8, the edge detection unit 151 includes a difference calculation unit (vertical) 161, a difference calculation unit (horizontal) 162, a difference calculation unit (upper right) 163, and a difference calculation unit (upper left) 164.
Further, a long exposure edge detection unit 165, a long exposure edge direction determination unit 166, a short exposure edge detection unit 167, and a short exposure edge direction determination unit 168 are provided.

The difference calculation unit (vertical) 161 applies the G pixel included in the edge detection image 141G shown in FIG. 8, that is, the 9 × 9 pixel region in the vicinity of the target pixel, and the following two pieces of vertical pixel value difference information Is calculated.
Average value (LDiff_v) of pixel value difference between two long-exposure G pixels that are close in the vertical direction,
An average value (SDiff_v) of pixel value differences between two short-time exposure G pixels adjacent in the vertical direction;
These are calculated.

The difference calculation unit (horizontal) 162 applies the G pixel included in the edge detection image 141G shown in FIG. 8, that is, the 9 × 9 pixel region in the vicinity of the target pixel, and the following two pieces of horizontal pixel value difference information Is calculated.
Average value (LDiff_h) of pixel value difference between two long-exposure G pixels adjacent in the horizontal direction,
An average value (SDiff_h) of pixel value differences between two short-time exposure G pixels adjacent in the horizontal direction;
These are calculated.

The difference calculation unit (upper right) 163 applies the edge detection image 141G illustrated in FIG. 8, that is, the G pixel included in the 9 × 9 pixel region in the vicinity of the target pixel, and the following two pieces of pixel value difference information in the upper right direction: Is calculated.
Average value (LDiff_a) of pixel value difference between two long-exposure G pixels adjacent in the upper right direction,
An average value (SDiff_a) of pixel value differences between two short-time exposure G pixels adjacent in the upper right direction;
These are calculated.

The difference calculation unit (upper left) 164 applies the G pixel included in the edge detection image 141G shown in FIG. 8, that is, the 9 × 9 pixel region in the vicinity of the target pixel, and the following two upper left pixel value difference information Is calculated.
Average value (LDiff_d) of pixel value difference between two long-exposure G pixels adjacent in the upper left direction,
Average value (SDiff_d) of pixel value difference between two short-exposure G pixels adjacent in the upper left direction,
These are calculated.

FIG. 9 is a diagram illustrating a specific example of vertical, horizontal, upper right, and upper left difference calculation processes regarding long exposure pixels executed by the difference calculation units 161 to 164.
(Lv) Long exposure vertical direction,
(Lh) Long exposure horizontal direction,
(La) Long exposure upper right direction,
(Ld) Long exposure upper left direction,
In each of these processes, a combination (pixel pair) of pixels for which a difference is calculated is indicated by an arrow.

The pixel value differences of a plurality of pixel pairs are calculated, and the average value is calculated to calculate the following values.
(1) Long exposure pixel vertical direction pixel value difference average value (LDiff_v),
(2) Long exposure pixel horizontal direction pixel value difference average value (LDiff_h),
(3) Long exposure pixel upper right direction pixel value difference average value (LDiff_a),
(4) Long exposure pixel upper left direction pixel value difference average value (LDiff_d),
The calculation formula of these values is as shown in the following (Formula 1).

... (Formula 1)
In the above formula, N, M. O and P are the calculated numbers of differences.

FIG. 10 is a diagram illustrating a specific example of vertical, horizontal, upper right, and upper left difference calculation processes relating to short exposure pixels executed by the difference calculation units 161 to 164.
(Sv) Short exposure vertical direction,
(Sh) horizontal direction of short exposure,
(Sa) Short exposure upper right direction,
(Sd) Short exposure upper left direction,
In each of these processes, a combination (pixel pair) of pixels for which a difference is calculated is indicated by an arrow.

The pixel value differences of a plurality of pixel pairs are calculated, and the average value is calculated to calculate the following values.
(1) Short exposure pixel vertical direction pixel value difference average value (SDiff_v),
(2) Short exposure pixel horizontal direction pixel value difference average value (SDiff_h),
(3) Short exposure pixel upper right direction pixel value difference average value (SDiff_a),
(4) Short exposure pixel upper left direction pixel value difference average value (S) Diff_d),
The calculation formulas for these values are as shown in (Formula 2) below.

... (Formula 2)
In the above formula, Q, R. S and T are the calculated numbers of differences.

  The difference calculation unit (vertical) 161, the difference calculation unit (horizontal) 162, the difference calculation unit (upper right) 163, and the difference calculation unit (upper left) 164 shown in FIG. A pixel value difference average value in four directions is calculated by applying the pixels. This calculated value is output to the long exposure edge detection unit 165, the long exposure edge direction determination unit 166, the short exposure edge detection unit 167, and the short exposure edge direction determination unit 168.

The long exposure edge detection unit 165 and the long exposure edge direction determination unit 166 input the pixel value difference average value in the four directions of the long exposure G pixel, and use the pixel value difference average value in each direction of the long exposure pixel, The presence / absence of an edge determined from the long exposure pixel and the edge direction are determined.
Further, the short exposure edge detection unit 167 and the short exposure edge direction determination unit 168 receive the pixel value difference average values in the four directions of the short exposure G pixels and use the pixel value difference average values in the respective directions of the short exposure pixels. Thus, the presence / absence of the edge and the edge direction determined from the short exposure pixel are determined.

  The long exposure edge detection unit 165 calculates a long exposure edge determination flag (LEEdge_flg) 171 indicating the presence / absence of an edge determined from the long exposure pixel according to the following (Equation 3).

... (Formula 3)

In the above (Formula 3),
TH is a predetermined threshold value.
The long exposure edge determination flag (LEEdge_flg) 171 is
When LEEdge_flg = 1, this indicates that the pixel of interest is an edge region,
When LEEdge_flg = 0, it indicates that the pixel of interest is not an edge region, that is, a flat region.

  The long exposure edge direction determination unit 166 calculates a long exposure edge direction flag (LDir_flg) 172 indicating the direction of the edge determined from the long exposure pixel according to the following (Equation 4).

(Formula 4)

In the above (Formula 4),
min {a, b, c, d} is a function for selecting the minimum value from a, b, c, d.

The long exposure edge direction flag (LDir_flg) 172 indicates the following edge direction according to each value 0 to 3 defined by (Equation 4).
LDir_flg = 0: the edge direction is the vertical direction,
LDir_flg = 1: the edge direction is the horizontal direction,
LDir_flg = 2: the edge direction is the upper left direction,
LDir_flg = 3: The edge direction is the upper right direction,

  Further, the short exposure edge detection unit 167 calculates a short exposure edge determination flag (SEdge_flg) 173 indicating the presence / absence of an edge determined from the short exposure pixel according to the following (Equation 5).

... (Formula 5)

In the above (Formula 5),
TH is a predetermined threshold value.
The short exposure edge determination flag (SEdge_flg) 173 is
When SEedge_flg = 1, it indicates that the target pixel is an edge region,
When SEedge_flg = 0, it indicates that the pixel of interest is not an edge region, that is, a flat region.

  The short exposure edge direction determination unit 168 calculates a long exposure edge direction flag (SDir_flg) 174 indicating the direction of the edge determined from the short exposure pixel according to the following (Equation 6).

... (Formula 6)

In the above (formula 6),
min {a, b, c, d} is a function for selecting the minimum value from a, b, c, d.

The short exposure edge direction flag (SDir_flg) 174 indicates the following edge direction according to each value 0 to 3 defined by the above (Expression 6).
SDir_flg = 0: the edge direction is the vertical direction,
SDir_flg = 1: Edge direction is horizontal,
SDir_flg = 2: the edge direction is the upper left direction,
SDir_flg = 3: The edge direction is the upper right direction,

Thus, the edge detection unit 151 is. The following information corresponding to the target pixel is generated as edge information and output to the centroid correction unit 152 and the pixel synthesis unit 153.
(1) A long exposure edge flag (LEEdge_flg) 171 indicating whether or not the target pixel determined based on the long exposure pixel is an edge region.
(2) Long exposure edge direction flag (LDir_flg) 172 indicating the edge direction of the target pixel determined based on the long exposure pixel
(3) A short exposure edge flag (SEedge_flg) 173 indicating whether or not the target pixel determined based on the short exposure pixel is an edge region.
(4) A short exposure edge direction flag (SDir_flg) 174 indicating the edge direction of the target pixel determined based on the short exposure pixel.

(2-2. Configuration and Processing of Center of Gravity Correction)
Next, the configuration and processing of the gravity center correction unit 152 will be described with reference to FIG.
FIG. 11 is a block diagram illustrating a detailed configuration of the gravity center correction unit 152.

The HDR image generation unit 131 illustrated in FIG. 7 combines the long exposure pixels and the short exposure pixels included in the input image (SVE array image) 141 illustrated in FIG. 7 which is an output image from the image sensor 102, and is an output image. One output pixel value of the HDR image 142 is calculated.
For example, the HDR image 142 that is the output image after the synthesis process is set to halve the number of pixels in the horizontal and vertical directions of the input image (SVE array image) 141 that is the input image before the synthesis process.

As shown in FIG. 12, the long exposure pixel and the short exposure pixel to be combined are at different pixel positions, and photographic pixel values at different subject positions are combined.
When such processing is performed, for example, jaggies in which boundaries such as edges are set in a staircase shape or aliasing such as false colors may occur.
In order to suppress the occurrence of such jaggy and aliasing, it is effective to perform adjustment so that the pixel positions of the long exposure pixel and the short exposure pixel to be combined are the same pixel position. This process is called center of gravity correction.

  The vertical centroid correction unit 181 of the centroid correction unit 152 illustrated in FIG. 11 has each pixel position when the pixel positions are the same for the long exposure pixels and the short exposure pixels included in the input image (SVE array image) 141. Pixel values are calculated and output to the correction result selection unit 183 as “long exposure pixel with center of gravity correction (L_w)” and “short exposure pixel with center of gravity correction (S_w)”.

The vertical centroid correction unit 181 of the centroid correction unit 152 executes processing for matching the pixel positions of the long-time exposure pixel and the short-time exposure pixel that are to be combined in this way. In particular,
The pixel value of the long-time exposure pixel output by the centroid correction unit is calculated by weighted addition using the pixel values of the plurality of long-time exposure pixels of the same color at different pixel positions.
In addition, a process of calculating the pixel value of the centroid correction unit output short-time exposure pixel is performed by weighted addition using the pixel values of a plurality of short-time exposure pixels of the same color at different pixel positions.

For example, processing as shown in FIG. 13 is performed.
FIG. 13 shows the following processing examples.
(A) Long-exposure pixel centroid correction processing;
(B) Short-exposure pixel centroid correction processing;
The processing example illustrated in FIG. 13 illustrates a processing example when the target pixel to be processed is the long exposure pixel in the fifth, sixth, ninth, and tenth rows in the input image (SVE array image).

Note that, as described above, each pixel of the HDR image 142 finally generated by the HDR image generation unit 131 illustrated in FIG. 7 is at least one long exposure pixel and short exposure pixel in the input image (SVE array image) 141. Is generated by combining pixel values of a plurality of pixels.
Accordingly, the HDR image generation unit 131 may be configured to sequentially select only the long exposure pixels in the input image (SVE array image) as the processing target pixel (target pixel). This is because the short exposure pixel is automatically selected as a pixel to be applied to synthesis with the long exposure pixel selected as the processing target pixel (target pixel).

The example illustrated in FIG. 13 illustrates the centroid correction process when the long exposure pixels in the fifth, sixth, ninth, and tenth rows in the input image (SVE array image) are selected as the processing target pixel (target pixel). In particular,
(A) Long exposure pixel with center of gravity correction (L_w),
(B) Short exposure pixel with center of gravity correction (S_w),
An example of the center of gravity correction processing of each of these exposure pixels is shown.

For example, in the long-exposure pixel centroid correction process shown in FIG. 13A, the following process is executed.
If any of the GRGR pixels in the fifth row in the input image (SVE array image) is the target pixel, the pixel value of the long exposure pixel (GRGR) at the pixel position in the first row of the input image (SVE array image) And the pixel value of the long-exposure pixel (GRGR) in the fifth row and the pixel value of the long-exposure pixel (GRGR) in the ninth row are added and averaged at 1: 6: 1, thereby obtaining FIG. ) The pixel value of each pixel (GRGR) in the uppermost stage (corresponding to the fifth row) of the long-time exposure gravity center corrected image is calculated.
In FIG. 13 (a2), the pixel value of the center-of-gravity correction pixel corresponds to the pixel value corresponding to which pixel position (the x-th row) in the input image. It shows.

  Further, when any of the BGBG pixels in the sixth row in the input image (SVE array image) is the target pixel, the long exposure pixel (BGBG) at the pixel position in the sixth row of the input image (SVE array image). The pixel value and the pixel value of the long exposure pixel (BGBG) in the tenth row are added and averaged at 3: 1, so that (a2) each second pixel (corresponding to the seventh row) of the long-time exposure centroid-corrected image The pixel value of (BGBG) is calculated.

Similarly, when any of the GRGR pixels in the ninth row in the input image (SVE array image) is the target pixel, the fifth row in the input image (SVE array image) is the same as the processing in the fifth row. The pixel value of the long exposure pixel (GRGR) at the pixel position, the pixel value of the long exposure pixel (GRGR) in the ninth row, and the pixel value of the long exposure pixel (GRGR) in the thirteenth row are 1: 6. The pixel value of each pixel (GRGR) of the third (equivalent to the 9th row) of the long-time exposure center-of-gravity correction image in FIG.
When any of the BGBG pixels in the 10th row in the input image (SVE array image) is the target pixel, the pixels in the 10th row of the input image (SVE array image) are the same as in the process in the 6th row. The pixel value of the long exposure pixel (BGBG) at the position and the pixel value of the long exposure pixel (BGBG) in the 14th row are added and averaged at a ratio of 3: 1. The pixel value of each pixel (BGBG) in the (11th row) is calculated.

In FIG. 13 (a2) long-time exposure centroid correction image,
When the pixel to be processed (target pixel) is a long exposure pixel in the fifth, sixth, ninth, and tenth rows of the input image (SVE array image), the pixel value of the center-of-gravity correction pixel is x line)).
When the processing target pixel (target pixel) is the GRGR pixel in the fifth row of the input image, the pixel position of the gravity center corrected image is the position of the fifth row in the input image.
When the pixel to be processed (target pixel) is the BGBG pixel in the sixth row of the input image, the pixel position of the center-of-gravity correction image is the position of the seventh row in the input image.
When the processing target pixel (target pixel) is the GRGR pixel in the ninth row of the input image, the pixel position of the gravity center corrected image is the position of the ninth row in the input image.
When the processing target pixel (target pixel) is the BGBG pixel in the 10th row of the input image, the pixel position of the gravity center corrected image is the 11th row position in the input image.
As described above, the center-of-gravity correction pixels are calculated as pixel values corresponding to pixels arranged at equal intervals in the vertical direction in the input image.

On the other hand, in the center of gravity correction processing of the short exposure pixel in FIG. 13B, the following processing is executed.
When any of the GRGR pixels in the fifth row in the input image (SVE array image) is the target pixel, the pixel value of the short exposure pixel (GRGR) at the pixel position in the third row in the input image (SVE array image) And the pixel value of the pixel value of the short exposure pixel (GRGR) in the seventh row are added and averaged by 1: 1, whereby the uppermost stage (corresponding to the fifth row) of FIG. ) For each pixel (GRGR).

  Further, when any of the BGBG pixels in the sixth row in the input image (SVE array image) is the target pixel, the short exposure pixel (BGBG) at the pixel position in the fourth row in the input image (SVE array image). The pixel value and the pixel value of the short exposure pixel (BGBG) in the eighth row are added and averaged by 1: 3, whereby (b2) each second pixel (corresponding to the seventh row) of the short-time exposure centroid-corrected image The pixel value of (BGBG) is calculated.

Similarly, when any of the GRGR pixels in the ninth row in the input image (SVE array image) is the target pixel, the seventh row in the input image (SVE array image) is the same as the processing in the fifth row. The pixel value of the short-exposure pixel (GRGR) at the pixel position and the pixel value of the short-exposure pixel (GRGR) in the eleventh row are added and averaged 1: 1, thereby FIG. A pixel value of each third pixel (corresponding to the ninth row) of the image (GRGR) is calculated.
When any of the BGBG pixels in the 10th row in the input image (SVE array image) is the target pixel, the pixels in the 8th row of the input image (SVE array image) are the same as in the process in the 6th row. The pixel value of the short-exposure pixel (BGBG) at the position and the pixel value of the short-exposure pixel (BGBG) in the twelfth row are added and averaged by 1: 3. The pixel value of each pixel (BGBG) in the (11th row) is calculated.

In FIG. 13 (b2) the short-time exposure centroid correction image,
When the pixel to be processed (target pixel) is a long exposure pixel in the fifth, sixth, ninth, and tenth rows of the input image (SVE array image), the pixel value of the center-of-gravity correction pixel is x line)).
When the processing target pixel (target pixel) is the GRGR pixel in the fifth row of the input image, the pixel position of the gravity center corrected image is the position of the fifth row in the input image.
When the pixel to be processed (target pixel) is the BGBG pixel in the sixth row of the input image, the pixel position of the center-of-gravity correction image is the position of the seventh row in the input image.
When the processing target pixel (target pixel) is the GRGR pixel in the ninth row of the input image, the pixel position of the gravity center corrected image is the position of the ninth row in the input image.
When the processing target pixel (target pixel) is the BGBG pixel in the 10th row of the input image, the pixel position of the gravity center corrected image is the 11th row position in the input image.
As described above, the center-of-gravity correction pixels are calculated as pixel values corresponding to pixels arranged at equal intervals in the vertical direction in the input image.

  As understood from the long-exposure and short-exposure center-of-gravity correction images shown in FIGS. 13A2 and 13B2, the pixel values of the pixels of each center-of-gravity correction image are the fifth, seventh, ninth, This corresponds to the pixel value at the position corresponding to the eleventh row.

As described above, the same row, for example, the fifth row of the long-time exposure centroid correction image shown in FIG. 13 (a2) and the short-time exposure centroid correction image shown in FIG. By combining these, processing equivalent to the combining processing of the short exposure pixels and the long exposure pixels at the same subject position is performed.
This synthesis result is the synthesis result shown in FIG.

  FIGS. 14A to 14D show processing examples of the vertical centroid correction unit 181 when the processing target pixel (target pixel) is a long exposure pixel corresponding to each of the fifth, sixth, ninth, and tenth rows of the input image. FIG. The specific processing of the pixel value calculation processing of the center-of-gravity correction pixels of the long exposure pixel and the short exposure pixel is shown.

(1) Processing when the long-time exposure pixel in the fifth row of the input image is the processing target pixel (target pixel) is as follows.
(1L) The pixel value of the long-time exposure centroid correction pixel is the ratio of the pixel values of the long exposure pixels in the first row, the fifth row, and the ninth row of the input image (SVE array image) at a ratio of 1: 6: 1. Calculate by adding.
(1S) The pixel value of the short-time exposure centroid correction pixel is calculated by adding the pixel values of the short-exposure pixels in the third row and the seventh row of the input image (SVE array image) at a ratio of 1: 1.

(2) Processing when the long-time exposure pixel in the sixth row of the input image is the processing target pixel (target pixel) is as follows.
(2L) The pixel value of the long-time exposure centroid correction pixel is calculated by adding the pixel values of the long exposure pixels in the sixth and tenth rows of the input image (SVE array image) at a ratio of 3: 1.
(2S) The pixel value of the short-time exposure centroid correction pixel is calculated by adding the pixel values of the short-exposure pixels in the fourth and eighth rows of the input image (SVE array image) at a ratio of 1: 3.

(3) Processing when the long-time exposure pixel in the ninth row of the input image is set as a processing target pixel (target pixel) is as follows.
(3L) The pixel value of the long-time exposure centroid correction pixel is the ratio of the pixel values of the long exposure pixels in the fifth row, the ninth row, and the thirteenth row of the input image (SVE array image) at a ratio of 1: 6: 1. Calculate by adding.
(3S) The pixel value of the short-time exposure centroid correction pixel is calculated by adding the pixel values of the short-exposure pixels in the seventh row and the eleventh row of the input image (SVE array image) at a ratio of 1: 1.

(4) Processing when the long-time exposure pixel in the tenth row of the input image is the processing target pixel (target pixel) is as follows.
(4L) The pixel value of the long-time exposure centroid correction pixel is calculated by adding the pixel values of the long exposure pixels in the 10th and 14th rows of the input image (SVE array image) at a ratio of 3: 1.
(4S) The pixel value of the short-time exposure centroid correction pixel is calculated by adding the pixel values of the short-exposure pixels in the eighth and twelfth rows of the input image (SVE array image) at a ratio of 1: 3.

  In this way, the respective pixel values when it is assumed that the long exposure pixel and the short exposure pixel are set at the same pixel position of the input image are calculated as the long exposure centroid correction pixel and the short exposure centroid correction pixel.

The vertical centroid correction unit 181 of the centroid correction unit 152 illustrated in FIG. 11 executes the processing described with reference to FIGS. 13 and 14 to generate the following data as illustrated in FIG.
Long exposure pixel with center of gravity correction (L_w),
Short exposure pixel with center of gravity correction (S_w),
The vertical centroid correction unit 181 calculates each piece of pixel value information and outputs it to the correction result selection unit 183.
Each of these pixel values is generated as centroid correction pixel value data corresponding to one long exposure pixel selected as one processing target pixel (target pixel), and is output to the correction result selection unit 183.

The correction result selection unit 183 further inputs the following pixel information included in the input image (SVE array image) 141.
Long exposure pixel without center of gravity correction (L_n)
Short exposure pixel without center of gravity correction (S_n)
These are pixel information of the input image (SVE array image) 141 itself.

The correction result selection unit 183 further receives a saturation flag (Sat_flg) 185 generated by the saturation determination unit 182.
The saturation determination unit 182 determines whether or not the pixel value of the long exposure pixel included in the input image (SVE array image) 141 is saturated, and indicates whether or not each long exposure pixel is saturated. (Sat_flg) 185 is set and output.

For example, if the pixel value of the long exposure pixel is the maximum value, it is determined that the pixel is saturated, and the value of the saturation flag (Sat_flg) 185 corresponding to the long exposure pixel is set to 1, that is,
Sat_flg = 1
Output as.
When the pixel value of the long exposure pixel is not saturated, the value of the saturation flag (Sat_flg) 185 corresponding to the long exposure pixel is set to 0, that is,
Sat_flg = 0
Output as.
The saturation flag (Sat_flg) 185 generated by the saturation determination unit 182 is output to the correction result selection unit 183 of the centroid correction unit 152, and is also output to the pixel synthesis unit 153 of the HDR image generation unit 131 shown in FIG. .

Processing of the correction result selection unit 183 of the gravity center correction unit 152 illustrated in FIG. 11 will be described.
The correction result selection unit 183 inputs the following data.
(1) Long exposure pixel without center of gravity correction (L_n)
(2) Short exposure pixel without center of gravity correction (S_n)
(3) Long exposure pixel with center of gravity correction (L_w),
(4) Short exposure pixel with center of gravity correction (S_w),
(5) Saturation flag (Sat_flg) 185

(1) Long exposure pixel without center of gravity correction (L_n), (2) Short exposure pixel without center of gravity correction (S_n), and these data are pixel information included in the input image (SVE array image) 141.
(3) Long exposure pixel with center-of-gravity correction (L_w), (4) Short exposure pixel with center-of-gravity correction (S_w), these data are obtained according to the processing described with reference to FIGS. The data generated by
(5) A saturation flag (Sat_flg) 185 is the saturation determination information of the long exposure pixel generated by the saturation determination unit 182.

  The correction result selection unit 183 includes a pixel whose saturation flag (Sat_flg) 185 is 1, that is, a saturated pixel, in the long exposure pixel applied to the generation of the “long exposure pixel with centroid correction (L_w)” in the vertical centroid correction unit 181. It is determined whether or not.

  The “long exposure pixel with centroid correction (L_w)” generated for the processing target pixel (target pixel) with the vertical centroid correction unit 181 is generated by the process of applying the long exposure pixel including the saturated pixel in the input image. In this case, the “long exposure pixel with center of gravity correction (L_w)” and “short exposure pixel with center of gravity correction (S_w)” generated for this processing target pixel (target pixel) are output pixels of the center of gravity correction unit 152. Do not select.

In this case, the pixel information of the processing target pixel (target pixel) itself, that is, the “long exposure pixel without center of gravity correction (L_n)” in the input image, and the processing target pixel (target pixel) when generating the HDR image are combined. The short exposure pixel “short exposure pixel without center of gravity correction (S_n)” is selected as the output of the center of gravity correction unit 152.
The “short exposure pixel without center of gravity correction (S_n)” is a short exposure pixel of the same color in the vicinity in the vertical direction with respect to the “long exposure pixel without center of gravity correction (L_n)” in the input image.

  On the other hand, when the saturated pixel is not included in the long exposure pixel applied to the generation of the “long exposure pixel with center of gravity correction (L_w)” generated by the vertical center of gravity correction unit 181, the “center of gravity correction generated by the vertical center of gravity correction unit 181 is used. “Long exposure pixel (L_w)” and “Short exposure pixel with center of gravity correction (S_w)” are output pixels of the center of gravity correction unit 152.

That is, the output of the center of gravity correction unit 152 shown in FIG.
(1) Center of gravity correction unit output length exposure pixel (Lo),
(2) Center of gravity correction unit output short exposure pixel (So),
Is selected and output as follows.

In the case where no saturated pixel is included in the long exposure pixel applied to the generation of “long exposure pixel with centroid correction (L_w)” in the vertical centroid correction unit 181,
(1) Center of gravity correction unit output long exposure pixel (Lo) = long exposure pixel with center of gravity correction (L_w),
(2) Center of gravity correction unit output short exposure pixel (So) = short exposure pixel with center of gravity correction (S_w),
When one or more saturated pixels are included in the long exposure pixel applied to the generation of the “long exposure pixel with centroid correction (L_w)” in the vertical centroid correction unit 181,
(1) Center of gravity correction unit output long exposure pixel (Lo) = long exposure pixel without center of gravity correction (L_n),
(2) Center of gravity correction unit output short exposure pixel (So) = short exposure pixel without center of gravity correction (S_n),
This is the setting.

Such selection output is executed for each processing target pixel (target pixel).
For example, when the processing target pixel (target pixel) is the long exposure pixel R shown in the fifth row of FIG. 14A, the following processing is performed.
In this case, the vertical center-of-gravity correction unit 181 applies a long exposure pixel R in the first row, a long exposure pixel R in the fifth row, and a long exposure pixel R in the ninth row to obtain a distribution ratio of 1: 6: 1. The weighted average is performed to calculate “long exposure pixel with center of gravity correction (L_w)”.
Similarly, the vertical centroid correction unit 181 applies a weighted average at a 1: 1 distribution ratio by applying the short exposure pixels R in the third row and the short exposure pixels R in the seventh row, and “short exposure with centroid correction” is performed. Pixel (S_w) "is calculated.

Here, at least one of the long exposure pixel R in the first row, the long exposure pixel R in the fifth row, and the long exposure pixel R in the ninth row applied to the generation of the “long exposure pixel with center of gravity correction (L_w)”. When one pixel is a saturated pixel, the correction result selection unit 183 calculates the “long exposure pixel with center of gravity correction (L_w)” and “short exposure pixel with center of gravity correction (S_w)” calculated by the vertical center of gravity correction unit 181. Are not selected as output pixels.
In this case, the pixel information of the processing target pixel (target pixel) itself, that is, the “long exposure pixel without center of gravity correction (L_n)” in the input image, and the processing target pixel (target pixel) and the synthesis target when generating the HDR image. The short exposure pixel “Short exposure pixel without center of gravity correction (S_n)” is selected and used as the output of the center of gravity correction unit 152.

  On the other hand, all of the long exposure pixels R in the first row, the long exposure pixels R in the fifth row, and the long exposure pixels R in the ninth row applied to the generation of the “long exposure pixel with center of gravity correction (L_w)” are saturated. When the pixel is not included, the correction result selection unit 183 calculates the “long exposure pixel with center of gravity correction (L_w)” and “short exposure pixel with center of gravity correction (S_w)” calculated by the vertical center of gravity correction unit 181. Output.

As described above, when the long exposure pixel applied to the generation of the “long exposure pixel with centroid correction (L_w)” in the vertical centroid correction unit 181 does not include any saturated pixel,
(1) Center of gravity correction unit output long exposure pixel (Lo) = long exposure pixel with center of gravity correction (L_w),
(2) Center of gravity correction unit output short exposure pixel (So) = short exposure pixel with center of gravity correction (S_w),
These pixel values are output to the pixel composition unit 153 of the HDR image generation unit 131 shown in FIG.

Further, when one or more saturated pixels are included in the long exposure pixel applied to the generation of “long exposure pixel with centroid correction (L_w)” in the vertical centroid correction unit 181,
(1) Center of gravity correction unit output long exposure pixel (Lo) = long exposure pixel without center of gravity correction (L_n),
(2) Center of gravity correction unit output short exposure pixel (So) = short exposure pixel without center of gravity correction (S_n),
This is the setting.
These pixel values are output to the pixel composition unit 153 of the HDR image generation unit 131 shown in FIG.

Furthermore, the center-of-gravity correction unit 152 also outputs a saturation flag (Sat_flg) 185 generated by the saturation determination unit 182 shown in FIG. 11 to the pixel composition unit 153 of the HDR image generation unit 131 shown in FIG.
The saturation flag (Sat_flg) 185 is information indicating whether or not the long exposure pixel in the input image (SVE array image) is saturated.

(2-3. Configuration and processing of pixel composition unit)
Next, the configuration and processing of the pixel composition unit 153 of the HDR image generation unit 131 illustrated in FIG. 7 will be described with reference to FIG.

FIG. 15 is a block diagram illustrating a detailed configuration of the pixel synthesis unit 153.
As illustrated in FIG. 15, the pixel synthesis unit 153 includes a flat part synthesis unit 211, an edge part synthesis unit 212, a saturation unit synthesis unit 213, and a long / short synthesis result selection & final synthesis result generation unit 214.

Each of the flat part synthesizing unit 211, the edge part synthesizing unit 212, and the saturation unit synthesizing unit 213 is an output signal generated by the centroid correcting unit 152 described above with reference to FIG.
(1) Center of gravity correction unit output length exposure signal (Lo),
(2) Center of gravity correction unit output short exposure signal (So),
These signals are combined in different manners, and the generated combined signal is output to the long / short combining result selection & final combining result generating unit 214.

The flat part synthesizing unit 211 generates a flat part synthesis result (Syn_f), which is a synthesized signal for the flat part, and outputs the result to the long / short synthesis result selection & final synthesis result generation unit 214.
The edge part synthesis unit 212 generates an edge part synthesis result (Syn_e) that is a synthesized signal for the edge part, and outputs the result to the long / short synthesis result selection & final synthesis result generation unit 214.
The saturation unit synthesis unit 213 generates a saturation unit synthesis result (Syn_s) that is a synthesized signal for the saturation unit, and outputs the result to the long / short synthesis result selection & final synthesis result generation unit 214.

The long / short synthesis result selection & final synthesis result generation unit 214 inputs the synthesis result of the long / short exposure pixels from these three synthesis units.
Further, from the edge detection unit 151 described with reference to FIG.
Long exposure edge flag (LEedge_flg) 171,
Short exposure edge flag (SEdge_flg) 173,
The edge information is input.
Furthermore, from the center of gravity correction unit 152 described with reference to FIG.
Saturation flag (Sat_flg) 185,
Enter this saturation information.

The long / short synthesis result selection & final synthesis result generation unit 214
(A) Long exposure edge flag (LEedge_flg) 171,
(B) a short exposure edge flag (SEdge_flg) 173;
(C) Saturation flag (Sat_flg) 185,
With reference to these pieces of information, a combination result of one of the following long and short exposure pixels is selected.
That is,
(1) Flat part synthesis result (Syn_f) which is a synthesized signal for the flat part,
(2) Edge portion synthesis result (Syn_e) which is a synthesized signal for the edge portion,
(3) Saturated portion synthesis result (Syn_s) which is a synthesized signal for the saturated portion,
The long / short synthesis result selection & final synthesis result generation unit 214 selects one of these three signals for each processing target pixel (target pixel).

  Further, based on the combination result of the selected long and short exposure pixels, horizontal pixel combination is performed to generate a final combination result, which is output as a combination result (Syn).

The processing of each component will be described.
The flat part synthesizing unit 211 receives the output signal generated by the centroid correcting unit 152 described with reference to FIG. 11 and generates a synthetic signal corresponding to the flat part according to these signals. That is,
(1) Center of gravity correction unit output length exposure signal (Lo),
(2) Center of gravity correction unit output short exposure signal (So),
A flat portion synthesis result (Syn_f) is generated for each of these signals to generate a flat portion synthesis result (Syn_f), which is output to the long / short synthesis result selection & final synthesis result generation unit 214.

The flat part synthesizing unit 211 receives an input signal, that is,
(1) Center of gravity correction unit output length exposure signal (Lo),
(2) Center of gravity correction unit output short exposure signal (So),
By applying a blend ratio according to the signal level of each of these signals, a flat portion as a composite result is obtained by blending (combining) the pixel values of these two pixels, specifically, the two pixel values after adjusting the exposure ratio. A synthesis result (Syn_f) is generated.

When the ratio of the exposure time of the short exposure pixels and the long exposure pixels that are constituent pixels of the input image (SVE array image) 141 photographed by the image sensor 102 is 1: E,
Each pixel value of the long exposure pixel and the short exposure pixel to be combined, that is,
(1) Center of gravity correction unit output length exposure signal (Lo),
(2) Center of gravity correction unit output short exposure signal (So),
These pixel values are in the relationship of the following relational expression (formula 7).
Lo≈So × E (Expression 7)

In a flat part, that is, a pixel region in which the pixel value varies little in a certain region,
If the pixel value is small, i.e.
The center of gravity correction unit output length exposure signal (Lo) which is the pixel value of the long exposure pixel,
The center of gravity correction unit output short exposure signal (So), which is the pixel value of the short exposure pixel,
If any of these pixel values are small,
The amount of noise included in the short exposure pixel is higher than the signal value. That is, there is a high possibility that the pixel value of the short exposure pixel is a signal with a poor SN ratio.
Accordingly, in such a case, a synthesis process (blend process) in which the blend ratio of the barycenter correction unit output length exposure signal (Lo), which is the pixel value of the long exposure pixel, is increased to execute the flat part synthesis result (Syn_f). Generate.

On the other hand, when the pixel value is large, that is,
The center of gravity correction unit output length exposure signal (Lo) which is the pixel value of the long exposure pixel,
The center of gravity correction unit output short exposure signal (So), which is the pixel value of the short exposure pixel,
If any of these pixel values are large,
The long exposure pixel is likely to be a saturated value or a pixel value close to the saturated value, and is more likely not to reflect an accurate pixel value.
Therefore, in such a case, a synthesis process (blend process) in which the blend ratio of the center-of-gravity correction unit output short exposure signal (So), which is the pixel value of the short exposure pixel, is increased to execute the flat part synthesis result (Syn_f). Generate.

A value corresponding to the contribution ratio of the centroid correction unit output short exposure signal (So) to the flat portion synthesis result (Syn_f) that is a pixel value generated as a result of the synthesis processing executed in the flat portion synthesis unit 211 is a blend rate (α). And
In addition, a value corresponding to the contribution ratio of the gravity center correction unit output length exposure signal (Lo) is defined as (1-α).

When this setting is made, the combined pixel value (Syn_f), which is the flat portion combining result calculated by the combining processing in the flat portion combining unit 211, is calculated by the following (Equation 8).
Syn_f = (1−α) × Lo + α × So × E (Equation 8)

In (Equation 8) above,
E is the exposure ratio between short exposure pixels and long exposure pixels,
α is a blend ratio that takes values from 0 to 1.
α varies depending on the values of Lo and So × E.
FIG. 16 is a diagram showing an example of setting the blend rate α.
The horizontal axis represents the pixel value of the center-of-gravity correction unit output length exposure signal (Lo), which is the pixel value of the long exposure pixel, and the vertical axis represents the blend ratio (α), that is, the contribution of the short exposure signal (So) in the combined pixel value. Rate.

As understood from FIG. 16 and (Equation 8), when the pixel value of the centroid correction unit output length exposure signal (Lo), which is a long exposure pixel, is large, the blend rate α is set large. When the blend ratio α is set to be large, in the calculation of the composite pixel value (Syn_h) according to (Equation 8), the reflection degree of the pixel value of the short-time exposure pixel becomes high.
That is, for example, when the pixel value of the long-time exposure pixel is close to the saturation pixel value, the pixel value of the short-time exposure pixel is greatly reflected.

On the other hand, when the pixel value of the gravity center correction unit output length exposure signal (Lo) is small, the blend rate α is set to a small value. When the blend rate α is set to be small, the reflection degree of the pixel value of the long-time exposure pixel increases as understood from (Equation 8).
That is, for example, when the pixel value of the short-time exposure pixel is small and the noise contained in the short-time exposure pixel is estimated to be large, the pixel value of the long-time exposure pixel is largely reflected.

  By executing such pixel synthesis, it is possible to set pixel values with high accuracy and low noise from low luminance to high luminance.

Next, the process of the edge part synthetic | combination part 212 is demonstrated.
The edge portion synthesis unit 212 receives the output signal generated by the centroid correction unit 152 described with reference to FIG. 11 and generates a synthesized signal corresponding to the edge portion according to these signals. That is,
(1) Center of gravity correction unit output length exposure signal (Lo),
(2) Center of gravity correction unit output short exposure signal (So),
An edge portion synthesis process (Syn_e) is generated on each of these signals to generate an edge portion synthesis result (Syn_e), which is output to the long / short synthesis result selection & final synthesis result generation unit 214.

The edge synthesis unit 212 receives an input signal, that is,
(1) Center of gravity correction unit output length exposure signal (Lo),
(2) Center of gravity correction unit output short exposure signal (So),
Regardless of the signal level of each of these signals, blend (synthesize) by applying a blend ratio (β) in which the pixel values of these two pixels, specifically, the two pixel values after adjusting the exposure ratio are fixed. To generate an edge synthesis result (Syn_e) as a synthesis result.

A value corresponding to the contribution ratio of the centroid correction unit output short exposure signal (So) to the edge portion synthesis result (Syn_e) which is a pixel value generated as a result of the synthesis processing executed in the edge portion synthesis unit 212 is a blend rate (β). And
In addition, a value corresponding to the contribution ratio of the gravity center correction unit output length exposure signal (Lo) is defined as (1-β).

When this setting is made, the combined pixel value (Syn_e), which is the edge portion combining result calculated by the combining processing in the edge portion combining unit 212, is calculated by the following (Equation 9).
Syn_e = (1−β) × Lo + β × So × E (Equation 9)

In (Equation 9) above,
E is the exposure ratio between short exposure pixels and long exposure pixels,
β is a value in the range of 0 to 1, and is a preset fixed value. For example,
β = 0.5
And

Next, the processing of the saturation unit combining unit 213 will be described.
The saturation unit synthesis unit 213 receives the output signal generated by the centroid correction unit 152 described with reference to FIG. 11 and generates a synthesis signal corresponding to the saturation unit according to these signals. That is,
(1) Center of gravity correction unit output length exposure signal (Lo),
(2) Center of gravity correction unit output short exposure signal (So),
A saturation part synthesis process is executed on each of these signals to generate a saturation part synthesis result (Syn_s) and output it to the long / short synthesis result selection & final synthesis result generation unit 214.

The saturation unit synthesis unit 213 receives an input signal, that is,
(1) Center of gravity correction unit output length exposure signal (Lo),
(2) Center of gravity correction unit output short exposure signal (So),
Of these signals, the center of gravity correction unit output short exposure signal (So) is selected, and the selected center of gravity correction unit output short exposure signal (So) is used as the saturated portion synthesis result (Syn_s) to select the long / short synthesis result and generate the final synthesis result. To the unit 214.

That is, the synthesized pixel value (Syn_s) that is the saturation portion synthesis result (Syn_s) is calculated by the following (Equation 10).
Syn_s = So × E (Equation 10)
In the above (Formula 10),
E is the exposure ratio between short exposure pixels and long exposure pixels,
It is.

The long / short synthesis result selection & final synthesis result generation unit 214 inputs the synthesis results from the three synthesis units 211 to 213 described above.
Further, from the edge detection unit 151 described with reference to FIG.
Long exposure edge flag (LEedge_flg) 171,
Short exposure edge flag (SEdge_flg) 173,
The edge information is input.
Furthermore, from the center of gravity correction unit 152 described with reference to FIG.
Saturation flag (Sat_flg) 185,
Enter this saturation information.

The long / short synthesis result selection & final synthesis result generation unit 214
(A) Long exposure edge flag (LEedge_flg) 171,
(B) a short exposure edge flag (SEdge_flg) 173;
(C) Saturation flag (Sat_flg) 185,
With reference to these pieces of information, one of the following synthesis results is selected.
That is,
(1) Flat part synthesis result (Syn_f) which is a synthesized signal for the flat part,
(2) Edge portion synthesis result (Syn_e) which is a synthesized signal for the edge portion,
(3) Saturated portion synthesis result (Syn_s) which is a synthesized signal for the saturated portion,
The long / short synthesis result selection & final synthesis result generation unit 214 selects one of these three signals for each processing target pixel (target pixel).
Further, based on the combination result of the selected long and short exposure pixels, horizontal pixel combination is further performed to generate a final combination result, which is output as a combination result (Syn).

  The synthesis result (Syn) generation process executed by the long / short synthesis result selection & final synthesis result generation unit 214 will be described with reference to the flowchart shown in FIG.

First, in step S101, the composition result of each composition part 211-213 and each flag information are inputted. That is, the following data is input.
(1) Flat part synthesis result (Syn_f) which is a synthesized signal for the flat part,
(2) Edge portion synthesis result (Syn_e) which is a synthesized signal for the edge portion,
(3) Saturated portion synthesis result (Syn_s) which is a synthesized signal for the saturated portion,
(A) Long exposure edge flag (LEedge_flg) 171,
(B) a short exposure edge flag (SEdge_flg) 173;
(C) Saturation flag (Sat_flg) 185,

Next, in step S102, the saturation flag (Sat_flg) 185 input from the centroid correction unit 152 described with reference to FIG. 11 is referred to, and the centroid correction unit output length exposure pixel which is the target pixel (target pixel) of the synthesis process It is determined whether or not the generation source pixel is a saturated pixel.
If it is a saturated pixel, that is, if the determination in step S102 is True, the process proceeds to step S104.
In step S104, the saturation portion combination result (Syn_s), which is the combined signal for the saturated portion generated by the saturation portion combining portion 213, is selected and used as the combination result of the long and short exposure pixels. The combination result of the long and short exposure pixels is a result of combining the pixels in the vertical direction, that is, intermediate data of the combining process. Further, in step S107, the intermediate data is used to combine the pixels in the horizontal direction. Execute and generate and output the final synthesis result (Syn).

  On the other hand, if it is determined in step S102 that the pixel from which the center-of-gravity correction unit output length exposure pixel that is the target pixel (target pixel) of the synthesis process is generated is not a saturated pixel, that is, if the determination in step S102 is false. The process proceeds to step S103.

  In step S103, the long exposure edge flag (LEEdge_flg) 171 and the short exposure edge flag (SEedge_flg) 173 input from the edge detection unit 151 described with reference to FIG. It is determined whether the pixel from which the center-of-gravity correction unit output pixel is generated is in the flat region or the edge region.

In particular,
Long exposure edge flag (LEedge_flg) 171,
Short exposure edge flag (SEdge_flg) 173,
If any of these flags is a value (1) indicating an edge region, it is determined that the flag is an edge region, and the process proceeds to step S105.

on the other hand,
Long exposure edge flag (LEedge_flg) 171,
Short exposure edge flag (SEdge_flg) 173,
If any of these flags is a value (0) indicating that the region is not an edge region, it is determined that the region is a flat region, and the process proceeds to step S106.

If it is determined in step S103 that the pixel from which the center-of-gravity correction unit output pixel that is the target pixel (target pixel) of the synthesis process is generated is in the edge region, the process proceeds to step S105.
In step S105, the edge portion combination result (Syn_e), which is the combined signal for the edge portion generated by the edge portion combining portion 212, is selected and used as the combination result of the long and short exposure pixels. The combination result of the long and short exposure pixels is a result of combining the pixels in the vertical direction, that is, intermediate data of the combining process. Further, in step S107, the intermediate data is used to combine the pixels in the horizontal direction. Execute and generate and output the final synthesis result (Syn).

On the other hand, if it is determined in step S103 that the pixel from which the center-of-gravity correction unit output pixel that is the target pixel (target pixel) of the synthesis process is generated is in a flat region, the process proceeds to step S106.
In step S106, the flat portion combining result (Syn_f), which is the combined signal for the flat portion generated by the flat portion combining portion 211, is selected and used as the combining result of the long and short exposure pixels. The combination result of the long and short exposure pixels is a result of combining the pixels in the vertical direction, that is, intermediate data of the combining process. Further, in step S107, the intermediate data is used to combine the pixels in the horizontal direction. Execute and generate and output the final synthesis result (Syn).

In step S107, the pixel synthesis result in the horizontal direction is further performed using the pixel synthesis result in the vertical direction generated by the processing in any of steps S104 to S106, and the final synthesis result (Syn) is generated and output. .
In step S101 to S106 in the flow of FIG. 17, intermediate synthesized data is generated by synthesizing long and short exposure pixels in the vertical direction, and final synthesized result (Syn) by horizontal pixel synthesis of the constituent pixels of the intermediate synthesized data executed in step S107. The generation process will be described.

FIG. 18 shows the following three data.
(1) Input image (SVE) image (2) Long / short exposure pixel composition result (intermediate composition data)
(3) Final synthesis result (Syn)

(1) The input image (SVE) image is image data of the image sensor, and shows an image that is captured when the exposure time is changed for each row in the Bayer array.
A white part is a long-time exposure pixel, and a gray part indicates a short-time exposure pixel.

(2) The long / short exposure pixel combination result (intermediate combination data) is intermediate combination data generated by the processes of steps S101 to S106 in the flowchart shown in FIG.
This (2) long / short exposure pixel synthesis result (intermediate synthesis data) is the synthesis result of steps S104 to S106 depending on whether the target pixel to be processed is a flat region, an edge region, or a saturated region. From the selected pixels.
However, the synthesis result is the output of the center of gravity correction unit described with reference to FIG.
(1) Center of gravity correction unit output length exposure signal (Lo),
(2) Center of gravity correction unit output short exposure signal (So),
The result of combining these signals is a combination of a plurality of vertical pixels. That is, it is intermediate composite data.

In step S107 of the flow shown in FIG. 17, the same color pixels in the horizontal direction of the intermediate combined data are added and averaged to generate and output a final combined result (Syn).
By the processing in step S107, a high dynamic range (HDR) image in which the number of pixels is halved in both the horizontal direction and the vertical direction is generated and output with respect to the input image (SVE image).

[3. Example 2: Example in which the configuration of the gravity center correction unit is changed]
Next, an embodiment in which the configuration of the gravity center correction unit is changed will be described as a second embodiment of the image processing apparatus of the present disclosure.
The second embodiment differs from the first embodiment described above in the configuration and processing of the centroid correction unit. Other configurations and processes are the same as those in the first embodiment. For example, the overall configuration of the image processing apparatus is the imaging apparatus 100 described with reference to FIG. 3, and the configuration of the signal processing unit 103 is the HDR image generation unit 131, the subsequent signal processing unit, as described with reference to FIG. 132 and an output unit 133. As illustrated in FIG. 7, the HDR image generation unit 131 includes an edge detection unit 151, a centroid correction unit 152, and a pixel synthesis unit 153.

FIG. 19 shows the configuration of the centroid correction unit 152 of the HDR image generation unit 131 in the image processing apparatus of the second embodiment.
In the first embodiment described above, as described above with reference to FIG. 11, the correction result selection unit 183 is based on the saturation flag (Sat_flg) input from the saturation determination unit 182, and the processing target pixel (target pixel) It is determined whether or not the long exposure pixel is saturated, and if it is saturated, the constituent pixel value of the input image (SVE image) 141 is selected and output as it is.

That is, the center-of-gravity correction unit 152 illustrated in FIG. 11 according to the first embodiment does not include any saturated pixel in the long exposure pixel applied to the generation of the “long exposure pixel with center of gravity correction (L_w)” in the vertical center-of-gravity correction unit 181. ,
(1) Center of gravity correction unit output long exposure pixel (Lo) = long exposure pixel with center of gravity correction (L_w),
(2) Center of gravity correction unit output short exposure pixel (So) = short exposure pixel with center of gravity correction (S_w),
These pixel values are output to the pixel composition unit 153 of the HDR image generation unit 131 shown in FIG.
Further, when one or more saturated pixels are included in the long exposure pixel applied to the generation of “long exposure pixel with centroid correction (L_w)” in the vertical centroid correction unit 181,
(1) Center of gravity correction unit output long exposure pixel (Lo) = long exposure pixel without center of gravity correction (L_n),
(2) Center of gravity correction unit output short exposure pixel (So) = short exposure pixel without center of gravity correction (S_n),
These pixel values are output to the pixel composition unit 153 of the HDR image generation unit 131 shown in FIG.
Such processing was performed.

The center-of-gravity correction unit 152 of the second embodiment is different in processing when one or more saturated pixels are included in the long exposure pixel applied to the generation of “long exposure pixel with center of gravity correction (L_w)” in the vertical center-of-gravity correction unit 181. .
In the present embodiment, when one or more saturated pixels are included in the long exposure pixel applied to the generation of “long exposure pixel with center of gravity correction (L_w)” in the vertical centroid correction unit 181, the short exposure pixel is illustrated in FIG. 19. The smoothed short exposure pixel (S_d) generated by the saturated portion short exposure pixel correction unit 301 is selected and output as the gravity center correction unit output short exposure pixel (So).
The saturated portion short exposure pixel correction unit 301 generates a smoothed short exposure pixel (S_d) by a short exposure pixel smoothing process according to the edge direction of the target pixel.

That is, the center-of-gravity correction unit 152 illustrated in FIG. 19 according to the second embodiment includes at least one saturated pixel in the long exposure pixels that the vertical center-of-gravity correction unit 181 applies to the generation of “long exposure pixel with center of gravity correction (L_w)”. If not,
(1) Center of gravity correction unit output long exposure pixel (Lo) = long exposure pixel with center of gravity correction (L_w),
(2) Center of gravity correction unit output short exposure pixel (So) = short exposure pixel with center of gravity correction (S_w),
These pixel values are output to the pixel composition unit 153 of the HDR image generation unit 131 shown in FIG.
These processes are the same as those in the first embodiment.

However, when one or more saturated pixels are included in the long exposure pixel applied to the generation of the “long exposure pixel with centroid correction (L_w)” by the vertical centroid correction unit 181,
(1) Center of gravity correction unit output long exposure pixel (Lo) = long exposure pixel without center of gravity correction (L_n),
(2) Center of gravity correction unit output short exposure pixel (So) = smoothed short exposure pixel (S_d),
These pixel values are output to the pixel composition unit 153 of the HDR image generation unit 131 shown in FIG.

  The saturated short exposure pixel correction unit 301 executes a pixel value correction process to which a smoothing filter corresponding to the edge direction is applied, and corrects the pixel value as a result of the smoothing process as a smoothed short exposure pixel (S_d). The result is output to the result selection unit 183.

The saturated portion short exposure pixel correction unit 301 receives the short exposure edge direction flag (SDir_flg) 174 generated by the edge detection unit 151 described with reference to FIG.
As described above, the short exposure edge direction flag (SDir_flg) 174 is information indicating the edge direction of the target pixel determined based on the short exposure pixel.

The saturated portion short exposure pixel correction unit 301 is applied from a plurality of smoothing filters stored in advance in the memory according to the edge direction determined based on the short exposure edge direction flag (SDir_flg) 174 input from the edge detection unit 151. Select the smoothing filter to be used.
The saturated portion short exposure pixel correction unit 301 executes a pixel value correction process that applies the selected smoothing filter to smooth the short exposure pixels that are to be combined with the target pixel along the edge direction, and performs smoothing. The pixel value as the processing result is output to the correction result selection unit 183 as a smoothed short exposure pixel (S_d).

A configuration example of a smoothing filter corresponding to the edge direction used by the saturated portion short exposure pixel correction unit 301 is shown in FIG.
FIG. 20 shows an input image to which the smoothing filter is applied and an example of coefficient setting of the smoothing filter according to the four edge directions. That is, the following data are shown.
(1) Input image (SVE image)
(2a) Coefficient setting example of applied filter when edge direction is vertical (2b) Coefficient setting example of applied filter when edge direction is horizontal (2c) Coefficient setting of applied filter when edge direction is upper left direction Example (2d) Application filter coefficient setting example when the edge direction is the upper right direction

  The example shown in FIG. 20 is an example in which a 9 × 9 pixel area in which a target pixel as a processing target pixel is a central pixel (a central G pixel in the example shown in FIG. 20A) is a smoothing filter application area. It is. Further, the short exposure G pixel on the vertical line including the target pixel G is set as a correction target pixel for performing pixel value correction.

As shown in FIGS. 20 (2a) to (2d), for example, the smoothing filter applied to the smoothing process is a filter in which a coefficient corresponding to the edge direction is set for each short-exposure G pixel.
By applying these filters, the pixel value of the short exposure pixel to be corrected is changed to a smoothed pixel value so as to change smoothly in the edge direction.
Note that the example of the filter coefficient shown in FIG. 20 is an example, and other coefficients can be set. Any smoothing filter may be used as long as the pixel value is changed smoothly along the edge direction.

  The saturated portion short exposure pixel correction unit 301 selects a correction result using, for example, the pixel value of the short exposure pixel corrected by the smoothing process using the smoothing filter having the coefficient illustrated in FIG. 20 as the smoothed short exposure pixel (S_d). To the unit 183.

As illustrated in FIG. 19, the correction result selection unit 183 inputs the following data.
(1) Long exposure pixel without center of gravity correction (L_n)
(2) Smoothed short exposure pixel (S_d)
(3) Long exposure pixel with center of gravity correction (L_w),
(4) Short exposure pixel with center of gravity correction (S_w),
(5) Saturation flag (Sat_flg) 185

(1) The long exposure pixel (L_n) without centroid correction is pixel information included in the input image (SVE array image) 141.
(2) The smoothed short exposure pixel (S_d) is short exposure pixel information having a pixel value smoothed in the edge direction generated by the saturated portion short exposure pixel correction unit 301.
(3) Long exposure pixel with center-of-gravity correction (L_w), (4) Short exposure pixel with center-of-gravity correction (S_w), these data are obtained according to the processing described with reference to FIGS. The data generated by
(5) A saturation flag (Sat_flg) 185 is the saturation determination information of the long exposure pixel generated by the saturation determination unit 182.

  The correction result selection unit 183 includes a pixel whose saturation flag (Sat_flg) 185 is 1, that is, a saturated pixel, in the long exposure pixel applied to the generation of the “long exposure pixel with centroid correction (L_w)” in the vertical centroid correction unit 181. It is determined whether or not.

  The “long exposure pixel with centroid correction (L_w)” generated for the processing target pixel (target pixel) with the vertical centroid correction unit 181 is generated by the process of applying the long exposure pixel including the saturated pixel in the input image. In this case, the “long exposure pixel with center of gravity correction (L_w)” and “short exposure pixel with center of gravity correction (S_w)” generated for this processing target pixel (target pixel) are output pixels of the center of gravity correction unit 152. Do not select.

  In this case, the pixel information of the processing target pixel (target pixel) itself, that is, “long exposure pixel without center of gravity correction (L_n)” in the input image, and the edge direction generated by the saturated portion short exposure pixel correction unit 301 are smoothed. The “smoothed short exposure pixel (S_d)” having the pixel value thus selected is selected as the output of the centroid correction unit 152.

  On the other hand, when the saturated pixel is not included in the long exposure pixel applied to the generation of the “long exposure pixel with center of gravity correction (L_w)” generated by the vertical center of gravity correction unit 181, the “center of gravity correction generated by the vertical center of gravity correction unit 181 is used. “Long exposure pixel (L_w)” and “Short exposure pixel with center of gravity correction (S_w)” are output pixels of the center of gravity correction unit 152.

That is, it is the output of the gravity center correction unit 152 shown in FIG.
(1) Center of gravity correction unit output length exposure pixel (Lo),
(2) Center of gravity correction unit output short exposure pixel (So),
Is selected and output as follows.

In the case where no saturated pixel is included in the long exposure pixel applied to the generation of “long exposure pixel with centroid correction (L_w)” in the vertical centroid correction unit 181,
(1) Center of gravity correction unit output long exposure pixel (Lo) = long exposure pixel with center of gravity correction (L_w),
(2) Center of gravity correction unit output short exposure pixel (So) = short exposure pixel with center of gravity correction (S_w),
When one or more saturated pixels are included in the long exposure pixel applied to the generation of the “long exposure pixel with centroid correction (L_w)” in the vertical centroid correction unit 181,
(1) Center of gravity correction unit output long exposure pixel (Lo) = long exposure pixel without center of gravity correction (L_n),
(2) Center of gravity correction unit output short exposure pixel (So) = smoothed short exposure pixel (S_d),
This is the setting.
Such selection output is executed for each processing target pixel (target pixel).

[4. Example 3: Example in which the configuration of the pixel synthesis unit is changed]
Next, an embodiment in which the configuration of the pixel composition unit is changed will be described as a third embodiment of the image processing apparatus of the present disclosure.
The third embodiment differs from the first embodiment described above in the configuration and processing of the pixel synthesis unit 153. Other configurations and processes are the same as those in the first embodiment. For example, the overall configuration of the image processing apparatus is the imaging apparatus 100 described with reference to FIG. 3, and the configuration of the signal processing unit 103 is the HDR image generation unit 131, the subsequent signal processing unit, as described with reference to FIG. 132 and an output unit 133. As illustrated in FIG. 7, the HDR image generation unit 131 includes an edge detection unit 151, a centroid correction unit 152, and a pixel synthesis unit 153.

As described in the first embodiment, when the ratio of the exposure time of the short exposure pixel and the long exposure pixel which are constituent pixels of the input image (SVE array image) 141 photographed by the image sensor 102 is 1: E,
Each pixel value of the long exposure pixel and the short exposure pixel to be combined, that is,
(1) Center of gravity correction unit output length exposure signal (Lo),
(2) Center of gravity correction unit output short exposure signal (So),
These pixel values are in the relationship of the following relational expression (formula 7) described above.
Lo≈So × E (Expression 7)

However, this (Equation 7) is
(1) Center of gravity correction unit output length exposure signal (Lo),
(2) Center of gravity correction unit output short exposure signal (So),
The condition is that they are shooting almost the same subject position.
However, there is a possibility that a moving subject (moving subject) may be included in the captured image by the imaging device.
In the moving subject region, even if the long exposure pixel and the short exposure pixel are close to each other, different subject positions are photographed, and the relational expression (Expression 7) is not satisfied.
In Example 3 described below, a composition process is performed in consideration of such a moving subject region.

FIG. 21 shows the configuration of the pixel composition unit 153 of the HDR image generation unit 131 in the image processing apparatus of the third embodiment.
In the present embodiment, the moving subject composition unit 321 in the pixel composition unit 153 shown in FIG. 21 executes long / short exposure pixel composition processing different from the other composition units 211 to 213.
The moving subject combining unit 321 executes combining processing of the centroid correction unit output length exposure signal (Lo) and the centroid correction unit output short exposure signal (So) by combining processing adapted to the moving subject region, and the moving subject unit length is short. A synthesis result (Syn_m) is generated and output to the long / short synthesis result selection & final synthesis result generation unit 214.

  Specifically, the moving subject combining unit 321 selects, for example, only one of (1) the center of gravity correction unit output length exposure signal (Lo) or the center of gravity correction unit output short exposure signal (So). Then, the process of setting the selection signal as the moving subject part length / short synthesis result (Syn_m) is performed.

As shown in FIG. 21, the pixel composition unit 153 of the third embodiment is similar to the pixel composition unit 153 of the first embodiment, and the edge detection unit 151 described with reference to FIG.
Long exposure edge flag (LEedge_flg) 171,
Short exposure edge flag (SEdge_flg) 173,
The edge information is input.
Furthermore, from the center of gravity correction unit 152 described with reference to FIG.
Saturation flag (Sat_flg) 185,
Enter this saturation information.
Furthermore, in this embodiment, in addition to these information,
A moving subject detection flag (Move_flg) 322 is input.

  The moving subject detection flag (Move_flg) 322 is input from, for example, a moving subject region detection unit provided in the imaging apparatus. The moving subject area detection unit discriminates the moving subject area in the image by, for example, image analysis, for example, block matching processing of continuously shot images. Further, a moving subject detection flag (Move_flg) 322 corresponding to a pixel indicating whether each pixel is a moving subject region is generated and output to the pixel composition unit 153.

The long / short synthesis result selection & final synthesis result generation unit 214 outputs pixel information from the outside, that is,
Long exposure edge flag (LEedge_flg) 171,
Short exposure edge flag (SEdge_flg) 173,
Saturation flag (Sat_flg) 185,
A moving subject detection flag (Move_flg) 322,
Based on these flags, the state of the processing target pixel (target pixel) is determined.

The long / short synthesis result selection & final synthesis result generation unit 214 further, based on the determination result,
(1) Flat part synthesis result (Syn_f) which is a synthesized signal for the flat part,
(2) Edge portion synthesis result (Syn_e) which is a synthesized signal for the edge portion,
(3) Saturated portion synthesis result (Syn_s) which is a synthesized signal for the saturated portion,
(4) Moving subject portion synthesis result (Syn_m), which is a synthesized signal for the moving subject region,
The long / short synthesis result selection & final synthesis result generation unit 214 selects one of these four signals in units of processing target pixels (target pixels).
Further, based on the combination result of the selected long and short exposure pixels, horizontal pixel combination is performed to generate a final combination result, which is output as a combination result (Syn).

  The synthesis result (Syn) generation process executed by the long / short synthesis result selection & final synthesis result generation unit 214 will be described with reference to the flowchart shown in FIG.

First, in step S21, the composition result of each composition part 211-213,321 and each flag information are inputted. That is, the following data is input.
(1) Flat part synthesis result (Syn_f) which is a synthesized signal for the flat part,
(2) Edge portion synthesis result (Syn_e) which is a synthesized signal for the edge portion,
(3) Saturated portion synthesis result (Syn_s) which is a synthesized signal for the saturated portion,
(4) Moving subject portion synthesis result (Syn_m), which is a synthesized signal for the moving subject region,
(A) Long exposure edge flag (LEedge_flg) 171,
(B) a short exposure edge flag (SEdge_flg) 173;
(C) Saturation flag (Sat_flg) 185,
(D) a moving subject detection flag (Move_flg) 322;

Next, in step S202, with reference to the saturation flag (Sat_flg) 185 input from the centroid correction unit 152 described with reference to FIG. 11, generation of a centroid correction unit output pixel that is a target pixel (target pixel) of the synthesis process It is determined whether or not a saturated pixel is included in the original pixel.
If a saturated pixel is included, that is, if the determination in step S102 is True, the process proceeds to step S205.
In step S205, the saturated portion synthesis result (Syn_s), which is the synthesized signal for the saturated portion generated by the saturated portion synthesizing unit 213, is selected and used as the combined result of the long and short exposure pixels. The combination result of the long and short exposure pixels is a result of combining the pixels in the vertical direction, that is, intermediate data of the combining process. Further, in step S09, the intermediate data is used to combine the pixels in the horizontal direction. Execute and generate and output the final synthesis result (Syn).

  On the other hand, when it is determined in step S202 that a saturated pixel is not included in the generation source pixel of the center-of-gravity correction unit output pixel that is the target pixel (target pixel) of the synthesis process, that is, when the determination in step S202 is false. Advances to step S203.

  In step S203, with reference to the moving subject flag (Move_flg) 322 input from the outside, whether or not the pixel that is the generation source of the center-of-gravity correction unit output pixel that is the target pixel (target pixel) of the synthesis process is in the moving subject region. Determine.

If it is determined in step S203 that the pixel from which the center-of-gravity correction unit output pixel is generated is a moving subject region, the process proceeds to step S206.
In step S206, a moving subject portion synthesis result (Syn_m), which is a synthesized signal for the moving subject area generated by the moving subject portion synthesis unit 321 is selected, and this is used as a synthesis result of the long and short exposure pixels. The combination result of the long and short exposure pixels is a result of combining the pixels in the vertical direction, that is, intermediate data of the combining process. Further, in step S209, the intermediate pixels are used to combine the pixels in the horizontal direction. Execute and generate and output the final synthesis result (Syn).

On the other hand, if it is determined in step S203 that the pixel from which the center-of-gravity correction unit output pixel is generated is not a moving subject area, the process proceeds to step S204.
In step S204, the long exposure edge flag (LEEdge_flg) 171 and the short exposure edge flag (SEedge_flg) 173 input from the edge detection unit 151 described with reference to FIG. It is determined whether the pixel is an edge region pixel.

Specifically, one of the pixels applied to generate the synthesis target pixel is
Long exposure edge flag (LEedge_flg) 171,
Short exposure edge flag (SEdge_flg) 173,
If any of these flags is a value (1) indicating an edge region, it is determined that the flag is an edge region, and the process proceeds to step S207.

on the other hand,
Long exposure edge flag (LEedge_flg) 171,
Short exposure edge flag (SEdge_flg) 173,
If any of these flags is a value (0) indicating that the region is not an edge region, it is determined that the region is a flat region, and the process proceeds to step S208.

If it is determined in step S204 that it is in the edge region, the process proceeds to step S207.
In step S207, the edge portion combination result (Syn_e), which is the combined signal for the edge portion generated by the edge portion combining portion 212, is selected and used as the combination result of the long and short exposure pixels. The combination result of the long and short exposure pixels is a result of combining the pixels in the vertical direction, that is, intermediate data of the combining process. Further, in step S209, the intermediate pixels are used to combine the pixels in the horizontal direction. Execute and generate and output the final synthesis result (Syn).

On the other hand, if it is determined in step S204 that the pixel from which the center-of-gravity correction unit output pixel is generated is in a flat region, the process proceeds to step S208.
In step S208, the flat portion combining result (Syn_f), which is the combined signal for the flat portion generated by the flat portion combining portion 211, is selected and used as the combining result of the long and short exposure pixels. The combination result of the long and short exposure pixels is a result of combining the pixels in the vertical direction, that is, intermediate data of the combining process. Further, in step S209, the intermediate pixels are used to combine the pixels in the horizontal direction. Execute and generate and output the final synthesis result (Syn).

In step S209, the pixel synthesis result in the horizontal direction is generated using the vertical pixel synthesis result generated by any one of steps S205 to S208, and the final synthesis result (Syn) is generated and output. .
The pixel synthesis in the horizontal direction is the process described above with reference to FIG.

  As described above, in the third embodiment, when a moving subject is included in a captured image, a moving subject region is extracted, and a combination process of long and short exposure pixels optimum for the moving subject region is executed.

[5. Example 4: Example of changing pixel arrangement of image sensor and long / short exposure pixel setting]
In the embodiment described above, as described above with reference to FIGS. 5 and 6, in the image pickup device having the Bayer array, the long exposure pixels and the short exposure pixels are alternately set in units of two rows. The processing example for the image photographed in FIG.

  However, the high dynamic range (HDR) image generation process in the above-described embodiment is not limited to the pixel arrangement and the long / short exposure pixel setting described above, but can be applied to other arrangements and configurations.

As a pixel array, for example, as shown in FIG.
(1) RGBW type pixel array having white (W) pixels that receive light of substantially all visible wavelength light;
(2) An array in which Bayer-type RGB pixels are set in units of four pixels,
These sequences may be applied.

Note that when the process of the present disclosure is applied to the configuration having the W pixels illustrated in FIG. 23A, it is preferable to perform edge detection using W pixels having a high pixel density.
In addition, when the processing of the present disclosure is applied to an array in which Bayer-type RGB pixels illustrated in FIG. 23B are set in units of four pixels, it is preferable to perform the following processing.

  In the pixel array shown in FIG. 23 (2), it is possible to align the gravity center positions of the short exposure pixels and the long exposure pixels by averaging the pixels with the same exposure time existing diagonally within the four pixels of the same color. It is. Therefore, the vertical centroid correction unit 181 in the centroid correction unit 152 of the HDR image generation unit 131 described in the first embodiment is set as an oblique addition centroid correction unit, and is changed to a configuration that executes centroid correction in an oblique direction. Perform processing.

In addition, the setting of the long and short exposure pixels is not a setting in units of two rows, and a configuration in which the long and short exposure pixels are alternately set in an oblique direction as shown in FIG. FIG. 24 shows the following setting examples.
(3) An example in which long and short exposure pixels are alternately arranged along a diagonal line in the Bayer array,
(4) An example in which long and short exposure pixels are alternately arranged along a diagonal line in the RGBW arrangement,
Examples of these are shown.

[6. Example 5: Example of generating HDR image without reducing the number of pixels]
Next, an embodiment for generating an HDR image without reducing the number of pixels will be described.
In the first embodiment described above, the HDR image generation unit 131 is configured such that the number of pixels of the output HDR image 142 is reduced to, for example, 1/4 with respect to the number of pixels of the input image (SVE image). This is because one pixel of the HDR image is generated by a synthesis process of a plurality of pixels of the input image.
Example 5 described below is an example in which an HDR image generation unit generates an HDR image having the same number of pixels as the input image without reducing the number of pixels.

  The fifth embodiment differs from the first embodiment described above in the configuration and processing of the HDR image generation unit 131. Other configurations and processes are the same as those in the first embodiment. For example, the overall configuration of the image processing apparatus is the imaging apparatus 100 described with reference to FIG. 3, and the configuration of the signal processing unit 103 is the HDR image generation unit 131, the post-stage signal processing, as described with reference to FIG. Part 132 and output part 133.

In the present embodiment, the HDR image generation unit 131 includes an edge detection unit 151, an array conversion centroid correction unit 351, a pixel synthesis unit 352, and a correlation processing unit 353 as illustrated in FIG.
In this embodiment, an HDR image 360 having the same number of pixels as the input image (SVE image) 350 from the image sensor 102 is generated and output.
In the present embodiment, an example will be described in which the input image (SVE image) 350 is an SVE image in which long and short exposure pixels are arranged in an oblique direction in the Bayer arrangement described with reference to FIG.

Hereinafter, processing of each component of the HDR image generation unit 131 illustrated in FIG. 25 will be described.
(6-1. Configuration and processing of edge detection unit)
The edge detection unit 151 performs the same process as the edge detection unit 151 in the first embodiment described above with reference to FIG.
That is, as illustrated in FIG. 8, the edge detection unit 151 includes a difference calculation unit (vertical) 161, a difference calculation unit (horizontal) 162, a difference calculation unit (upper right) 163, and a difference calculation unit (upper left) 164.
Further, a long exposure edge detection unit 165, a long exposure edge direction determination unit 166, a short exposure edge detection unit 167, and a short exposure edge direction determination unit 168 are provided.

The difference calculation unit (vertical) 161 applies the G pixel included in the edge detection image 141G shown in FIG. 8, that is, the 9 × 9 pixel region in the vicinity of the target pixel,
Average value (LDiff_v) of pixel value difference between two long-exposure G pixels that are close in the vertical direction,
An average value (SDiff_v) of pixel value differences between two short-time exposure G pixels adjacent in the vertical direction;
These are calculated.

The difference calculation unit (horizontal) 162 applies the G pixel included in the edge detection image 141G shown in FIG. 8, that is, the 9 × 9 pixel region in the vicinity of the target pixel,
Average value (LDiff_h) of pixel value difference between two long-exposure G pixels adjacent in the horizontal direction,
An average value (SDiff_h) of pixel value differences between two short-time exposure G pixels adjacent in the horizontal direction;
These are calculated.

The difference calculation unit (upper right) 163 applies the edge detection image 141G shown in FIG. 8, that is, the G pixel included in the 9 × 9 pixel region in the vicinity of the target pixel,
Average value (LDiff_a) of pixel value difference between two long-exposure G pixels adjacent in the upper right direction,
An average value (SDiff_a) of pixel value differences between two short-time exposure G pixels adjacent in the upper right direction;
These are calculated.

The difference calculation unit (upper left) 164 applies the G pixel included in the edge detection image 141G shown in FIG. 8, that is, the 9 × 9 pixel region in the vicinity of the target pixel,
Average value (LDiff_d) of pixel value difference between two long-exposure G pixels adjacent in the upper left direction,
Average value (SDiff_d) of pixel value difference between two short-exposure G pixels adjacent in the upper left direction,
These are calculated.

FIG. 26 is a diagram illustrating a specific example of vertical, horizontal, upper right, and upper left difference calculation processes regarding long exposure pixels executed in the difference calculation units 161 to 164.
(Lv) Long exposure vertical direction,
(Lh) Long exposure horizontal direction,
(La) Long exposure upper right direction,
(Ld) Long exposure upper left direction,
In each of these processes, a combination (pixel pair) of pixels for which a difference is calculated is indicated by an arrow.

FIG. 26 is a diagram showing a specific example of each vertical difference, horizontal, upper right, and upper left difference calculation process regarding long exposure pixels executed in the difference calculation units 161 to 164 as in FIG. 9 described in the first embodiment. .
However, the example shown in FIG. 9 is a processing example for an SVE image in which long exposure pixels and short exposure pixels are alternately arranged in units of two rows in the horizontal direction.
The example shown in FIG. 26 is an example of processing for an SVE image in which long and short exposure pixels are arranged in an oblique direction in the Bayer arrangement described in FIG.

The edge detection unit 151 calculates pixel value differences of a plurality of pixel pairs, calculates the average value, and calculates the following values.
(1) Long exposure pixel vertical direction pixel value difference average value (LDiff_v),
(2) Long exposure pixel horizontal direction pixel value difference average value (LDiff_h),
(3) Long exposure pixel upper right direction pixel value difference average value (LDiff_a),
(4) Long exposure pixel upper left direction pixel value difference average value (LDiff_d),
The calculation formulas for these values are as shown in (Formula 11) below.

... (Formula 11)
In the above formula, N, M. O and P are the calculated numbers of differences.

FIG. 27 is a diagram illustrating a specific example of vertical, horizontal, upper right, and upper left difference calculation processes relating to short exposure pixels executed in the difference calculation units 161 to 164.
(Sv) Short exposure vertical direction,
(Sh) horizontal direction of short exposure,
(Sa) Short exposure upper right direction,
(Sd) Short exposure upper left direction,
In each of these processes, a combination (pixel pair) of pixels for which a difference is calculated is indicated by an arrow.

The pixel value differences of a plurality of pixel pairs are calculated, and the average value is calculated to calculate the following values.
(1) Short exposure pixel vertical direction pixel value difference average value (SDiff_v),
(2) Short exposure pixel horizontal direction pixel value difference average value (SDiff_h),
(3) Short exposure pixel upper right direction pixel value difference average value (SDiff_a),
(4) Short exposure pixel upper left direction pixel value difference average value (S) Diff_d),
The calculation formula of these values is as shown in the following (Formula 12).

(Equation 12)
In the above formula, Q, R. S and T are the calculated numbers of differences.

  The difference calculation unit (vertical) 161, the difference calculation unit (horizontal) 162, the difference calculation unit (upper right) 163, and the difference calculation unit (upper left) 164 shown in FIG. A pixel value difference average value in four directions is calculated by applying the pixels. This calculated value is output to the long exposure edge detection unit 165, the long exposure edge direction determination unit 166, the short exposure edge detection unit 167, and the short exposure edge direction determination unit 168.

The long exposure edge detection unit 165 and the long exposure edge direction determination unit 166 input the pixel value difference average value in the four directions of the long exposure G pixel, and use the pixel value difference average value in each direction of the long exposure pixel, The presence / absence of an edge determined from the long exposure pixel and the edge direction are determined.
Further, the short exposure edge detection unit 167 and the short exposure edge direction determination unit 168 receive the pixel value difference average values in the four directions of the short exposure G pixels and use the pixel value difference average values in the respective directions of the short exposure pixels. Thus, the presence / absence of the edge and the edge direction determined from the short exposure pixel are determined.

These processes are the same as the processes described in the first embodiment.
According to (Equation 3) to (Equation 6) described above, the following information corresponding to the target pixel is generated as edge information, and is output to the array conversion centroid correction unit 351 and the pixel synthesis unit 352.
(1) A long exposure edge flag (LEEdge_flg) 171 indicating whether or not the target pixel determined based on the long exposure pixel is an edge region.
(2) Long exposure edge direction flag (LDir_flg) 172 indicating the edge direction of the target pixel determined based on the long exposure pixel
(3) A short exposure edge flag (SEedge_flg) 173 indicating whether or not the target pixel determined based on the short exposure pixel is an edge region.
(4) A short exposure edge direction flag (SDir_flg) 174 indicating the edge direction of the target pixel determined based on the short exposure pixel.

(6-2. Configuration and Processing of Array Conversion Center of Gravity Correction)
Next, the configuration and processing of the array conversion centroid correction unit 351 will be described.
FIG. 28 is a diagram illustrating the configuration and processing of the array conversion centroid correction unit 352.
As illustrated in FIG. 28, the array conversion center-of-gravity correction unit 352 includes a center-of-gravity correction high-frequency signal generation unit 361 and a low-frequency signal generation unit 362.

The center-of-gravity correction high-frequency signal generation unit 361 inputs an input image (SVE image) 350 in which long and short exposure pixels are alternately arranged in an oblique direction that is an output of the image sensor 102, and corrects the center of gravity of G pixels in the input image 350. To generate a G signal high frequency component.
In addition, the low-frequency signal generation unit 362 calculates an average value of each RGB pixel of the 9 × 9 pixel input image 350.

The center-of-gravity correction high-frequency signal generation unit 361 performs processing for estimating and setting G pixels in all pixels of the 9 × 9 pixel input image 350. As the G pixel value, two G pixel values of a long exposure G pixel value and a short exposure G pixel value are estimated for each pixel.
In this process, the edge direction information input from the edge detection unit 151 is used, and a G pixel value is estimated at each target pixel position using a filter corresponding to the edge direction in which the weight of the G pixel along the edge direction is set large. And set.

Processing executed by the center-of-gravity correction high-frequency signal generation unit 361 will be described with reference to FIGS. 29 to 31.
First, the center-of-gravity correction high-frequency signal generation unit 361 extracts long exposure G pixels and short exposure G pixels from an input image 350 of 9 × 9 pixels, as shown in FIG.
The following data shown in FIG. 29 is extracted.
(2) Long exposure G pixel data (3) Short exposure G pixel data

Furthermore, the G pixel value of the unset position of the G pixel is estimated using the G pixel data of each of FIGS. 29 (2) and 29 (3).
30 and 31 show an example of G pixel value estimation processing by applying a filter according to each estimated position and edge direction.
FIG. 30 shows an example of filter coefficients used in the following cases.
(1) Edge direction is vertical,
(2) The edge direction is horizontal,
It is a setting example of the filter coefficient in these cases.
In the figure, a, b, c, and d are patterns of positions where G pixel values are estimated.
The filter to be applied is changed according to each estimated position.

FIG. 31 shows an example of filter coefficients used in the following cases.
(3) The edge direction is the upper left direction,
(4) The edge direction is the upper right direction,
It is a setting example of the filter coefficient in these cases.
In the figure, a, b, c, and d are patterns of positions where G pixel values are estimated.
The filter to be applied is changed according to each estimated position.

30 and 31 show processing examples for the long exposure G pixel data, but the same processing is performed for the short exposure G pixel data.
By this processing, the long exposure G pixel value and the short exposure G pixel value at all pixel positions are estimated.
The center-of-gravity correction high-frequency signal generation unit 361 outputs the two G pixel estimated values to the pixel synthesis unit 352.

On the other hand, the low-frequency signal generation unit 362 calculates the average value of the long exposure RGB pixels and the average value of the short exposure RGB pixels in the input image (SVE image) 350 including the 9 × 9 pixel region.
The low-frequency signal generation unit 362 outputs the calculated average value of the exposure RGB pixels and the average value of the short exposure RGB pixels to the pixel synthesis unit 352.

(6-3. Configuration and Processing of Pixel Combining Unit)
Next, the configuration and processing of the pixel composition unit 352 of the HDR image generation unit 131 illustrated in FIG. 25 will be described.
With reference to FIG. 32, the configuration and processing of the pixel composition unit 352 will be described.

As illustrated in FIG. 32, the pixel synthesis unit 352 includes a center-of-gravity correction high-frequency signal synthesis unit 371 and a low-frequency signal synthesis unit 372.
The center-of-gravity correction high-frequency signal synthesis unit 371 and the low-frequency signal synthesis unit 372 both have the configuration shown in FIG. 15 previously named in the first embodiment, and correspond to the flat portion, the edge portion, and the saturation portion. A process of generating a synthesized signal and selectively outputting one of the synthesized signals according to the pixel region of the processing target pixel (target pixel) is executed.

However, in the present embodiment, the center-of-gravity correction high-frequency signal synthesis unit 371 and the low-frequency signal synthesis unit 372 execute processing with the following signals as synthesis processing targets.
The center-of-gravity correction high-frequency signal combining unit 371 executes a combination process of the long exposure G pixel signal and the short exposure G pixel signal generated by the center-of-gravity correction high-frequency signal generation unit 361 of the array conversion center-of-gravity correction unit 351.
The low-frequency signal synthesis unit 372 executes synthesis processing of the long exposure RGB average value signal and the short exposure RGB average value signal generated by the low-frequency signal generation unit 362 of the array conversion centroid correction unit 351.

As a result of this synthesis processing, the following signals are generated and output to the correlation processing unit 353 as shown in FIG.
The center-of-gravity correction high-frequency signal combining unit 371 executes a combination process of the long exposure G pixel signal and the short exposure G pixel signal generated by the center-of-gravity correction high-frequency signal generation unit 361 of the array conversion center-of-gravity correction unit 351 to perform high-frequency HDR. A signal (G) is generated and output to the correlation processing unit 353.
The low-frequency signal synthesis unit 372 performs a synthesis process on the long-exposure RGB average value signal and the short-exposure RGB average value signal generated by the low-frequency signal generation unit 362 of the array conversion centroid correction unit 351, and performs a low-frequency HDR signal. (MR, mG, mB) is generated and output to the correlation processing unit 353.

(6-4. Configuration and processing of correlation processing unit)
Next, the configuration and processing of the correlation processing unit 353 of the HDR image generation unit 131 illustrated in FIG. 25 will be described.
The configuration and processing of the correlation processing unit 353 will be described with reference to FIG.

As illustrated in FIG. 33, the correlation processing unit 353 includes an output color selection unit 381 and a correlation calculation unit 382.
The output color selection unit 381 generates a low-frequency HDR signal (mR, mG) that is a synthesis result of the long exposure RGB average value signal and the short exposure RGB average value signal that are generated by the low-frequency signal synthesis unit 372 of the pixel synthesis unit 352. , MB) and sequentially select the low-frequency HDR signal corresponding to the output color according to the pixel position of the HDR image 360 to be output from the low-frequency HDR signals (mR, mG, mB) of these three colors. And output to the correlation calculation unit 382. FIG. 33 shows this output as mC. mC is either mR, mB, or mG.

The correlation calculation unit 382
(1) mC from the output color selection unit 381;
(2) a low-frequency HDR signal (mG) that is a generated signal of the low-frequency signal synthesis unit 372 of the pixel synthesis unit 352,
(3) A high-frequency HDR signal (G) that is a generated signal of the center-of-gravity correction high-frequency signal combining unit 371 of the pixel combining unit 352,
Each of these signals is input.

  The correlation calculation unit 382 calculates and outputs each synthesized pixel value of the HDR image 360 that is output by executing a calculation process using these input signals.

The calculation of the output pixel value executed by the correlation calculation unit 382 is executed by the following calculation.
mC = mR,
That is, the calculation of the output pixel value (R_out) when the output color is an R pixel is executed according to (Equation 13) shown below.
R_out = mR / mG × G (Formula 13)

mC = mG,
That is, the calculation of the output pixel value (G_out) when the output color is a G pixel is executed according to (Equation 14) shown below.
G_out = mG / mG × G (Expression 14)

mC = mB,
That is, the calculation of the output pixel value (B_out) when the output color is a B pixel is executed according to (Equation 15) shown below.
B_out = mB / mG × G (Equation 15)

  By the above calculation, an RGB signal including a high frequency component is generated. By appropriately selecting the color signal to be output, it is possible to generate and output an HDR image 360 having the same pixel arrangement as that of the input image, for example, a Bayer arrangement.

[7. Example 6: Configuration Example of Imaging Device Having Gradation Conversion Unit and Camera Signal Processing Unit]
Any of the embodiments described above can be executed in the imaging apparatus 100 described above with reference to FIG.
However, the processing of the present disclosure can be executed in various apparatuses in addition to the imaging apparatus illustrated in FIG. The present invention is not limited to the imaging apparatus, and can be executed by an information processing apparatus such as a PC that inputs an image captured by the imaging apparatus and executes image processing.

Further, the configuration of the imaging apparatus is not limited to the configuration illustrated in FIG. 3, and the processing of the present disclosure can be executed in imaging apparatuses having various configurations.
FIG. 34 is a diagram illustrating an example of an imaging apparatus 500 having a configuration different from that of the imaging apparatus illustrated in FIG. 3 that executes the processing of the present disclosure.

A configuration and processing of the imaging apparatus 500 illustrated in FIG. 34 will be described.
Light incident through the optical lens 501 is incident on an image sensor 502 constituted by an imaging unit, for example, a CMOS image sensor, and outputs image data by photoelectric conversion. The output image data is, for example, the SVE image 521 composed of the long exposure pixels and the short exposure pixels described above with reference to FIGS. Or it is the SVE image 521 demonstrated with reference to FIG. 23, FIG.

The SVE image 521 is input to the high dynamic range (HDR) image generation unit 503.
The high dynamic range (HDR) image generation unit 503 generates and outputs an HDR image 522 according to the processing described in the above embodiments.
The high dynamic range (HDR) image generation unit 503 has, for example, the configuration described above with reference to FIG.

The HDR image 504 generated by the high dynamic range (HDR) image generation unit 503 is input to the gradation conversion unit 504.
The gradation conversion unit 504 converts the bit information constituting the pixel value of each pixel of the HDR image 522 into the number of bits that can be processed by a normal camera signal processing unit. For example, when the bit value indicating the pixel value of the HDR image 522 is 10 bits, gradation conversion is performed to convert the bit value to 8 bits, an NDR (Normal Dynamic Range) image 523 is generated, and output to the camera signal processing unit 505. .

  The camera signal processing unit 505 generates signal 530 by executing signal processing in a general camera such as white balance (WB) adjustment and gamma correction. The output image 530 is stored in a storage unit (not shown). Or it outputs to a display part.

  The control unit 510 outputs a control signal to each unit according to a program stored in a memory (not shown), for example, and controls various processes.

[8. Summary of composition of the present disclosure]
As described above, the embodiments of the present disclosure have been described in detail with reference to specific embodiments. However, it is obvious that those skilled in the art can make modifications and substitutions of the embodiments without departing from the gist of the present disclosure. In other words, the present invention has been disclosed in the form of exemplification, and should not be interpreted in a limited manner. In order to determine the gist of the present disclosure, the claims should be taken into consideration.

The technology disclosed in this specification can take the following configurations.
(1) having a signal processing unit that executes pixel value synthesis processing of pixels included in an input image including long exposure pixels and short exposure pixels to generate an output image;
The signal processing unit
An edge detection unit that performs edge detection on each component pixel of the input image and outputs edge information corresponding to each pixel;
A centroid correction unit that executes centroid correction processing of the long-time exposure pixel and the short-time exposure pixel to be combined, and outputs the centroid correction unit output long-time exposure pixel and centroid correction unit output short-time exposure pixel;
An image processing apparatus comprising: a pixel synthesis unit that performs a synthesis process of the center-of-gravity correction unit output long-time exposure pixels generated by the barycentric correction unit and the center-of-gravity correction unit output short-time exposure pixels.

  (2) The center-of-gravity correction unit is configured to execute processing for matching the pixel positions of the long-time exposure pixel and the short-time exposure pixel that are to be combined as the center-of-gravity correction processing. Calculates the pixel value of the long-time exposure pixel output from the centroid correction unit by weighted addition using the pixel value of the long-time exposure pixel of the color, and applies the pixel value of multiple short-time exposure pixels of the same color at different pixel positions The image processing apparatus according to (1), wherein a process of calculating a pixel value of the short-time exposure pixel output from the centroid correction unit is performed by performing the weighted addition.

  (3) The center-of-gravity correction unit includes a saturation determination unit that determines whether or not the pixel applied to the center-of-gravity correction process is saturated, and when the pixel applied to the center-of-gravity correction process is saturated, If the pixel value of the input image is the pixel value of the centroid correction unit output long exposure pixel and the centroid correction unit output short exposure pixel, and the pixel applied to the centroid correction process is not saturated, A pixel value calculated by weighted addition using pixel values of a plurality of long-time exposure pixels of the same color is used as the pixel value of the center-of-gravity correction unit output long-time exposure pixel, and a plurality of short-time exposures of the same color at different pixel positions are performed. The image processing apparatus according to (2), wherein a pixel value calculated by weighted addition using a pixel value of the pixel is selectively output as a pixel value of the centroid correction unit output short-time exposure pixel.

(4) The centroid correction unit includes a saturation determination unit that determines whether or not the pixel applied to the centroid correction process is saturated, and when the pixel applied to the centroid correction process is saturated, The pixel value of the center-of-gravity correction unit output long-time exposure pixel is set as the pixel value of the long-time exposure pixel of the input image, and the pixel value of the center-of-gravity correction unit output short-time exposure pixel is set to a plurality of short-time exposure pixels of the input image. The image processing apparatus according to (2) or (3), wherein the image value is set as a pixel value calculated by a smoothing process based on a pixel value.
(5) The image processing according to (4), wherein the center-of-gravity correction unit performs the smoothing process by applying a smoothing filter set with a coefficient corresponding to an edge direction input from the edge detection unit. apparatus.

  (6) The pixel synthesizing unit is a synthesis unit that performs synthesis processing of the centroid correction unit output long-time exposure pixel and the centroid correction unit output short-time exposure pixel. A synthesis unit, an edge synthesis unit that executes synthesis processing adapted to the edge region, a saturation synthesis unit that executes synthesis processing adapted to the saturation region, and a synthesis result of each synthesis unit A selection unit configured to generate a synthesis result image, and the selection unit receives the edge information and a saturation flag indicating whether a pixel applied to the centroid correction process is a saturated pixel, and the centroid correction process When the pixel applied to is a saturated pixel, the combination result of the saturation unit combining unit is selected, and when the pixel applied to the centroid correction process is not a saturated pixel but an edge region, the edge unit combining unit Select synthesis result The center of gravity correction processing pixel is not saturated pixel applied to, if not an edge region, wherein selecting the combined result of the flat portion combining unit (1) to (5) The image processing apparatus according to any one.

  (7) The flat portion synthesis unit outputs the centroid correction unit output long-time exposure pixel and the centroid correction unit output according to a blend ratio (α) that changes according to a pixel value of the centroid correction unit output long-time exposure pixel. The pixel value of the short-time exposure pixel is blended to calculate a pixel value as a synthesis result, and the edge synthesis unit responds to a fixed blend rate (β) that does not depend on the pixel value of the long-time exposure pixel output from the centroid correction unit. The centroid correction unit output long-time exposure pixel and the centroid correction unit output short-time exposure pixel are blended to calculate a pixel value as a synthesis result, and the saturation unit synthesis unit outputs the centroid correction unit output short-time The image processing apparatus according to (6), wherein a pixel value of an exposure pixel is calculated as a pixel value as a synthesis result.

(8) The pixel synthesizing unit further includes a moving subject unit synthesizing unit that executes a synthesizing process adapted to the moving subject region, and the selection unit is such that the pixel applied to the centroid correction processing is the moving subject region. When the moving subject detection flag indicating whether or not the pixel applied to the center-of-gravity correction processing is a moving subject region, the synthesis result of the moving subject synthesis unit is selected in (6) or (7). The image processing apparatus described.
(9) The input image is a Bayer array image configured by RGB pixels, and the edge detection unit executes edge detection processing using only G pixels, according to any one of (1) to (8). Image processing apparatus.

(10) having a signal processing unit that generates an output image by executing pixel value synthesis processing of pixels included in an input image including long-time exposure pixels and short-time exposure pixels;
The signal processing unit
An edge detection unit that performs edge detection on each component pixel of the input image and outputs edge information corresponding to each pixel;
An array conversion centroid correction unit that generates a specific color high-frequency signal and an all-color low-frequency signal of the input image;
A pixel synthesizer that inputs the specific color high-frequency signal and the all-color low-frequency signal, and generates a specific color high-frequency high-dynamic range (HDR) signal and an all-color low-frequency high dynamic range (HDR) signal;
Correlation processing for generating the high dynamic range (HDR) image having the same number of pixels as the input image by inputting the specific color high frequency range high dynamic range (HDR) signal and the all color low frequency range high dynamic range (HDR) signal. An image processing apparatus having a unit.

(11) The input image is a Bayer array image composed of RGB pixels,
The array conversion center-of-gravity correction unit generates a G-color high-frequency signal and RGB low-frequency signals of the input image, and the pixel composition unit inputs the G-color high-frequency signal and RGB low-frequency signals. , A G color high frequency range high dynamic range (HDR) signal and RGB color low frequency range high dynamic range (HDR) signal, and the correlation processing unit, the G color high frequency range high dynamic range (HDR) signal, The image processing apparatus according to (10), wherein the RGB low color high dynamic range (HDR) signal is input to generate a high dynamic range (HDR) image having a Bayer array having the same number of pixels as the input image.

  Furthermore, the configuration of the present disclosure also includes a method of processing executed in the above-described apparatus and system, a program that executes the processing, and a recording medium that records the program.

  The series of processing described in the specification can be executed by hardware, software, or a combined configuration of both. When executing processing by software, the program recording the processing sequence is installed in a memory in a computer incorporated in dedicated hardware and executed, or the program is executed on a general-purpose computer capable of executing various processing. It can be installed and run. For example, the program can be recorded in advance on a recording medium. In addition to being installed on a computer from a recording medium, the program can be received via a network such as a LAN (Local Area Network) or the Internet and can be installed on a recording medium such as a built-in hard disk.

  Note that the various processes described in the specification are not only executed in time series according to the description, but may be executed in parallel or individually according to the processing capability of the apparatus that executes the processes or as necessary. Further, in this specification, the system is a logical set configuration of a plurality of devices, and the devices of each configuration are not limited to being in the same casing.

As described above, according to the configuration of an embodiment of the present disclosure, an apparatus and a method for generating a high-quality, high dynamic range image by pixel synthesis of an image including a long-time exposure pixel and a short-time exposure pixel are realized. Is done.
Specifically, an edge detection unit that performs edge detection for each component pixel of the input image including the long exposure pixel and the short exposure pixel, and a centroid correction process of the long exposure pixel and the short exposure pixel to be combined And a pixel composition unit that performs composition processing of the generated pixels of the center of gravity correction unit. The pixel compositing unit is a flat part compositing result obtained by applying a blend ratio (α) corresponding to the pixel value of the long-time exposure pixel, an edge part compositing result applying a fixed blend ratio, and a saturated part compositing adapted to the saturated part region. From the result, an optimum synthesis result corresponding to the situation of each pixel is selected, and a high dynamic range (HDR) image is generated and output based on the selected synthesis result.
By applying the above method, it is possible to obtain an optimum synthesis result corresponding to the pixel in accordance with the situation of each constituent pixel, and to generate a high-quality, high dynamic range image.

DESCRIPTION OF SYMBOLS 100 Image pick-up apparatus 101 Optical lens 102 Image pick-up element 103 Signal processing part 105 Control part 120 Output image 131 HDR pixel production | generation part 132 Later stage signal processing part 133 Output part 141 Input image (SVE image)
142 High Dynamic Range (HDR) Image 151 Edge Detection Unit 152 Center of Gravity Correction Unit 153 Pixel Composition Unit 161-164 Difference Calculation Unit 165 Long Exposure Edge Detection Unit 166 Long Exposure Direction Determination Unit 167 Short Exposure Edge Detection Unit 168 Short Exposure Direction Determination Unit 181 Vertical center of gravity correction unit 182 Saturation determination unit 183 Correction result selection unit 211 Flat part synthesis unit 212 Edge unit synthesis unit 213 Saturation unit synthesis unit 214 Long / short synthesis result selection & final synthesis result generation unit 301 Saturation unit short exposure pixel correction unit 321 Subject portion synthesis unit 351 Array conversion centroid correction unit 352 Pixel synthesis unit 353 Correlation processing unit 361 Center of gravity correction high-frequency signal generation unit 362 Low-frequency signal generation unit 371 Center-of-gravity correction high-frequency signal synthesis unit 372 Low-frequency signal synthesis unit 381 Output color selection Unit 382 Correlation calculation unit 500 Imaging device 501 Optical lens 5 02 Image sensor 503 High dynamic range (HDR) image generation unit 504 Gradation conversion unit 505 Camera signal processing unit 510 Control unit

Claims (13)

A signal processing unit that generates an output image by executing pixel value synthesis processing of pixels included in an input image including the long-time exposure pixel and the short-time exposure pixel;
The signal processing unit
An edge detection unit that performs edge detection on each component pixel of the input image and outputs edge information corresponding to each pixel;
A centroid correction unit that executes centroid correction processing of the long-time exposure pixel and the short-time exposure pixel to be combined, and outputs the centroid correction unit output long-time exposure pixel and centroid correction unit output short-time exposure pixel;
An image processing apparatus comprising: a pixel synthesis unit that performs a synthesis process of the center-of-gravity correction unit output long-time exposure pixels generated by the barycentric correction unit and the center-of-gravity correction unit output short-time exposure pixels. The center of gravity correction unit is
As the centroid correction process, a process of executing the process of matching the pixel positions of the long-time exposure pixel and the short-time exposure pixel to be combined,
By calculating the pixel value of the centroid correction unit output long exposure pixel by weighted addition applying the pixel values of a plurality of long exposure pixels of the same color at different pixel positions,
The image processing according to claim 1, wherein a process of calculating a pixel value of the centroid correction unit output short-time exposure pixel is performed by weighted addition using pixel values of a plurality of short-time exposure pixels of the same color at different pixel positions. apparatus. The center of gravity correction unit is
A saturation determination unit that determines whether or not a pixel applied to the centroid correction process is saturated;
When the pixel applied to the centroid correction process is saturated, the pixel value of the input image is the pixel value of the centroid correction unit output long exposure pixel and the centroid correction unit output short exposure pixel,
If the pixel applied to the centroid correction process is not saturated,
A pixel value calculated by weighted addition applying pixel values of a plurality of long-time exposure pixels of the same color at different pixel positions is set as a pixel value of the centroid correction unit output long-time exposure pixel,
The pixel value calculated by weighted addition using pixel values of a plurality of short-time exposure pixels of the same color at different pixel positions is selectively output as a pixel value of the centroid correction unit output short-time exposure pixel. Image processing device. The center of gravity correction unit is
A saturation determination unit that determines whether or not a pixel applied to the centroid correction process is saturated;
If the pixel applied to the centroid correction process is saturated,
The pixel value of the center-of-gravity correction unit output long exposure pixel is the pixel value of the long exposure pixel of the input image,
The image processing apparatus according to claim 2, wherein the pixel value of the centroid correction unit output short-time exposure pixel is set as a pixel value calculated by a smoothing process based on a plurality of pixel values of the short-time exposure pixel of the input image. The center of gravity correction unit is
The image processing apparatus according to claim 4, wherein the smoothing process is executed by applying a smoothing filter set with a coefficient corresponding to an edge direction input from the edge detection unit. The pixel composition unit
As a synthesizing unit that executes a synthesizing process between the center-of-gravity correction unit output long exposure pixel and the center of gravity correction unit output short-time exposure pixel,
A flat part synthesis unit that executes a synthesis process adapted to the flat part region;
An edge synthesis unit that executes synthesis processing adapted to the edge region;
A saturation unit synthesis unit that performs synthesis processing adapted to the saturation unit region;
A selection unit that selects a synthesis result of each of the synthesis units and generates a synthesis result image;
The selection unit inputs the edge information and a saturation flag indicating whether a pixel applied to the centroid correction process is a saturated pixel,
If the pixel applied to the centroid correction process is a saturated pixel, select the combined result of the saturated portion combining unit,
If the pixel applied to the centroid correction processing is not a saturated pixel but an edge region, select the synthesis result of the edge synthesis unit,
The image processing apparatus according to claim 1, wherein when the pixel applied to the centroid correction processing is not a saturated pixel and is not an edge region, a synthesis result of the flat portion synthesis unit is selected. The flat part synthesizing unit outputs the centroid correction unit output long-time exposure pixel and the centroid correction unit output short-time exposure according to a blend ratio (α) that changes according to a pixel value of the centroid correction unit output long-time exposure pixel. Blend the pixel values of the pixels to calculate the resulting pixel value,
The edge synthesizing unit is configured to output the centroid correction unit output long-time exposure pixel and the centroid correction unit output short-time exposure pixel according to a fixed blend ratio (β) that does not depend on a pixel value of the centroid correction unit output long-time exposure pixel. To calculate the pixel value of the composite result by blending the pixel values of
The image processing apparatus according to claim 6, wherein the saturation unit combination unit calculates a pixel value of the centroid correction unit output short-time exposure pixel as a pixel value of a combination result. The pixel composition unit further includes:
A moving subject unit synthesizing unit that performs synthesis processing adapted to the moving subject region;
The selection unit inputs a moving subject detection flag indicating whether a pixel applied to the gravity center correction process is a moving subject region,
The image processing apparatus according to claim 6, wherein when the pixel applied to the center-of-gravity correction process is a moving subject region, a synthesis result of the moving subject synthesis unit is selected. The input image is a Bayer array image composed of RGB pixels,
The image processing apparatus according to claim 1, wherein the edge detection unit performs an edge detection process using only G pixels. A signal processing unit that generates an output image by executing pixel value synthesis processing of pixels included in an input image including the long-time exposure pixel and the short-time exposure pixel;
The signal processing unit
An edge detection unit that performs edge detection on each component pixel of the input image and outputs edge information corresponding to each pixel;
An array conversion centroid correction unit that generates a specific color high-frequency signal and an all-color low-frequency signal of the input image;
A pixel synthesizer that inputs the specific color high-frequency signal and the all-color low-frequency signal, and generates a specific color high-frequency high-dynamic range (HDR) signal and an all-color low-frequency high dynamic range (HDR) signal;
Correlation processing for generating the high dynamic range (HDR) image having the same number of pixels as the input image by inputting the specific color high frequency range high dynamic range (HDR) signal and the all color low frequency range high dynamic range (HDR) signal. An image processing apparatus having a unit. The input image is a Bayer array image composed of RGB pixels,
The array conversion center-of-gravity correction unit generates a G color high frequency signal and an RGB color low frequency signal of the input image,
The pixel synthesizing unit inputs the G color high-frequency signal and the RGB color low-frequency signals, and generates a G color high-frequency high dynamic range (HDR) signal and an RGB color low-frequency high dynamic range (HDR) signal. And
The correlation processing unit receives the G color high-frequency high dynamic range (HDR) signal and the RGB low-color high-frequency dynamic range (HDR) signals, and includes a Bayer array having the same number of pixels as the input image. The image processing apparatus according to claim 10, wherein the image processing apparatus generates a high dynamic range (HDR) image. An image processing method executed in an image processing apparatus,
The signal processing unit executes a signal processing step of generating an output image by performing pixel value synthesis processing of pixels included in the input image including the long exposure pixels and the short exposure pixels,
In the signal processing step,
Edge detection processing for executing edge detection for each component pixel of the input image and outputting edge information corresponding to each pixel;
The center-of-gravity correction processing for executing the center-of-gravity correction processing of the long-time exposure pixel and the short-time exposure pixel to be combined and outputting the center-of-gravity correction unit output long-time exposure pixel and the center-of-gravity correction unit output
An image processing method for executing a synthesis process of the centroid correction unit output long-time exposure pixel and the centroid correction unit output short-time exposure pixel generated by the centroid correction unit. A program for executing image processing in an image processing apparatus;
Causing the signal processing unit to execute a signal processing step of generating an output image by performing pixel value synthesis processing of pixels included in the input image including the long exposure pixels and the short exposure pixels;
In the signal processing step,
Edge detection processing for executing edge detection for each component pixel of the input image and outputting edge information corresponding to each pixel;
The center-of-gravity correction processing for executing the center-of-gravity correction processing of the long-time exposure pixel and the short-time exposure pixel to be combined and outputting the center-of-gravity correction unit output long-time exposure pixel and the center-of-gravity correction unit output
A program for executing a synthesis process of the center-of-gravity correction unit output long-time exposure pixel and the center-of-gravity correction unit output short-time exposure pixel generated by the barycentric correction unit.