JP2013118492A - Image processing apparatus, imaging apparatus and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate a wide dynamic range image of high resolution.SOLUTION: An image processing apparatus is provided in which, in a first line group, imaging is performed with a first exposure. In a second line group, for a captured image picked up with a second exposure, a pixel value of the first line group is estimated from a second image formed in the second line group to generate a first estimated image, luminance of the first image is adjusted to luminance almost equal to the second image to generate a first adjusted image, a first blended image resulting from blending the first estimated image and the first adjusted image and the second image are combined to generate a first synthetic image, a pixel value of the second line group is estimated from the first image formed in the first line group to generate a second estimated image, luminance of the second image is adjusted to luminance almost equal to the first image to generate a second adjusted image, and a second blended image resulting from blending the second estimated image and the second adjusted image and the first image are combined to generate a second synthetic image.

Description

  The present invention relates to an image processing device, an imaging device, and an image processing method.

  Imaging elements such as CMOS (Complementary Metal Oxide Semiconductor) image sensors and CCD (Charge Coupled Device) image sensors tend to have a narrower dynamic range than photographic films. Therefore, when imaging using such an image sensor, white exposure may occur in pixels with a large amount of incident light when the exposure amount is increased, while blackout occurs in pixels with a small amount of incident light when the exposure amount is decreased. May occur. To solve these problems, a method has been devised in which images obtained by imaging the same subject with a plurality of different exposure amounts are combined to obtain an image (hereinafter referred to as an HDR image) having an image quality equivalent to that obtained by imaging with an imaging device having a wide dynamic range. It was done.

  For example, Patent Document 1 below discloses a method for obtaining an HDR image by combining an image captured with a high exposure amount and an image captured with a low exposure amount. Further, in Patent Documents 2 and 3 below, images are captured with different exposure amounts for each line of the image, and the interpolation processing using the lines captured with the high exposure amount corresponds to the lines captured with the low exposure amount. The pixel value of the line is calculated to generate a high-intensity interpolation image, and the pixel value of the line corresponding to the line imaged at a high exposure amount is calculated by interpolation processing using the line imaged at a low exposure amount. A configuration is disclosed in which a low-brightness interpolation image is generated and both the interpolation images are combined.

JP 2001-016499 A JP 2008-118573 A JP-A-2005-348301

  Certainly, when the technique described in Patent Document 1 is used, an HDR image including an image component imaged at a high exposure amount and an image component imaged at a low exposure amount is obtained. However, since the timing for imaging with a high exposure amount and the timing for imaging with a low exposure amount are not simultaneous, blurring occurs if the subject is not completely stationary. Further, as in the techniques described in Patent Documents 2 and 3 above, when an interpolation image is generated by an interpolation process using adjacent lines, the line obtained by the interpolation process includes only information on the adjacent lines. . As a result, the resolution is reduced by the amount of information that should be included in the line to be interpolated is missing.

  Therefore, the present invention has been made in view of the above problems, and an object of the present invention is to provide a new and improved image processing apparatus capable of generating a high resolution wide dynamic range image, An imaging device and an image processing method are provided.

  In order to solve the above-described problem, according to an aspect of the present invention, the imaging surface of the imaging device is divided into a first line group and a second line group, and the first exposure is performed for the first line group. An image acquisition unit configured to acquire a captured image obtained by imaging with a second exposure amount different from the first exposure amount for the second line group; and a second line of the captured image A first estimated image generation unit configured to generate a first estimated image by estimating a pixel value of each pixel constituting the first line group from a second image formed by a group; A first adjusted image generation unit that adjusts the luminance of the image to the same luminance as the second image to generate a first adjusted image, and blends the first estimated image and the first adjusted image. A first blend image generating unit for generating a first blend image and the first blend image From the first image formed by the first image synthesizing unit that synthesizes the second image and generates a first synthesized image, and the first line group of the photographed image, the second line A second estimated image generation unit that estimates a pixel value of each pixel constituting the group and generates a second estimated image; and adjusts the luminance of the second image to the same luminance as the first image A second adjusted image generating unit that generates a second adjusted image; and a second blended image generating unit that generates a second blended image by blending the second estimated image and the second adjusted image. A second image composition unit that synthesizes the second blend image and the first image to generate a second composite image, and the first composite image and the second composite image. A wide dynamic range image generation unit that generates a wide dynamic range image using the image, Management apparatus is provided.

  If the first estimated image and the second image are combined to generate the first combined image, the first line group information is not used when generating the first estimated image. The resolution of the first composite image is reduced by the amount of missing information. Similarly, when the second estimated image and the first image are combined to generate the second combined image, the information of the second line group is not used when generating the second estimated image. Therefore, the resolution of the second composite image is reduced by the amount of missing information. However, the image processing apparatus uses the first blend image when generating the first composite image. The first blend image is generated using a first adjusted image obtained by adjusting the luminance of the first image, and includes information on the first line group. Therefore, the first composite image generated by the image processing apparatus is a high-resolution image that includes information on the first line group. The same applies to the second composite image. Therefore, the wide dynamic range image generated by the image processing apparatus has a high resolution.

  In addition, the first estimated image generation unit performs interpolation processing using pixel values of pixels that form the second line group of the captured image, and thereby each image that forms the first line group. The second estimated image generation unit executes the interpolation process using the pixel values of the pixels constituting the first line group of the captured image, thereby performing the second line group. May be configured to estimate the pixel value of each image constituting the image. As described above, the processing load can be reduced by using the interpolation processing with a relatively low load.

  Further, the interpolation processing may be neighborhood interpolation using a pixel value of a neighboring pixel located in the vicinity of a pixel to be estimated as a pixel value of the pixel to be estimated. With this configuration, it is possible to estimate the pixel value with a low processing load.

  Further, the interpolation processing may be linear interpolation that estimates a pixel value of a pixel to be estimated by linear approximation using luminance information of neighboring pixels located in the vicinity of the pixel to be estimated. With such a configuration, it is possible to estimate the pixel value with a low processing load while realizing a certain degree of accuracy.

  The interpolation processing may be polynomial interpolation that estimates the pixel value of the pixel to be estimated by polynomial approximation using luminance information of neighboring pixels located in the vicinity of the pixel to be estimated. With such a configuration, it is possible to estimate the pixel value with a low processing load while realizing a certain degree of accuracy.

  The first and second estimated image generation units are configured to perform the interpolation processing using pixel values weighted by edge information of neighboring pixels located in the vicinity of the pixel to be estimated. Also good. With this configuration, it is possible to estimate the pixel value with higher accuracy.

  Further, the first blend image generation unit generates the first blend image by alpha-blending the first estimated image and the first adjustment image, and the second blend image generation unit The second estimated image and the second adjusted image may be alpha-blended to generate the second blended image.

  In addition, the first blended image generation unit uses the first estimated image for a pixel region where the brightness of the subject exceeds a predetermined threshold, and the first adjusted image for a pixel region below the predetermined threshold. The first estimated image and the first adjusted image are blended to generate the first blend image so that the brightness of the subject is a predetermined value. The second estimated image and the second adjusted image are used so that the second estimated image is used for a pixel region below a threshold and the second adjusted image is used for a pixel region above the predetermined threshold. May be configured to generate the second blended image. With this configuration, it is possible to suppress missing of information and generate a higher resolution blend image.

  In order to solve the above-described problem, according to another aspect of the present invention, the imaging surface of the imaging device is divided into a first line group and a second line group, and the first line group is the first line group. An imaging unit that captures an image with an exposure amount of 1 and captures the second line group with a second exposure amount different from the first exposure amount, and a second line group of the captured image A first estimated image generation unit configured to generate a first estimated image by estimating a pixel value of each pixel constituting the first line group from the second image formed by the first line group; and the first image A first adjusted image generation unit that generates a first adjusted image by adjusting the luminance of the first image to the same luminance as the second image, and blends the first estimated image and the first adjusted image. A first blend image generating unit for generating a first blend image; the first blend image; and the second blend image The second line group is composed of a first image synthesizing unit that synthesizes the image and generates a first synthesized image, and a first image formed by the first line group of the photographed image. A second estimated image generation unit that estimates a pixel value of each pixel to generate a second estimated image; and a second adjustment by adjusting the luminance of the second image to the same luminance as the first image A second adjusted image generating unit that generates an image; a second blended image generating unit that generates a second blended image by blending the second estimated image and the second adjusted image; A second dynamic image combining unit that generates a second composite image by combining the two blend images and the first image, and a wide dynamic image using the first composite image and the second composite image. Provided by an imaging device comprising a wide dynamic range image generation unit for generating a range image It is.

  The imaging device uses the first blend image when generating the first composite image. The first blend image is generated using a first adjusted image obtained by adjusting the luminance of the first image, and includes information on the first line group. For this reason, the first composite image generated by the imaging apparatus is a high-resolution image including information on the first line group. The same applies to the second composite image. Therefore, the wide dynamic range image generated by the imaging apparatus has a high resolution.

  In order to solve the above-described problem, according to another aspect of the present invention, the imaging surface of the imaging device is divided into a first line group and a second line group, and the first line group is the first line group. An image acquisition step of acquiring a captured image obtained by imaging with an exposure amount of 1 and capturing an image of the second line group with a second exposure amount different from the first exposure amount; A first estimated image generating step of generating a first estimated image by estimating a pixel value of each pixel constituting the first line group from a second image formed by two line groups; A first adjusted image generating step of generating a first adjusted image by adjusting the luminance of the first image to the same luminance as the second image; the first estimated image; and the first adjusted image; A first blend image generation step of generating a first blend image by blending From the first image composition step of combining the blend image and the second image to generate a first composite image, and the first image formed by the first line group of the captured image, the first image A second estimated image generation step of generating a second estimated image by estimating a pixel value of each pixel constituting the second line group, and setting the luminance of the second image to the same luminance as the first image A second adjusted image generating step of adjusting and generating a second adjusted image; and a second blend for generating a second blended image by blending the second estimated image and the second adjusted image An image generating step, a second image synthesizing step of synthesizing the second blend image and the first image to generate a second synthesized image, and the first synthesized image and the second synthesized image. Wide dynamic range image that generates a wide dynamic range image using image Comprising a forming step, the image processing method is provided.

  The image processing method uses the first blend image when generating the first composite image. The first blend image is generated using a first adjusted image obtained by adjusting the luminance of the first image, and includes information on the first line group. For this reason, the first composite image generated by the imaging apparatus is a high-resolution image including information on the first line group. The same applies to the second composite image. Therefore, the wide dynamic range image generated by the above image processing method has a high resolution.

  As described above, according to the present invention, it is possible to generate a high resolution wide dynamic range image.

It is explanatory drawing which showed the function structure of the imaging device which concerns on one Embodiment of this invention. It is explanatory drawing which showed the detailed functional structure about the image signal process part of the imaging device which concerns on the embodiment. It is explanatory drawing which showed the production | generation method of the HDR image which concerns on the same embodiment. It is explanatory drawing which showed the flow of the image processing which concerns on the same embodiment. It is explanatory drawing which showed an example of the interpolation processing method which concerns on the same embodiment. It is explanatory drawing which showed an example of the interpolation processing method which concerns on the same embodiment. It is explanatory drawing which showed an example of the interpolation processing method which concerns on the same embodiment. It is explanatory drawing which showed an example of the interpolation processing method which concerns on the same embodiment. It is explanatory drawing which showed an example of the interpolation processing method which concerns on the same embodiment. It is explanatory drawing for demonstrating the blend process which concerns on the same embodiment. It is explanatory drawing for demonstrating the blend process which concerns on the same embodiment. It is explanatory drawing for demonstrating the blend process which concerns on the same embodiment. It is explanatory drawing for demonstrating the blend process which concerns on the same embodiment.

  Exemplary embodiments of the present invention will be described below in detail with reference to the accompanying drawings. In addition, in this specification and drawing, about the component which has the substantially same function structure, duplication description is abbreviate | omitted by attaching | subjecting the same code | symbol.

[About the flow of explanation]
Here, the flow of explanation regarding the embodiment of the present invention described below will be briefly described. First, an overall configuration of the imaging apparatus 100 according to the present embodiment will be described with reference to FIG. Next, a detailed functional configuration of the image signal processing unit 103 included in the imaging apparatus 100 according to the present embodiment will be described with reference to FIG. Next, an HDR image generation method according to the present embodiment will be described with reference to FIGS.

[Overview]
The present embodiment relates to a method for generating an image (HDR image) having a dynamic range wider than a dynamic range that can be expressed by an image sensor when an image is captured with a constant exposure amount. In particular, the present embodiment relates to a method for generating a high-resolution HDR image using a single photographed image. In this paper, the dynamic range means the difference between the saturation signal output level of the image sensor and the maximum noise value.

<1: Configuration of Imaging Device 100>
Hereinafter, the configuration of the imaging apparatus 100 capable of realizing the HDR image generation method according to the present embodiment will be described.

[1-1: Overall Configuration of Imaging Device 100]
First, the overall configuration of the imaging apparatus 100 according to the present embodiment will be described with reference to FIG. FIG. 1 is an explanatory diagram illustrating the overall configuration of the imaging apparatus 100 according to the present embodiment. The overall configuration shown in FIG. 1 is an example, and some components may be omitted, added, or changed. In addition, the imaging apparatus 100 includes, for example, a digital still camera that can shoot a still image, a video camera that can shoot a moving image, or a mobile phone, game machine, or the like equipped with an imaging function equivalent to a digital still camera or a video camera. Information terminal, personal computer, etc.

  As shown in FIG. 1, the imaging apparatus 100 mainly includes a lens 101, an imaging element 102, an image signal processing unit 103, a CPU 104, a DRAM 105, a ROM 106, a control unit 107, an operation unit 108, The compression processing unit 109, the monitor driving unit 110, the LCD monitor 111, the memory interface 112, and the memory card 113 are included. However, the memory card 113 may be connected in a detachable state.

  The lens 101 forms part of an optical system that forms an image of external light information on the image sensor 102, and guides light from the subject to the image sensor 102. The optical system including the lens 101 includes, for example, a zoom lens, a diaphragm, and a focus lens. The zoom lens is a lens that changes the angle of view by changing the focal length. The diaphragm is a mechanism that adjusts the amount of light transmitted. The focus lens moves in the optical axis direction to focus the subject image on the imaging surface of the image sensor 102. The focus lens is driven and controlled by the control unit 107.

  The image sensor 102 includes a plurality of photoelectric conversion elements that convert incident light that has passed through the lens 101 into electrical signals. Each photoelectric element generates an electrical signal corresponding to the amount of light received. Examples of the image sensor 102 that can be used include a CCD image sensor and a CMOS image sensor. In addition, in order to control the exposure amount of the image sensor 102, the image pickup apparatus 100 is equipped with a mechanical shutter (not shown) and / or an electronic shutter. The mechanical shutter and the electronic shutter operate in response to an operation of the operation unit 108 by the user (such as pressing a shutter button). The exposure control in the image sensor 102 will be described later.

  The image sensor 102 is connected to a CDS / AMP circuit (not shown). Further, the CDS / AMP circuit is connected to an A / D conversion circuit (not shown). CDS is an abbreviation for Correlated Double Sampling, and AMP is an abbreviation for Amplifier. The CDS / AMP circuit removes reset noise and amplifier noise included in the electrical signal output from the image sensor 102. Further, the CDS / AMP circuit amplifies the electric signal from which the reset noise and the amplifier noise are removed to a predetermined level. On the other hand, the A / D conversion circuit converts an analog electrical signal output from the CDS / AMP circuit into a digital signal. The digital signal generated by the A / D conversion circuit is input to the image signal processing unit 103.

  The image signal processing unit 103 generates an HDR image using the input digital signal. A detailed functional configuration of the image signal processing unit 103 and a method for generating an HDR image will be described later. First, when a digital signal is input, the image signal processing unit 103 performs preprocessing such as pixel defect correction, black level correction, and shading correction of the image sensor 102. The image signal processing unit 103 converts the preprocessed digital signal into a luminance signal and a color signal. Further, the image signal processing unit 103 corrects the digital signal based on the white balance control value, the γ value, the edge enhancement control value, and the like, and generates an image signal that can be used for generating an HDR image.

  This image signal is recorded on the DRAM 105 or the memory card 113 or is compressed and encoded (for example, JPEG format or MPEG format) by the compression processing unit 109 and then recorded on the memory card 113. Note that recording to the memory card 113 is executed via the memory interface 112. Further, this image signal is displayed on the LCD monitor 111 as a through image, or is displayed on the LCD monitor 111 in accordance with a user operation using the operation unit 108. The display on the LCD monitor 111 is executed via the monitor driving unit 110. Further, although the LCD monitor 111 is illustrated here as a display means, an organic EL display or the like can also be used.

  By the way, the image signal processing unit 103, the control unit 107, the compression processing unit 109, and the like operate according to the control of a CPU (Central Processor Unit) 104. The CPU 104 is an arithmetic processing device that executes arithmetic processing according to a program stored in the ROM 106 or the like, and functions as a control unit that controls processing of each component constituting the imaging device 100. For example, the CPU 104 inputs a control signal to the control unit 107 to drive the focus lens. Further, the CPU 104 may be configured to execute an operation related to the HDR image generation process executed by the image signal processing unit 103.

  In the above description, the memory card 113 is exemplified as the recording means. However, a non-card semiconductor storage medium, magnetic recording medium, optical recording medium, magneto-optical recording medium, etc. may be used for recording image signals. Good. The operation unit 108 may be configured with up / down / left / right keys, a power switch, a mode dial, a shutter button, and the like. The shutter button may be capable of detecting a pressed state such as half-pressed, fully-pressed, or released. For example, the focus control may be started when the shutter button is half-pressed, and the focus control may be ended when the half-press is released. Furthermore, it may be configured such that shooting is started when the shutter button is fully pressed.

(About exposure control)
Here, the exposure control according to the present embodiment will be described. Note that the exposure control by the mechanical shutter and / or the electronic shutter is realized by the function of the control unit 107.

  The control unit 107 according to the present embodiment has a function of performing exposure control so that an imaging surface is divided into a plurality of imaging areas and imaging is performed with different exposure amounts for each imaging area. For example, the control unit 107 sets one or a plurality of lines as a unit of the imaging region (hereinafter, a line group), sets the exposure amount for the odd-numbered line group low, and sets the exposure amount for the even-numbered line group high. . With this setting, like the captured image PIN schematically shown in FIG. 3, the pixels constituting the odd-numbered line group have low luminance, and the pixels constituting the even-numbered line group have high luminance. As described above, the control unit 107 according to the present embodiment performs exposure control so as to capture images with different exposure amounts for each imaging region. In the following description, for the convenience of explanation, the exposure amount for the odd-numbered line group is set low, and the description is advanced on the assumption that the exposure control for setting the exposure amount for the even-numbered line group is set high. The application range of the technology according to the present embodiment is not limited to this.

  The overall configuration of the imaging apparatus 100 has been described above.

[1-2: Configuration of Image Signal Processing Unit 103]
Next, a detailed functional configuration of the image signal processing unit 103 will be described with reference to FIG. FIG. 2 is an explanatory diagram showing a detailed functional configuration of the image signal processing unit 103. Note that FIG. 2 mainly shows constituent elements related to generation of an HDR image, and description of other constituent elements is omitted.

  As shown in FIG. 2, the image signal processing unit 103 mainly includes an image separation unit 131, a high luminance image interpolation unit 132, a luminance reduction unit 133, a luminance increase unit 134, and a low luminance image interpolation unit 135. The high luminance image blending unit 136, the high luminance image merging unit 137, the low luminance image blending unit 138, the low luminance image merging unit 139, and the HDR processing unit 140.

  The image signal processing unit 103 receives an image signal (hereinafter, “captured image PIN”) captured with different exposure amounts for each imaging region. Here, the description will be made on the assumption that the odd-numbered line group constituting the photographed image PIN is captured with a low exposure amount and the even-numbered line group is captured with a high exposure amount. The captured image PIN is first input to the image separation unit 131. When the captured image PIN is input, the image separation unit 131, as shown in FIG. 3, the input captured image PIN is an image composed of an even-numbered line group (hereinafter referred to as a high brightness image P <b> 1) and an odd number. The image is separated into an image composed of the th line group (hereinafter referred to as a low luminance image P2).

  As shown in FIG. 2, the high luminance image P1 is input to the high luminance image interpolation unit 132, the luminance reduction unit 133, and the high luminance image merge unit 137. On the other hand, the low luminance image P2 is input to the luminance increasing unit 134, the low luminance image interpolating unit 135, and the low luminance image merging unit 139.

  As shown in FIG. 3, the high-brightness image interpolating unit 132 to which the high-brightness image P1 is input performs an interpolation process using the even-numbered line group constituting the input high-brightness image P1, and the odd-numbered Pixel information corresponding to the line group is estimated to generate a high-intensity interpolation image PIP1. Details of the interpolation processing will be described later. The high-intensity interpolation image PIP1 is input to the high-intensity image blending unit 136. Further, as shown in FIG. 3, the luminance reduction unit 133 to which the high luminance image P1 is input reduces (gains down) the luminance of the high luminance image P1 to the same luminance as that of the low luminance image P2, and generates the luminance reduced image PGD. Generate. Then, the reduced brightness image PGD is input to the low brightness image blend unit 138.

  On the other hand, as illustrated in FIG. 3, the low-brightness image interpolation unit 135 to which the low-brightness image P2 has been input performs an interpolation process using the odd-numbered line groups that constitute the input low-brightness image P2. The pixel information corresponding to the even-numbered line group is estimated to generate the low luminance interpolation image PIP2. Details of the interpolation processing will be described later. Further, the low-brightness interpolated image PIP2 is input to the low-brightness image blending unit 138. Further, the luminance increasing unit 134 to which the low luminance image P2 is input increases (gains up) the luminance of the low luminance image P2 to the same luminance as the high luminance image P1, as shown in FIG. Generate. Then, the brightness increase image PGU is input to the high brightness image blend unit 136.

  As described above, the high-intensity image blending unit 136 receives the high-intensity interpolation image PIP1 and the luminance-increasing image PGU. When the high luminance interpolation image PIP1 and the luminance increase image PGU are input, the high luminance image blend unit 136 blends the input high luminance interpolation image PIP1 and the luminance increase image PGU, and generates a high luminance blend image PBL1. . Details of the blending process will be described later. The high luminance blend image PBL1 is input to the high luminance image merging unit 137. At this time, the high-intensity image merge unit 137 receives the high-intensity image P1 composed of even-numbered lines and the high-intensity blended image PBL1 composed of odd-numbered lines.

  When the high-intensity image P1 and the high-intensity blend image PBL1 are input, the high-intensity image merge unit 137 merges the input high-intensity image P1 and the high-intensity blend image PBL1 to generate a high-intensity merge image PM1. . That is, the even-numbered line group constituting the high-intensity merged image PM1 is formed by the even-numbered line group constituting the high-intensity image P1. On the other hand, the odd-numbered line group constituting the high-intensity merged image PM1 is formed of the odd-numbered line group constituting the high-intensity blended image PBL1. The high-intensity merged image PM1 generated in this way is input to the HDR processing unit 140.

  On the other hand, the low luminance interpolated image PIP2 and the luminance reduced image PGD are input to the low luminance image blending unit 138. When the low luminance interpolation image PIP2 and the luminance reduced image PGD are input, the low luminance image blending unit 138 blends the input low luminance interpolation image PIP2 and the luminance reduced image PGD to generate a low luminance blended image PBL2. . Details of the blending process will be described later. The low luminance blend image PBL2 is input to the low luminance image merge unit 139. At this time, the low-brightness image merge unit 139 receives the low-brightness image P2 configured by the odd-numbered line group and the low-brightness blended image PBL2 configured by the even-numbered line group.

  When the low-intensity image P2 and the low-intensity blend image PBL2 are input, the low-intensity image merge unit 139 merges the input low-intensity image P2 and the low-intensity blend image PBL2 to generate a low-intensity merge image PM2. . That is, the odd-numbered line group constituting the low-luminance merged image PM2 is formed of the odd-numbered line group constituting the low-luminance image P2. On the other hand, the even-numbered line group constituting the low-intensity merged image PM2 is formed of the even-numbered line group constituting the low-intensity blended image PBL2. The low-intensity merged image PM2 generated in this way is input to the HDR processing unit 140.

  When the high-intensity merged image PM1 and the low-intensity merged image PM2 are input, the HDR processing unit 140 combines the input high-intensity merged image PM1 and the low-intensity merged image PM2 to generate an HDR image PHDR. The HDR image PHDR generated in this way is output from the image signal processing unit 103 and subjected to compression processing by the compression processing unit 109, displayed on the LCD monitor 111, or recorded on the memory card 113.

  The detailed functional configuration of the image signal processing unit 103 has been described above. Note that interpolation processing, blend processing, and the like, which are not described in detail in the above description, will be described later.

<2: HDR image generation method>
Next, an HDR image generation method according to this embodiment will be described.

[2-1: Overall flow]
First, an overall flow of HDR image generation processing according to the present embodiment will be described with reference to FIG. FIG. 4 is an explanatory diagram for explaining the overall flow of HDR image generation processing according to the present embodiment. Note that the processing shown in FIG. 4 is realized by the function of the image signal processing unit 103 described above.

  As shown in FIG. 4, first, the image signal processing unit 103 separates the captured image PIN into a high luminance image P1 and a low luminance image P2 (S101). Next, the image signal processing unit 103 generates a high luminance interpolation image PIP1 from the high luminance image P1 by interpolation processing (S102). Next, the image signal processing unit 103 reduces the gain of the high luminance image P1 to generate a luminance reduced image PGD (S103).

  Next, the image signal processing unit 103 generates a low luminance interpolation image PIP2 from the low luminance image P2 by interpolation processing (S104). Next, the image signal processing unit 103 gains up the low luminance image P2 to generate a luminance increased image PGU (S105). Next, the image signal processing unit 103 blends the high-intensity interpolation image PIP1 and the luminance-increasing image PGU to generate a high-intensity blended image PBL1 (S106). Next, the image signal processing unit 103 blends the low luminance interpolation image PIP2 and the luminance reduced image PGD to generate a low luminance blend image PBL2 (S107).

  Next, the image signal processing unit 103 merges the high luminance blend image PBL1 and the high luminance image P1 to generate a high luminance merge image PM1 (S108). Next, the image signal processing unit 103 merges the low luminance blend image PBR2 and the low luminance image P2 to generate a low luminance merge image PM2 (S109). Next, the image signal processing unit 103 combines the high-intensity merge image PM1 and the low-intensity merge image PM2 to generate an HDR image PHDR (S110), and ends a series of processes.

  The overall flow of the HDR image generation process has been described above. Note that the processing order of some processing steps may be changed.

(Linterpolation: Fig. 5)
Here, a method for generating a blend image (the high-intensity blend image PBL1 or the low-intensity blend image PBL2) will be described more specifically with reference to FIG.

  FIG. 5 shows a specific method for generating the low-intensity blend image PBL2 as an example. As shown in FIG. 5, the blend image generation method according to this embodiment is a method that uses neighboring pixels and pixels at the same position with different luminance values. In the example of FIG. 5, the pixel Q included in the line group captured with a high exposure amount and the pixels 1 to 6 included in the neighboring line group imaged with a low exposure amount are used. A pixel (low luminance pixel) imaged with a low exposure amount corresponding to the position is estimated.

  As shown in FIG. 5, when a low-luminance pixel at the position of the pixel Q is estimated, pixels in the same color and in the vicinity (pixels 1 to 6 in this example) are selected. Then, the pixel value of the pixel at the position of the pixel Q is calculated by interpolation processing using the pixel values of the pixels 1 to 6. As the interpolation process used here, for example, nearest neighbor interpolation (see FIG. 6), linear interpolation (see FIG. 7), polynomial interpolation (see FIG. 8), or the like can be used. Further, the edge information at the positions of the pixels 1 to 6 may be used for the interpolation process (see FIG. 9).

  On the other hand, a high-luminance pixel at the position of the pixel Q (a pixel imaged with a high exposure amount) is gain-downed to the same luminance as a pixel imaged with a low exposure amount. Then, the pixel after the gain reduction and the pixel generated by the above interpolation process are blended to generate an interpolation pixel at the position of the pixel Q (an estimated pixel imaged with a low exposure amount). Thus, the pixel estimation method combining the interpolation process and the blend process may be referred to as “interpolation” in this paper. By using this pixel estimation method, it is possible to suppress a decrease in resolution and a positional deviation.

[2-2: Method for generating interpolation image]
Here, details of the interpolation processing will be described with reference to FIGS.

(Nearest neighbor interpolation: Fig. 6)
As the simplest interpolation method, there is nearest neighbor interpolation as shown in FIG. Nearest neighbor interpolation is an interpolation method in which a point closest to the position to be interpolated is selected and the value of the selected point is used as the value of the point at the position to be interpolated. In other words, this is an interpolation method in which a pixel closest to a pixel whose pixel value is to be estimated by interpolation processing (hereinafter referred to as an estimated pixel) is selected and the pixel value of the selected pixel is used as the pixel value of the estimated pixel. Since this interpolation method uses the pixel value of the nearest pixel as it is, the estimation accuracy is inferior, but the processing load is very low.

(Linear interpolation: Fig. 7)
As a relatively simple interpolation method, there is a linear interpolation as shown in FIG. Linear interpolation is an interpolation method in which two adjacent points are connected by a straight line, and the value on the straight line is the value of the point at the position to be interpolated. For example, the pixel value PIP of the pixel at the position x is calculated based on the following formula (1). However, the coefficients A and B are calculated from the positions and pixel values of neighboring pixels. This interpolation method has a higher estimation accuracy than the nearest neighbor interpolation and uses a linear expression, so that the processing load is relatively low.

PIP (x) = A * x + B
... (1)

(Polynomial interpolation: Fig. 8)
A somewhat advanced interpolation method is polynomial interpolation as shown in FIG. Polynomial interpolation is an interpolation method in which a plurality of neighboring points are fitted with a polynomial, and points on the polynomial are used as point values at positions to be interpolated. For example, the pixel value PIP of the pixel at the position x is calculated based on the following equation (2). However, the coefficients A0,..., An are calculated from the positions and pixel values of neighboring pixels. This interpolation method has higher estimation accuracy than linear interpolation, but uses a polynomial, so that the processing load increases as the order n increases.

PIP (x) = A0 + A1 * x + ... + An * xn
... (2)

(Interpolation considering edge information: Fig. 9)
As a more advanced interpolation method, as shown in FIG. 9, an interpolation method that considers edge information of neighboring pixels (hereinafter, an edge consideration method) can be considered. This interpolation method is used in combination with the linear interpolation described above. Here, the explanation will be made with an example in combination with linear interpolation in mind.

  First, the image signal processing unit 103 calculates the pixel value of the estimated pixel from the pixel value of the neighboring pixel by linear interpolation. For example, the image signal processing unit 103 calculates the pixel value PIP3 of the estimated pixel by linear interpolation using the pixel px3 and the pixel px3 ′. Further, the image signal processing unit 103 extracts edge information for each direction from the pixel values of neighboring pixels. For example, the image signal processing unit 103 extracts the edge information e3 in the direction connecting the pixel px3 and the pixel px3 ′. Then, the image signal processing unit 103 calculates the pixel value PIP of the estimated pixel using the pixel value PIPd calculated by linear interpolation and the edge information ed (d = −3 to 3 in the example of FIG. 9).

  The pixel value PIP is calculated based on the following equation (3). A function w (•) included in the following equation (3) is defined by the following equation (4). However, σd represents the size of the kernel.

…（３）

…（４）

... (3)

(4)

  The details of the interpolation processing have been described above.

[2-3: Method for generating blend image]
Next, details of the blending process will be described with reference to FIGS.

(Blend method using threshold)
As shown in FIG. 10, in the case of long exposure, the sensitivity is obtained in the region where the brightness of the scene is bl1 to bl2. However, when the brightness of the scene exceeds bl2, the saturation output level of the image sensor is exceeded, and the output level becomes constant and sensitivity cannot be obtained. On the other hand, in the case of short-time exposure, sensitivity can be obtained in a region where the brightness of the scene is bl3 (bl3> bl1) to bl4 (bl4> bl2). In other words, the area where the brightness of the scene is less than bl3 is blacked out, but even if the brightness of the scene exceeds bl2, the sensitivity up to the area up to bl4 can be obtained without whiteout. Therefore, an image with a wide dynamic range can be obtained by synthesizing an image captured with long exposure and an image captured with short exposure.

  Here, let us consider a case where the gain of an image captured by long exposure (hereinafter, long exposure image) is reduced. Reducing the gain of the long exposure image is equivalent to suppressing the output signal characteristics as a whole when the image is captured with long exposure as shown in FIG. 11 (see the alternate long and short dash line). The output signal characteristic corresponding to the long-exposure image with the gain reduced approximates the output signal characteristic when the image is captured with short-time exposure in an area where the scene brightness is up to bl2, but the brightness of the scene is higher than that of bl2. In large areas. Specifically, in the area where the brightness of the scene exceeds bl2, the output signal corresponding to the long-exposure image with the gain reduced is suppressed to a level lower than the saturation signal level. That is, bright scene information is lost due to gain reduction.

  Therefore, as shown in FIG. 12, the scene brightness bl2 is used as a threshold value, a short-time exposure image is used for an area where the scene brightness is larger than the threshold value, and gain reduction is performed for an area where the scene brightness is smaller than the threshold value. A blending technique using long exposure images was devised. When this blending method is applied to the generation of the low-intensity blended image PBL2 (application to step S107 in FIG. 4), the region where the brightness of the scene exceeds the threshold uses the low-intensity interpolation image PIP2, and the region falls below the threshold For, both images are blended to utilize the reduced brightness image PGD. By applying the blend method using the threshold value in this way, information loss is suppressed and a high-resolution blend image is obtained.

(Α blend)
As shown in FIG. 13, a method of α blending a long-exposure image with reduced gain and an image captured by short-time exposure is also conceivable. The α blend is a method for calculating the pixel value of the blend image using the coefficient α as in the following formula (5). For example, assuming that the pixel value of the low luminance interpolation image PIP2 is pxIP and the pixel value of the corresponding luminance reduced image PGD is pxGD, the pixel value pxBL of the low luminance blend image PBL2 is calculated based on the following equation (5). The

pxBL = α * pxIP + (α−1) * pxGD
... (5)

  The details of the blending process have been described above. In the above description, the method for generating the low-intensity blended image PBL2 has been described as an example. However, both the blending method using the threshold and the α blend can be applied to the generation of the high-intensity blended image PBL1. It is.

  As described above, when the HDR image generation method and the configuration of the imaging apparatus 100 according to the present embodiment are applied, a high-resolution HDR image can be obtained. In particular, when the technique of the present embodiment is applied, not only a wide dynamic range image can be obtained, but also HDR image generation can be realized by one shooting, so that adaptability to moving object shooting can be obtained. Further, there is no position shift and no shaggy occurs. In the synthesized image, the resolution of the overlapping dynamic range is maintained.

  As mentioned above, although preferred embodiment of this invention was described referring an accompanying drawing, it cannot be overemphasized that this invention is not limited to the example which concerns. It will be apparent to those skilled in the art that various changes and modifications can be made within the scope of the claims, and these are naturally within the technical scope of the present invention. Understood.

  For example, in the above description, the imaging surface is divided into two types of line groups, and the method of generating an HDR image using images captured with different exposure amounts has been introduced. However, the imaging surface is divided into three or more types of line groups. Alternatively, the way of dividing the line group can be variously modified. Furthermore, the number of lines constituting the line group is also arbitrary. Moreover, the combination of the interpolation method and the blend method is also free, and a plurality of methods may be combined. In the above description, the description has been made with an apparatus equipped with an imaging function such as a digital still camera or a digital video camera in mind, but an image processing apparatus that generates an HDR image using an image captured by such an apparatus. Can also be realized. In that case, from the above description, a computer program capable of realizing each function of the image processing apparatus and a method for realizing a recording medium on which such a computer program is recorded can be conceived.

DESCRIPTION OF SYMBOLS 100 Imaging device 101 Lens 102 Image pick-up element 103 Image signal processing part 104 CPU
105 DRAM
106 ROM
DESCRIPTION OF SYMBOLS 107 Control part 108 Operation part 109 Compression processing part 110 Monitor drive part 111 LCD monitor 112 Memory interface 113 Memory card 131 Image separation part 132 High brightness image interpolation part 133 Brightness reduction part 134 Brightness increase part 135 Low brightness image interpolation part 136 High brightness Image blending unit 137 High luminance image merging unit 138 Low luminance image blending unit 139 Low luminance image merging unit 140 HDR processing unit

Claims (10)

The imaging surface of the imaging device is divided into a first line group and a second line group, the first line group is imaged with a first exposure amount, and the second line group is the first line group. An image acquisition unit for acquiring a captured image obtained by imaging with a second exposure amount different from the exposure amount;
A first estimated image for generating a first estimated image by estimating a pixel value of each pixel constituting the first line group from a second image formed by the second line group of the photographed image A generator,
A first adjusted image generating unit that generates a first adjusted image by adjusting the luminance of the first image formed by the first line group of the captured image to the same luminance as the second image;
A first blend image generation unit that blends the first estimated image and the first adjustment image to generate a first blend image, and combines the first blend image and the second image. A first image composition unit for generating a first composite image;
A second estimated image generation unit configured to estimate a pixel value of each pixel constituting the second line group from the first image and generate a second estimated image;
A second adjusted image generation unit configured to generate a second adjusted image by adjusting the luminance of the second image to the same luminance as the first image;
A second blend image generation unit that generates a second blend image by blending the second estimated image and the second adjustment image, and combines the second blend image and the first image. A second image composition unit for generating a second composite image;
A wide dynamic range image generating unit that generates a wide dynamic range image using the first composite image and the second composite image;
An image processing apparatus comprising: The first estimated image generation unit performs an interpolation process using pixel values of each pixel that constitutes the second line group of the captured image, so that the pixels of each image that constitute the first line group Estimate the value,
The second estimated image generation unit performs interpolation processing using the pixel values of the pixels that form the first line group of the captured image, so that the pixels of the images that form the second line group The image processing apparatus according to claim 1, wherein a value is estimated. 3. The image according to claim 2, wherein the interpolation processing is neighborhood interpolation using a pixel value of a neighboring pixel located in the vicinity of a pixel to be estimated as a pixel value of the pixel to be estimated. Processing equipment. The interpolation processing is linear interpolation in which a pixel value of a pixel to be estimated is estimated by linear approximation using luminance information of neighboring pixels located in the vicinity of the pixel to be estimated. 2. The image processing apparatus according to 2. The interpolation process is a polynomial interpolation for estimating a pixel value of a pixel to be estimated by polynomial approximation using luminance information of neighboring pixels located in the vicinity of the pixel to be estimated. 2. The image processing apparatus according to 2. The said 1st and 2nd estimated image generation part performs the said interpolation process using the pixel value weighted by the edge information of the neighboring pixel located in the vicinity of the pixel used as estimation object, The said interpolation process is characterized by the above-mentioned. The image processing apparatus according to any one of 2 to 5. The first blend image generation unit generates the first blend image by alpha blending the first estimated image and the first adjustment image,
The second blended image generation unit generates the second blended image by alpha blending the second estimated image and the second adjusted image. The image processing apparatus according to any one of the above. The first blended image generation unit uses the first estimated image for a pixel region where the brightness of the subject exceeds a predetermined threshold, and uses the first adjusted image for a pixel region below the predetermined threshold. Blending the first estimated image and the first adjusted image to generate the first blended image,
The second blended image generation unit uses the second estimated image for a pixel region where the brightness of the subject is below a predetermined threshold, and uses the second adjusted image for a pixel region above the predetermined threshold. The image according to any one of claims 1 to 6, wherein the second blended image is generated by blending the second estimated image and the second adjusted image. Processing equipment. The imaging surface of the imaging device is divided into a first line group and a second line group, the first line group is imaged with a first exposure amount, and the second line group is the first line group. An imaging unit that obtains a captured image by imaging with a second exposure amount different from the exposure amount;
A first estimated image for generating a first estimated image by estimating a pixel value of each pixel constituting the first line group from a second image formed by the second line group of the photographed image A generator,
A first adjusted image generation unit that adjusts the luminance of the first image to the same luminance as the second image to generate a first adjusted image;
A first blend image generation unit that blends the first estimated image and the first adjustment image to generate a first blend image;
A first image synthesizing unit that synthesizes the first blend image and the second image to generate a first synthesized image;
A second estimated image that generates a second estimated image by estimating a pixel value of each pixel constituting the second line group from a first image formed by the first line group of the captured image. A generator,
A second adjusted image generation unit configured to generate a second adjusted image by adjusting the luminance of the second image to the same luminance as the first image;
A second blended image generator for blending the second estimated image and the second adjusted image to generate a second blended image;
A second image composition unit that composes the second blend image and the first image to generate a second composite image;
A wide dynamic range image generating unit that generates a wide dynamic range image using the first composite image and the second composite image;
An imaging apparatus comprising: The imaging surface of the imaging device is divided into a first line group and a second line group, the first line group is imaged with a first exposure amount, and the second line group is the first line group. An image acquisition step of acquiring a captured image obtained by imaging with a second exposure amount different from the exposure amount;
A first estimated image for generating a first estimated image by estimating a pixel value of each pixel constituting the first line group from a second image formed by the second line group of the photographed image Generation process;
A first adjusted image generating step of adjusting the luminance of the first image to the same luminance as the second image to generate a first adjusted image;
A first blend image generating step of generating a first blend image by blending the first estimated image and the first adjustment image;
A first image combining step of combining the first blend image and the second image to generate a first combined image;
A second estimated image that generates a second estimated image by estimating a pixel value of each pixel constituting the second line group from a first image formed by the first line group of the captured image. Generation process;
A second adjusted image generation step of adjusting the luminance of the second image to the same luminance as the first image and generating a second adjusted image;
A second blended image generating step of generating a second blended image by blending the second estimated image and the second adjusted image;
A second image composition step of combining the second blend image and the first image to generate a second composite image;
A wide dynamic range image generating step for generating a wide dynamic range image using the first composite image and the second composite image;
An image processing method comprising the steps of:

Images