JP2010273001A - Image processor, imaging apparatus, and synthetic image generating method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide technique for generating a synthetic image with a wide dynamic range by setting the optimum threshold corresponding to an acquired reference image. <P>SOLUTION: An imaging apparatus 1A includes an HDR (high dynamic range) image generating part 35 for synthesizing the two reference images obtained by photographing the same subject under different photographing conditions, and generating an HDR image. The HDR image generating part 35 has: an exposure level order determining part 352 for determining the level order of exposure in the two reference images; a cumulative frequency calculating part 353 for calculating cumulative frequencies respectively in the two reference images, which are obtained by adding the numbers of pixels equal to or less than a prescribed pixel value; a setting means for setting a boundary luminance value to be used as reference of selection respectively concerning the two reference images, based on the cumulative frequencies, when the pixel values to be used for generating the HDR image are selected from the two reference images; and a pixel value determining part 356 for determining the pixel values of each pixel constituting the HDR image, based on the level order of exposure and the boundary luminance value. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image composition technique.

  In general, it is known that the dynamic range of an image pickup device such as a CCD is narrower than the dynamic range of a landscape (ratio of light and darkness of the brightest part to the darkest part).

  Therefore, the same subject is photographed multiple times under different photographing conditions, a plurality of images having different dynamic ranges (also referred to as “reference images”) are acquired, and the reference images are synthesized to synthesize the dynamic range. There exists a technique for generating a wide image (also referred to as a “high dynamic range image”).

  For example, Patent Document 1 acquires a long-time exposure image and a short-time exposure image as reference images, divides each image into a proper exposure region and an improper exposure region, and performs tone correction for each proper exposure region. An image processing apparatus has been proposed that generates a single high dynamic range image by synthesizing the appropriate exposure areas subjected to gradation correction.

JP 2000-228747 A

  However, since it is determined based on the comparison between the pixel value of the target pixel and a predetermined threshold value indicating the upper limit of the appropriate exposure, whether or not the acquired reference image is a proper exposure area is a synthesized image. The pass / fail is influenced by a predetermined threshold.

  Since the predetermined threshold value used for such image synthesis is normally manually set to a fixed value by the designer of the imaging apparatus, there is a low possibility that the predetermined threshold value is suitable for each acquired reference image. Although it is conceivable that the user is allowed to set a threshold value, it is difficult for the user to set an optimal threshold value.

  Therefore, an object of the present invention is to provide a technique capable of automatically setting an optimum threshold value according to an acquired reference image and generating a composite image having a wide dynamic range.

  According to a first aspect of the present invention, there is provided an image processing apparatus for acquiring a first image relating to a predetermined subject and a second image obtained by photographing the predetermined subject under photographing conditions different from the first image. And image synthesizing means for generating a synthesized image having a wide dynamic range using the pixel values of the pixels constituting the first image and the pixel values of the pixels constituting the second image. A means for discriminating a level of exposure between the first image and the second image; and a cumulative sum obtained by adding the number of pixels equal to or less than a predetermined pixel value in each of the first image and the second image. Based on the cumulative frequency, a cumulative frequency calculation means for calculating a frequency and a threshold value used as a reference for the selection when selecting a pixel value to be used for generating the composite image from the first image or the second image. Setting means for setting each of the first image and the second image, and pixel value determining means for determining a pixel value of each pixel constituting the composite image based on the exposure level and the threshold value. Have.

  According to a second aspect of the present invention, there is provided an imaging apparatus, which is configured to capture an image of the same subject under different imaging conditions and acquire a first image and a second image, and a pixel constituting the first image. Image synthesizing means for generating a synthesized image having a wide dynamic range using pixel values and pixel values of pixels constituting the second image, wherein the image synthesizing means comprises the first image and the second image. Discriminating means for discriminating the level of exposure with respect to each other; cumulative frequency calculating means for calculating a cumulative frequency obtained by adding together the number of pixels equal to or less than a predetermined pixel value in each of the first image and the second image; When selecting a pixel value to be used for generating the composite image from one image or the second image, a threshold value used as a reference for the selection is set for each of the first image and the second image based on the cumulative frequency. A setting means for setting, based on the height ranking and the threshold value of the exposure, the pixel value determining means for determining a pixel value of each pixel constituting the composite image.

  According to the present invention, a high dynamic range image can be generated by automatically setting an optimal threshold value according to the acquired reference image.

1 is a block diagram illustrating a functional configuration of an imaging apparatus according to a first embodiment of the present invention. It is a figure which shows the function structure in a high dynamic range image generation part. FIG. 6 is a frequency distribution diagram illustrating a frequency distribution related to two reference images. It is a figure which shows the histogram regarding a high exposure image, the histogram regarding a low exposure image, and the synthetic | combination histogram obtained by synthesize | combining these histograms. It is a figure for demonstrating the pixel value determination method of a high dynamic range image. It is a figure for demonstrating the pixel value determination method of a high dynamic range image. It is a flowchart which shows operation | movement of an imaging device. It is the figure which conceptualized the mode of the production | generation of the high dynamic range image performed with the imaging device which concerns on 2nd Embodiment using the histogram. It is a block diagram which shows the function structure of the image processing apparatus which concerns on 3rd Embodiment.

  Embodiments of the present invention will be described below with reference to the drawings.

<1. First Embodiment>
[1-1. Configuration Overview]
FIG. 1 is a block diagram showing a functional configuration of an imaging apparatus 1A according to the first embodiment of the present invention.

  As shown in FIG. 1, the imaging apparatus 1A includes an imaging unit 101, an AFE (analog front end) 102, an image processing unit 103, an image memory 104, a removable recording device 105, an operation unit 106, an overall control unit 110, and the like. have.

  The imaging unit 101 includes an imaging element such as a COMS sensor or a CCD sensor, for example, and has a function of receiving light (subject light) that forms a subject image and generating an image signal related to the subject image. ing.

  The AFE 102 provides a timing pulse for executing a predetermined operation to the image sensor, performs predetermined signal processing on an image signal output from the image sensor, converts the image signal to a digital signal, and outputs the digital signal to the image processing unit 103 have.

  Specifically, the AFE 102 includes a timing control circuit, a signal processing unit, an A / D conversion unit, and the like.

  The timing control circuit generates a predetermined timing pulse (a pulse for generating a vertical scanning pulse φVn, a horizontal scanning pulse φVm, a reset signal φVr, etc.) based on a reference clock output from the overall control unit 110 and outputs the generated timing pulse to the image sensor. The imaging operation of the imaging device is controlled. In addition, the operation of the signal processing unit and the A / D conversion unit is controlled by outputting predetermined timing pulses to the signal processing unit and the A / D conversion unit, respectively.

  The signal processing unit performs predetermined analog signal processing on the analog image signal output from the image sensor. The signal processing unit includes a CDS (correlated double sampling) circuit, an AGC (auto gain control) circuit, a clamp circuit, and the like.

  The A / D conversion unit outputs a plurality of analog red (R), green (G), and blue (B) image signals output from the signal processing unit based on timing pulses output from the timing control circuit. The digital image signal is composed of bits (for example, 8 bits). The image signal converted into the digital signal is input to the image processing unit 103.

  The image signal input to the image processing unit 103 is temporarily stored as image data in the image memory 104. Thereafter, the image data stored in the image memory 104 is accessed, and various processes of the image processing unit 103 are executed.

  The image memory 104 is a high-speed accessible memory for temporarily storing generated image data, and has a capacity capable of storing image data for a plurality of frames.

  Here, various processes realized by the image processing unit 103 will be described in detail. The image processing unit 103 includes a white balance (WB) control unit 31, a pixel interpolation unit 32, a gamma correction unit 33, a YCC conversion unit 34, and a high dynamic range (HDR) image generation unit 35.

  The white balance control unit 31 performs white balance correction for converting the digital signal level of each color component of R (red), G (green), and B (blue). In this white balance correction, a portion that is originally white from a photographic subject is estimated from brightness, saturation data, and the like, and an average value, a G / R ratio, and a G / B ratio for each of R, G, and B are obtained. The level is corrected as R and B correction gains.

  In the image data stored in the image memory 104, the pixel interpolation unit 32 obtains a color component having a shortage of pixels by interpolation using color information of peripheral pixels adjacent to the pixel.

  The gamma correction unit 33 corrects the gradation characteristics of the image signal subjected to WB adjustment. Specifically, the gamma correction unit 33 performs non-linear conversion and offset adjustment for the level of the image signal for each color component, using a preset gamma correction table.

  The YCC converter 34 converts the color space of the image data. Specifically, the YCC conversion unit 34 converts a color space having RGB primary color components into a color space having a luminance component and a color difference component by matrix calculation using the following equation (1).

  Note that “R (i, j)”, “G (i, j)”, “B (i, j)”, “Y (i, j)”, “Cb (i, j)” in Expression (1) ) ”And“ Cr (i, j) ”are R signals and G signals of pixels existing at the i-th position in the X direction and the j-th position in the Y direction in the reference image when the lower left of the reference image is the origin. , B signal, luminance signal, and two color difference signals, respectively.

  By this color space conversion, the signal value (Y) of the luminance component of each pixel and the signal value (Cr, Cb) of the color difference component of each pixel that constitute the image data are acquired. Hereinafter, the signal value of the luminance component is also referred to as “luminance signal” or “luminance value”, and the signal value of the color difference component is also referred to as “color difference signal” or “color difference signal value”.

  The high dynamic range image generation unit 35 combines a plurality of images acquired by shooting the same subject under different shooting conditions to generate a combined image having a wide dynamic range (also referred to as a “high dynamic range image”). It functions as an image composition means.

  An image used for generating a high dynamic range image (HDR image) is also referred to as a reference image. Shooting conditions for shooting the reference image include shutter speed and aperture value exposure conditions, and flash emission amount, and each reference image shot under different shooting conditions has a different exposure amount. . A method for generating a high dynamic range image will be described later.

  The operation unit 106 includes various buttons and switches including a release button, a menu button for setting a shooting mode, and the like. In response to a user input operation on the operation unit 106, the overall control unit 110 or the like causes various operations to be executed. For example, when a pressing operation of the release button is detected, a shooting operation for the actual captured image is executed.

  The overall control unit 110 is configured as a microcomputer and mainly includes a CPU, a RAM, a ROM, and the like. The overall control unit 110 reads out a program stored in the ROM and executes the program on the CPU, thereby causing the exposure control unit 111, the instruction control unit 112, the imaging control unit 113, the display control unit 114, and the like to function. Realize.

  The exposure control unit 111 performs exposure control for adjusting the shutter speed and the aperture value. Specifically, the exposure control unit 111 determines the exposure value based on the luminance information of the subject acquired by the image sensor, and further sets the shutter speed and the aperture value based on the determined exposure value. As described above, the imaging apparatus 1 </ b> A can automatically calculate the exposure time by the exposure control by the exposure control unit 111.

  The instruction control unit 112 detects a user input instruction using the operation unit 106 and issues an instruction to execute a predetermined operation according to the input instruction.

  For example, when the high dynamic range image generation function is enabled in the shooting mode by operating the menu button or the like, the instruction control unit 112 causes the exposure control unit 111 to capture the shutter speed and the shutter speed when shooting the reference image. Instructs to set the aperture value. When the pressing operation of the release button is detected, the instruction control unit 112 issues a reference image shooting instruction to the shooting control unit 113 described below.

  The shooting control unit 113 has a function of controlling various shooting operations in the selected shooting mode.

  For example, when a reference image shooting instruction is received from the instruction control unit 112 with the high dynamic range image generation function enabled in the shooting mode, the shooting control unit 113 is set in the exposure control unit 111. Control is performed such that exposure is performed a plurality of times based on the shutter speed and the aperture value, and a plurality of reference images are acquired.

  The display control unit 114 controls display contents on a display unit such as the monitor 12. For example, the display control unit 114 reads the image data stored in the recording device 105 in the playback mode and displays the image.

[1-2. About high dynamic range image generation]
Here, a high dynamic range image generation process executed by the imaging apparatus 1A will be described in detail.

  As described above, the imaging apparatus 1A acquires a plurality of captured images (reference images) of the same subject under different shooting conditions in a state where the high dynamic range image generation function is enabled in the shooting mode. By synthesizing a plurality of reference images, a high dynamic range image can be generated.

  The high dynamic range image generation process is mainly executed by the high dynamic range image generation unit 35. FIG. 2 is a diagram illustrating a functional configuration in the high dynamic range image generation unit 35.

  As shown in FIG. 2, the high dynamic range image generation unit 35 corresponds to a frequency distribution acquisition unit 351, an exposure level ranking determination unit (discrimination unit) 352, a cumulative frequency calculation unit (cumulative frequency calculation unit) 353, and A luminance value specifying unit (specifying unit) 354, a boundary luminance value searching unit (searching unit) 355, and a pixel value determining unit (pixel value determining unit) 356 are provided.

  When a plurality of reference images are acquired by the imaging unit 101, each reference image is input to the image processing unit 103 via the AFE 102 and stored in the image memory 104. Then, after color space conversion or the like is performed by the YCC conversion unit 34 of the image processing unit 103, the reference image is input to the high dynamic range image generation unit 35. In the following, a case where a high dynamic range image is generated by combining two reference images acquired under different shooting conditions will be exemplified. The pixel value of each pixel constituting the reference image includes a luminance value and a color difference signal value, and each of the luminance value and the color difference signal value is represented by 8 bits.

  The frequency distribution acquisition unit 351 compares the luminance values of the pixels constituting the two reference images, and acquires a frequency distribution representing the frequency (number) of pixels with respect to the luminance value. FIG. 3 is a frequency distribution diagram (histogram) HG1 and HG2 showing the frequency distribution of two reference images.

  The exposure level determination unit 352 determines the order (also referred to as “exposure level” or “exposure level”) as to which of the two input reference images is an image with a large exposure amount. To do. The reference image exposure level can be determined by comparing the feature amounts of the reference images.

  As the feature amount of the reference image, for example, an average luminance value of the reference image can be adopted. Specifically, the exposure level ranking determining unit 352 calculates the average brightness value of each reference image, and compares the average brightness value to determine the exposure level. That is, an image that is determined to have a high average luminance value by comparing the average luminance values of two reference images is a high-exposure image KH with a relatively large exposure amount, and an image that is determined to have a low average luminance value is The low exposure image KL has a relatively small exposure amount. For example, in FIG. 3, the reference image configured with the luminance value represented by the histogram HG1 is determined as the high exposure image KH, and the reference image configured with the luminance value represented by the histogram HG2 is the low exposure image KL. Is determined.

Based on the frequency distribution acquired by the frequency distribution acquisition unit 351, the cumulative frequency calculation unit 353 calculates a cumulative frequency (also referred to as “cumulative pixel number”) that is the sum of all frequencies below a predetermined luminance value. Specifically, the cumulative frequency “P _KN (q)” below the predetermined luminance value “q” related to the reference image KN is calculated based on the following equation (2).

In Equation (2), H _KN (Y) indicates the frequency of the pixel having the luminance value “Y” in the reference image KN.

  The cumulative frequency calculation unit 353 calculates the cumulative frequency for each luminance value for each reference image. Therefore, when the luminance value is represented by 8 bits, 256 from the luminance value “0” to the luminance value “255”. The cumulative frequency is calculated for each reference image.

The corresponding luminance value specifying unit 354 compares the cumulative frequency “P _KN (Y)” of each reference image acquired by the cumulative frequency calculating unit 353, and the cumulative frequency between the two reference images is the closest. A combination of luminance values (also referred to as “corresponding luminance values”) is specified.

  Specifically, in the corresponding luminance value specifying unit 354, of the two reference images, the other reference image (for example, for the cumulative frequency at a predetermined luminance value of one reference image (for example, the high-exposure image KH)). In the low-exposure image KL), the luminance value that is the closest cumulative frequency is specified. Such identification of the corresponding luminance value is performed for each luminance value of one reference image. When the luminance value is represented by 8 bits, 256 sets of corresponding luminance values are specified between the two reference images. The

In addition, such a process for specifying the corresponding luminance value can also be expressed as a process for specifying a set of luminance values when the difference between the cumulative frequencies in the two reference images is minimized. That is, the identification process of the corresponding luminance value is expressed by the equation (3) expressed using the cumulative frequency “P _K1 (Y1)” of the reference image _K1 and the cumulative frequency “P _K2 (Y2)” of the reference image K2. It can be considered that this is a process for obtaining a set of luminance values that minimizes the value calculated by.

  When the set of corresponding luminance values is specified using Expression (3), first, the luminance value “Y1” in one of the two reference images is fixed to a certain value, and the above Expression (3) is used. The luminance value “Y2” that minimizes the value calculated in step) is specified. When the brightness value corresponding to the fixed-side brightness value “Y1” is specified, the corresponding brightness value specifying process using the formula (3) is performed again by changing the fixed value. By repeating such a series of processes, 256 sets of corresponding luminance values are specified.

  The boundary luminance value search unit (also referred to as “joint luminance value search unit”) 355 is a frequency distribution related to two reference images from among a plurality of combinations of corresponding luminance values specified by the corresponding luminance value specifying unit 354. Search for a combination of boundary luminance values when combining.

  A search for a set of boundary luminance values is performed when a frequency distribution (also referred to as a “composite frequency distribution”) is obtained by synthesizing a frequency distribution related to two reference images using the corresponding luminance value set. This is done by specifying a set of corresponding luminance values that maximizes the dynamic range of the luminance values that represent the class of.

  Specifically, the boundary luminance value search process is conceptualized and described with reference to FIG. FIG. 4 shows a histogram HG1 related to the high-exposure image KH and a histogram HG2 related to the low-exposure image KL, and a histogram (also referred to as “pseudo-composite histogram” or “composite histogram”) HG3 obtained by combining the histograms HG1 and HG2. Is shown.

  As shown in FIG. 4, for example, the luminance value “a” in the high-exposure image KH and the luminance value “b” in the low-exposure image KL are specified as a set of corresponding luminance values, and the set of the corresponding luminance values The frequency distribution is synthesized using In this case, when the frequency distribution composition processing is represented on the histogram, the frequency distribution composition is performed for each frequency from the luminance value “0” to the luminance value “a” in the high-exposure image KH and the luminance value in the low-exposure image KL. This corresponds to generating the combined histogram HG3 by combining the frequencies from “b” to the luminance value “255”.

  Note that the number of pixels (number of pixels) constituting the synthesis frequency distribution is substantially equal to the number of pixels of each reference image. This is because the corresponding luminance value specifying unit 354 specifies a set of luminance values having the closest cumulative frequency between two reference images as a set of corresponding luminance values, and a combined frequency distribution is generated using the set of corresponding luminance values. It is because it is obtained. Therefore, the corresponding luminance value specifying unit 354 can be expressed as a set of corresponding luminance values in which a set of luminance values in which the number of pixels of the reference image and the number of pixels constituting the synthesis frequency distribution substantially match is specified.

  As described above, the composite frequency distribution uses the determination result in the exposure level determination unit 352 and the corresponding luminance value, and the pixel information of the low luminance band in the high exposure image KH from the two reference images and the low exposure image KL. The pixel information of the high luminance band is extracted, and the extracted information is combined.

  At this time, in the combined histogram HG3, the range of the luminance value indicating the class is “0” to “255 + ab”, so that the dynamic range of the combined histogram HG3, that is, the dynamic range D (a, b) is expressed by the equation (4).

  Here, in the process of searching for the boundary luminance value pair, as described above, the process of specifying the corresponding luminance value pair that maximizes the dynamic range of the luminance value of the composite frequency distribution as the boundary luminance value pair is executed. The Therefore, in the boundary luminance value search unit 355, the corresponding luminance values “a” and “b” that maximize the dynamic range “D (a, b)” expressed by the above equation (4) are the boundary luminance value “L1”. , “L2”.

  As described above, the boundary luminance value search unit 355 combines the corresponding luminance values when the value obtained by subtracting the corresponding luminance value “b” in the low exposure image KL from the corresponding luminance value “a” in the high exposure image KH is the largest. Are searched from a plurality of combinations of corresponding luminance values. Then, the searched pair of corresponding luminance values is set as the boundary luminance value “L1” in the high exposure image KH and the boundary luminance value “L2” in the low exposure image KL, respectively.

  In the above description, the histogram is used to facilitate understanding of the search for the boundary luminance value. However, in the actual boundary luminance value search process, the histogram need not be generated.

  The pixel value determination unit 356 has pixel values of the pixels of the boundary luminance values “L1” and “L2” specified by the boundary luminance value search unit 355 and two reference images (high exposure image KH and low exposure image KL). Based on the above, the pixel value of each pixel of the high dynamic range image is determined.

  Specifically, the pixel value determination unit 356 uses a pixel value smaller than the boundary luminance value “L1” in the high exposure image KH and a pixel value larger than the boundary luminance value “L2” in the low exposure image KL. The pixel value of the high dynamic range image is determined.

More specifically, the luminance value “Y _H (i, j) of a pixel existing at the i-th position in the X direction and the j-th position in the Y direction in the high dynamic range image when the lower left corner of the high dynamic range image is the origin. "Can be calculated using equation (5).

  Here, the brightness value determination method using Formula (5) will be described in detail. When determining the luminance value of the pixel at the position (i, j) in the high dynamic range image, first, the luminance value of the pixel at the position (i, j) of each of the high exposure image KH and the low exposure image KL is acquired. .

When the luminance value “Y _KH (i, j)” of the pixel at the position (i, j) of the high exposure image KH is smaller than the boundary luminance value “L1”, the luminance value “Y _KH of the high exposure image _KH. (I, j) ”is adopted as the luminance value“ Y _H (i, j) ”of the high dynamic range image. When the luminance value “Y _KL (i, j)” of the pixel at the position (i, j) of the low exposure image KL is larger than the boundary luminance value “L2”, the luminance value “Y _KL ” of the low exposure image _KL. A value obtained by correcting (i, j) ”with the boundary luminance value is adopted as the luminance value“ Y _H (i, j) ”of the high dynamic range image. Here, even when the luminance value “Y _KL (i, j)” of the pixel at the position (i, j) of the low-exposure image KL is equal to the boundary luminance value “L2”, the luminance value of the low-exposure image KL. A value obtained by correcting “Y _KL (i, j)” with the boundary luminance value is adopted as the luminance value “Y _H (i, j)” of the high dynamic range image.

Further, the color difference signal value “Cb _H (i, j)” of the pixel at the position (i, j) in the high dynamic range image can be calculated using Expression (6).

More specifically, when the color difference signal value “Cb _H (i, j)” of the pixel at the position (i, j) in the high dynamic range image is determined, the pixel of the pixel at the position (i, j) of the high exposure image KH is determined. When the luminance value “Y _KH (i, j)” is smaller than the boundary luminance value “L1”, the color difference signal value “Cb _KH (i, j)” of the high exposure image KH is the color difference signal of the high dynamic range image. It is adopted as the value “Cb _H (i, j)”. Further, when the luminance value “Y _KL (i, j)” of the pixel at the position (i, j) of the low-exposure image KL is larger than the boundary luminance value “L2”, the color difference signal value “Cb” of the low-exposure image KL. _KL (i, j) ”is adopted as the color difference signal value“ Cb _H (i, j) ”of the high dynamic range image. Here, even when the luminance value “Y _KL (i, j)” of the pixel at the position (i, j) of the low-exposure image KL is equal to the boundary luminance value “L2”, the color difference signal of the low-exposure image KL. It is assumed that the value “Cb _KL (i, j)” is adopted as the color difference signal value “Cb _H (i, j)” of the high dynamic range image.

The pixel value determination unit 356 also uses the same principle as the above-described method for determining the color difference signal value “Cb _H (i, j)” for the other color difference signal value “Cr _H (i, j)”. And is calculated using the following equation (7).

  As described above, the pixel value determination unit 356 uses the pixel value used to generate the high dynamic range image from each reference image based on the magnitude relationship between the boundary luminance value of the reference image and the luminance value of each pixel of the reference image. Is selected.

  Note that the boundary luminance value set for each reference image is used as a selection criterion when selecting a pixel value to be used for generating a high dynamic range image from each reference image. Based on such a viewpoint, the corresponding luminance value specifying unit 354 and the boundary luminance value searching unit 355 serve as setting means for setting a reference value when selecting a pixel value used for generating a high dynamic range image from each reference image. It can also be expressed as functioning.

  Further, in the pixel value determination method as described above, in a pixel of a high dynamic range image, a pixel value whose pixel value is adopted redundantly from the high exposure image KH and the low exposure image KL (also referred to as “overlapping pixel”). ) Or pixels whose pixel values are not determined (also referred to as “blank pixels”). 5 and 6 are diagrams for explaining a pixel value determination method for a high dynamic range image.

  Specifically, when determining the luminance value of the pixel at a certain position (iw, jw) of the high dynamic range image, the luminance value “Yw1” of the pixel at the position (iw, jw) of the high-exposure image KH is Assume that the luminance value “Yw2” of the pixel at the position (iw, jw) of the low-exposure image KL is smaller than the boundary luminance value “L1” and larger than the boundary luminance value “L2”. In this case, according to the above equation (5), the two luminance values of the luminance value “Yw1” based on the high exposure image KH and the luminance value “Yw2” based on the low exposure image KL are the luminance values of the high dynamic range image. Will be calculated.

  As described above, when the pixel values are calculated overlappingly, the difference between the calculated pixel value and the boundary luminance values “L1” and “L2” in the frequency distribution of each of the high exposure image KH and the low exposure image KL. May be calculated, and the pixel value of the high dynamic range image may be determined based on the pixel value having the larger absolute value. For example, in the histograms HG1 and HG2 of the high exposure image KH and the low exposure image KL shown in FIG. 5, the difference between the luminance value Yw1 and the boundary luminance value “L1” in the high exposure image KH is lower. It is larger than the difference between the luminance value Yw2 and the boundary luminance value “L2” in the image KL. In this case, since the pixel value having a large difference with respect to the boundary luminance value is the luminance value Yw1 in the high exposure image KH, the luminance value Yw1 in the high exposure image KH is adopted as the pixel value of the high dynamic range image.

  When determining the luminance value of the pixel at a certain position (iv, jv) in the high dynamic range image, the luminance value “Yv1” of the pixel at the position (iv, jv) of the high-exposure image KH is the boundary luminance value. Assume that the luminance value “Yv2” of the pixel at a position (iv, jv) of the low-exposure image KL that is larger than “L1” is smaller than the boundary luminance value “L2”. In this case, according to the above formula (5), the luminance value “Yv1” based on the high exposure image KH and the luminance value “Yv2” based on the low exposure image KL are both calculated as the luminance values of the high dynamic range image. Will not be.

  Thus, when the pixel value is not calculated, the absolute value of the difference between the calculated pixel value and the boundary luminance values “L1” and “L2” in the frequency distribution of each of the high exposure image KH and the low exposure image KL is calculated. The pixel value of the high dynamic range image may be determined based on the calculated pixel value with the smaller absolute value. For example, in the histograms HG1, HG2 of the high exposure image KH and the low exposure image KL shown in FIG. 6, the difference between the luminance value Yv2 and the boundary luminance value “L2” in the low exposure image KL is higher. It is smaller than the difference between the luminance value Yv1 and the boundary luminance value “L1” in the image KH. In this case, since the pixel value having a small difference with respect to the boundary luminance value is the luminance value Yv2 in the low-exposure image KL, the pixel value of the high dynamic range image is determined based on the luminance value Yv2 in the low-exposure image KL. become.

[1-3. Operation]
Next, the operation of the imaging apparatus 1A will be described. FIG. 7 is a flowchart showing the operation of the imaging apparatus 1A.

  As shown in FIG. 7, when the high dynamic range image generation function is enabled in the shooting mode and the release button is pressed in step SP11, the process proceeds to step SP12.

  In step SP12, a photographing operation for photographing the same subject twice under different photographing conditions is executed, and two reference images are acquired.

  In step SP13, predetermined image processing such as WB correction and YCC conversion is performed on the two reference images.

  In step SP14, the frequency distribution acquisition unit 351 acquires a frequency distribution representing the frequency of the pixel with respect to the luminance value for each reference image.

  In step SP15, the exposure height order determination unit 352 determines the exposure level order of the two reference images, and determines the high exposure image KH and the low exposure image KL.

  In step SP16, the cumulative frequency calculation unit 353 calculates the cumulative frequency at each luminance value for each reference image.

  In step SP17, the corresponding luminance value specifying unit 354 specifies the luminance value (corresponding luminance value) that is the cumulative frequency corresponding to each other between the two reference images.

  In step SP18, the boundary luminance value search unit 355 searches the corresponding luminance value for a boundary luminance value for combining the frequency distribution of the high exposure image KH and the frequency distribution of the low exposure image KL.

  In step SP19, the pixel value determination unit 356 determines the pixel value of each pixel of the high dynamic range image based on the pixel value of each pixel of the high exposure image KH and the pixel value of each pixel of the low exposure image KL. The

  In step SP20, the pixel value of the high dynamic range image is adjusted according to the output format of the high dynamic range image. For example, when a high dynamic range image is output with 8 bits, the pixel value determined in step SP19 is subjected to compression processing so that the maximum luminance value of the high dynamic range image is 255.

  As described above, the imaging apparatus 1A includes the high dynamic range image generation unit 35 that generates a high dynamic range image by combining two reference images obtained by shooting the same subject under different shooting conditions, and includes a high dynamic range image. The generation unit 35 calculates the cumulative frequency obtained by adding the number of pixels equal to or less than a predetermined pixel value in each of the two reference images and the exposure level determination unit 352 that determines the level of exposure in the two reference images. When selecting a pixel value to be used for generating a high dynamic range image from two reference images and a cumulative luminance calculating unit 353, the boundary luminance value used as the selection criterion is referred to based on the cumulative frequency. A high dynamic range image is constructed on the basis of setting means for setting each image, the level of exposure and the boundary luminance value. And a pixel value determining unit 356 determines the pixel value of the pixel.

  According to such an imaging apparatus 1A, it is possible to automatically set an optimum threshold value each time according to the acquired reference image and generate a high dynamic range image.

<2. Second Embodiment>
Next, a second embodiment of the present invention will be described. In the imaging apparatus 1A according to the first embodiment, two reference images are combined to generate a high dynamic range image. However, in the imaging apparatus 1B according to the second embodiment, the high reference range is obtained from three reference images. Generate a dynamic range image. Note that the imaging apparatus 1B has substantially the same structure and function (see FIGS. 1 and 2) as the imaging apparatus 1A, and common portions are denoted by the same reference numerals and description thereof is omitted.

  In the imaging apparatus 1B of the second embodiment, the imaging unit 101 (see FIG. 1) captures the same subject three times under different imaging conditions, and acquires three reference images.

  Then, the imaging apparatus 1B generates a high dynamic range image by combining the three reference images with the high dynamic range image generation unit 35.

  Specifically, as shown in FIG. 2, in the high dynamic range image generation unit 35, the frequency distribution for each of the three reference images is acquired by the frequency distribution acquisition unit 351.

  The exposure level ranking determination unit 352 determines the level of exposure level for the three reference images. Specifically, the exposure level ranking determination unit 352 calculates the average luminance value of each of the three reference images. Then, by comparing the calculated average luminance values, the high exposure image KH having the highest average luminance value, the low exposure image KL having the lowest average luminance value, and the medium exposure image KM having an intermediate average luminance value. Are identified.

  The cumulative frequency calculation unit 353 calculates the cumulative frequency at each luminance value for each reference image.

  In the corresponding luminance value specifying unit 354, luminance values (corresponding luminance values) corresponding to the cumulative frequencies corresponding to each other between the high exposure image KH and the medium exposure image KM and between the medium exposure image KM and the low exposure image KL, respectively. ) Is identified.

  The boundary luminance value search unit 355 combines the frequency distribution of the high exposure image KH and the frequency distribution of the medium exposure image KM from the set of corresponding luminance values between the high exposure image KH and the medium exposure image KM. A set of boundary luminance values “L11” and “L12” is searched. Further, the boundary luminance value search unit 355 combines the frequency distribution of the medium exposure image KM and the frequency distribution of the low exposure image KL from the set of corresponding luminance values between the medium exposure image KM and the low exposure image KL. A set of boundary luminance values “L13” and “L14” is searched. Thus, in the imaging device 1B, the high-exposure image KH and the low-exposure image are captured by the imaging device 1A between the high-exposure image KH and the medium-exposure image KM and between the medium-exposure image KM and the low-exposure image KL, respectively. The same processing as the search processing for the set of boundary luminance values performed with KL is performed.

  In the pixel value determination unit 356, each pixel of the high dynamic range image is determined based on the pixel value of each pixel of the high exposure image KH, the pixel value of each pixel of the medium exposure image KM, and the pixel value of each pixel of the low exposure image KL. The pixel value of the pixel is determined.

  Here, the high dynamic range image generation processing performed in the imaging apparatus 1B will be conceptualized and described using a histogram. FIG. 8 is a diagram conceptualized using a histogram of the generation of a high dynamic range image executed by the imaging apparatus 1B.

  As shown in FIG. 8, the boundary luminance value search process between the high exposure image KH and the medium exposure image KM is searched as “L11” and “L12” by the boundary luminance value search process, and the medium exposure image KM. Assume that a set of boundary luminance values between the image and the low-exposure image KL is searched as “L13” and “L14”. In this case, the histogram (composite histogram) HG4 of the high dynamic range image includes histogram information in the range from the minimum luminance value “0” to the boundary luminance value “L11” in the high exposure image KH and the boundary luminance value “ Histogram information in the range from “L12” to the boundary luminance value “L13” and histogram information in the range from the boundary luminance value “L14” to the maximum luminance value “255” in the low-exposure image KL.

  In other words, the high dynamic range image in the imaging apparatus 1B includes each pixel from the minimum luminance value “0” to the boundary luminance value “L11” in the high exposure image KH and the boundary luminance value from the boundary luminance value “L12” in the medium exposure image KM. It can also be expressed as being generated using each pixel of “L13” and each pixel of the maximum luminance value “255” from the boundary luminance value “L14” in the low exposure image KL.

The following formulas (8) to (10) are used to determine the pixel value of each pixel in the high dynamic range image. Specifically, the luminance value “Y _HB (i, j) of a pixel existing at the i-th position in the X direction and the j-th position in the Y direction in the high dynamic range image when the lower left of the high dynamic range image is the origin. "Is calculated using equation (8).

Further, the color difference signal values “Cb _HB (i, j)” and “Cr _HB (i, j)” of the pixel at the position (i, j) in the high dynamic range image are expressed by equations (9) and (10), respectively. Is used to calculate.

  As described above, in the imaging apparatus 1B, the order of exposure level of the three reference images is determined, and two sets of reference images having similar levels of exposure level are specified based on the determination result. Then, a process similar to the process of combining the two reference images executed by the imaging apparatus 1A is performed twice with different combinations, and a high dynamic range image is generated.

<3. Third Embodiment>
Next, a third embodiment of the present invention will be described. In each of the above embodiments, the high dynamic range image is generated in the imaging devices 1A and 1B. However, in the third embodiment, an image processing device (for example, a personal computer (PC) is used with the captured reference image as input information ND. ), The portable information terminal) 50, and the image processing apparatus 50 executes a high dynamic range image generation process. FIG. 9 is a block diagram illustrating a functional configuration of the image processing apparatus 50 according to the third embodiment. Note that the image processing apparatus 50 has many functions that are common to the imaging apparatus 1A according to the first embodiment, and the common functions are denoted by the same reference numerals and description thereof is omitted.

  The image processing apparatus 50 according to the third embodiment includes a data acquisition unit (image acquisition unit) 501, a data storage unit 502, a high dynamic range (HDR) image generation unit 35 included in the image processing unit 503, and overall control. Part 510.

  In the image processing apparatus 50, the reference image input as the input information ND is stored in the data storage unit 502 via the data acquisition unit 501. Then, the reference image stored in the data storage unit 502 is accessed in accordance with an instruction from the overall control unit 510, and the high dynamic range image generation unit 35 performs high dynamic range image generation processing.

  The high dynamic range image generation unit 35 has the same configuration as that of the first embodiment (see FIG. 2), and generates a high dynamic range image by combining a plurality of reference images.

<4. Modification>
Although the embodiments of the present invention have been described above, the present invention is not limited to the contents described above.

  For example, in each of the above embodiments, the pixel value represented by 8 bits is used as the pixel value of each pixel constituting the reference image. However, the present invention is not limited to this, and the number of bits of the pixel value of the reference image is It may be changed.

  Specifically, the number of bits of the pixel value may be increased before starting the generation process of the high dynamic range image. For example, when the bit number of the pixel value is expressed by 10 bits, the cumulative frequency calculation unit 353 calculates 1024 cumulative frequencies from the luminance value “0” to the luminance value “1023” for each reference image. . Then, the corresponding luminance value specifying unit 354 specifies 1024 sets of corresponding luminance values between two reference images, and the boundary luminance value searching unit 355 searches for boundary luminance values from among the 1024 sets of corresponding luminance values. Will be. That is, when the number of bits of the pixel value of the reference image is increased, it is possible to accurately set the boundary luminance value at the time of combining the frequency distribution related to the two reference images.

  As described above, when the number of bits of the pixel value in the reference image is increased before the generation process of the high dynamic range image is started, it is possible to generate a smooth high dynamic range image with less distortion due to synthesis.

  On the other hand, the number of bits of the pixel value may be reduced before starting the generation process of the high dynamic range image. For example, when the bit number of the pixel value is expressed by 6 bits, the cumulative frequency calculation unit 353 calculates 64 cumulative frequencies from the luminance value “0” to the luminance value “63” for each reference image. . Then, the corresponding luminance value specifying unit 354 specifies 64 sets of corresponding luminance values between the two reference images, and the boundary luminance value searching unit 355 searches for the boundary luminance value from among the 64 sets of corresponding luminance values. Will be.

  That is, if the number of bits of the pixel value in the reference image is reduced before starting the generation process of the high dynamic range image, it is possible to reduce the amount of computation required for the search for the boundary luminance value. Processing can be performed at high speed.

  As described above, by changing the number of bits of the pixel value in the reference image before starting the generation process of the high dynamic range image, the high dynamic range according to the use of the high dynamic range image and / or the shooting environment. The image generation process can be optimized.

  Note that the process of changing the number of bits of the pixel value can be performed by, for example, the YCC conversion unit 34. For example, when the number of bits of the pixel value is increased, fraction processing that increases the number of bits may be performed at the time of color space conversion using Expression (1). On the other hand, when reducing the number of bits of the pixel value, a process of compressing the number of bits after color space conversion may be performed.

  Further, in the exposure level determination unit 352 of each of the above embodiments, the average luminance value of the reference image is used as the feature amount of the reference image used for the determination of the exposure level. However, the present invention is not limited to this.

  Specifically, the median value of the histogram relating to the luminance value of the reference image (also referred to as “central gradation value” or “central luminance value”) can be used as the feature amount of the reference image.

  Further, the exposure height order determination unit 352 may determine the exposure height order based on the exposure value determined by the exposure control unit 111.

  In the first embodiment and the second embodiment, the case where the high dynamic range image is generated by synthesizing the two reference images and the three reference images, respectively, is illustrated, but the present invention is not limited to this.

  Specifically, the high dynamic range image can be generated by synthesizing N reference images. When generating a high dynamic range image based on N reference images, N-1 sets of combinations of reference images based on exposure level are specified, and in each set, the same as the synthesis of two reference images Processing based on the principle of is executed.

1A, 1B Imaging device 50 Image processing device 103, 503 Image processing unit 34 YCC conversion unit 35 High dynamic range (HDR) image generation unit 351 Frequency distribution acquisition unit 352 Exposure high / low rank discrimination unit 353 Cumulative frequency calculation unit 354 Corresponding luminance value specification 355 Boundary luminance value search unit 356 Pixel value determination unit 110, 510 Overall control unit 111 Exposure control unit 112 Instruction control unit 113 Shooting control unit 114 Display control unit HG1 to HG4 Frequency distribution diagram (histogram)
KH High exposure image KL Low exposure image KM Medium exposure image

Claims (10)

Image acquisition means for respectively acquiring a first image relating to a predetermined subject and a second image obtained by photographing the predetermined subject under photographing conditions different from those of the first image;
Image synthesizing means for generating a synthesized image having a wide dynamic range using a pixel value of a pixel constituting the first image and a pixel value of a pixel constituting the second image;
With
The image composition means includes
Discriminating means for discriminating the order of exposure of the first image and the second image;
A cumulative frequency calculating means for calculating a cumulative frequency obtained by adding together the number of pixels equal to or less than a predetermined pixel value in each of the first image and the second image;
When selecting a pixel value to be used for generating the composite image from the first image or the second image, a threshold value used as a reference for the selection is set based on the cumulative frequency, and the first image and the second image, respectively. Setting means to set about,
Pixel value determining means for determining a pixel value of each pixel constituting the composite image based on the exposure level and the threshold;
An image processing apparatus. The cumulative frequency calculating means calculates the cumulative frequency for each pixel value of the first image and the second image,
The setting means includes
A specification that compares the cumulative frequency for each pixel value in the first image with the cumulative frequency for each pixel value in the second image, and identifies a plurality of combinations of corresponding pixel values that are close to each other. Means,
Search means for searching for a combination of corresponding pixel values that maximizes the dynamic range of the composite image from among a plurality of combinations of the corresponding pixel values specified by the specifying means;
Including
The image processing according to claim 1, wherein the setting unit sets a combination of corresponding pixel values searched by the search unit as a set of a first threshold value in a high exposure image and a second threshold value in a low exposure image. apparatus. The discriminating unit discriminates a high-exposure image with a relatively large exposure amount and a low-exposure image with a relatively small exposure amount from the first image and the second image based on the level of exposure. And
The search means uses a combination of the corresponding pixel value that maximizes the value obtained by subtracting the corresponding pixel value in the low-exposure image from the corresponding pixel value in the high-exposure image as the second threshold in the low-exposure image. The image processing apparatus according to claim 2, wherein the image processing apparatus is set as a pair with a threshold value. The pixel value determining means includes
The pixel value of the composite image is determined using a pixel value smaller than the first threshold value in the high-exposure image and a pixel value larger than the second threshold value in the low-exposure image. The image processing apparatus described. The pixel value determining means includes
When the pixel value of a pixel existing at a predetermined position in the high exposure image is smaller than the first threshold value, the pixel value exists at the same position as the predetermined position in the composite image based on the pixel value in the high exposure image. Determine the pixel value of the pixel to be
When a pixel value of a pixel existing at the same position as the predetermined position in the low-exposure image is larger than the second threshold value, the predetermined position in the composite image is determined based on the pixel value in the low-exposure image. The image processing apparatus according to claim 3 or 4, wherein a pixel value of a pixel existing at the same position is determined. The pixel value determining means includes
A pixel value of a pixel existing at a predetermined position in the high exposure image is larger than the first threshold value, and a pixel value of a pixel existing at the same position as the predetermined position in the low exposure image is the second value. When it is smaller than the threshold value, the absolute value of the difference between the pixel value of the pixel existing at the predetermined position in the high exposure image and the first threshold value is present at the same position as the predetermined position in the low exposure image. Comparing the absolute value of the difference between the pixel value of the pixel and the second threshold value, and based on the pixel value of the smaller absolute value, the pixel value of the pixel existing at the same position as the predetermined position in the composite image The image processing device according to claim 3, wherein the image processing device is determined. The pixel value determining means includes
A pixel value of a pixel existing at a predetermined position in the high exposure image is smaller than the first threshold value, and a pixel value of a pixel existing at the same position as the predetermined position in the low exposure image is the second value. When larger than the threshold, the absolute value of the difference between the pixel value of the pixel existing at the predetermined position in the high exposure image and the first threshold is present at the same position as the predetermined position in the low exposure image. Comparing the pixel value of the pixel with the absolute value of the difference between the second threshold value, and based on the pixel value having the larger absolute value, the pixel value of the pixel existing at the same position as the predetermined position in the composite image The image processing device according to claim 3, wherein the image processing device is determined. Changing means for changing the number of bits of the pixel value of each pixel in the first image and the second image;
The image processing apparatus according to claim 1, wherein the cumulative frequency calculation unit calculates the cumulative frequency for each changed pixel value. Photographing means for photographing the same subject under different photographing conditions and obtaining a first image and a second image, respectively;
Image synthesizing means for generating a synthesized image having a wide dynamic range using a pixel value of a pixel constituting the first image and a pixel value of a pixel constituting the second image;
With
The image composition means includes
Discriminating means for discriminating the order of exposure of the first image and the second image;
A cumulative frequency calculating means for calculating a cumulative frequency obtained by adding together the number of pixels equal to or less than a predetermined pixel value in each of the first image and the second image;
When selecting a pixel value to be used for generating the composite image from the first image or the second image, a threshold value used as a reference for the selection is set based on the cumulative frequency, and the first image and the second image, respectively. Setting means to set about,
Pixel value determining means for determining a pixel value of each pixel constituting the composite image based on the exposure level and the threshold;
An imaging apparatus having a) obtaining a first image relating to a predetermined subject and a second image obtained by photographing the predetermined subject under photographing conditions different from the first image;
b) generating a composite image having a wide dynamic range using a pixel value of a pixel constituting the first image and a pixel value of a pixel constituting the second image;
With
The step b)
b-1) determining the level of exposure of the first image and the second image;
b-2) calculating a cumulative frequency obtained by adding together the number of pixels equal to or less than a predetermined pixel value in each of the first image and the second image;
b-3) When selecting a pixel value to be used for generating the composite image from the first image or the second image, a threshold value used as a reference for the selection is set based on the cumulative frequency and the first image and the Setting for each of the second images;
b-4) determining a pixel value of each pixel constituting the composite image based on the exposure level and the threshold;
A method for generating the composite image.

Images