JP5487884B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, an imaging apparatus, an electronic apparatus, and the like suitable for generating image data with an extended dynamic range by combining a plurality of types of captured image data having different exposure amounts.

  Conventionally, image data (HDR image) in which the same subject is imaged with a plurality of types of exposure amounts (exposure times), and pixel signals corresponding to the plurality of types of exposure amounts obtained by the imaging are combined to expand the dynamic range. Data). In such a technique, noise (false color, pseudo contour, etc.) generated in a boundary region between an image portion corresponding to each exposure amount and an image portion corresponding to another exposure amount in the composite image is often a problem. .

  For example, the same subject is imaged at exposure times T1 and T2 (T1 <T2) corresponding to the exposure amounts L1 and L2 (L1 <L2), and pixel signals D1 (corresponding to T1) and D2 (corresponding to T2) are obtained. Suppose that In this case, as shown in FIGS. 12A and 12B, normalization is performed by multiplying the pixel signal D1 by a coefficient “T2 / T1” in order to match the slope of D1 with the slope of D2. Next, the normalized “D1 × T2 / T1” and “D2” are linearly synthesized. As a result, as shown in FIG. 12C, a captured image signal with an extended dynamic range (good S / N) is obtained.

However, if there is a slight difference between the slope of “D1 × T2 / T1” and the slope of D2, the linearity of the captured image signal after synthesis may be lost. If the linearity is lost, noise that may cause false colors or pseudo contours may occur in the boundary region between the “D1 × T2 / T1” image and the D2 image.
In order to solve such a problem, weighted image synthesis has been conventionally used. For example, in Patent Document 1, functions f and g whose weights change based on lighter brightness information among two sets of synthesized images are prepared, and a synthesized image D0 is obtained by the following equation (1). ing.

D0 = (D1 × T2 / T1) × f (D2) + D2 × g (D2) (1)

Further, in Patent Document 2, an overlap region (80% to 100% of a signal) is set in advance, and when two images are combined, an image having a longer exposure time reaches the overlap region. That is, weighting synthesis is performed with an image having a shorter exposure time that is less likely to be saturated in the highlight portion.

Japanese Patent No. 3084189 Japanese Patent No. 2970440

  However, in the prior art disclosed in Patent Document 1, when four images having different exposure amounts are weighted and combined, the combining method using the above equation (1) only supports two images. Absent. Therefore, for example, in order to perform weighted synthesis of four pixels D1 to D4, it is necessary to perform synthesis processing six times in total as shown in FIG. 13, and the calculation amount jumps as the number of images to be synthesized increases. Increase.

Moreover, in the prior art of the above-mentioned Patent Document 2, since the area where the weighted composition is performed is limited to the overlap area, an unnatural joint is formed between the highlighted part that is weighted and the part that is not. May occur.
Therefore, the present invention has been made paying attention to such unsolved problems of the conventional technology. According to some aspects of the present invention, it is possible to provide an image processing device, an image processing method, an imaging device, and an electronic apparatus suitable for weighted synthesis of images obtained by multistage exposure.

In order to achieve the above object, one image processing apparatus according to the present invention is an image processing apparatus that performs image composition using three or more types of images obtained by imaging a subject with three or more types of exposure amounts. A weight calculation unit that calculates the weight of the first image in the image composition based on the luminance of the first image and the luminance of the second image, and the weight calculated by the weight calculation unit. An image synthesis unit that performs the image synthesis; and an image analysis unit that generates information for adjusting the three or more types of exposure amounts based on statistical information obtained from an image obtained as a result of the image synthesis. The exposure amount of the image is an exposure amount other than the minimum exposure amount and the maximum exposure amount among the three or more types of exposure amounts, and the exposure amount of the second image is greater than the exposure amount of the first image. Larger than the above three types Wherein the of the next large exposure amount of the exposure amount of the first image in the exposure amount.
In the one image processing apparatus described above, in the image captured with the three or more types of exposure amounts, the image captured with the minimum exposure amount is set as a third image, and the next largest exposure amount after the third image When the image captured in step 4 is the fourth image, the weight calculation unit calculates the weight of the third image based on the luminance of the fourth image and captures the image with the maximum exposure amount. When the image is the fifth image and the image captured with the second smallest exposure amount after the fifth image is the sixth image, the weight calculation unit determines that the weight of the fifth image is the first image. It is preferable to calculate based on the luminance of the six images.
In the one image processing apparatus described above, it is preferable that the statistical information is an average luminance obtained from a plurality of results of the image synthesis.
In order to achieve the above object, one image processing method according to the present invention is an image processing method for performing image composition using three or more images obtained by imaging a subject with three or more exposure amounts. A weight calculation step of calculating a weight of the first image in the image composition based on the luminance of the first image and the luminance of the second image, and the weight calculated by the weight calculation unit. Image synthesis for performing the image synthesis, and image analysis for generating information for adjusting the three or more types of exposure amounts based on statistical information obtained from the image as a result of the image synthesis. The exposure amount of the first image is an exposure amount other than the minimum exposure amount and the maximum exposure amount among the three or more types of exposure amounts, and the exposure amount of the second image is the exposure amount of the first image. More than exposure Hear, and characterized in that it is a next highest exposure amount of the exposure amount of the first image among the three or more exposure.
In the one image processing method described above, in the image captured with the three or more types of exposure amounts, the image captured with the minimum exposure amount is set as a third image, and the next largest exposure amount after the third image In the weight calculation step, the weight of the third image is calculated based on the luminance of the fourth image, and is captured with the maximum exposure amount. In the weight calculation step, the weight of the fifth image is set to be the fifth image, and the sixth image is the image captured with the next smallest exposure amount after the fifth image. It is preferable to calculate based on the luminance of the sixth image.
In the one image processing method described above, it is preferable that the statistical information is an average luminance obtained from a plurality of results of the image synthesis.
[Mode 1] In order to achieve the above object, an image processing apparatus according to mode 1 includes:
Among the three or more types of images obtained by imaging the subject with three or more types of exposure amounts, the luminance of the first image and the luminance of the second image having the next largest exposure amount after the first image Weight calculating means for calculating a weight to be used at the time of image synthesis for the first image,
Image synthesis means for performing weighted addition of the three or more types of images using the weights calculated by the weight calculation means and synthesizing the images.

With such a configuration, the luminance of the first image among the three or more types of images obtained by imaging the subject with the three or more types of exposure amounts by the weight calculation unit, and the first image Next, based on the brightness of the second image having the large exposure amount, a weight used for image synthesis of the first image is calculated.
For example, when calculating weights for images S1 to S4 obtained by imaging a subject with four types of exposure amounts L1, L2, L3, and L4 (L1 <L2 <L3 <L), for example, As for the weight, the weight is calculated based on the luminance of S2 and the luminance of S3.

Here, the calculation of the weight is performed for all images having an exposure amount that needs to be weighted. For example, when there are two or more types of images having exposure amounts that need to be weighted, the weights are calculated by setting each of the images as the first image and the second image having the next largest exposure amount as the second image.
When the weights are calculated, the image combining means performs weighted addition of three or more types of images using the calculated weights to combine the images.
From the above, among the images to be combined, the weight of the image with the smaller exposure amount can be calculated based on the brightness of the two images whose exposure amounts are adjacent to each other. As a result, it is possible to obtain an effect that smoother linear composition is possible than calculating the weight based only on the brightness of the image with the larger exposure amount.

[Mode 2] Furthermore, the imaging apparatus of mode 2 is the imaging apparatus of mode 1,
The weight calculation means is configured to set the luminance of the first image and the luminance of the second image for images other than the two types of images having the minimum and maximum exposure amounts among the three or more types of images. Based on this, the weight is calculated.
With such a configuration, the weight calculation means excludes the two types of images having the minimum and maximum exposure amounts, and the remaining intermediate exposure amounts are the luminances of the first image and the second image. The weight is calculated based on
As a result, it is possible to obtain an effect that smoother linear composition is possible than calculating the weight based only on the brightness of the image with the larger exposure amount.

[Mode 3] Furthermore, the imaging device of mode 3 is the imaging device of mode 1 or 2,
The weight calculation means calculates the weight using a value calculated by a function F (x) in which the luminance of the image is an element x and the function value increases according to the luminance.
With such a configuration, the weight for the first image can be calculated by the weight calculation means using the value calculated by the function F (x) that increases according to the magnitude of the luminance x.

As a result, the greater the luminance of the second image, the greater the weight for the first image. For example, the luminance of the first image is non-saturated and the luminance of the second image is saturated. In this case, the saturated portion can be complemented with the first image.
Therefore, it is possible to obtain an effect that smoother linear synthesis is possible than calculating the weight based only on the brightness of the image with the larger exposure amount.

[Mode 4] Furthermore, the imaging device of mode 4 is the imaging device of mode 3,
The weight calculation means switches, as the function, a first function F1 (x) in which a calculated value for the element x changes more rapidly and a second function F2 (x) in which the calculation value changes more gradually. Use.
With such a configuration, the weight calculation unit can switch between the first function F1 (x) and the second function F2 (x) for calculation of the weight.
That is, when the function F1 (x) that changes more steeply is used, the degree of mixing of the corresponding image can be made stronger, and when the function F2 (x) that changes more slowly is used, The degree of image mixing can be made weaker.
As a result, it is possible to change the degree of mixing of a plurality of types of images and to obtain an effect that a more appropriate weight can be calculated.

[Mode 5] On the other hand, in order to achieve the above object, an image processing method according to mode 4 includes:
Among the three or more types of images obtained by imaging the subject with three or more types of exposure amounts, the luminance of the first image and the luminance of the second image having the next largest exposure amount after the first image A weight calculating step for calculating a weight to be used for image synthesis on the first image, based on
An image synthesis step of performing weighted addition of the three or more types of images using the weights calculated in the weight calculation step, and synthesizing the images.
With such a configuration, operations and effects equivalent to those of the image processing apparatus described in aspect 1 can be obtained.

[Mode 6] In order to achieve the above object, an image processing program according to mode 5
Among the three or more types of images obtained by imaging the subject with three or more types of exposure amounts, the luminance of the first image and the luminance of the second image having the next largest exposure amount after the first image A weight calculating step for calculating a weight to be used for image synthesis on the first image, based on
A program for causing a computer to execute a process including a weighted addition of the three or more types of images using the weight calculated in the weight calculating step and an image combining step of combining the images.
With such a configuration, when the program is executed by a computer, the same operations and effects as those of the image processing apparatus described in the first embodiment can be obtained.

[Mode 7] In order to achieve the above object, an imaging apparatus according to mode 7 includes:
Imaging means for imaging a subject with N types of exposures (N is a natural number of 3 or more);
An image processing apparatus according to any one of forms 1 to 4,
The image processing apparatus includes a weight for the first image of at least three types of images used for image synthesis among N types of images corresponding to the N types of exposure amounts obtained by imaging with the imaging unit. Is calculated.
With such a configuration, operations and effects equivalent to those of the image processing apparatus described in aspect 1 can be obtained.
[Mode 8] In order to achieve the above object, an electronic device according to mode 6
The imaging apparatus according to the seventh aspect is provided.
With such a configuration, operations and effects equivalent to those of the imaging device described in aspect 7 can be obtained.

1 is a block diagram illustrating a configuration of an imaging apparatus 1 according to the present invention. (A) is a schematic diagram which shows the internal structure of the image pick-up element 10, (b) is a block diagram which shows the internal structure of HDR sensor 10d. 2 is a block diagram showing an internal configuration of a video signal processing unit 16. FIG. It is a figure which shows an example of the exposure for every line of each pixel in a sensor cell array, and the read-out operation of a pixel signal. 3 is a flowchart illustrating an example of operation processing of the imaging apparatus 1. It is a flowchart which shows an example of a HDR synthetic | combination process. It is a flowchart which shows an example of an exposure ratio adjustment process. (A)-(d) is a figure which shows an example of response signal S1-S4 with respect to incident light in exposure time T1-T4 determined so that a ratio might become equal. (A)-(d) is a figure which shows an example of synthetic | combination weight W1-W4 with respect to incident light. FIG. 10 is a diagram in which FIGS. 9A to 9D are displayed in an overlapping manner. It is a figure at the time of expressing the axis | shaft of the incident light in Fig.8 (a)-(d) by the logarithm. It is a figure which shows an example of the light-signal characteristic of the image of the synthetic | combination object in the case of the conventional non-weighting, and the image after HDR synthetic | combination processing. It is a figure which shows an example of the flow of the calculation process which concerns on the HDR synthetic | combination process in the case of the conventional weighting.

Hereinafter, a first embodiment of the present invention will be described with reference to the drawings. 1 to 11 are diagrams illustrating embodiments of an image processing apparatus, an image processing method, an image processing program, an imaging apparatus, and an electronic apparatus according to the present invention.
First, the configuration of the imaging apparatus according to the present invention will be described with reference to FIG. FIG. 1 is a block diagram showing a configuration of an imaging apparatus 1 according to the present invention.
As shown in FIG. 1, the imaging apparatus 1 includes an imaging device 10, a SW (switch) 11, first to fourth memories 12 to 15, a video signal processing unit 16, a frame memory 17, and a control unit 18. And an image analysis unit 19 and an operation unit 20.

Next, based on FIG. 2, the detailed structure of the image pick-up element 10 is demonstrated. Here, FIG. 2A is a schematic diagram showing the internal configuration of the image sensor 10, and FIG. 2B is a block diagram showing the internal configuration of the HDR sensor 10d.
The imaging element 10 has a function of imaging a subject with a plurality of types of exposure amounts, and includes a lens 10a, a microlens 10b, a color filter array 10c, and an HDR sensor 10d as shown in FIG. Consists of including.

Hereinafter, for convenience of description, the image sensor 10 has exposure times T1, T2, T3, and T4 (T1 <T2 <T3) corresponding to four types of exposure amounts L1, L2, L3, and L4 (L1 <L2 <L3 <L4). The configuration will be described assuming that imaging is performed at <T4). Of course, the exposure amount is not limited to four, and may be three or five or more.
The lens 10a collects reflected light from the subject and guides it to the microlens. There are types such as a single focus lens, a zoom lens, and an auto iris lens depending on the imaging conditions.

The microlens 10b condenses light transmitted through the lens on each sensor cell (pixel) of the sensor cell array included in the HDR sensor 10d.
The color filter array 10c separates light having a wavelength corresponding to one predetermined type of color element from light transmitted through the microlens 10b, and makes the separated light incident on each corresponding pixel (hereinafter referred to as CF). Part) and at least the number of pixels.

The HDR sensor 10d controls the exposure times T1 to T4 by an electronic shutter method and outputs four types of digital image data having different exposure amounts.
As shown in FIG. 2B, the HDR sensor 10d includes a reference timing generator 50, a scan line scanner 54, a sensor cell array 56, and a horizontal transfer unit 58.
The reference timing generator 50 generates a reference timing signal based on the vertical synchronization signal and the horizontal synchronization signal from the control unit 18 and outputs the reference timing signal to the scanning line scanner 54.

The scanning line scanner 54 generates a reset line selection signal that validates a line on which reset processing is performed based on various signals from the reference timing generator 50 and the control unit 18. Then, the generated reset line selection signal is output to the sensor cell array 56.
Further, the scanning line scanner 54 generates a readout line selection signal that enables the line on which charges are accumulated after the resetting and for the set exposure times T1 to T4 as the readout line of the pixel signal. Then, the generated read line selection signal is output to the sensor cell array 56.
The sensor cell array 56 includes a light receiving region having a configuration in which a plurality of sensor cells (pixels) including light receiving elements (photodiodes and the like) are arranged in a two-dimensional matrix, which is configured using CMOS technology. On the other hand, the address line, the reset line, and the readout line are connected in common.

Then, various drive signals (selection signals) are transmitted to the sensor cells constituting each line through the three control lines, and when the address line and the readout line become valid, the accumulated charge (pixel signal) is transmitted through the signal line. Transfer (output) to the horizontal transfer unit 58.
With such a configuration, the sensor cell array 56 validates (selects) a pixel line on which a reset operation or a read operation is performed by an address line based on a selection signal supplied from the scanning line scanner 54. When a reset operation is performed on each pixel on the line selected by the selection signal, a signal instructing the reset operation is input via the reset line, and when a pixel signal is read out, a read line is input. A signal for instructing the transfer of the accumulated charge is input via the. Further, in each pixel selected by the selection signal, the reset operation is performed when a signal instructing the reset operation is input, and the signal line is connected when the signal instructing the transfer of the accumulated charge is input. The accumulated charge is transferred to the horizontal transfer unit 58 via the via.

The horizontal transfer unit 58 performs A / D conversion on pixel signal (analog signal) data read from each pixel of the sensor cell array 56, and sequentially outputs the data serially to the memories 12 to 15 in line units.
Returning to FIG. 1, when the image data S <b> 1 to S <b> 4 are input from the image sensor 10, the SW 11 switches a switch based on the identification signal input corresponding to the image data S <b> 1 to S <b> 4. The input image data is output to a memory corresponding to the exposure time indicated by the identification signal. Specifically, the image data S1 of the exposure time T1 is stored in the first memory 12, the image data S2 of the exposure time T2 is stored in the first memory 13, the image data S3 of the exposure time T3 is stored in the third memory 14, and the exposure data of the exposure time T4 is stored. The image data S4 is output to the fourth memory 15.

  The first to fourth memories 12 to 15 store image data S1 to S4 corresponding to the four types of exposure amounts L1 to L4, which are input through the SW11 and obtained by imaging the subject with the exposure times T1 to T4. It is a memory that temporarily stores any corresponding one. Specifically, the memory serves as a buffer that outputs the image data S1 to S4 sequentially input via the SW 11 to the video signal processing unit 16 in synchronization.

As shown in FIG. 3, the video signal processing unit 16 includes a weight calculation unit 16a, a normalization unit 16b, and a synthesis unit 16c. Here, FIG. 3 is a block diagram showing an internal configuration of the video signal processing unit 16.
The weight calculation unit 16a uses the weights W1 to W4 (x, y) used at the time of image composition for the pixel data S1 to S4 corresponding to the exposure times T1 to T4 (hereinafter referred to as composition weights W1 to W4 (x, y)). It has a function to calculate

  Specifically, S2 (x, y) is used when calculating the combined weight W1 (x, y) of S1 (x, y). Further, when calculating the composite weight W2 (x, y) of S2 (x, y), two pixel data of S2 (x, y) and S3 (x, y) are used. Further, when calculating the combined weight W3 (x, y) of S3 (x, y), the weight is calculated using two pixel data of S3 (x, y) and S4 (x, y). Further, when calculating the combined weight W4 (x, y) of S4 (x, y), the calculation is performed using the fact that the total sum of the combined weights W1 to W4 (x, y) is “1”.

In the present embodiment, the weight calculation unit 16a calculates composite weights W1 to W4 (x, y) based on the following equations (2) to (6).

F (x) = (x / 255) ⁿ (2)
W1 (x, y) = F (S2 (x, y)) (3)
W2 (x, y) = F (S3 (x, y)) − W1
= F (S3 (x, y))-F (S2 (x, y)) (4)
W3 (x, y) = F (S4 (x, y))-W2-W1
= F (S4 (x, y))-F (S3 (x, y)) (5)
W4 (x, y) = 1-W3 (x, y) -W2 (x, y) -W1 (x, y)
= 1-F (S4 (x, y)) (6)

However, the above expression (2) is an expression corresponding to x in the 8-bit gradation range (0 to 255). For example, if the gradation range is 10 bits, “255” in the above expression (2). "Is set to" 1023 ". As described above, it is necessary to change the value according to the number of bits of the data as the element.
The weight calculation unit 16a outputs information on the combination weights W1 to W4 (x, y) obtained by calculation according to the above equations (2) to (6) to the combination unit 16c.
The normalization unit 16b performs normalization processing of the pixel data S1 to S4 (x, y) based on the pixel data S1 to S4 (x, y) from the memories 12 to 15 and the exposure ratio information from the control unit 18. . In the present embodiment, the exposure ratio information is information on exposure times T1 to T4.

Specifically, in the present embodiment, the normalization unit 16b performs normalization of S1 to S4 (x, y) using the exposure times T1 to T4 based on the following expressions (7) to (10), and normalization is performed. Generated pixel data N1 to N4 (x, y) are generated.

N1 (x, y) = S1 (x, y) × T4 / T1 (7)
N2 (x, y) = S2 (x, y) × T4 / T2 (8)
N3 (x, y) = S3 (x, y) × T4 / T3 (9)
N4 (x, y) = S4 (x, y) × T4 / T4 (10)

In addition, the normalization part 16b outputs the information of the normalization pixel data N1-N4 (x, y) obtained by calculating according to said Formula (7)-(10) to the synthetic | combination part 16c.

The combining unit 16c performs HDR combining processing based on the combined weights W1 to W4 (x, y) acquired from the weight calculating unit 16a and the normalized pixel data N1 to N4 (x, y) acquired from the normalizing unit 16b. And has a function of generating HDR pixel data.
Specifically, in the present embodiment, the combining unit 16c combines the normalized pixel data N1 to N4 (x, y) with weights W1 to W4 (x, y) based on the following equation (11). , HDR pixel data HDR (x, y) is generated.

HDR (x, y) = W1 (x, y) × N1 (x, y) + W2 (x, y) × N2 (x, y) + W3 (x, y) × N3 (x, y) + W4 (x, y) × N4 (x, y)
(11)

The synthesizing unit 16c stores the HDR pixel data HDR (x, y) obtained by calculation according to the above equation (11) in the frame memory 17.
Returning to FIG. 1, the frame memory 17 is a memory for storing the HDR image data after the synthesis in the video signal processing unit 16.

The control unit 18 has a function of generating exposure ratio information based on input information from the operation unit 20, a function of adjusting the current exposure ratio to an appropriate one based on statistical information acquired from the image analysis unit 19, and And a function of outputting the generated exposure ratio information to the video signal processing unit 16.
Further, the control unit 18 generates an exposure amount signal based on the exposure ratio information corresponding to the adjusted exposure ratio or the newly generated exposure ratio information, and transmits the generated exposure amount signal and various control signals to the image sensor 10. It also has a function to do. Here, the various control signals include a vertical synchronization signal, a horizontal synchronization signal, a pixel clock, and the like.
Specifically, the image sensor 10 of the present embodiment images the same subject with exposure times T1 to T4 corresponding to four types of exposure amounts L1 to L4 corresponding to the exposure amount signals output from the control unit 18.

In the present embodiment, the control unit 18 sets the ratios “L2 / L1”, “L3 / L2”, and “L4 / L3” of two adjacent exposure amounts in the exposure amounts L1, L2, L3, and L4 arranged in ascending order. "Is equal (L2 / L1 = L3 / L2 = L4 / L3), the exposure times T1 to T4 corresponding to the exposure amounts L1 to L4 are set.
Further, in the present embodiment, the control unit 18 adjusts the exposure amount ratio so as to be an optimum ratio according to the brightness of the imaging environment. For this purpose, the control unit 18 compares the average luminance Lav of the HDR image data obtained from the image analysis unit 19 with the threshold value Lth on the high luminance side and the threshold value Ltl on the low luminance side prepared in advance.

  Based on the comparison result, the control unit 18 determines that the imaging environment is relatively bright if the average luminance Lav is greater than Lth, and determines that the imaging environment is relatively dark if Lav is less than Ltl. Further, based on the determination result, the currently set exposure times T1 to T4 are increased / decreased while maintaining a uniform exposure amount ratio, so that the optimum exposure amount for the brightness of the imaging environment is obtained. Make adjustments to achieve the ratio.

The image analysis unit 19 calculates statistical information related to the brightness of the imaging environment based on the HDR image data generated by the video signal processing unit 16 and stored in the frame memory 17, and the calculated statistical information is sent to the control unit 18. It has a function to output. In the present embodiment, the image analysis unit 19 statistically analyzes the HDR image data (including at least the first frame) obtained from the frame memory 17 during continuous imaging of a plurality of frames. In this embodiment, in order to obtain information relating to the brightness of the imaging environment, the average luminance Lav of the HDR image data is calculated as statistical information. Then, the calculated Lav is output to the control unit 18.
The operation unit 20 is operated by the user at the time of imaging, when viewing a captured image, or when selecting an arbitrary mode from among a plurality of types of imaging modes. Information corresponding to the user's operation content is sent to the control unit 18. It has a function to output.

Next, a method for controlling the exposure time of the HDR sensor 10d of the image sensor 10 and a method for reading a pixel signal from the sensor cell array 56 will be described with reference to FIG. Here, FIG. 4 is a diagram illustrating an example of exposure and pixel signal read operations for each pixel line in the sensor cell array of the HDR sensor.
In the present embodiment, a non-destructive readout line L1 for performing non-destructive readout of a pixel signal at an exposure time T1 and non-destructive readout of a pixel signal at an exposure time T2 with respect to an exposure region (scanning region) of the sensor cell array 56. A nondestructive read line L2 to be performed is set. Further, a nondestructive readout line L3 for performing nondestructive readout of the pixel signal at the exposure time T3 is set. Further, a read & reset line L4 & R for resetting the accumulated charge of each pixel line and reading the pixel signal at the exposure time T4 is set. The relationship between T1 and T4 is “T1 <T2 <T3 <T4”. After the reset, first, the nondestructive read line L1 is set when T1 has elapsed. Next, the nondestructive read line L2 is set when T2 elapses, the nondestructive read line L3 is set when T3 elapses, and the read & reset line L4 & R is set when T4 elapses.

  Specifically, as shown in FIG. 4, the non-destructive readout lines L1, L2, and L3 and the readout & reset line L4 & R are read out when charges corresponding to the exposure time T4 are sequentially accumulated in the pixel lines in the exposure region. The reset line L4 & R is set to sequentially read out the pixel signals of the lines of each pixel and to sequentially reset the accumulated charges. On the other hand, in the line of each pixel after resetting the exposure area, the pixel signals of the line of each pixel are sequentially read in a non-destructive manner during the exposure times T1, T2, and T3 during the period in which the charge for the exposure time T4 is accumulated. Non-destructive read lines L1, L2 and L3 are set respectively.

  For example, it is assumed that the pixel signal S4 of the exposure time T4 is read and reset on the first line that is the first line of the exposure region. Thereafter, every time the pixel signal S4 is read from the fourth line memory, scanning of the read & reset lines L4 & R is sequentially performed for each line in the scan direction in FIG. At this time, when the read & reset line L4 & R reaches the first line again, scanning is performed so that the exposure time T4 has just passed. In such a procedure, readout of pixel signals at the time of exposure and reset of accumulated charges are sequentially performed for each pixel line for each pixel line in the exposure region of the sensor cell array.

  On the other hand, when the accumulated charge is reset, non-destructive readout of the pixel signal S1 of the pixel that has been exposed for the exposure time T1 in the non-destructive readout line L1 is performed on the pixel line after the reset, In the nondestructive readout line L2, nondestructive readout of the pixel signal S2 of the pixel that has been exposed for the exposure time T2 is performed. Subsequently, non-destructive readout of the pixel signal S3 of the pixel that has been exposed for the exposure time T3 in the non-destructive readout line L3 is performed. In such a procedure, the pixel signals S1, S2, and S3 at the time of exposure are sequentially read for each line of each pixel of the sensor cell array at the exposure times T1, T2, and T3.

  In the present embodiment, as shown in FIG. 4, pixel signal data (analog data) S1 corresponding to the exposure time T1 is read into the first line memory, and the pixel signal data corresponding to the exposure time T2 is read. Data (analog data) S2 is read to the second line memory. Further, pixel signal data (analog data) S3 corresponding to the exposure time T3 is read to the third line memory, and pixel signal data (analog data) S4 corresponding to the exposure time T4 is read to the fourth line memory. Read out. Then, the read pixel signal data S1 to S4 (hereinafter referred to as pixel data S1 to S4) are sequentially output to the ADC in the order of S1 to S4 through the selection circuit, as shown in FIG. There, it is converted into digital data. The converted pixel data S1 to S4 are in the order of conversion (S1 to S4), S1 in the first memory 12, S2 in the second memory 13, S3 in the third memory 14, and S4 in the first. 4 is output to each of the memories 15.

Next, based on FIG. 5, the flow of operation processing of the imaging apparatus 1 will be described. Here, FIG. 5 is a flowchart illustrating an example of operation processing of the imaging apparatus 1.
When the imaging apparatus 1 is powered on, the process proceeds to step S100 as shown in FIG.
In step S100, the control unit 18 sets initial values of the shortest exposure time Tmin and the longest exposure time Tmax, and the process proceeds to step S102. For the initial value setting, a value input or selected by the user via the operation unit 20 may be used, or an initial value set in advance as the standard exposure time may be used.

  In step S102, the control unit 18 uses the initial value set in step S100 to calculate an exposure time Tmid that is an exposure time corresponding to the intermediate exposure amount, and calculates the calculated Tmid and the initial value Tmin. And exposure ratio information are generated based on Tmax. Then, an exposure amount signal is generated based on the generated exposure ratio information, the generated exposure ratio information is output to the video signal processing unit 16, and the generated exposure amount signal and various control signals are transmitted to the image sensor 10. The process proceeds to step S104.

In the present embodiment, since there are four types of exposure amounts L1 to L4, the shortest exposure time Tmin corresponding to the exposure amount L1 is T1, and the longest exposure time Tmax corresponding to the exposure amount L4 is T4. The intermediate exposure times Tmid corresponding to the exposure amounts L2 and L3 are T2 and T3. Then, the control unit 18 calculates intermediate exposure times T2 and T3 that satisfy “T2 / T1 = T3 / T2 = T4 / T3” according to the following equations (12) and (13).

T2 = (T1 ² × T4) ^1/3 (12)
T3 = (T1 × T4 ² ) ^1/3 (13)

For example, when the shortest exposure time Tmin that can be set is 1H and the longest exposure time Tmax is 1000H, T1 = 1H and T3 = 1000H, and from the above equation (12), “T2 = (1 ² × 1000) ^1/3 = 10 ". Further, it can be calculated from the above equation (13) as “T3 = (1 × 1000 ² ) ^1/3 = 100”. Thereby, the exposure amount ratio = exposure time ratio = “1: 10: 100: 1000” is obtained. In this case, the dynamic range is “60 [dB]”.
Note that the scanning time for one line in the sensor cell array is abbreviated as HSYNC, which is the name of the horizontal synchronization signal, and is represented as 1H. For example, when the frame rate is 30 [fps] with respect to the total number of lines of the sensor cell array of 500 lines, the 1H period is the time obtained by dividing 1/30 seconds by 500 lines, that is, 1/15000 seconds, in other words. 1/15 milliseconds (about 67 microseconds).

In step S104, the video signal processing unit 16 determines whether or not the imaging process has been started. If it is determined that the imaging process has started (Yes), the process proceeds to step S106. If not (No), the imaging process is performed. The determination process is repeated until is started.
When the imaging process is started, the image sensor 10 captures an image of the subject with the exposure amounts L1 to L4 (exposure times T1 to T4) based on the exposure amount signal and various control signals from the control unit 18. Then, the image data S1 to S4 obtained by imaging are sequentially output to the first to fourth memories 12 to 15, and the process proceeds to step S106.

In step S106, the video signal processing unit 16 sequentially outputs image data S1 to S4 output from the first to fourth memories 12 to 15, and exposure ratio information (T1: T2: T3: T4) from the control unit 18. Then, the HDR synthesizing process is executed, and the process proceeds to step S108.
In step S108, the control unit 18 determines whether or not the imaging process has been completed. If it is determined that the imaging process has ended (Yes), the series of processes is terminated. If not (No), the process proceeds to step S110. Transition.

When the process proceeds to step S110, the control unit 18 executes the exposure ratio adjustment process based on the HDR image data generated by the HDR synthesizing process stored in the frame memory 17, and the process proceeds to step S106.
When the exposure ratio adjustment process is executed, the image sensor 10 executes the image pickup process based on the adjusted exposure amount signal.

Next, the flow of HDR synthesis processing in the video signal processing unit 16 will be described with reference to FIG. Here, FIG. 6 is a flowchart illustrating an example of the HDR synthesis process.
When the process proceeds to step S106, the HDR synthesis process is started in the video signal processing unit 16, and first, the process proceeds to step S200 as shown in FIG.
In step S200, the video signal processing unit 16 obtains one line of pixel data S1 to S4 from the first to fourth memories 12 to 15, and proceeds to step S202.

Specifically, the video signal processing unit 16 acquires S1 to S4 at the timing when the pixel data S1 to S4 for one line at the same pixel position are stored in the first to fourth memories 12 to 15.
In the present embodiment, the pixel data S1 to S4 (x, y) are data including luminance information (luminance values) and pixel position information (row and column position information). . Further, the pixel data S1 to S4 may have no position information, and the video signal processing unit 16 may generate the pixel position information by counting the horizontal synchronization signal and the pixel clock.

Further, the video signal processing unit 16 may perform preprocessing such as fixed pattern noise removal processing and clamping processing on the acquired pixel data S1 to S4 as necessary.
In step S202, the video signal processing unit 16 outputs pixel data for one pixel for which the calculation processing of the combined weights W1 to W4 and the normalized pixel data N1 to N4 is unprocessed from the pixel data S1 to S4 for one line. S1 to S4 (x, y) are selected. Then, the selected pixel data S1 to S4 (x, y) are output to the normalization unit 16b, and S2 to S4 (x, y) among the selected pixel data S1 to S4 (x, y) are weight calculation unit 16a. And the process proceeds to step S204.

In step S204, the weight calculation unit 16a uses the pixel data S2 (x, y) selected in step S202 to obtain the combined weight W1 (x, y) from the above equations (2) and (3) as “ W1 (x, y) = F (S2 (x, y)) ”is calculated, and the process proceeds to step S206.
In step S206, the weight calculation unit 16a first calculates F (S3 (x, y)) from the above formula (2) based on the pixel data S3 (x, y) selected in step S202. Next, from the calculated F (S3 (x, y)), F (S2 (x, y)) calculated in step S204, and the above equation (4), the combined weight W2 (x, y) is obtained. , “W2 (x, y) = F (S3 (x, y)) − F (S2 (x, y))” is calculated, and the process proceeds to step S208.

In step S208, the weight calculation unit 16a first calculates F (S4 (x, y)) from the above equation (2) based on the pixel data S4 (x, y) selected in step S202. Next, from the calculated F (S4 (x, y)), F (S3 (x, y)) calculated in step S206, and the above equation (5), the combined weight W3 is “W3 (x , Y) = F (S4 (x, y)) − F (S3 (x, y)) ”, and the process proceeds to step S210.
In step S210, the weight calculation unit 16a calculates “W4 (x, y) as a combined weight W4 (x, y) from F (S4 (x, y)) calculated in step S208 and the above equation (6). ) = 1−F (S4 (x, y)) ”, and the process proceeds to step S212.
Note that the combined weights W1 to W4 (x, y) calculated in steps S204 to S210 are output to the combining unit 16c.

  In step S212, the normalization unit 16b converts the pixel data S1 (x, y) selected in step S202 and the exposure ratio information “T1: T2: T3: T4” from the control unit 18 to T1 and T4. Based on the above equation (7), the pixel data S1 (x, y) is normalized. Specifically, “N1 (x, y) = S1 (x, y) × T4 / T1” is calculated as the normalized pixel data N1 (x, y), and the process proceeds to step S214.

  In step S214, the normalizing unit 16b converts the pixel data S2 (x, y) selected in step S202 and the exposure ratio information “T1: T2: T3: T4” from the control unit 18 to T2 and T4. Based on the above equation (8), the pixel data S2 (x, y) is normalized. Specifically, “N2 (x, y) = S2 (x, y) × T4 / T2” is calculated as the normalized pixel data N2 (x, y), and the process proceeds to step S216.

In step S216, the normalization unit 16b converts the pixel data S3 (x, y) selected in step S202 and the exposure ratio information “T1: T2: T3: T4” from the control unit 18 to T3 and T4. Based on the above equation (9), the pixel data S3 is normalized. Specifically, “N3 (x, y) = S3 (x, y) × T4 / T3” is calculated as the normalized pixel data N3 (x, y), and the process proceeds to step S218.
Note that the normalized pixel data N1 to N3 (x, y) calculated in steps S212 to S216 and the normalized pixel data N4 (x, y) that is “N4 (x, y) = S4 (x, y)”. Is output to the combining unit 16c.

In step S218, the synthesis unit 16c uses the synthesis weights W1 to W4 (x, y) acquired from the weight calculation unit 16a and the normalized pixel data N1 to N4 (x, y) acquired from the normalization unit 16b. Then, N1 to N4 (x, y) are synthesized based on the above equation (11), and the process proceeds to step S220.
Specifically, the synthesizing unit 16c obtains “HDR (x, y) = W1 (x, y) × N1 (x, y) + W2 ( x, y) * N2 (x, y) + W3 (x, y) * N3 (x, y) + W4 (x, y) * N4 (x, y) "is calculated.

In step S220, the synthesizing unit 16c stores the HDR pixel data HDR (x, y) obtained by synthesizing in step S218 in the frame memory 17, and proceeds to step S222.
In step S222, the video signal processing unit 16 determines whether or not the HDR synthesizing process is completed for the pixel data S1 to S4 for one line. If it is determined that the process is completed (Yes), the process proceeds to step S224. If not (No), the process proceeds to step S202.

When the process proceeds to step S224, in the video signal processing unit 16, it is determined whether or not the HDR synthesis process is completed for the pixel data S1 to S4 for one frame (one image), and it is determined that the process has ended. (Yes) terminates the series of processing and returns to the original processing. On the other hand, if it is determined that the process has not been completed (No), the process proceeds to step S200.
Note that the processing in steps S204 to S210 and the processing in steps S212 to S216 may be performed sequentially or in parallel.

Next, the flow of the exposure ratio adjustment process in step S110 will be described with reference to FIG. Here, FIG. 7 is a flowchart showing an example of the exposure ratio adjustment processing.
When the process proceeds to step S110 and the exposure ratio adjustment process is started, the process proceeds to step S300 as shown in FIG.
In step S300, the control unit 18 acquires HDR image data for one frame (for one image) generated by the HDR synthesizing process from the frame memory 17, calculates an average luminance Lav from the acquired HDR image data, The process proceeds to step S302.

In step S302, the control unit 18 compares the average luminance Lav calculated in step S300 with a threshold Lth on the high luminance side of a preset average luminance value, and determines whether or not Lav is larger than Lth. . If it is determined that Lav is greater than Lth (Yes), the process proceeds to step S304. If not (No), the process proceeds to step S308.
When the process proceeds to step S304, the control unit 18 decreases the shortest exposure time Tmin by a preset time, and the process proceeds to step S306. That is, since the imaging environment is relatively bright, adjustment is performed to reduce the shortest exposure time Tmin in order to obtain an image that does not blow out. It should be noted that an appropriate value may be calculated for each decrease time according to the average luminance, or a value corresponding to the magnitude of the average luminance may be prepared in advance as a table.

In step S306, the control unit 18 recalculates the intermediate exposure time Tmid using Tmin decreased in step S304 and Tmax at the initial setting, and recalculated Tmid, the decreased Tmin, and the initial setting time. The exposure ratio information is generated based on Tmax.
For example, in the case of four types of exposure times T1 to T4, “T1: T2: T3: T4 = Tmin (decrease): Tmid1 (re): Tmid2 (re): Tmax (first)” is generated as exposure ratio information. To do. It should be noted that (decrease) attached after each exposure time indicates that the exposure time after the reduction adjustment, (re) indicates after recalculation, and (initial) indicates each exposure time at the time of initial setting. Further, the control unit 18 generates an exposure amount signal based on the generated exposure ratio information. Then, the generated exposure ratio information is output to the video signal processing unit 16, the generated exposure amount signal and various control signals are transmitted to the image sensor 10, the series of processes is terminated, and the original process is restored.

  On the other hand, when Lav is equal to or less than Lth in step S302 and the process proceeds to step S308, the control unit 18 calculates the average luminance Lav calculated in step S300 and the threshold Ltl on the low luminance side of the preset average luminance value. And compare. Furthermore, based on the comparison result, it is determined whether or not Lav is smaller than Ltl. If it is determined that Lav is smaller than Ltl (Yes), the process proceeds to step S310. If not (No), the process proceeds to step S314.

  When the process proceeds to step S310, the control unit 18 increases the longest exposure time Tmax by a preset time, and the process proceeds to step S312. That is, since the imaging environment is in a relatively dark state, adjustment is performed to increase the longest exposure time Tmax in order to obtain an image that is not blackened. Note that an appropriate value for the increasing time may be calculated each time according to the average luminance, or a value corresponding to the magnitude of the average luminance may be prepared in advance as a table.

In step S312, the control unit 18 recalculates the intermediate exposure time Tmid using Tmax increased in step S310 and the initial setting Tmin, and recalculates the Tmid, the initial setting Tmin, and the increased value. Exposure ratio information is generated based on Tmax.
For example, when there are four types of exposure times T1 to T4, the exposure ratio information is “T1: T2: T3: T4 = Tmin (initial): Tmid1 (re): Tmid2 (re): Tmax (increase)”. Become. Note that (increased) added after the exposure time Tmax indicates the exposure time after the increase adjustment. Further, the control unit 18 generates an exposure amount signal based on the generated exposure ratio information. Then, the generated exposure ratio information is output to the video signal processing unit 16, the generated exposure amount signal and various control signals are transmitted to the image sensor 10, the series of processes is terminated, and the original process is restored.

In step S308, if Lav is equal to or greater than Ltl and the process proceeds to step S314, the control unit 18 sets the normal exposure time as the shortest exposure time Tmin and the longest exposure time Tmax, and the process proceeds to step S316. In the present embodiment, the initial exposure time is set as the normal exposure time.
In step S316, the control unit 18 recalculates the intermediate exposure time Tmid using the Tmin and Tmax set in step S314. Based on the recalculated Tmid, the initial setting Tmin, and the increased Tmax. Exposure ratio information is generated.

  For example, when there are four types of exposure times T1 to T4, the exposure ratio information is “T1: T2: T3: T4 = Tmin (first): Tmid1 (re): Tmid2 (re): Tmax (first)”. Become. Further, the control unit 18 generates an exposure amount signal based on the generated exposure ratio information. Then, the generated exposure ratio information is output to the video signal processing unit 16, the generated exposure amount signal and various control signals are transmitted to the image sensor 10, the series of processes is terminated, and the original process is restored.

Next, the operation of the present embodiment will be described with reference to FIGS.
Here, FIGS. 8A to 8D are diagrams showing examples of response signals S1 to S4 with respect to incident light at exposure times T1 to T4 determined so that the ratios are equal. FIGS. 9A to 9D are diagrams illustrating examples of combined weights W1 to W4 for incident light. FIG. 10 is a diagram in which FIGS. 9A to 9D are displayed in an overlapping manner. FIG. 11 is a diagram showing the incident light axis in FIGS. 8A to 8D in logarithm.

When the power is turned on, the imaging apparatus 1 first generates exposure ratio information and an exposure amount signal in the control unit 18 before the imaging process is started, and the generated exposure ratio information is used as the video signal processing unit 16. The generated exposure amount signal and various control signals are transmitted to the image sensor 10.
Specifically, first, initial values of the exposure times T1 and T3 are set based on an instruction from the user via the operation unit 20 or a preset value (step S100). Here, it is assumed that “T1 = 2H, T4 = 500H” is set as the initial value.

Next, the control unit 18 calculates intermediate exposure times T2 and T3 having a relationship of “T2 / T1 = T3 / T2 = T4 / T3” according to the above equations (6) and (7) (step S102). Specifically, since “T2 = 2H, T3 = 500H” is set, the control unit 18 automatically calculates “T2 = (4 × 500) ^1/3 ≈12.6”. Subsequently, the control unit 18 automatically calculates “T3 = (2 × 250,000) ^1/3 ≈79.4”.

Here, rounding is performed and T2 is set to “13”, and T3 is set to “79”. As a result, “T2 / T1 = 13/2 = 6.5”, “T3 / T2 = 79 / 13≈6.1”, and “T4 / T3 = 500 / 79≈6.3” are obtained, and the ratios are substantially equal. The exposure ratio information “T1: T2: T3: T4 = 2: 13: 79: 500” can be obtained.
Further, the control unit 18 generates an exposure amount signal including information of the exposure ratio information “T1: T2: T3: T4 = 2: 13: 79: 500”, and sends the generated exposure ratio information to the video signal processing unit 16. The generated exposure amount signal and various control signals are transmitted to the image sensor 10.

Thereby, information of the exposure time ratio “T1: T2: T3: T4 = 2: 13: 79: 500” is written in the register of the HDR sensor 10d of the image sensor 10.
Subsequently, when imaging of the subject is started in the image sensor 10, the light reflected from the subject is collected by the lens 10a and is incident on the microlens 10b. Incident light from the lens 10a is collimated by the microlens 10b and is incident on each pixel of the sensor cell array via the color filter array 10c.

On the other hand, in the HDR sensor 10d, the reset line R is set one line at a time from the start line, and the accumulated charge of each pixel is reset. Subsequently, the non-destructive readout lines L1, L2, and L3 are set for each scanning line at the elapse timing of the exposure times T1, T2, and T3 after the resetting of each pixel. Thereby, the pixel signals S1 to S3 of the optical response as shown in FIGS. 8A to 8C are sequentially read out. Subsequently, a read & reset line L4 & R is set in order from the start line, and a photoresponse pixel signal S4 as shown in FIG. 8D is read from each pixel of the scanned line, and thereafter, accumulation of each pixel is performed. The charge is reset.
Thereafter, while the imaging is being performed, the setting of L1 to L3 & R, the reading of the pixel signals S1 to S3, and the reset process are repeatedly performed according to the above procedure.

The pixel signals S1 to S4 read out in this way are stored in the first line memory to the fourth line memory for each line and output to the selection circuit in line units. From the selection circuit, analog pixel data S1 to S4 are output to the ADC in the order of S1 to S4. The ADC converts the analog pixel data S1 to S4 into digital pixel data S1 to S4. Then, the ADC sequentially outputs the pixel data to the first to fourth memories 12 to 15 in units of lines and in the order of the pixel data S1 to S4. Here, the pixel data S1 to S4 are data indicating luminance values (0 to 255) in an 8-bit gradation range.
As described above, the pixel signals S1 to S4 obtained by imaging with the exposure times T1, T2, T3, and T4 in which the ratios are substantially equal are shown in a graph representing incident light in a logarithm as shown in FIG. Line up at approximately equal intervals.

On the other hand, when the video signal processing unit 16 starts imaging (the branch of “Yes” in step S104) and the pixel data S1 to S4 for one line are stored in the first to fourth memories 12 to 15, the HDR is performed. The synthesizing process is started (step S106).
When the HDR synthesizing process is started, the video signal processing unit 16 acquires pixel data S1 to S4 for one line from the first to fourth memories 12 to 15 (step S200).
The video signal processing unit 16 out of the acquired pixel data S1 to S4 for one line, the pixel data S1 to S4 for one pixel for which the calculation processing of the combined weights W1 to W4 and the normalized pixel data N1 to N4 is unprocessed. Select (x, y). Then, the pixel data S2 to S4 (x, y) are output to the weight calculation unit 16a, and the pixel data S1 to S4 (x, y) are output to the normalization unit 16b (step S202).

On the other hand, when the weight calculation unit 16a acquires the pixel data S2 to S4 (x, y), first, the pixel data S2 (x, y) is substituted into the above equation (2), and F (S2 (x, y) y)) is calculated. Further, “W1 (x, y) = F (S2 (x, y))” is calculated from the above equation (3) (step S204). That is, when calculating the composite weight W1 (x, y), only the pixel data S2 (x, y) is used.
For example, it is assumed that the luminance value of the pixel data S2 (x, y) is “10”. Here, n in the above equation (2) is “2”. In this case, “F (S2 (x, y)) = (10/255) ² ≈1.538 × 10 ⁻³ ”, and from the above equation (3), “W1 (x, y) = 1.538 × 10 ⁻³ ”.

Note that the value of the function F (x) in the above equation (2) is switched so as to change more steeply or more gradually by appropriately switching the value of n according to the imaging conditions and the like. Or switch to
Here, as shown in FIG. 9A, the combined weight W1 (x, y) changes relatively slowly as the amount of incident light is small, and becomes a constant value with a certain amount of incident light as a boundary. In other words, W1 (x, y) increases as the brightness (luminance value) of S2 (x, y) increases, and W1 (x, y) when the brightness of S2 (x, y) is saturated. ) Is the maximum.

  Next, the weight calculation unit 16a substitutes S3 (x, y) into the above equation (2) to calculate F (S3 (x, y)), and calculates F (S3 (x, y)) ) And F (S2 (x, y)) calculated earlier are substituted into the above equation (4). Thus, “W2 (x, y) = F (S3 (x, y)) − F (S2 (x, y))” is calculated (step S206). That is, when calculating the composite weight W2 (x, y), two pieces of pixel data S2 (x, y) and S3 (x, y) are used.

For example, it is assumed that the luminance value of the pixel data S3 (x, y) is “40”. In this case, “F (S3 (x, y)) = (40/255) ² ≈0.0246”, and from the above equation (4), “W2 = F (S3 (x, y)) − F ( S2 (x, y)) = 0.0246-1.538 × 10 ⁻³ ≈0.02307 ”.
Here, as shown in FIG. 9B, the combined weight W2 (x, y) rises sharply as compared to W1 (x, y) with respect to the incident light amount, and at a certain incident light amount as a boundary. descend. In other words, W2 (x, y) increases as the brightness (luminance value) of S3 (x, y) increases, but W2 (x, y) decreases when the brightness of S2 (x, y) is also large. . In addition, when both S2 (x, y) and S3 (x, y) are saturated, W2 (x, y) becomes the minimum value “0”.

  Next, the weight calculation unit 16a substitutes S4 (x, y) into the above equation (2) to calculate F (S4 (x, y)), and calculates F (S4 (x, y)) ) And F (S3 (x, y)) calculated earlier are substituted into the above equation (5). Thus, “W3 (x, y) = F (S4 (x, y)) − F (S3 (x, y))” is calculated (step S208). That is, when calculating the composite weight W3 (x, y), two pieces of pixel data S3 (x, y) and S4 (x, y) are used.

For example, it is assumed that the luminance value of the pixel data S4 (x, y) is “160”. In this case, “F (S4 (x, y)) = (160/255) ² ≈0.3937”, and from the above equation (5), “W3 = F (S4 (x, y)) − F ( S3 (x, y)) = 0.3937−0.02307≈0.3706 ”.
Here, as shown in FIG. 9C, the combined weight W3 (x, y) rises sharply with respect to the incident light amount as compared with W2 (x, y), and at a certain incident light amount as a boundary. Compared with W2 (x, y), it decreases sharply. Specifically, W3 (x, y) increases as the brightness (luminance value) of S4 (x, y) increases, but W3 (x, y) decreases when the brightness of S3 (x, y) also increases. Become. Further, when both S3 (x, y) and S4 (x, y) are saturated, W3 (x, y) has the minimum value “0”.

Next, the weight calculation unit 16a substitutes the previously calculated F (S4 (x, y)) into the above equation (6), and “W4 (x, y) = 1−F (S4 (x, y))”. Is calculated (step S210). That is, when calculating the combined weight W4 (x, y), the calculation is performed using the fact that “W1 + W2 + W3 + W4 = 1”.
For example, when the value of the above numerical example is used, “W4 = 1−F (S4 (x, y)) = 1−0.3706 = 0.6294”.

  Here, as shown in FIG. 9D, the combined weight W4 (x, y) increases as the amount of incident light decreases, and is compared with W3 (x, y) as the amount of incident light increases. Suddenly decreases. Specifically, W4 (x, y) increases as the brightness (luminance value) of S4 (x, y) decreases, but W4 (x, y) decreases when the brightness of S4 (x, y) increases. Become. Further, when S4 (x, y) is saturated, W4 (x, y) becomes the minimum value “0”.

When the combined weights W1 to W4 (x, y) calculated as described above are displayed in an overlapping manner, as shown in FIG. 10, W1 to W4 (x, y) overlap with the incident light amount. Appears. In this embodiment, when calculating W2 (x, y), two brightness values S2 (x, y) and S3 (x, y) are used, and when calculating W3 (x, y), S3 Two brightness values (x, y) and S4 (x, y) are used. Therefore, in the overlapping region, more appropriate composite weights W2 (x, y) and W3 (x, y) than calculating using only S3 (x, y) or only S4 (x, y) ) Can be calculated.
The weight calculator 16a outputs the combined weights W1 to W4 (x, y) calculated as described above to the combiner 16c.

Further, when the normalization unit 16b acquires the pixel data S1 to S4 (x, y), first, the normalization unit 16b converts the pixel data S1 (x, y) into T1 and T4 among the exposure ratio information acquired from the control unit 18. Based on the above equation (7), the pixel data S1 (x, y) is normalized (step S212).
Specifically, the pixel data S1 (x, y) and the exposure times T1 and T4 are substituted into the above equation (7) to obtain “N1 (x, y) as normalized pixel data N1 (x, y). ) = S1 (x, y) × T4 / T1 ”.
Using the above numerical example as the exposure times T1 and T4, when the luminance value of the pixel data S1 (x, y) is “2”, “N1 (x, y) = 2 × 500/2 = 500”. .

Next, the normalization unit 16b calculates the pixel data S2 (x, y) from the above equation (8) based on the pixel data S2 (x, y) and T2 and T4 of the exposure ratio information acquired from the control unit 18. y) is normalized (step S214).
Specifically, the pixel data S2 (x, y) and the exposure times T2 and T4 are substituted into the above equation (8) to obtain “N2 (x, y) as normalized pixel data N2 (x, y). ) = S2 (x, y) * T4 / T2 ".
When the above numerical examples are used as the exposure times T2 and T4 and the pixel data S2 (x, y), “N2 (x, y) = 10 × 500 / 13≈385”.

Next, the normalization unit 16b calculates the pixel data S3 (x, y) from the above equation (9) based on the pixel data S3 (x, y) and T3 and T4 of the exposure ratio information acquired from the control unit 18. y) is normalized (step S216).
Specifically, the pixel data S3 (x, y) and the exposure times T3 and T4 are substituted into the above equation (9) to obtain “N3 (x, y) as normalized pixel data N3 (x, y). ) = S3 (x, y) * T4 / T3 ".

When the above numerical example is used as the exposure times T3 and T4 and the pixel data S3 (x, y), “N3 (x, y) = 40 × 500 / 79≈253”.
Since the exposure time T4 is used as a reference, the pixel data S4 (x, y) uses the same value (the above formula (10)). In the above numerical example, “N4 (x, y) = 160”.
The normalizing unit 16b outputs the normalized pixel data N1 to N4 (x, y) calculated as described above to the synthesizing unit 16c.

When the combining unit 16c acquires the combined weights W1 to W4 (x, y) from the weight calculating unit 16a and the normalized pixel data N1 to N4 (x, y) from the normalizing unit 16b, the above formula (11) Is used to execute HDR synthesis processing of N1 to N4 (step S218).
Specifically, as shown in the above equation (11), the normalized pixel data N1 to N4 (x, y) are weighted with the combined weights W1 to W4 and added to add the HDR pixel data HDR (x, y). Calculate That is, as HDR pixel data HDR (x, y), “HDR (x, y) = W1 (x, y) × N1 (x, y) + W2 (x, y) × N2 (x, y) + W3 (x , Y) * N3 (x, y) + W4 (x, y) * N4 (x, y) ".

Here, the above numerical example is used as the combined weights W1 to W4 (x, y) and the normalized pixel data N1 to N4 (x, y).
When the above numerical values are substituted into the above equation (11), “HDR (x, y) = 1.538 × 10 ⁻³ × 500 + 0.02307 × 385 + 0.3706 × 253 + 0.6294 × 160≈0.769 + 8.88 + 93.8 + 101 ≈ 204 ”.

The HDR pixel data HDR (x, y) generated in this way is stored in a memory area at an address corresponding to each pixel position in the frame memory 17 (step S220).
As described above, the composite weights W1 to W4 (x, y), normalized pixel data N1 to N4 (x, y), and HDR pixel data HDR (x, y) of each pixel are sequentially calculated. This HDR synthesizing process is performed until the HDR synthesizing process is completed for one frame of pixels (“Yes” branch in steps S222 and S224).

Then, when HDR pixel data for one frame is generated and the imaging process is continued (“No” branch of step S108), the exposure ratio adjustment process is executed (step S110).
When the exposure ratio adjustment process is started, the image analysis unit 19 acquires the HDR pixel data for one frame from the frame memory 17. Based on the acquired HDR pixel data, the average luminance Lav for one frame is calculated as statistical information related to the brightness of the imaging environment, and the calculated Lav is output to the control unit 18 (step S300). The average luminance for a plurality of frames may be calculated using the HDR pixel data for a plurality of frames, not limited to one frame.
When acquiring the average luminance Lav from the image analysis unit 19, the control unit 18 compares Lav with the threshold Lth on the high luminance side and determines whether or not Lav is greater than Lth. Here, the threshold Lth on the high luminance side is “192”, and the threshold Ltl on the low luminance side is “64”.

For example, when the average luminance Lav is “61”, since Lav (61) is smaller than Lth (192) (the “No” branch in Step S302), the control unit 18 next calculates Lav and Ltl. A comparison is made to determine whether Lav is smaller than Ltl. Here, since Lav (61) is smaller than Ltl (64) (branch “Yes” in step S308), the control unit 18 determines that the imaging environment is dark, and currently sets the longest exposure time “Tmax”. Adjustment to increase (T3) = 500H ”is performed (step S310). For example, it is assumed that the number increases to “T3 = 1000H”. As a result, the ratio that is substantially uniform is lost, and the control unit 18 substitutes “T1 = 2H (initial), T4 = 1000H (increase)” in the above equations (6) and (7). Then, the intermediate exposure time Tmid (T2 and T3) is recalculated. As a result, “T2 = (4 × 1000) ^1/3 ≈16” is calculated, and “T3 = (2 × 1000000) ^1/3 ≈126” is calculated. Then, “T1: T2: T3: T4 = 2: 16: 126: 1000” is generated as new exposure ratio information, and a new exposure information signal is generated based on this exposure ratio information. The generated exposure ratio information is output to the video signal processing unit 16, and the generated exposure information signal is transmitted to the image sensor 10 together with each control signal. Thereby, in the image sensor 10, an imaging process is performed based on new exposure ratio information suitable for the brightness of the imaging environment.

When the calculated average luminance Lav is “221”, since Lav (221) is larger than Lth (192) (“Yes” branch in step S302), the control unit 18 has a bright imaging environment. And the adjustment is performed to reduce the currently set shortest exposure time “Tmin (T1) = 2H” (step S114). For example, it is assumed that the value decreases to “T1 = 1H”. As a result, the substantially uniform ratio is lost, and the control unit 18 substitutes “T1 = 1H (decrease), T4 = 500H (initial)” in the above equations (12) and (13). Then, the intermediate exposure time Tmid (T2 and T3) is recalculated. Thus, “T2 = (1 × 500) ^1/3 ≈8” is calculated, and “T3 = (1 × 250,000) ^1/3 ≈63” is calculated. Then, “T1: T2: T3: T4 = 1: 8: 63: 500” is generated as new exposure ratio information, and a new exposure information signal is generated based on this exposure ratio information. The generated exposure ratio information is output to the video signal processing unit 16, and the generated exposure information signal is transmitted to the image sensor 10 together with each control signal. Thereby, in the image sensor 10, an imaging process is performed based on new exposure ratio information suitable for the brightness of the imaging environment.

As described above, the imaging apparatus 1 according to the present embodiment calculates the combination weights W1 to W4 when weighting and combining the pixel data S1 to S4 obtained by imaging with the exposure times T1 to T4. Is calculated using two of S2 and S3. Furthermore, W3 is calculated using two of S3 and S4.
In other words, the combined weight W2 for S2 is calculated using the brightness (luminance value) of both S2 and S3, which has the longest exposure time, and therefore is more natural than the case where only the brightness of S2 is used. An appropriate composite weight W2 can be calculated.
Similarly, the composite weight W3 for S3 is calculated using the brightness (luminance value) of both S3 and S4, which has the longest exposure time, and therefore more natural than the case where only the brightness of S3 is used. Thus, an appropriate composite weight W3 can be calculated.

In the image sensor 10, the ratios “L2 / L1”, “L3 / L2”, and “L4 / L3” of each two adjacent exposure amounts are equally “L2 / L1 = L3 / L2 = L4 / L3”. The subject can be imaged at exposure times T1 to T4 corresponding to the exposure amounts L1 to L4.
As a result, the ratio of the exposure amounts of the pixel data S1 to S4 used in the HDR synthesizing process is always constant, so that it is possible to obtain a synthesized image with uniform luminance and less noise from the dark part to the bright part.

Further, as the statistical information related to the brightness of the imaging environment, the average luminance Lav is calculated from the HDR pixel data for one frame, and the Lav is compared with the high luminance side threshold value Lth and the low luminance side threshold value Ltl, thereby obtaining the imaging environment. It is possible to determine the brightness state.
Further, when the imaging environment is determined to be bright, adjustment is performed to decrease the shortest exposure time Tmin (T1), and when the imaging environment is determined to be dark, adjustment is performed to increase the longest exposure time Tmax (T4). It can be carried out.

Furthermore, by using T1 and T4 after adjustment and recalculating T2 and T3 according to the above formulas (12) and (13), the ratio collapsed by adjusting the exposure time can be returned evenly.
Thus, the exposure amount ratio (exposure time ratio) can be automatically adjusted to an exposure amount ratio suitable for the brightness of the imaging environment.
In the above embodiment, the imaging device 10 and the control unit 18 correspond to the imaging unit described in the form 7, and the weight calculation unit 16a corresponds to the weight calculation unit described in any one of the forms 1 to 4, and The combining unit 16b and the combining unit 16c correspond to the image combining unit described in the first embodiment.

Moreover, in the said embodiment, step S204-S210 respond | corresponds to the weight calculation step as described in form 5 or 6, and step 212-s218 respond | corresponds to the image composition step as described in form 5 or 6.
Note that the imaging device 1 in the above embodiment can be combined with other devices (not shown) such as a display device and a memory device to constitute an electronic device such as a digital camera or a digital video camera.
Moreover, in the said embodiment, when using the synthetic | combination weight, although the structure using the function F (x) shown to said Formula (2) was mentioned as an example, it is not restricted to this structure.

For example, as long as the value of the element x (in the above embodiment, the luminance value) increases, the function increases as long as the maximum value is “1” or another constant, and another function may be used. For example, a function that sets an offset value and sets all values to “0” when the value of x is equal to or smaller than the offset value may be used.
Moreover, in the said embodiment, although it was set as the structure which implement | achieves the multiple types of exposure amount from which a ratio becomes equal by controlling exposure time, according to the function of the image pick-up element 10 not only in this structure, for example, It may be configured to be realized by controlling iris (aperture), imaging sensitivity, and the like.

Moreover, in the said embodiment, although it was set as the structure which calculates intermediate | middle exposure time T2 and T3 using the said Formula (12) and (13), it is not restricted to this structure. For example, T2 and T3 for predetermined T1 and T4 are calculated in advance using the above equations (12) and (13) to generate a data table (LUT), and T2 and T3 are determined using the generated LUT. It is good also as a structure.
In the above embodiment, the HDR sensor 10d of the image sensor 10 has the sensor cell array 56 configured using the CMOS technology. However, the configuration is not limited thereto. For example, other configurations such as a configuration having a sensor cell array composed of CCDs may be used.

The above embodiments are preferable specific examples of the present invention, and various technically preferable limitations are given. However, the scope of the present invention is described in particular in the above description to limit the present invention. As long as there is no, it is not restricted to these forms. In the drawings used in the above description, for convenience of illustration, the vertical and horizontal scales of members or parts are schematic views different from actual ones.
Further, the present invention is not limited to the above-described embodiment, and modifications, improvements, and the like within the scope that can achieve the object of the present invention are included in the present invention.

DESCRIPTION OF SYMBOLS 1 ... Image pick-up device, 10 ... Image pick-up element, 10a ... Lens, 10b ... Micro lens, 10c ... Color filter array, 10d ... HDR sensor, 12-15 ... 1st-4th memory, 16 ... Image signal processing part, 16a ... Weight calculation unit, 16b ... normalization unit, 16c ... synthesis unit, 17 ... frame memory, 18 ... control unit, 19 ... image analysis unit, 20 ... operation unit, 50 ... reference timing generator, 54 ... scanning line scanner, 56 ... Sensor cell array, 58 ... Horizontal transfer unit

Claims (6)

An image processing apparatus that performs image composition using three or more types of images obtained by imaging a subject with three or more types of exposure amounts,
A weight calculation unit for calculating a weight of the first image in the image composition based on the luminance of the first image and the luminance of the second image;
An image synthesis unit that performs the image synthesis using the weight calculated in the weight calculation unit;
And an image analysis unit that generates information for adjusting the three or more types of exposure amounts based on statistical information obtained from the image obtained as a result of the image synthesis. The exposure amount of the first image includes the three or more types of exposure amounts. The exposure amount other than the minimum exposure amount and the maximum exposure amount in the exposure amount,
The exposure amount of the second image is larger than the exposure amount of the first image, and is the second largest exposure amount after the exposure amount of the first image among the three or more types of exposure amounts. An image processing apparatus. In an image captured with the three or more exposure amounts,
When the image captured with the minimum exposure amount is the third image, and the image captured with the next largest exposure amount after the third image is the fourth image,
In the weight calculation unit, the weight of the third image is calculated based on the luminance of the fourth image,
The image captured at the maximum exposure amount when the fifth image,
The weight calculation unit calculates the weight of the fifth image based on the luminance of an image other than the third image among images captured with the three or more types of exposure amounts. Item 8. The image processing apparatus according to Item 1.   The image processing apparatus according to claim 1, wherein the statistical information is an average luminance obtained from a plurality of image synthesis results. An image processing method for performing image composition using three or more images obtained by imaging a subject with three or more exposure amounts,
A weight calculation step of calculating a weight of the first image in the image composition based on the luminance of the first image and the luminance of the second image;
An image synthesis step of performing the image synthesis using the weight calculated in the weight calculation unit;
And an image analysis step of generating information for adjusting the three or more types of exposure amounts based on statistical information obtained from the image obtained as a result of the image synthesis. The amount of exposure of the first image is three or more types. The exposure amount other than the minimum exposure amount and the exposure amount other than the maximum exposure amount,
The exposure amount of the second image is larger than the exposure amount of the first image, and is the second largest exposure amount after the exposure amount of the first image among the three or more types of exposure amounts. An image processing method characterized by the above. In an image captured with the three or more exposure amounts,
When the image captured with the minimum exposure amount is the third image, and the image captured with the next largest exposure amount after the third image is the fourth image,
In the weight calculation step, the weight of the third image is calculated based on the luminance of the fourth image;
The image captured at the maximum exposure amount when the fifth image,
In the weight calculation step, the weight of the fifth image is calculated based on the luminance of an image other than the third image among images captured with the three or more types of exposure amounts. The image processing method according to claim 4.   6. The image processing method according to claim 4, wherein the statistical information is an average luminance obtained from a plurality of results of the image synthesis.

Images