JP5852385B2 - Imaging device - Google Patents

Description

The present invention combines a plurality of captured images with different exposure amounts, an image relating to that imaging device generates a wide dynamic range.

  In general, there has been proposed an image processing apparatus that synthesizes a plurality of captured images with different exposure amounts, expands the dynamic range, and then performs gradation compression in accordance with the dynamic range of the output device (Patent Document 1). Image processing that converts the input image into multiple resolution images and uses them to optimize the tone compression characteristics for each small area around the pixel of interest in the input image and compress the dynamic range of the input image Has been proposed (Patent Document 2).

Japanese Patent Application Laid-Open No. 07-131704 Japanese Patent No. 03731577

  However, in the technique described in Patent Document 1, an image with a wide dynamic range obtained by combining a plurality of images with different exposure amounts is output with uniform characteristics according to the luminance in the screen as shown in FIG. Tone compression is performed to fit within the dynamic range of the device. For this reason, there is a problem that a luminance region in which gradation reproducibility is impaired occurs.

  On the other hand, the technique described in Patent Document 2 requires multi-resolution processing in order to generate gradation compression characteristics. That is, it is necessary to refer to images of a plurality of types of resolutions generated from the input image at the same timing. Therefore, it is necessary to adjust the timing by buffering the high resolution image in the frame memory while generating the low resolution image. The buffering of an image using a frame memory has a problem that the load on the imaging system becomes high particularly when shooting a moving image or when the image size and the bit width are large.

  The present invention has been made in view of the above problems, and an object thereof is to suppress an increase in system load when a plurality of captured images with different exposure amounts are combined to generate an image with a wide dynamic range. It is in.

In order to achieve the above object, an imaging apparatus according to an aspect of the present invention has the following arrangement. That is,
An imaging apparatus that generates an image with an expanded dynamic range by combining two images having different exposure amounts,
Imaging means for outputting a first image and a second image following the first image by photographing in which two different exposure amounts are sequentially applied;
Synthesizing means for synthesizing the first image and the second image to generate a synthesized image having an extended dynamic range;
Generating means for generating gradation compression characteristics based on the first image and an image obtained by reducing the resolution of the first image;
Gradation compression means for compressing the gradation of the composite image using the gradation compression characteristic generated by the generation means;
Analyzing means for analyzing the first image in order to determine which of the two exposure amounts is appropriate as an exposure amount of the image used for generating the gradation compression characteristic;
Control means for controlling correspondence between the first and second images and the two exposure amounts so that the generation unit uses an image taken with an exposure amount determined to be appropriate by the analysis unit. .

  According to the present invention, it is possible to suppress an increase in system load when a plurality of captured images with different exposure amounts are combined to generate an image with a wide dynamic range.

The block diagram which shows the structure of the imaging device of 1st and 2nd embodiment. The block diagram which shows the structure of a gradation compression characteristic production | generation part. 6 is a timing chart showing image composition processing according to the first embodiment. The schematic diagram which shows an example of the luminance distribution of the image in imaging | photography. The figure explaining the characteristic at the time of producing | generating a gradation compression characteristic with a low exposure image or a high exposure image. The timing chart which shows the synthetic | combination process of the image by 2nd Embodiment. The block diagram which shows the structure of the imaging device by 3rd Embodiment. 10 is a timing chart showing image composition processing according to the third embodiment. The figure which shows an example of a gradation compression characteristic. The schematic diagram which shows pixel arrangement | positioning of an image pick-up element.

  Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings.

[First Embodiment]
In the first embodiment, an imaging apparatus having an image processing apparatus that synthesizes images of two frames with different exposure amounts and outputs an image of one frame with an expanded dynamic range will be described.

  FIG. 1 is a block diagram illustrating a configuration of the imaging apparatus 100 according to the first embodiment. In FIG. 1, an optical system 1 includes optical elements such as an imaging lens and a diaphragm. The imaging device 2 converts an optical image formed on the imaging surface by the optical system 1 into an electrical signal and outputs an image signal. The imaging device 2 of the present embodiment can perform imaging by switching a plurality of types of exposure amounts, and the switching of the exposure amounts is realized by switching the charge accumulation time in the photoelectric conversion element, for example. The frame memory 3 is a memory for buffering one frame of the image output from the image sensor 2. The synthesizer 4 synthesizes the image read from the frame memory 3 and the image of the current frame output from the image sensor 2 by a predetermined calculation to generate an image with a wide dynamic range. The gradation compression characteristic generation unit 5 generates gradation compression characteristics for the image synthesized by the synthesis unit 4. The gradation compression unit 6 performs gradation compression using the gradation compression characteristic output from the gradation compression characteristic generation unit 5 so that the output image of the synthesis unit 4 falls within a predetermined dynamic range. The image analysis unit 7 outputs from the image sensor 2 in order to determine which of the two kinds of exposure amount images output from the image sensor 2 should be used for generating the gradation compression characteristics. Analyzing the resulting image. The system control unit 8 controls the overall operation of the imaging apparatus 100.

  Next, an outline of the operation of the imaging apparatus 100 according to the first embodiment will be described. When shooting is started in the imaging apparatus 100, the system control unit 8 controls the optical system 1 and the imaging element 2 so that an image with the instructed exposure amount is output from the imaging element 2 for each frame. An image output from the image sensor 2 via the optical system 1 is input to the frame memory 3, the synthesis unit 4, the gradation compression characteristic generation unit 5, and the image analysis unit 7. The frame memory 3 holds one frame of the output image from the image sensor 2.

  The synthesizing unit 4 synthesizes the image held in the frame memory 3 and the image output from the image sensor 2 so as to extend the dynamic range. As a result, the output image of the image sensor 2 one frame before read from the frame memory 3 and the output image of the current frame of the image sensor 2 currently output from the image sensor 2 are combined. The amount of exposure differs between the output image of the image sensor 2 one frame before read from the frame memory 3 and the output image of the current frame from the image sensor 2. Therefore, a gain process corresponding to the difference in exposure amount is performed before synthesis, and the level adjustment of the output image of the image sensor 2 of the current frame and the previous frame is performed.

  The gradation compression characteristic generation unit 5 uses the output image of the previous frame read from the frame memory 3 and the output image of the current frame from the image sensor 2 to generate a floor for the output image of the synthesis unit 4. Generate a tonal compression characteristic. The gradation compression unit 6 uses the gradation compression characteristic output from the gradation compression characteristic generation unit 5 to synthesize the image being captured so that the desired gradation is reproduced and within the dynamic range of the output video format. Tone compression of the image output from the unit 4 is performed.

  Next, the configuration of the gradation compression characteristic generation unit 5 will be described with reference to FIG. The gradation compression characteristic generation unit 5 determines the gradation compression characteristic for each pixel by referring to the luminance level of the small area including the target pixel.

  In FIG. 2, the output image of the frame memory 3 is input to the input terminal 58, and the output image of the image sensor 2 is input to the input terminal 59. The luminance generation unit 51 and the luminance generation unit 52 generate luminance signals from images input from the input terminal 58 and the input terminal 59, respectively. Both the output image of the frame memory 3 and the output image of the image sensor 2 are RAW data, and there is a level difference due to a color filter for each pixel such as RGB Bayer. Therefore, the luminance generation units 51 and 52 perform interpolation processing and matrix calculation to generate a luminance signal in which the level difference due to the color filter is eliminated.

  The image reduction unit 53 reduces the luminance image for one frame output from the luminance generation unit 52 at a high reduction ratio such as × 1/64 or × 1/128, and the result is a memory 54 as a second memory. To store. Note that the capacity of the memory 54 for storing the reduced image is sufficiently smaller than the frame memory 3 for buffering one frame of the input image, and even if the memory 54 is used, the load on the imaging system does not matter. . Further, the frame memory 3 and the memory 54 may be separate memories, or may be different memory areas of the same memory. The image enlarging unit 55 enlarges the reduced image stored in the memory 54 using linear interpolation or the like so as to have an image size equal to the luminance image output from the luminance generating unit 51.

The local luminance level estimation unit 56 calculates the luminance of the small region including the target pixel by calculation using the high-resolution luminance image output from the luminance generation unit 51 and the low-resolution luminance image output from the image enlargement unit 55. Estimate the level. As an example of such an estimation operation,
・ Compare the output of multiple images with different resolutions for each pixel,
When the difference between the high resolution image and the low resolution image is small, the output of the low resolution image is output as the luminance level at the pixel of interest,
-When the difference between the high-resolution image and the low-resolution image is large, a method of weighting and adding the low-resolution image and the high-resolution image and outputting the luminance level at the pixel of interest can be used. By using such an estimation calculation, in an image in which small areas of various luminance levels are mixed as shown in FIGS. The separation accuracy of each region can be increased. In the present embodiment, the local luminance level is estimated using images of two types of resolutions, but the present invention is not limited to this, and images of three or more types of different resolutions may be used.

The gradation compression characteristic determination unit 57 refers to the output of the local luminance level estimation unit 56 and determines an appropriate gradation compression characteristic for each pixel. For example,
When the brightness level of the small area including the target pixel is darker than the appropriate exposure, a gradation compression characteristic that increases the brightness level of the target pixel is generated,
When the luminance level of the small area including the target pixel is brighter than the appropriate exposure, a gradation compression characteristic that reduces the luminance level of the target pixel is generated.

  Incidentally, the local luminance level estimation unit 56 needs to refer to the low resolution image and the high resolution image at the same timing. Therefore, while the image reduction unit 53 is reducing the image for one frame, it is necessary to buffer and delay the high-resolution image for one frame. Such image buffering increases the system load of the imaging device, particularly when the image size is large or when the frame rate of the captured image is high. Therefore, in the imaging apparatus 100 according to the present embodiment, the frame memory 3 used in the synthesizing unit 4 is shared as a delay adjustment buffer in the gradation compression characteristic generating unit 5 to prevent an increase in system load. .

  On the other hand, an image for one frame can be buffered in the frame memory 3 used in the synthesis unit 4. In the present embodiment, when a low exposure image and a high exposure image are combined, the first image of the two frames of the low exposure image and the high exposure image, that is, the first frame image is the frame memory 3. Buffered. Then, the second frame image output from the image sensor 2 and the first frame image buffered in the frame memory 3 are combined. As described above, the images of the two frames are images taken by applying different exposure amounts by the optical system 1 and the image sensor 2. Therefore, when the image buffered for composition is shared by the gradation compression characteristic generation unit 5, the first frame image before composition, that is, the low-exposure image or high-exposure image before composition is scaled. A tonal compression characteristic is generated.

  FIG. 5 shows merits and demerits when the gradation compression characteristic is generated using each of the low exposure image and the high exposure image. The low-exposure image has a dark S / N lower than that of the high-exposure image, but is not saturated even in the bright part. A wide range of gradation compression characteristics can be generated. On the other hand, a high-exposure image cannot generate a gradation compression characteristic in a saturated area in a bright part, but a dark part has a higher S / N than a low-exposure image. Can generate tone compression characteristics with good S / N. In other words, generating gradation compression characteristics using a low-exposure image results in gradation compression processing suitable for bright areas, and generating gradation compression characteristics using a high-exposure image results in gradation compression processing suitable for dark areas. .

  Therefore, in the imaging apparatus 100 according to the present embodiment, it is determined whether the dominant luminance region is a dark part as shown in FIG. 4A or a bright part as shown in FIG. To do. In the image being captured, the image used in the gradation compression characteristic generation unit 5, that is, the frame memory 3, so that gradation compression processing is performed with preferable gradation compression characteristics at least in the dominant luminance region. Determine the images to be retained.

  As a method for determining a dominant luminance region in an image being shot, as described below, the image analysis unit 7 may determine by analyzing the luminance distribution of the image being shot. The system control unit 8 may determine the shooting conditions such as settings and the user's instructions. Hereinafter, the operation of the image analysis unit 7 that analyzes the luminance distribution in the image and determines whether the dark part is the dominant image or the bright part is the dominant image will be described.

  As an example of the analysis method in the image analysis unit 7, the area of the pixel region having the luminance level is calculated for each predetermined luminance level, and the luminance level with the largest area is regarded as the dominant luminance level in the screen. There is a way. In this case, when the dominant luminance level is lower than the predetermined threshold, it is determined that the dark part is dominant in the image being shot. When the dominant luminance level is higher than a predetermined value, it is determined that the bright part is dominant in the image being shot.

  As another analysis method by the image analysis unit 7, an average value of luminance is obtained in a predetermined pixel area at the center of the screen, and when the average luminance of the pixel area is lower than the predetermined value, a dark part is captured in the image being shot. Is determined to be dominant. When the average luminance of the pixel area is higher than a predetermined value, it is determined that the bright part is dominant in the image being shot.

  Further, as another analysis method by the image analysis unit 7, it is detected from the brightness, color, movement, edge amount, etc. of the image whether a specific subject such as a face or a pet exists in the screen, and the average luminance of the specific subject region is determined. Ask. Then, when the average brightness of the specific subject area is lower than a predetermined value, it is determined that the dark portion is dominant in the image being shot, and when the average brightness of the specific subject is higher than the predetermined value, It is determined that the bright portion is dominant in the image being shot.

  As described above, the image analysis unit 7 analyzes the image being shot and sends the analysis result to the system control unit 8. The system control unit 8 controls the driving of the image sensor 2 based on the analysis result output of the image analysis unit 7. For example, as a result of analysis by the image analysis unit 7, when it is determined that the dark part is dominant in the image being shot, the high-exposure image is held in the frame memory 3. In the present embodiment, the image pickup device 2 is driven and controlled so that an image with a higher exposure amount is captured first among images of two consecutive frames to be synthesized with respect to an image in which the dark portion is dominant. . On the other hand, as a result of the analysis by the image analysis unit 7, when it is determined that the bright part is dominant in the image being shot, the low-exposure image is held in the frame memory 3. In the present embodiment, the image pickup device 2 is driven and controlled so that an image with a lower exposure amount is captured first among images of two consecutive frames to be synthesized.

  An example of the imaging operation when the dominant region in the image being shot changes from the bright part to the dark part will be described with reference to the timing chart of FIG. In FIG. 3, the intervals between times t1, t2, t3,..., T7 each indicate one frame period (hereinafter referred to as 1V). A low-exposure image or a high-exposure image is output from the image sensor 2 for each 1V, and the image synthesized by the synthesis unit 4 is a combination surrounded by a broken line in FIG.

  That is, in the period from time t1 to t3, the low exposure image L1 and the high exposure image H1 output from the image sensor 2 are combined to generate a combined image (H1 + L1). Similarly, in the period from time t3 to t5, the low exposure image L2 output from the image sensor 2 and the high exposure image H2 are combined, and in the period from time t5 to t7, the low exposure image L3 output from the image sensor 2 is high. The exposure image H3 is synthesized.

  The control signals S1 to S3 are signals that the system control unit 8 controls and outputs. The control signal S <b> 1 is a binary control signal that controls writing of an image into the frame memory 3 and on / off of processing in the gradation compression characteristic generation unit 5. When the control signal S1 is “1”, the output image of the image sensor 2 is written into the frame memory 3 over a period of 1V. At the same time, the output image of the image sensor 2 is input to the gradation compression characteristic generation unit 5, and the reduced image is generated over a 1 V period and held in the memory 54. When the control signal S1 is “0”, the writing of the image to the frame memory 3 is stopped, and the gradation compression characteristic generation unit 5 holds the image held in the frame memory 3 and the memory 54. A gradation compression characteristic is generated using the reduced image and sequentially output.

  The control signal S <b> 2 is a binary signal that controls reading of an image from the frame memory 3, processing on / off in the synthesis unit 4, and processing on / off in the gradation compression unit 6. . When the control signal S2 is “0”, reading of the image from the frame memory 3 is stopped, and the processing in the synthesis unit 4 and the gradation compression unit 6 is turned off. When the control signal S2 is “1”, the image held in the frame memory 3 is read. Then, the synthesis unit 4 performs a synthesis process for extending the dynamic range using the current frame image output from the image sensor 2 and the image read from the frame memory 3 (the image immediately before the current frame). . Then, the gradation compression unit 6 uses the output of the gradation compression characteristics sequentially output from the gradation compression characteristic generation unit 5 for the combined image sequentially output from the combining unit 4 to perform gradation compression. Process.

  The control signal S3 is a binary control signal that determines which of the two frames to be combined, that is, the low-exposure image or the high-exposure image, is to be captured first by the image sensor 2. The control signal S3 is updated based on the analysis result in the image analysis unit 7, for example. The control signal S3 becomes “0” when the bright portion is determined to be dominant in the image being shot as a result of the analysis by the image analysis unit 7. When the control signal S3 is “0”, the image sensor 2 is controlled so that the low-exposure image is captured first when the next composition image is captured. On the other hand, the control signal S3 becomes “1” when the dark portion is determined to be dominant in the image being photographed as a result of the analysis by the image analyzing portion 7. When the control signal S3 is “1”, the imaging device 2 is controlled so that a high-exposure image is captured first when the next composition image is captured.

  In FIG. 3, since the control signal S3 is “0” in the period from time t1 to t2, the images of the two frames to be combined are the low exposure image L1 for the first frame and the high exposure image H1 for the second frame. Are output from the image sensor 2 in the order of Then, in the 1V period from time t1 to time t2, the low exposure image L1 is written into the frame memory 3, and in parallel with this, a reduced image of the low exposure image L1 is generated and held in the memory 54. From time t2 to t3, the synthesizing unit 4 synthesizes the low-exposure image L1 written in the frame memory 3 and the high-exposure image H1 output from the image sensor 2, and outputs an image H1 + L1. At the same time, the gradation compression characteristic generation unit 5 sequentially generates gradation compression characteristics g (L1) from the low-exposure image L1 held in the frame memory 3 and the reduced image held in the memory 54. Then, the gradation compression unit 6 performs gradation compression processing on the combined image H1 + L1 using the gradation compression characteristic g (L1) to generate a final image.

  In the period from time t1 to time t3, the image analysis unit 7 analyzes the low-exposure image L1 that is captured first of the low-exposure image L1 and the high-exposure image H1 to be synthesized, and the analysis result A (L1). ) Is output. Then, the system control unit 8 updates the control signal S3 referred to in the synthesis from time t3 to “0” based on the analysis result A (L1). This is because the bright part is determined to be dominant as a result of analyzing the image currently being shot. Therefore, in the next compositing process, a low-exposure image is taken first, and the tone compression characteristics in the bright part are obtained. This is because it is considered that a better image quality is obtained by optimizing.

  Since the control signal S3 is “0” also during the period from the time t3 to the time t5, the images of the two frames to be synthesized are output from the image sensor 2 in the order of the low-exposure image L2 and the high-exposure image H2. Then, in the 1V period from time t3 to time t4, the low-exposure image L2 is written into the frame memory 3, and at the same time, a reduced image of the low-exposure image L1 is generated and held in the memory 54. From time t4 to t5, the synthesizing unit 4 synthesizes the low-exposure image L2 written in the frame memory 3 and the high-exposure image H2 output from the image sensor 2, and outputs an image H2 + L2. In parallel, the gradation compression characteristic generation unit 5 sequentially generates gradation compression characteristics g (L2) from the low-exposure image L2 held in the frame memory 3 and the reduced image held in the memory 54. Then, the gradation compression unit 6 performs gradation compression processing on the combined image H2 + L2 using the gradation compression characteristic g (L2) to generate a final image.

  In the period from time t3 to time t5, the image analysis unit 7 analyzes the low-exposure image L2 that has been captured first, of the two frames to be combined, that is, the low-exposure image L2 and the high-exposure image H2. The analysis result A (L2) is output. Now, assume that the dark portion is determined to be dominant in the image analysis unit 7. In this case, as a result of analyzing the image currently being photographed, it can be determined that the dark part is dominant. Therefore, in the next compositing process, a high-exposure image is photographed first, and gradation compression in the dark part is performed. It is considered that a better image quality is obtained when the characteristics are optimized. Therefore, the system control unit 8 updates the control signal S3 referred to in the synthesis from time t5 to “1” based on A (L2).

  Since the control signal S3 is “1” during the period from time t5 to time t7, the images of the two frames to be combined are output from the image sensor 2 in the order of the high exposure image H3 and the low exposure image L3. Then, in the 1V period from time t5 to t6, the high-exposure image H3 is written into the frame memory 3, and at the same time, a reduced image of the high-exposure image H3 is generated and held in the memory 54. From time t6 to t7, the synthesizing unit 4 synthesizes the high-exposure image H3 written in the frame memory 3 and the low-exposure image L3 output from the image sensor 2, and outputs an image H3 + L3. In parallel with this, the gradation compression characteristic generation unit 5 sequentially generates the gradation compression characteristic g (H3) using the high-exposure image H3 held in the frame memory 3 and the reduced image held in the memory 54. To do. Then, the gradation compression unit 6 performs gradation compression processing on the combined image H3 + L3 using the gradation compression characteristic g (H3) to generate a final image.

  In the period from time t5 to t7, the image analysis unit 7 uses the high-exposure image H3 previously captured out of the two frames to be combined, that is, the low-exposure image L3 and the high-exposure image H3. The analysis result A (H3) is output. Then, the system control unit 8 updates the control signal S3 referred to in the synthesis from the time t7 to 1 based on the analysis result A (H3). As a result of analyzing the image currently being photographed, it can be determined that the dark part is dominant. Therefore, in the next compositing process, a high-exposure image is taken first, and the tone compression characteristics in the dark part are optimized. This is because it is considered that better image quality is obtained.

  By performing a series of imaging control as described above, multiple images with different exposure amounts are combined, and gradation compression is performed with gradation compression characteristics suitable for the combined image, and a video signal is output. can do.

  As described above, in the imaging apparatus 100 according to the present embodiment, tone compression is performed on the synthesized image using the same image as the image buffered in the frame memory 3 for the synthesis process for generating an image with a wide dynamic range. Properties can be generated. Therefore, an increase in system load due to memory access can be avoided. In addition, by analyzing the characteristics of the image being shot and determining the image used to generate the tone compression characteristics, tone compression processing suitable for the image being shot can be performed.

[Second Embodiment]
Next, an imaging apparatus according to the second embodiment will be described. In the first embodiment, when the exposure amounts of the first frame and the second frame of the two frames to be synthesized are switched based on the analysis result of the image analysis unit 7, A configuration is shown in which the order of application of the exposure amounts of frames is changed. For example, a configuration is adopted in which the exposure amount corresponding to the first frame and the second frame is switched by switching the order from the low exposure image L → the high exposure image H to the high exposure image H → the low exposure image L. However, the configuration for switching the exposure amount is not limited to the first embodiment. For example, the exposure amount of the first frame and the second frame may be switched by shifting the frame held by the frame memory 3 by one frame. In the second embodiment, such a configuration will be described.

  The second embodiment is also an imaging apparatus that synthesizes images for two frames with different exposure amounts and outputs an image for one frame with an expanded dynamic range. The configuration of the imaging apparatus 100 of the second embodiment is the same as that of the first embodiment and is as shown in FIG. As in the first embodiment, since the synthesizing unit 4 and the gradation compression characteristic generating unit 5 share the frame memory 3, the image to be buffered in the frame memory 3 among the images of the two frames to be synthesized. A tone compression characteristic for the combined image is generated.

  However, in the second embodiment, as in the first embodiment, a mechanism for selecting whether to shoot a low-exposure image or a high-exposure image first is based on the result of the image analysis unit 7. It is unnecessary. That is, in the image pickup device 2 of the second embodiment, the drive of the image pickup device 2 is controlled so that a low exposure image or a high exposure image is always taken first. In the second embodiment, the frame memory 3 is configured so that an image that can generate a preferable gradation compression characteristic among the images of two frames to be synthesized based on the result of the image analysis unit 7 is buffered in the frame memory 3. The timing of image writing to the memory 3 is controlled.

  As a result of the analysis by the image analysis unit 7, the system control unit 8 buffers the high-exposure image in the frame memory 3 when the dark part is dominant in the image being shot, and when the bright part is dominant. The low-exposure image is controlled to be buffered in the frame memory 3. Here, in the image being shot, when the dominant luminance region changes from a dark part to a bright part, or from a bright part to a dark part, two high-exposure images and two low-exposure images output from the image sensor 2 are displayed. The frame is continuously buffered in the frame memory 3.

  Therefore, the image analysis processing in the image analysis unit 7 is completed within the 1V period in which the output image of the image sensor 2 is buffered in the frame memory 3, and the image is buffered in the frame memory 3 in the next 1V period. It is necessary to judge whether it is necessary. For this reason, in the second embodiment, signal reading from the image sensor 2 is performed by thinning out two lines at a time and dividing it into two times.

  For example, when the pixel arrangement of the image sensor 2 is as shown in FIG. 10, in the first half of the 1V period, the signals of the lines L0, L1, L4, L5,. In the latter half, the signals of the lines L2, L3, L6, L7,. The image analysis unit 7 inputs an image thinned out in the vertical direction 1/2 read from the image sensor 2 in the first half of the 1V period, and performs an analysis process in the second half of the 1V period. Therefore, while an image for one frame is being written in the frame memory 3, it can be determined whether the image should be buffered in the frame memory 3 in the next 1V period.

  Hereinafter, with reference to FIG. 6, control of the imaging apparatus of the second embodiment will be described. The image sensor 2 always outputs the pre-combination images in the order of the low exposure image and the high exposure image. The control signal S1 is a binary control signal for controlling the on / off of the image writing to the frame memory 3 and the processing in the gradation compression characteristic generation unit 5, and is updated every 1V period. When the control signal S1 is “1”, the output image of the image sensor 2 is written into the frame memory 3 over a period of 1V. At the same time, the output image of the image sensor 2 is input to the gradation compression characteristic generation unit 5, and a reduced image is generated over a 1 V period and held in the memory 54. When the control signal S1 is “0”, the writing of the image to the frame memory 3 is stopped, and the reduced image generation process in the gradation compression characteristic generation unit 5 is turned off.

  Here, when the control signal S1 is “0” in a certain 1V period, the control signal S1 is always “1” in the next 1V period. In addition, when the control signal S1 is “1” in a certain 1V period, whether the control signal S1 takes “0” or “1” in the next 1V period is input to the image analysis unit 7. It depends on the analysis result of the image.

  When the control signal S1 for a certain 1V period is “1”, the input image of the image analysis unit 7 is a low-exposure image, and the bright portion is determined to be dominant by the analysis result of the image analysis unit 7, the system The controller 8 sets the control signal S1 to “0” in the next 1V period. This is because if the bright part is dominant in the image being shot, the tone compression characteristic may be generated with the low-exposure image buffered in the frame memory 3. This is because it is not necessary to buffer the exposure image.

  On the other hand, when the control signal S1 for a certain 1V period is “1”, the input image of the image analysis unit 7 is a low-exposure image, and the dark portion is determined to be dominant by the analysis result of the image analysis unit 7, The system control unit 8 sets the control signal S1 to “1” in the next 1V period. This is because when the dark part is dominant in the image being shot, it is necessary to buffer the high-exposure image in the frame memory 3 in the next 1V period and generate a gradation compression characteristic that emphasizes the dark part. .

  Further, when the control signal S1 for a certain 1V period is “1”, the input image of the image analysis unit 7 is a high exposure image, and it is determined that the dark part is dominant in the analysis result of the image analysis unit 7, The system control unit 8 sets the control signal S1 to “0” in the next 1V period. This is because if the bright part is dominant in the image being shot, the gradation compression characteristic may be generated using the high-exposure image buffered in the frame memory 3, so in the next 1V period, This is because it is not necessary to buffer a low exposure image.

  Furthermore, when the control signal S1 for a certain 1V period is “1”, the input image of the image analysis unit 7 is a high-exposure image, and the bright portion is determined to be dominant by the analysis result of the image analysis unit 7 The system control unit 8 sets the control signal S1 to “1” in the next 1V period. This is because when the bright part is dominant in the image being shot, it is necessary to buffer the low-exposure image in the frame memory 3 in the next 1V period and generate gradation compression characteristics that emphasize the bright part. It is.

  The control signal S <b> 2 is a binary signal that controls reading of an image from the frame memory 3, processing on / off in the synthesis unit 4, and processing on / off in the gradation compression unit 6. . When the control signal S2 is “0”, reading of the image from the frame memory 3 is stopped, and the processing in the synthesis unit 4 and the gradation compression unit 6 is turned off. When the control signal S2 is “1”, the image written in the frame memory 3 is read out, and the image output from the image sensor 2 and the image read out from the frame memory 3 in the synthesis unit 4 are used to dynamically Performs composition processing to extend the range. Then, the gradation compression characteristic unit 5 generates a gradation characteristic using the image read from the frame memory 3 and the reduced image read from the memory 54 and outputs the gradation characteristic. The tone compression unit 6 performs tone compression processing on the image after synthesis from the synthesis unit 4 using the tone characteristics from the tone compression characteristic generation unit 5.

  Further, the reading process from the frame memory 3 controlled by the control signal S2, the processing by the synthesizing unit 4 and the gradation compressing unit 6 are the image writing process and the gradation to the frame memory 3 controlled by the control signal S1. It is executed 1V after the processing in the compression characteristic generation unit 5 is completed. Therefore, the control signal S2 is obtained by delaying the waveform of the control signal S1 by 1V.

  In FIG. 6, since the control signal S1 is 1 at time t1, the low-exposure image L1 is written to the frame memory 3 in the 1V period from time t1 to t2, and at the same time, the control signal S1 is low by time t2. A reduced image of the exposure image L 1 is generated and held in the memory 54. From time t2 to t3, the synthesizing unit 4 synthesizes the low-exposure image L1 held in the frame memory 3 and the high-exposure image H1 output from the image sensor 2, and outputs an image H1 + L1. In parallel with this, the gradation compression characteristic generation unit 5 generates a gradation compression characteristic g (L1) from the low-exposure image L1 held in the frame memory 3 and the reduced image held in the memory 54. Then, the gradation compression unit 6 performs gradation compression processing on the combined image H1 + L1 using the gradation compression characteristic g (L1).

  Further, the low-exposure image L1 thinned out in the vertical direction 1/2 in the first half of the 1V period from the time t1 to the time t2 is input to the image analysis unit 7, and the image analysis is performed in the second half of the 1V period from the time t1 to the time t2. Result A (L1) is generated. Based on A (L1), the control signal S1 at time t2 is updated to “0”. This is because, as a result of analyzing the currently captured image, it can be determined that the bright part is dominant, it is preferable to optimize the tone compression characteristics using the low-exposure image even in the next composition processing. This is because it is determined that it is not necessary to buffer the high exposure image during the period from t2 to t3.

  Since the control signal S1 is “0” in the 1V period from the time t2 to the time t3, the control signal S1 is updated to “1” in the 1V period from the time t3 to the time t4. Since the control signal S1 is “1” during the period from the time t3 to the time t4, the low exposure image L2 output from the image sensor 2 is written into the frame memory 3, and at the same time, by the time t4, the low exposure is performed. A reduced image of the image L2 is generated and held in the memory 54.

  Further, in the first half of the 1V period from time t3 to t4, the low-exposure image L2 thinned out in the vertical direction 1/2 is input to the image analysis unit 7, and the image analysis result is obtained in the second half of the 1V period from time t3 to t4. A (L2) is generated. Then, based on the analysis result A (L2), the system control unit 8 updates the control signal S1 at time t4 to “1”. As a result of analyzing the currently captured image, it is determined that the dark portion has become dominant. In the next composition processing, it is preferable to optimize the tone compression characteristics using the high-exposure image, and the time t4 This is because it was determined that it is necessary to buffer the high-exposure image during the period from t to t5.

  From time t4 to t5, the synthesizing unit 4 synthesizes the low-exposure image L2 written in the frame memory 3 and the high-exposure image H2 output from the image sensor 2, and outputs an image H2 + L2. In parallel with this, the gradation compression characteristic generation unit 5 generates a gradation compression characteristic g (L2) from the low-exposure image L1 held in the frame memory 3 and the reduced image held in the memory 54. Then, the gradation compression unit 6 performs gradation compression processing on the combined image H2 + L2 using the gradation compression characteristic g (L2).

  Further, since the control signal S1 is “1” from time t4 to t5, when the low-exposure image L2 is read out for the above-described composition processing, the frame memory 3 is sequentially overwritten with the high-exposure image H2. . In parallel, a reduced image of the low-exposure image L2 is generated and held in the memory 54 by time t5. Note that the reduced image is written to the memory 54 after the compressed image generated at times 43 to t4 is read from the memory 54 and then overwritten sequentially.

  Further, the low-exposure image H2 thinned out in the vertical direction 1/2 in the first half of the 1V period from time t4 to t5 is input to the image analysis unit 7, and image analysis is performed in the second half of the 1V period from time t4 to t5. Result A (H2) is generated. Then, based on the analysis result A (H2), the system control unit 8 updates the control signal S1 at time t5 to “0”. As a result of analyzing the currently captured image, it can be determined that the dark portion is dominant, and it is preferable to optimize the tone compression characteristics using the high-exposure image in the next synthesis processing, from time t5 to t6. This is because it has been determined that it is not necessary to buffer the low-exposure image during this period.

  From time t5 to t6, the synthesizing unit 4 synthesizes the high-exposure image H2 written in the frame memory 3 and the low-exposure image L3 output from the image sensor 2, and outputs an image H2 + L3. In parallel with this, the gradation compression characteristic generation unit 5 generates a gradation compression characteristic g (H2) from the high-exposure image H2 held in the frame memory 3 and the reduced image held in the memory 54. Then, the gradation compression unit 6 performs gradation compression processing on the combined image H2 + L3 using the gradation compression characteristic g (H2). Since the control signal S1 is “0” in the 1V period from time t5 to t6, the control signal S1 is updated to “1” in the 1V period from time t6 to t7.

  By performing a series of imaging control as described above, a plurality of images with different exposure amounts are combined in the second embodiment, and gradation compression is performed with gradation compression characteristics suitable for the combined image. Video signals can be output.

  As described above, also in the imaging apparatus according to the second embodiment, it is possible to generate the gradation compression characteristics for the combined image using the same image as the image buffered in the frame memory 3 for the combining process. Thus, an increase in system load due to memory access can be avoided.

  In addition, by analyzing the characteristics of the image being shot and determining the image used to generate the tone compression characteristics, tone compression processing suitable for the image being shot can be performed. Further, the image sensor 2 does not need to have a configuration for switching the acquisition order of the low exposure image and the high exposure image.

  In the second embodiment, it is determined whether or not to switch the exposure amounts of the first frame and the second frame until the input of the image of the first frame is completed. (For example, t3 to t4). With this configuration, useless frames that are not used for synthesis are not generated. However, from the viewpoint of switching the road light amounts of the first frame and the second frame according to the analysis result of the image analysis unit 7, the configuration is not limited to such a configuration. For example, the analysis by the image analysis unit 7 may be performed over a period of one frame, and the order of the road light quantity in the two frames may be switched after the second frame. For example, in FIG. 6, when the image is analyzed in the period from t3 to t4 and it is determined to hold the high exposure amount image in the frame memory 3, the control signal S1 is set to “1” at the timing of t6. You may do it.

[Third Embodiment]
Next, an imaging apparatus according to a third embodiment will be described. In the third embodiment, an imaging apparatus that synthesizes images of n frames with different exposure amounts and outputs an image of one frame with an expanded dynamic range will be described. Here, n is a natural number of 3 or more, and in this embodiment, a case where n = 3 will be described as an example.

  FIG. 7 shows the configuration of the imaging apparatus 101 according to the third embodiment. In the imaging device 101, a selection unit 9 is added to the configuration of the imaging device 100 of the first embodiment. Moreover, the point from which the output of the selection part 9 is input into the frame memory 3 and the gradation compression characteristic production | generation part 5 differs from 1st Embodiment.

  Next, an outline of the operation of the imaging apparatus 101 according to the third embodiment will be described. When shooting is started, the system control unit 8 controls the optical system 1 and the image sensor 2 so that the exposure amount of the image output from the image sensor 2 changes in a cycle of 3V. An image output from the imaging image sensor 2 via the optical system 1 is input to the selection unit 9, the synthesis unit 4, and the image analysis unit 7.

  In addition to the output image of the image sensor 2, the output image from the synthesizing unit 4 is also input to the selection unit 9, and the output image from the synthesizing unit 4 or the output from the image sensor 2 based on the control signal from the system control unit 8. Select one of the images. The output image selected by the selection unit 9 is input to the frame memory 3 and the gradation compression characteristic generation unit 5. The frame memory 3 buffers the output image from the selection unit 9 for one frame. The selection unit 9 holds the signal from the image sensor 2 in the frame memory 3 at the timing of the first frame among the n frames to be combined, and the second and subsequent n-1th frames are combined. The composite image output from the unit 4 is held in the frame memory 3. The combining unit 4 combines the image read from the frame memory 3 and the output image from the image sensor 2. As a result, the frame memory 3 finishes combining the image of the first frame or the frame immediately before the current frame among the n frames to be combined that are continuously output from the image sensor 2. Retained images.

  The tone compression characteristic generation unit 5 uses the image read from the frame memory 3 and the output image from the selection unit 9 to generate tone compression characteristics for the output image of the synthesis unit 4. The gradation compression unit 6 uses the gradation compression characteristic output from the gradation compression characteristic generation unit 5 to achieve preferable gradation reproduction for an image being shot and to be within the dynamic range of the output video format. Tone compression is performed on the output image of the synthesis unit 4. That is, when the image that has been combined up to the (n-1) th frame is held in the frame memory 3, the gradation compression characteristic generation unit 5 reduces the resolution of the image held in the frame memory 3 and the held image. A gradation compression characteristic is generated based on the image. Then, the tone compression unit 6 compresses the tone of the image synthesized up to the nth frame output from the synthesis unit 4 using the generated tone compression characteristic.

  Also in the third embodiment, as in the first embodiment, the frame memory 3 is configured to buffer an image for one frame. Therefore, when synthesizing images of three frames or more, the operation is such that the images are sequentially synthesized for every two frames. In the case of combining three frames, after combining the first frame and the second frame, the combined result of the first frame and the second frame and the input image of the third frame are combined. Therefore, the frame memory 3 of the present embodiment needs a capacity capable of holding an image for one frame having an expanded dynamic range after synthesis, that is, an image for one frame with an increased number of bits per pixel. is there. In the case of synthesizing images for n frames, tone compression processing is performed after two frames are synthesized n-1 times.

  For this reason, in the third embodiment, as shown in FIG. 7, the selection unit 9 is provided in the previous stage of the frame memory 3. When the images for n frames are synthesized, the output image of the image sensor 2 is written in the frame memory 3 at the time of the first synthesis, and the output image of the synthesis unit 4 is framed at the time of the synthesis from the second time to the (n-1) th time. The data is written in the memory 3.

  The internal configuration of the gradation compression characteristic generation unit 5 is the same as that of the first embodiment, but the input signal is partially different, the output image of the frame memory 3 is input to the input terminal 58 of FIG. The output image of the selection unit 9 is input.

  As described above, when images for n frames are combined in the present embodiment, the image being combined is buffered in the frame memory 3 and combined for every two frames, and all n frames are combined. After the completion, gradation compression processing is performed. Therefore, the gradation compression characteristic is generated using an image generated by the synthesis process up to the (n-1) th frame (the (n-2) th synthesis process). The input terminal 59 (FIG. 2) receives the image after the (n−2) th combining process output from the selection unit 9, performs a reduction process, and stores the reduced image in the memory 54. The frame memory 3 stores an image after the (n-2) th combining process.

  The processing from the input terminal 59 to the image enlargement unit 55 takes 1V period as in the first embodiment. The low-resolution image and the high-resolution image input from the frame memory 3 via the input terminal 58 and obtained through the luminance generation unit 51 are input to the local luminance level estimation unit 56 in a state where the timings coincide with each other. . The processes of the local luminance level estimation unit 56 and the gradation compression characteristic determination unit 57 are the same as those in the first embodiment.

  Also in the third embodiment, the frame memory 3 for buffering an image for one frame is shared by the synthesizing unit 4 and the gradation compression characteristic unit 5. Therefore, based on the analysis result in the image analysis unit 7, the synthesized image up to the (n−1) th frame used in the gradation compression characteristic generation unit 5 can be generated so that the gradation compression characteristic suitable for the image being shot can be generated. Determine the composition.

  For example, as shown in FIG. 4A, when the bright part is dominant, the drive of the image sensor 2 is controlled so that images are taken in ascending order of exposure amount at the time of the next composition image capturing. As a result, the n-2th post-combination images (post-combination images up to the (n-1) th frame) referred to by the gradation compression characteristic generation unit 5 are generated based on the low-exposure image. Thus, tone compression characteristics suitable for the bright part can be generated.

  Also, as shown in FIG. 4B, when the dark part is dominant, the drive of the image sensor 2 is controlled so that images with the highest exposure amount are photographed in order at the time of the next composition image photographing. As a result, the post-combination images up to the (n-2) th time (post-combination images up to the (n-1) th frame) referred to by the gradation compression characteristic generation unit 5 are generated based on the high exposure image. Therefore, tone compression characteristics suitable for dark areas can be generated.

  As described above, the change in the order of the exposure amounts of the n frames to be combined is at least the exposure amount of the image of the nth frame among the n frames becomes the maximum exposure amount or the minimum exposure amount. Thus, the order of exposure amounts may be switched. For example, when the synthesis target is three frames as in the present embodiment, whether the image of at least the third frame has the maximum exposure amount or the minimum exposure amount is analyzed by the image analysis unit 7. It may be determined based on the result.

  In the present embodiment, the image analysis unit 7 refers to the output image from the image sensor 2 before synthesis, but is not limited thereto. Image analysis may be performed with reference to an image in the middle of synthesis output from the synthesis unit 4. The analysis method by the image analysis unit 7 is the same as in the first embodiment.

  Next, an example of the imaging control of the third embodiment will be described with reference to the timing chart of FIG. Images with different exposure amounts are output from the image sensor 2 every 1V period, and the combination of the pre-combination images used to generate one composite image is surrounded by a broken line in FIG. Become. That is, the output images L1, M1, and H1 of the image sensor 2 are combined during the period from time t1 to t4, and the output images H2, M2, and L2 of the image sensor 2 are combined during the period from time t4 to t7. Of the combinations of images to be combined, an image with L is the smallest exposure, an image with H is the largest, and an image with M is , The exposure amount is intermediate between L and H.

  The control signal S <b> 1 is a binary control signal that controls writing of an image into the frame memory 3. When the control signal S1 is “0”, the writing of the image to the frame memory 3 is stopped, and when the control signal S1 is “1”, the output image from the selection unit 9 is 1 V over time. It is written in the frame memory 3.

  The identification signal I is an identification signal indicating how many frames of the n frames of images have been combined. In the case of combining three images, 0, 1, 2, 0, 1, 2,.・ ・ Changes periodically. The identification signal I is also a signal controlled by the system control unit 8. When the identification signal I is “0”, the selection unit 9 selects and outputs an image from the image sensor 2, and the image output from the image sensor 2 is written in the frame memory 3. When the identification signal I is “1”, the selection unit 9 selects and outputs an output image from the synthesis unit 4 (an image in the middle of synthesis), and the output image of the synthesis unit 4 is written in the frame memory 3. It is. When the identification signal I is “2”, the control signal S1 becomes “0”, and the writing of the image to the frame memory is stopped.

  The identification signal I is also referred to in the gradation compression characteristic generation unit 5. When the identification signal I is 1 (when the compositing target is n frames, when I is “n-2”), the gradation compression characteristic generation unit 5 generates a composite image (n−1th image) from the selection unit 9. (Composite image up to the first frame) is generated, a reduced image is generated, and stored in the memory 54. Therefore, when gradation compression is performed on the result of combining images of n frames having different exposure amounts, gradation compression characteristics are created by referring to the result of combining the images up to the (n-1) th frame. Become. In the case of combining three images as in the present embodiment, a gradation compression characteristic is generated from the result of combining up to the second image.

  The control signal S <b> 2 is a binary signal that controls reading of an image from the frame memory 3 and on / off of processing in the synthesis unit 4 and the gradation compression unit 6. When the control signal S2 is “1”, the synthesizing unit 4 reads the image written in the frame memory 3, and uses the image output from the image sensor 2 and the image read from the frame memory 3 to perform the synthesizing process. Do. Further, when the control signal S2 is “1”, the gradation compression unit 6 refers to the identification signal I described above, and the identification signal I is “n−1” in the n-sheet synthesis (in the case of the three-sheet synthesis, When I = “2”), gradation compression processing is performed on the output of the synthesis unit 4.

  The control signal S3 is a control signal that determines in what order the images for three frames with different exposure amounts to be combined are taken, and is determined based on the result of the image analysis unit 7. The control signal S3 becomes “0” when it is determined that the bright part is dominant in the image being shot, and the control signal S3 becomes “1” in the image being shot. This is a case where the dark part is determined to be dominant. When the control signal S3 is “0”, the image sensor 2 is controlled so that two frames with a small exposure amount among the three frames to be combined are captured first. Further, when the control signal S3 is “1”, the image sensor 2 is controlled so that, of the three frames to be combined, two frames having a large exposure amount are shot first.

  In FIG. 8, when the identification signal I is “0”, the image analysis unit 7 performs processing using the image output from the image sensor 2, but the identification signal I is “1”. Sometimes an image output from the image sensor 2 or an image in the middle of synthesis may be used.

  In FIG. 8, since the control signal S3 is “0” in the period from time t1 to time t4, the images for three frames to be combined are L1, M1, H1, and the image sensor in order of increasing exposure amount. 2 is output.

  In the period from time t1 to t2, the image L1 is written in the frame memory 3, and in the period from time t2 to t3, the synthesis unit 4 outputs the image L1 read from the frame memory 3 and the image sensor 2. The image M1 is synthesized. The combined image L1 + M1 is overwritten in the frame memory 3 and input to the gradation compression characteristic generation unit 5, and a low-resolution image is generated over a 1V period and held in the memory 54.

  During the period from time t3 to t4, the synthesizer 4 reads the image L1 + M1 being synthesized from the frame memory 3, synthesizes it with the image H1 output from the image sensor 2, and outputs the synthesized image L1 + M1 + H1. . The tone compression characteristic generation unit 5 uses the image L1 + M1 being synthesized read out from the frame memory 3 and the reduced image (held in the memory 54 before 1V) to use the tone compression characteristic g (L1 + M1). ) Is generated and output. The gradation compression unit 6 performs gradation compression on the combined image L1 + M1 + H1 output from the composition unit 4 using the gradation compression characteristic g (L1 + M1) from the gradation compression characteristic generation unit 5.

  In the period from time t1 to time t4, the image analysis unit 7 performs analysis using the image L1 first output from the image sensor 2 among the images of the plurality of frames to be combined, and the analysis result A (L1 ) Is output. Then, based on the analysis result A (L1), the control signal S3 referred to in the synthesis from time t4 is updated to “1”. As a result of analyzing the currently captured image, it can be determined that the dark portion has become dominant, and the composition processing from time t4 is to capture the high-exposure image first, and to improve the tone compression characteristics in the dark portion. This is because optimization is considered preferable.

  Since the control signal S3 is “1” during the period from time t4 to t7, the image for the three frames to be synthesized is image sensor 2 in order from the largest exposure amount, that is, in the order of H2, M2, and L2. Is output from.

  In the period from time t4 to t5, the image H2 is written in the frame memory 3, and in the period from time t5 to t6, the synthesizer 4 outputs the image H2 read from the frame memory 3 and the image sensor 2. The image M2 is synthesized. The combined image H2 + M2 is overwritten in the frame memory 3 and is input to the gradation compression characteristic generation unit 5, and a low-resolution image is generated over a 1V period and held in the memory 54.

  In the period from time t6 to t7, the synthesizer 4 synthesizes the image H2 + M2 read from the frame memory 3 and the image L2 output from the image sensor 2, and outputs a synthesized image H2 + M2 + L2. Further, the gradation compression characteristic generation unit 5 uses the image H2 + M2 being synthesized read out from the frame memory 3 and a reduced image thereof (stored in the memory 54 before 1V) to generate the gradation compression characteristic g (H2 + M2). ) Is generated and output. The gradation compression unit 6 performs gradation compression on the combined image H2 + M2 + L2 output from the combination unit 4 using the gradation compression characteristic g (L1 + M1) from the gradation compression characteristic generation unit 5.

  In the period from time t4 to time t7, the image analysis unit 7 performs an analysis process using the image H2 first output from the image sensor 2 among the images of a plurality of frames to be combined, and the analysis result A ( H2) is output. Based on A (H2), the system control unit 8 updates the control signal S3 referred to in the synthesis from time t7. By performing a series of imaging control as described above, multiple images with different exposure amounts are combined, and gradation compression is performed with gradation compression characteristics suitable for the combined image, and the video signal is converted. Can be output.

  As described above, even when an image of n frames is selected as a synthesis target, tone compression characteristics for the synthesized image are generated using the same image as the image buffered in the frame memory 3 for the synthesis process. be able to. Therefore, an increase in system load due to memory access can be avoided. In addition, by analyzing the characteristics of the image being shot and determining the image used to generate the tone compression characteristics, tone compression processing suitable for the image being shot can be performed.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (8)

An imaging apparatus that generates an image with an expanded dynamic range by combining two images having different exposure amounts,
Imaging means for outputting a first image and a second image following the first image by photographing in which two different exposure amounts are sequentially applied;
Synthesizing means for synthesizing the first image and the second image to generate a synthesized image having an extended dynamic range;
Generating means for generating gradation compression characteristics based on the first image and an image obtained by reducing the resolution of the first image;
Gradation compression means for compressing the gradation of the composite image using the gradation compression characteristic generated by the generation means;
Analyzing means for analyzing the first image in order to determine which of the two exposure amounts is appropriate as an exposure amount of the image used for generating the gradation compression characteristic;
Control means for controlling correspondence between the first and second images and the two exposure amounts so that the generation unit uses an image taken with an exposure amount determined to be appropriate by the analysis unit. An imaging apparatus characterized by that. A holding means for holding the first image in a memory;
The imaging apparatus according to claim 1, wherein the generation unit generates the reduced-resolution image in parallel with the first image being held in the memory. The imaging means continuously outputs images obtained by imaging by alternately applying the two exposure amounts,
It said control means, the image pickup apparatus according to claim 1 or 2, characterized in that switching the order of application of the two exposure amount in the imaging means. The imaging means continuously outputs images obtained by imaging by alternately applying the two exposure amounts,
Wherein, imaging according to claim 1 or 2, characterized in that shifting one image to extract image used as the first and second images from images continuously output from the imaging means apparatus. The analysis unit analyzes whether the bright part is dominant or the dark part is dominant in the first image, and determines which of the two exposure amounts is appropriate based on a result of the analysis. The imaging apparatus according to any one of claims 1 to 4 . The analysis means determines that the image with the smaller exposure amount is appropriate if the bright portion is dominant, and determines that the image with the larger exposure amount is appropriate if the dark portion is dominant. The imaging apparatus according to claim 5 . An imaging apparatus that generates an image with an expanded dynamic range by combining n images (n is a natural number of 3 or more) with different exposure amounts,
imaging means for continuously outputting images obtained by sequentially applying and shooting n different exposure amounts;
Synthesis means for sequentially synthesizing n images continuously output from the imaging means to obtain a synthesized image with an extended dynamic range;
Generating means for generating a gradation compression characteristic based on an intermediate composite image obtained at the stage before the (n−1) th from the synthesis means, and an image obtained by reducing the resolution of the intermediate composite image;
A gradation compression means for compressing the gradation of the composite image obtained by the composition means using the gradation compression characteristic generated by the generation means;
It is determined whether the nth image among the n images is an image to which the maximum exposure amount among the n exposure amounts is applied or an image to which the minimum exposure amount is applied. Therefore, analysis means for analyzing the first frame image of the n frames or the intermediate synthesized image output by the synthesizing means,
An imaging apparatus comprising: control means for controlling correspondence between the n images and the n exposure amounts based on the determination by the analysis means . A holding unit that holds the first image of the n images to be combined output from the imaging unit or the intermediate combined image output from the combining unit in a memory;
The synthesizing unit synthesizes an image currently output from the imaging unit and an image held in the memory, and outputs the synthesized image to the holding unit.
The imaging apparatus according to claim 7 , wherein the generation unit generates an image obtained by reducing the resolution of the intermediate composite image in parallel with the intermediate composite image being held in the memory.

Images