JP2013225724A - Imaging device, control method therefor, program, and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To realize high dynamic range composition capable of suppressing noise and deterioration in color reproduction.SOLUTION: An imaging device for combining a plurality of images captured while changing exposure conditions comprises: imaging means that captures a plurality of images while changing exposure conditions; image composition means that combines the plurality of images with the different exposure conditions to generate a composite image; and control means that, when the imaging means captures a first image, controls exposure so as to make ISO sensitivity lower than in the case of capturing a second image with a brighter exposure condition than that of the first image.

Description

  The present invention relates to a technique for generating an image having a wide dynamic range by combining a plurality of images shot with different exposures.

  There is a method of generating an image having a wide dynamic range by photographing a plurality of images with few whiteouts and images with little blackout under different exposure conditions. This is called high dynamic range (HDR) synthesis.

  Here, for example, a case where three images having different exposure amounts are combined will be described.

  By performing weighted addition for each pixel in accordance with the composition ratio corresponding to the luminance level shown in FIG. 8 with reference to the luminance level of an intermediate brightness image (appropriate image) among three images having different exposure amounts. One HDR composite image is generated. In HDR synthesis, an image after gamma processing described later is used.

  In FIG. 8, 801 is the brightest image (over image) of the three images, 802 is the appropriate image, and 803 is the composition ratio of the darkest image (under image). For example, the composition ratio corresponding to the pixel value of the luminance level 255 count is 100% for the under image 803, 0% for the other images, and 100% of the pixel value of the under image 803 is used. The composition ratio corresponding to the pixel value of the luminance level 175 count is 50% for the under image 803, 50% for the appropriate image 802, and 0% for the under image 801, and the pixel value of the under image is 50%. Pixel values are used at a rate of 50%.

  Next, consider a case where gamma conversion processing is applied to three images having different exposure amounts by applying the gamma curve shown in FIG.

  In the gamma curve of FIG. 9A, the output value B (8 bits) is a value expressed as appropriate brightness when a uniform luminance surface of 18% gray is photographed, and the input value at this time is A (12 Bit). FIG. 9B is a graph in which the horizontal axis represents the number of exposure levels with respect to FIG. 9A, and 0 on the horizontal axis corresponds to the input value A which is the appropriate brightness. Further, 1 on the horizontal axis corresponds to A × 2 brighter by A, and −1 corresponds to A / 2 darker by A.

  When three images having different exposure amounts subjected to gamma conversion processing with such a gamma curve are synthesized according to the luminance level by the above-described method, for example, as shown by 1001 in FIG. The output result is equivalent to the curve. In FIG. 10, 1002 indicates an over image, 1003 indicates a proper image, and 1004 indicates an under image.

  On the other hand, in order to maintain the gradation connection, different gamma curves as shown in FIG. 11A are applied to the three images having different exposure amounts. In FIG. 11B, the horizontal axis is represented by the number of stages in the same manner as FIG. 10 described above with reference to FIG. In FIG. 11B, curves 1103 to 1104 represent over images, curves 1104 to 1105 represent appropriate images, and 1105 to 1106 represent gamma curves of the under image. As shown in FIG. 11B, by connecting the respective gamma curves when the number of horizontal axis stages is displayed, the connection of the gradation of the image after HDR synthesis can be made smooth.

  For example, Patent Document 1 describes a method of performing multiple shootings with different exposure amounts by fixing the ISO sensitivity of an imaging apparatus and changing the aperture and shutter speed of a lens.

JP 2003-259200 A

  However, the HDR synthesis disclosed in Patent Document 1 has the following problems.

  When the gamma conversion processing is performed using the gamma curve illustrated in FIG. 11, among the three images with different exposures, a darker image has a higher digital gain. The feeling is different. When HDR synthesis is performed in this state, an image with a different noise feeling depending on the brightness is obtained.

  In addition, in HDR synthesis, if a moving subject exists in an image, a composite image having different luminances according to the position of the moving subject is generated as shown in FIG. 12A to 12C are images obtained by performing gamma conversion processing on the under image, the appropriate image, and the over image, respectively, using the gamma curve of FIG. In FIG. 12, reference numeral 1201 denotes a moving subject having partially different luminances. When three images are synthesized according to the brightness by the above-described method, the same subject as shown in FIG. Becomes an image synthesized at partially different positions.

  In order to eliminate such an inconvenience, a method of setting the under image usage ratio to 100% for pixels having a difference in brightness in the image after gamma processing apart from the synthesis processing according to the brightness can be considered. . According to this method, as a result of combining FIGS. 12A to 12C, the darkest image is used for the moving subject as shown in FIG. 12E, and the same subject is partially combined at different positions. Can be prevented.

  However, when such a method is used, the areas 1202 and 1203 in FIG. 12 are subjected to gamma processing and digital gain is applied to the under image, so that only that area becomes a noisy image. In addition, since a signal brightened with a digital gain is used for a dark image, when the linearity of the imaging element is poor, colors may not be correctly expressed due to color bending or the like.

  In particular, when this method is used in Patent Document 1, a very noisy image is obtained particularly at high sensitivity.

  The present invention has been made in view of the above problems, and an object thereof is to realize high dynamic range synthesis that can suppress deterioration of noise and color reproduction.

  In order to solve the above-described problems and achieve the object, the present invention provides an image pickup unit that picks up a plurality of images by changing the exposure conditions in an image pickup apparatus that combines a plurality of images picked up by changing the exposure conditions. And an image composition unit that generates a composite image by combining a plurality of images having different exposure conditions, and when the imaging unit captures the first image, the exposure condition is brighter than that of the first image. Control means for controlling the exposure so that the ISO sensitivity is lowered as compared with the case of capturing the second image.

  According to the present invention, it is possible to perform high dynamic range synthesis while suppressing noise and deterioration of color reproduction.

1 is a block diagram illustrating a configuration of an imaging apparatus according to an embodiment. The flowchart which shows the HDR synthetic | combination process of this embodiment. The figure which shows the program diagram for HDR imaging | photography. The block diagram which shows the structure of the image process part of this embodiment. The figure which illustrates a position shift detection result. The block diagram which shows the structure of the mobile body area | region detection part of this embodiment. The figure which illustrates the synthetic result of the moving subject of this embodiment. The figure which illustrates an HDR synthetic | combination ratio. The figure which illustrates the gamma curve of normal imaging | photography. The figure which shows the example which applied the gamma curve of normal imaging | photography to HDR synthetic | combination. The figure which illustrates the gamma curve for HDR composition. The figure which illustrates the HDR synthetic result of a moving subject.

  Hereinafter, embodiments for carrying out the present invention will be described in detail. The embodiment described below is an example for realizing the present invention, and should be appropriately modified or changed according to the configuration and various conditions of the apparatus to which the present invention is applied. It is not limited to the embodiment. Moreover, you may comprise combining suitably one part of each embodiment mentioned later.

  [Embodiment 1] An embodiment in which the present invention is applied to an image pickup apparatus such as a digital camera for taking an image will be described below. Since the present invention can be applied to an image composed of a plurality of color planes photographed through a photographing lens, the image targeted by the present invention is limited to RAW data, JPEG data after development processing, and the like. Is not to be done.

  <Apparatus Configuration> With reference to FIGS. 1 to 7, an outline of a configuration and functions of an imaging apparatus according to an embodiment of the present invention will be described.

  In FIG. 1, reference numeral 100 denotes a photographing lens that forms a subject image L on an image sensor, 101 denotes an aperture for adjusting the amount of light, 102 denotes a shutter, and 103 denotes an image sensor that converts the subject image into an electrical signal. Reference numeral 104 denotes an A / D converter that converts an analog signal output from the image sensor 103 into digital data. The timing generation unit 105 is controlled by the memory control unit 108 and the system control unit 130 to supply a clock signal and a control signal to the image sensor 103, the A / D converter 104, and the D / A converter 106. Control the operation timing.

  The image processing unit 113 performs image processing on the data from the A / D converter 104 or the data from the memory control unit 108. The image processing includes white balance processing, color interpolation processing for converting a signal of a well-known RGB Bayer array into an RGB 3-plane signal, gamma correction processing, saturation correction, hue correction, and the like. Further, as will be described later, the image processing unit 113 also performs a process of creating an HDR composite image by combining a plurality of captured images.

  The memory control unit 108 controls the A / D converter 104, the timing generation unit 105, the image processing unit 113, the image display memory 109, the D / A converter 106, the memory 110, and the compression / decompression unit 111, respectively. Thereby, the digital data A / D converted by the A / D converter 104 is sent to the image display memory 109 or the memory 110 via the image processing unit 113 and the memory control unit 108 or directly via the memory control unit 108. Is written to.

  The display memory 109 stores data to be displayed on the first display unit 107, and the data stored in the display memory 109 is transmitted to the first display unit such as a TFT / LCD via the D / A converter 106. It is output to 107 and displayed. The first display unit displays various menus for controlling the camera such as a white balance selection menu. These menus are displayed and selected by operating the first operation unit 124.

  The memory 110 has a storage capacity sufficient to store a predetermined number of still images in order to store captured still images. The memory 110 can also be used as a work area for the system control unit 130 and the image processing unit 113. A compression / decompression unit 111 compresses and decompresses image data. The compression / decompression unit 111 reads image data stored in the memory 110 and performs compression processing, or reads compressed image data and performs decompression processing, and writes the processed data to the memory 110. Can do. The external storage device 112 is a detachable external recording medium such as a CF card or an SD card. The image data temporarily recorded in the memory 110 is finally recorded in the external storage device 112.

  The photometric sensor 119 measures the luminance of each pixel associated with the imaging screen in a conjugate manner. When an appropriate exposure amount is calculated by the system control unit 130 according to the output of the photometric sensor 119, the exposure control unit 114 detects the ISO sensitivity of the aperture 101, the shutter 102, and the image sensor 103 according to the exposure amount. To control.

  The distance measuring sensor 118 detects distance information of a distance measuring point arbitrarily selected by the user. The distance measurement control unit 115 controls the focusing of the lens 100 by the output of the distance measurement sensor 118.

  The zoom control unit 116 detects a zooming amount (focal length) of the lens 100 manually performed. Alternatively, the zoom amount of the lens is controlled when zooming is automatically performed using the second operation unit 125 of the camera.

  The automatic flash unit 126 also has an AF auxiliary light projecting function and a flash light control function.

  The angular velocity sensor 117 is a sensor that detects camera shake in the horizontal / vertical direction, and is used for known camera shake correction processing, vertical position shooting, and determination of horizontal position shooting.

  The system control unit 130 controls the operation of the entire imaging apparatus. The memory 120 stores constants for operating the system control unit 130, variable programs, image processing parameters, and the like, and is also used as a work memory. The memory 120 also records a program diagram for automatic exposure control for HDR photography, which will be described later.

  The second display unit 121 includes an LCD panel unit, a speaker, and the like that display an operation state, a message, and the like using characters, images, sounds, and the like according to execution of the program by the system control unit 130. The display content of the second display unit 121 includes single shot / continuous shooting display, self-timer display, compression rate display, recording pixel number display, recording number display, remaining recordable number display, shutter speed display, aperture value display, etc. , Exposure compensation display, battery remaining amount display, error display, external recording memory 112 attachment / detachment state display, and the like.

  The nonvolatile memory 122 is an electrically erasable and recordable memory. For example, an EEPROM or the like is used. This is an operation unit for inputting various operation instructions of the release button 123, the second operation unit 125, and the system control unit 130. The release button 123 includes a two-stage switch, SW1 and SW2. SW1 is turned on by the first stroke of the release button 123, and serves as a switch for starting photometry and distance measurement. SW2 is turned on by the second stroke of the release button 123 and serves as a switch for starting an exposure operation. The second operation unit 125 is used for single shot / continuous shooting / self-timer switching, manual focus / autofocus switching, shutter speed, aperture value, exposure correction setting, and the like.

  The power switch 127 is a switch for turning on / off the main power supply of the imaging apparatus. The power control unit 128 includes a battery detection unit, a DC-DC converter, a switch unit that switches a block to be energized, and the like. The power supply control unit 128 detects the presence / absence of a battery, the type of battery, the remaining battery level, controls the DC-DC converter based on the detection result and the support of the system control unit 130, and requires the necessary voltage. It is supplied to each part including the recording medium for a period. The power source 129 includes a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li battery, or an AC adapter.

  [HDR Compositing Process] The HDR synthesizing process of this embodiment will be described with reference to FIG.

  In the following, a process will be described in which three images are taken under different exposure conditions, and one HDR composite image is created by combining these images. 2 is realized by the system control unit 130 expanding and executing the control program stored in the ROM 122 in the work area of the RAM 120.

  In FIG. 2, in step S <b> 201, the system control unit 130 performs a photometric operation by the photometric sensor 119 when the SW <b> 1 of the release button 123 is pressed. By this photometric operation, the BV value of the subject is calculated. The system control unit 130 uses the calculated BV value of the subject to determine three different exposure conditions for HDR shooting according to the HDR shooting program diagram (FIG. 3) recorded in the memory 120. To do. As shown in FIG. 3, in the present embodiment, shooting is performed with an exposure difference of ± 2 steps with respect to intermediate exposure (appropriate exposure) as shooting conditions for three images for HDR shooting. The exposure difference condition is not limited to this, and may be ± 1, ± 3 or more. Further, it may be ± 1/3 step instead of one step. Note that the program diagram differs for each exposure difference.

  Here, the program diagram of FIG. 3 will be described in detail.

  FIG. 3A shows the relationship between the shutter speed (y axis) and the BV value (x axis). Reference numeral 301 in FIG. 3A indicates the shutter speed of intermediate exposure (appropriate exposure) among the three exposure conditions. Similarly, reference numeral 302 denotes a dark side exposure (under exposure), and 303 denotes a shutter time at a bright side exposure (over exposure).

  FIG. 3B shows the relationship of the lens aperture (y-axis) to the BV value (x-axis). Reference numeral 304 in FIG. 3B indicates that the aperture condition is common to the appropriate, under, and over exposure conditions.

  FIG. 3C shows the relationship of ISO sensitivity (y axis) to BV value (x axis). In FIG. 3C, reference numeral 305 denotes a proper and over ISO sensitivity condition, and reference numeral 306 denotes an under ISO sensitivity condition.

  According to the program diagram of FIG. 3, when the BV value is 2 or less, the shutter speed conditions for proper and under shooting are equal, and the ISO sensitivity condition is set lower by two steps for under shooting than for proper shooting. The conditions for overshooting are the same as for proper shooting.

  When the BV value is 4 or more, the ISO sensitivity is the same for both proper, under, and over. Under shutter time, under shooting is set at two steps faster than proper shooting, and over shooting is two steps higher than proper shooting. Set for slow seconds.

  When the BV value is 2 to 4, the control is set so that the control of the BV value 2 or less and the control of 4 or more are connected by linear interpolation.

  Note that the aperture is set to be common for all the BV values, appropriate, under and over. This is because the background blur changes depending on the aperture, and if the aperture is changed, an HDR composite image described later becomes an image with a sense of incongruity.

  According to this program diagram, the ISO sensitivity of the under image is set to a relatively low ISO sensitivity when the ISO sensitivity of the over image is higher than ISO 100, which is the minimum ISO sensitivity that can be set in this embodiment. To create. Thereby, control is performed so that the noise of the under image is relatively reduced.

  Also, under the condition that the ISO sensitivity of the under image is ISO 100, which is the minimum ISO sensitivity that can be set in this embodiment, the under image is controlled by setting the shooting time to the high speed side.

  Note that noise is increased if the ISO sensitivity is set higher than the appropriate image when shooting the over image. Therefore, by setting the same ISO sensitivity as the appropriate image and setting the shutter time longer than the appropriate exposure, the over image Have control to get.

  In this embodiment, the exposure condition is automatically determined from the BV value of the subject. However, the user may set the shooting condition manually. In this case, it is assumed that the user can set only a photographing condition with appropriate exposure and an exposure difference between three images for HDR composition.

  Even in the case of manual setting, the under image is realized by lowering the ISO sensitivity relatively by the amount of the exposure difference condition set for the appropriate image, as in the control shown by the program diagram of FIG. However, as in the program diagram of FIG. 3, when the ISO sensitivity of the under image is ISO 100, which is the lower limit value of the ISO sensitivity that can be set in the imaging apparatus of the present embodiment, the shutter speed is set to the high speed side. do it. Similar to the program diagram of FIG. 3, the over image is also realized by setting the shutter time longer than the appropriate image.

  The exposure conditions are determined as described above (S201). Note that the first image, the second image, and the third image of the present invention correspond to the under image, the appropriate image, and the over image of the present embodiment, respectively.

  Returning to FIG. 2, in step S202, when the SW2 of the release button 123 is pressed, the system control unit 130 captures an under image under the under image capturing conditions determined in step S201. The captured image is converted to digital data by the A / D converter 104 and temporarily held in the memory 110 as a 12-bit RAW image.

  In step S203, the system control unit 130 shoots an appropriate image for HDR composition under the shooting conditions determined in step S201 in accordance with SW2 of the release button 123 pressed in step S202. The appropriate image is also temporarily held in the memory 110 as a 12-bit RAW image, like the under image in step S202.

  Similarly, in step S204, an HDR over-image is shot under the shooting conditions determined in step S201, and is stored in the memory 110.

  In step S <b> 205, the system control unit 130 synthesizes three images for HDR synthesis held in the memory 110.

  Here, the operation from step S206 to S215 will be described with reference to the configuration of the image processing unit 113 in FIG.

  In step S <b> 206, the development processing unit 4010 performs development processing for the under image recorded in the memory 110. In the development processing unit 4010, first, white balance processing is performed in the white balance processing unit 4011. Specifically, a gain is applied to each of R, G, and B so that R, G, and B of the region that should be white in the image have the same signal value. Next, a noise reduction processing unit 4012 reduces noise caused by the sensor that is not derived from the subject image of the input image. A subsequent color interpolation unit 4013 generates a color image in which the color information of the R, G, and B3 planes is aligned in all pixels by interpolating the color mosaic image. A basic color image is generated from the generated color image by the matrix conversion unit 4014 and the gamma conversion unit 4015. Thereafter, the color adjustment unit 4016 performs correction processing of the saturation, hue, and brightness of the color image.

  In the gamma conversion unit 4015, the gamma curve indicated by 1103 in FIG.

  Similarly, in steps S207 and S208, appropriate images and over-images are developed. Here, only the gamma curve of the gamma conversion unit 4015 is different, but the other processes are the same as those for the under image. As for the gamma curve, 1102 in FIG. 11A is applied to the appropriate image, and 1101 is applied to the over image.

  With the image developed according to the gamma curve as described above, an HDR composite image with smooth gradation can be created as described above.

  Note that, as described above, the gamma curve is a curve that requires more gain for an under image, and therefore, the noise increases for an under image after development processing. In particular, the higher the sensitivity, the more conspicuous the effect of image quality degradation. However, in the present embodiment, when the exposure condition is determined in step S201, the ISO sensitivity at the time of under image shooting is set lower than the ISO sensitivity at the time of shooting an appropriate image or over image, so the difference in noise feeling with respect to the proper image is It is suppressed.

  Next, in steps S209 to S212, the misregistration detection unit 4040 and the misregistration correction unit 4030 align each image that has undergone development processing.

  Here, details of the alignment processing will be described.

  First, an image is divided into a plurality of block areas, and edge detection processing is performed for each block. As an edge detection method, a low frequency image is created by applying a low-pass filter to the input image, and an edge is detected by subtracting the low frequency image from the input image. Alternatively, a method using a known differential filter, a pre-wit filter, or the like can be considered. In edge detection, it is possible to improve the positional deviation detection accuracy by detecting only the edge of the subject image, not the noise caused by the sensor.

  Next, the shift amount detection is performed on the block area in which the edge is detected in the block area of the image. By detecting the shift amount only for the block area where the edge can be detected, the positional shift detection accuracy can be improved. As a displacement amount detection method, a sum of absolute values (Sum of Absolute Difference) of the difference between the pixel value (luminance) of the position reference image of all the pixels in the block region and the pixel value (luminance) of the displacement detection image is calculated. There is a method in which the movement amount and the movement direction that minimize the total sum are used as the motion vector of the block. FIG. 5 illustrates a deviation amount detection method. FIG. 5A shows the pixel value of the target block area of the position reference image.

  On the other hand, FIG. 5B shows the pixel value of the misregistration detection image. In the case of FIG. 5, the movement amount with which the absolute value of the difference between the pixel values becomes the smallest is obtained as (x, y) = (1, 2). Similar processing is performed on all block areas of the image to obtain motion vectors of all block areas. In displacement amount detection, it is possible to improve the displacement detection accuracy when the same subject area of the position reference image and the displacement detection image has the same brightness.

  Finally, a geometric transformation coefficient is calculated. An affine coefficient is used as the geometric transformation coefficient. The affine coefficient is a matrix used for affine transformation combining linear transformation and parallel movement, and is expressed by the following Expression 1.

  Here, the coordinates of the image before correction are (x, y), the coordinates of the image after correction are (x ′, y ′), and the 3 × 3 matrix is called an affine coefficient. An affine coefficient is obtained using a motion vector obtained from each block region.

  The above is the positional deviation detection method in the positional deviation detection unit 4020, but the present invention is not limited to this. There are various other misalignment detection methods, such as detecting the misalignment between two images from frequency analysis.

  Returning to the description of FIG. 2, in steps S211 to S212, the misalignment correction unit 4030 corrects (affine transform) the appropriate image and the over image based on the affine coefficient calculated in the misalignment detection process.

  In steps S213 to S214, the moving body region detection unit 4040 detects the moving body region with respect to the under image, the appropriate image after alignment, and the over image. FIG. 6 shows the configuration of the moving object region detection unit 4040. In FIG. 6, the reference image 4041 is an under image, and is a scene as shown in FIG. On the other hand, the aligned image 4042 is a properly aligned image or an over-aligned image, and is a scene moved slightly to the right with respect to (a) as shown in FIG. 7B. The moving object detection unit 4043 detects a moving object region from the reference image 4041 and the aligned image 4042. There are several methods for detecting the moving object region. For example, a method for obtaining a difference between two images is conceivable. The difference Diff is calculated by the following equation 2 using the color and luminance signals.

  Here, Y represents a luminance signal, and U and V represent color signals. Therefore, the difference Diff means a color difference.

  Next, the overexposure blackout area exclusion unit 4044 excludes the underexposure or appropriate image overexposure brightness areas and the overexposure overexposure areas overout of the difference Diff. This is performed for the purpose of preventing an overexposure or blackout area from appearing as a difference in the moving object detection unit 4043 and being erroneously determined as a moving object area. The signal area with the blackout brightness th1 or less from the reference image 4041 and the signal area with the whiteout brightness th2 or more in the aligned image 4042 are excluded from the difference Diff (the signal value is set to 0). The result is as shown in FIG.

  Thereafter, the process proceeds to the process of correcting the moving object region while referring to the result of the difference Diff. Prior to that, an isolated region such as a minimal moving object or a non-moving object in the moving object is detected as an isolated region due to erroneous detection from the difference Diff. By removing by the removing unit 4045, the boundary of the moving body region can be rendered smooth and natural. There are several methods for removing the isolated region. For example, consider reducing and expanding the difference Diff. A small area erroneously determined as a moving body at the stage of reduction can be removed, and a non-moving body area existing inside the moving body area can be contracted at the stage of enlargement to the original size.

  The moving body region extracted by the above moving body region detection process is as shown in FIG. In the image signal, the non-moving body area is represented as 0, and the moving body area is represented as 255 (in the case of 8 bits). The boundary between the moving body region and the non-moving body region is represented by 1 to 244.

  In step S215, the moving body region combining unit 4050 combines the under image and the appropriate image moving body region, and the under image and the over image moving body region output from the moving body region detection unit 4040. Specifically, a moving object detection image of an under image and an appropriate image, and a moving object detection image of an under image and an over image are selected to a large value. Since the moving object area is represented by one or more signals, an image in which all moving object areas are detected can be obtained by selecting a large value.

  In step S <b> 216, the image composition unit 4060 separately performs luminance-specific composition processing using the under image, the appropriate image that has been aligned, and the over image, and generates an HDR composite image. Specifically, as described in the “Background Art” section, the under image, the appropriate image, and the over image are combined according to the combining ratio shown in FIG. 8 using the luminance level of the appropriate image as the reference luminance. As described in the column “Problems to be Solved by the Invention”, the image generated by the synthesis process is not correctly synthesized for the moving object region.

  Therefore, in step S217, the moving object processing unit 4070 corrects the moving object region in the HDR composite image while referring to the moving object detection image. Specifically, the moving body region calculated in step S215 is replaced with an under image for the HDR composite image created in step S216.

  Here, if the moving object region is simply replaced with an under image, an image in a dark region of the under image is used when the moving object is dark. Particularly in the dark area of the under image, a large gain is applied by the gamma conversion process. Therefore, not only the noise increases but the linearity of the signal of the image sensor 103 cannot be sufficiently obtained, and the color of the subject is appropriate due to color covering. May not be reproduced.

  Therefore, when the under image development processing in step S206 is performed, the noise reduction processing unit 4012 performs a noise reduction process that is relatively stronger than the appropriate image / over image, or the color adjustment unit 4016 performs saturation saturation of the dark portion. The process which lowers is performed.

  Further, if only the saturation of the under image is reduced, the saturation of only the moving body portion is reduced. Therefore, the processing for reducing the saturation similarly in the appropriate image and over image development processing in steps S207 and S208. It is good also as a structure which gives.

  Finally, in step S218, after the HDR composite image is held in the memory 110, the compression / decompression unit 111 performs JPEG compression processing, records it in the external recording device 112, and the series of processing ends.

  According to the present embodiment, an increase in the noise of the under image due to the gamma conversion process can be suppressed by photographing the ISO sensitivity of the under image relatively lower than that of the appropriate image. In addition, by performing the process of reducing the saturation on the dark part of the under image, it is possible to suppress the influence of the color curve of the moving body region.

  [Other Embodiments] The present invention is also realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads and executes the program code. It is processing to do. In this case, the program and the storage medium storing the program constitute the present invention.

Claims (11)

In an imaging device that synthesizes multiple images that are captured under different exposure conditions,
An imaging means for capturing a plurality of images under different exposure conditions;
Image combining means for generating a combined image by combining a plurality of images having different exposure conditions;
Control means for controlling the exposure so that the ISO sensitivity is lowered when the image pickup means picks up the first image, compared with the case of picking up the second image with a lighter exposure condition than the first image; An imaging apparatus comprising:   The control means changes the exposure condition by changing the ISO sensitivity without changing the shutter speed and the aperture value between the first image and the second image, and sends the first image to the imaging means. The image pickup apparatus according to claim 1, wherein the image and the second image are picked up.   The control unit causes the imaging unit to capture a third image with an exposure condition brighter than that of the second image, and for imaging under the exposure condition of the third image, the shutter time is set to other exposure. The imaging apparatus according to claim 1, wherein the exposure is controlled by setting the exposure time longer than the condition.   The control unit causes the imaging unit to capture a third image with a lighter exposure condition than the second image, and an ISO sensitivity and an aperture value between the second image and the third image. 3. The imaging apparatus according to claim 1, wherein an exposure condition is changed by changing a shutter time without changing the shutter speed, and the imaging unit is caused to capture the second image and the third image. 4.   5. The control unit according to claim 1, wherein when the ISO sensitivity reaches a lower limit value that can be set by the imaging apparatus, the control unit controls exposure by setting a shutter speed to a high speed side. The imaging device as described in any one.   The imaging apparatus according to claim 1, wherein noise reduction processing for the first image is made stronger than an image captured under other exposure conditions. The image composition means includes
Developing means for developing the plurality of images;
A misregistration detecting means for detecting misregistration of the developed plurality of images;
Correction means for correcting the detected displacement;
Moving body region detecting means for detecting a moving body region from the plurality of images;
Moving body region combining means for combining the detected moving body region,
The developing means performs a different gamma conversion process on each image in order to align the luminance levels of the plurality of images,
The image combining means replaces a moving body region of a composite image generated by combining the plurality of images with an image captured under the darkest exposure condition among the plurality of images. The imaging device according to any one of 1 to 6.   The image synthesizing unit replaces the moving body area of the synthesized image generated by synthesizing the plurality of images with an image captured under the darkest exposure condition among the plurality of images, and the darkest exposure is performed. The image pickup apparatus according to claim 7, wherein the saturation of an image picked up under conditions is lower than that of an image picked up under other exposure conditions. A method for controlling an image pickup apparatus that combines a plurality of images picked up by changing exposure conditions,
An imaging step of capturing a plurality of images under different exposure conditions;
An image combining step of generating a combined image by combining a plurality of images having different exposure conditions;
When capturing the first image in the imaging step, a control step for controlling the exposure so that the ISO sensitivity is lower than when capturing the second image with a brighter exposure condition than the first image; The control method characterized by having.   A program for causing a computer to function as each unit of the imaging apparatus according to any one of claims 1 to 8.   A computer-readable storage medium storing a program for causing a computer to function as each unit of the imaging apparatus according to any one of claims 1 to 8.