JP6541501B2 - IMAGE PROCESSING APPARATUS, IMAGING APPARATUS, AND IMAGE PROCESSING METHOD - Google Patents

Description

  The present invention relates to an image processing apparatus that performs image processing on a captured image, an imaging apparatus that captures an image, an image processing method, and a program.

  In recent years, many imaging devices such as digital cameras capable of combining a plurality of images and recording the combined image have been commercialized. Among these imaging devices, an image captured with a shorter exposure time (referred to as a low exposure image) and a longer exposure time than the appropriate exposure time for capturing an exposed image in which the brightness of the subject is appropriate There is an apparatus provided with a function of combining captured images (high exposure images). This function is called a high dynamic range (HDR) function, and by combining a low exposure image and a high exposure image, it is possible to improve so-called whiteout and dark part noise. Hereinafter, the composite image by HDR is described as an HDR image.

  In general, the dynamic range of the imaging device is narrower than the dynamic range of brightness in the natural world, and for example, when shooting in a backlit scene, overexposure may occur. On the other hand, when shooting with an exposure time shorter than the proper exposure time, the entire image is captured dark, so it is easy to suppress overexposure, but the noise in the dark part increases and the image quality decreases . On the other hand, when shooting is performed with an exposure time longer than the proper exposure time, since the entire image is captured brightly, the noise in the dark part is improved, but overexposure easily occurs. On the other hand, according to the HDR, weighting is applied to a low-exposure image in a bright area (an area where whiteout or the like occurs), and conversely, in a dark area, a weight is applied to a high-exposure image. This makes it possible to suppress overexposure and improve the noise in the dark part.

  Further, in Patent Document 1, a luminance level of a pixel or the like in a plurality of captured images is obtained as an evaluation value, and one of the captured image and the HDR image is selected based on the evaluation value of the luminance level. Such techniques are disclosed. According to the technique described in this patent document 1, it is possible to improve the image quality of the recorded image by avoiding the unnaturalness and false contour of the image while obtaining the image with little whiteout and black crushing by the composition by the HDR. It has been made.

  Furthermore, when an image of a subject or the like is photographed, the user often confirms, for example, the degree of appearance of the photographed image immediately after photographing, and often confirms the effect of the synthesis also on the HDR image. In general, confirmation of a photographed image or an HDR image is performed by, for example, confirmation by thumbnail display or preview display. However, in particular, since the HDR image is generated by combining processing of a plurality of images such as a low exposure image and a high exposure image as described above, it is very important that the processing is completed and displayed. It will take a long time. In particular, when an HDR image is generated and displayed immediately after shooting, the user has to wait for a long time before it becomes possible to confirm the effect of image composition by HDR. As a solution in such a case, a reduced image is generated from each of a plurality of photographed images, and the processing time is shortened by combining the plurality of reduced images to create an HDR image for display. A similar method is conceivable.

JP, 2013-93786, A

  By the way, when confirming the effect of the image composition by HDR as mentioned above, not only the whole of the image but confirmation of details is often performed, and it is necessary to enlarge a display image for confirmation of details. is there. However, when the display HDR image generated from the reduced image is enlarged and displayed as described above, the enlarged image is rough and can not confirm details because the original image size is small. On the other hand, if it is an HDR image generated from an image that has not been reduced, detailed confirmation by enlarged display is possible, but as described above, it takes a long processing time to combine the HDR image from the image that has not been reduced It will be necessary.

  In the case of the technique described in the above-mentioned Patent Document 1, if it is determined that the effect of the combining process by HDR is small or the unnaturalness is noticeable when combining process based on the evaluation value of the luminance level, shooting Since one image is selected, the display image can be generated in a short time. However, when an HDR image is selected, a long processing time is required as described above. For example, although it is conceivable to generate an HDR image for display from a reduced image in order to shorten the processing time, in this case it is not possible to cope with the detail confirmation by the enlarged display as described above.

  The present invention has been made in view of such problems, and it is an image processing apparatus that enables confirmation of details by enlarged display without requiring a long processing time when the user confirms the effect of composition by HDR. It is an object of the present invention to provide an imaging device, an image processing method, and a program.

  The image processing apparatus according to the present invention comprises: reduction means for reducing a plurality of photographed images photographed at different exposure values for high dynamic range synthesis to generate a plurality of reduced images; and at least the brightness of the plurality of photographed images And combining the plurality of reduced images based on the combined evaluation value, and generating means for generating a combined evaluation value representing a combination ratio of pixels when combining the plurality of reduced images in the high dynamic range According to the combining means for generating a composite image for display, and the input position of the enlargement command and the composite evaluation value at the time of generating the composite image for display when an enlargement instruction for the composite image for display is input. And selecting means for switching and selecting the photographed image before being reduced to obtain a magnified image for display.

  ADVANTAGE OF THE INVENTION According to this invention, when a user confirms the effect of the synthesis | combination by HDR, long processing time is not required, and the detail confirmation by expansion display is also attained.

It is a figure showing an example of composition of a digital camera of an embodiment. It is a flowchart of the process performed with the digital camera of embodiment. It is explanatory drawing of the brightness matching process of two image signals from which exposure differs. It is explanatory drawing of the brightness ratio map of a reference | standard image and a non-reference | standard image. It is explanatory drawing of the switching process based on the synthetic | combination ratio of the image for expansion, and a center coordinate. It is explanatory drawing of the brightness ratio map in case a non-reference image is two sheets. It is explanatory drawing of a detection of a movement area | region. It is explanatory drawing in the case of referring the brightness ratio map of the past frame.

  Hereinafter, as an application example of the image processing apparatus according to the embodiment, a digital camera which is an imaging apparatus will be described as an example. The image processing apparatus according to the present embodiment receives, for example, in real time a captured image signal captured by a digital camera, and performs image processing on the captured image signal, such as a tablet terminal, a smartphone, or a personal computer. It may be an information processing apparatus. When the image processing apparatus according to the present embodiment is the information processing apparatus, various image processing according to the present embodiment for the captured image signal as described later is performed by the CPU or the like in the information processing apparatus according to the present embodiment. This is realized by executing a computer program of

First Embodiment
Hereinafter, a schematic configuration of the digital camera 100 according to the first embodiment will be described with reference to FIG. In the digital camera 100 of FIG. 1, the image processing apparatus according to the present embodiment corresponds to the image processing unit 103 and the system control unit 101. The system control unit 101 is, for example, a CPU. The system control unit 101 controls the operation of each unit included in the digital camera 100 and performs various signal processing. Specifically, the system control unit 101 executes, for example, a program for operation control stored in the non-volatile memory 102 to control the operation of each unit included in the digital camera 100, and executes a program for signal processing. It realizes various signal processing by doing. The nonvolatile memory 102 is, for example, an electrically erasable / recordable memory such as an EEPROM. The non-volatile memory 102 stores, in addition to a program for operation control and a program for signal processing, parameters necessary for operation control of each part and signal processing.

  The image processing unit 103 performs various image conversion processing on an image signal output from the imaging unit 110 or an image signal read from a memory 104 described later. For various image conversion processing, predetermined pixel interpolation processing, resizing processing for image reduction, color conversion processing, high dynamic range combining (HDR combining) processing, generation processing of an image signal displayed on the screen of the display unit 105 Etc. are included. Further, the image processing unit 103 executes arithmetic processing using an image signal captured by the image sensor 113 and digitized by the A / D conversion unit 114, and the system control unit 101 performs exposure control and distance measurement control. Generate information to be used when performing. Information on the calculation processing result by the image processing unit 103 is sent to the system control unit 101.

  The system control unit 101 operates the photographing lens 111 and the shutter 112 via a drive system (not shown) based on the information of the calculation result by the image processing unit 103 to perform exposure control and distance measurement control, and Control operations such as flash pre-flash processing. In addition, the system control unit 101 controls writing and reading of an image signal output from the A / D conversion unit 114 or an image signal subjected to various processing by the image processing unit 103 to the memory 104. .

  In addition, the system control unit 101 controls the imaging unit 110 immediately before or after imaging of a proper exposure image by proper exposure time to obtain an image to be subjected to HDR combining processing, and an exposure time different from the proper exposure time Take multiple images by. In the case of the present embodiment, the system control unit 101 controls, for example, the imaging unit 110 to capture a low exposure image with an exposure time shorter than the appropriate exposure time and capture a high exposure image with an exposure time longer than the proper exposure time. And Then, the image processing unit 103 generates an HDR image using a plurality of images captured at the respective exposure times. However, although details will be described later, the digital camera 100 according to the present embodiment matches the display resolution of the display unit 105 when generating an HDR image for live view display or thumbnail display by the display unit 105. The HDR combining process is performed using the reduced image. Although details will be described later, the digital camera 100 according to the present embodiment captures an image by the imaging unit 110 when generating an HDR image to be recorded to a removable memory (not illustrated) or output to the outside. HDR combining processing is performed using the captured image (captured image by main shooting). In addition, since the size of the captured image by main imaging is large and a somewhat long processing time is required, it is desirable that the HDR combining processing using the captured image by main imaging is performed, for example, after the end of image capturing.

  A memory 104 is used as an expansion area of a program for operation control of each part of the digital camera 100 and a program for signal processing, and is a storage area for temporarily storing intermediate data etc. generated in the operation of each part and signal processing. It is also used as In addition to the image signals of the plurality of frames taken and the image signals for display, the memory 104 is also written with the information of the sound being taken, so the memory 104 is designed to have a sufficient storage capacity for storing the information. It is done. The image signal for display written in the memory 104 is read from the memory 104 and sent to the display unit 105 under the control of the system control unit 101. The display unit 105 displays an image based on an image signal for display.

  Further, in the digital camera 100 of the present embodiment, on the display surface of the display unit 105, a touch panel 106 capable of detecting a touch by a finger or a pen of a user who is the user of the camera is provided. The system control unit 101 can detect the following user operation and the like on the touch panel 106 by receiving a detection signal from the touch panel 106. Examples of user operations that can be detected by the system control unit 101 via the touch panel 106 include touch-down operations when the touch panel 106 is touched with a finger or the like, and touch-on operations in a state where the touch panel 106 is touched with a finger or the like. Be Further, as detectable user operations, there are a move operation in which the touch panel is moved while touching with a finger or the like, a touch-up operation in which the finger touching the touch panel is released, a touch-off operation in which nothing is touched on the touch panel, and the like.

  The touch panel 106 notifies the system control unit 101 of detection signals of these operations and information of position coordinates where a finger or the like touches the touch panel 106. The system control unit 101 analyzes what kind of operation has been performed on the touch panel 106 based on the information notified from the touch panel 106. The system control unit 101 determines the x-axis component and y-axis component on the touch panel 106 based on the change in the position coordinates of (x, y) with respect to the movement direction of the finger or the like moving on the touch panel 106 by the move operation. It can be judged every time. In addition, when a touch-up operation is performed on the touch panel 106 after a certain move operation from the touch-down operation, the system control unit 101 determines that a stroke is drawn by a finger or the like. The operation of drawing a stroke quickly is called a flick operation. The flick operation is an operation in which a finger or the like is touched on the touch panel 106 quickly by a certain distance and then released as it is. In other words, the flick operation is an operation of tracing quickly on the touch panel 106 as if it is flicked by the finger. When it is detected that the move operation is performed at a predetermined speed or more at a predetermined distance or more and the touch-up operation is detected as it is, the system control unit 101 determines that the flick operation is performed. When it is detected that the move operation is performed at a predetermined distance or more at a speed less than the predetermined speed, the system control unit 101 determines that the drag operation is performed.

  The imaging unit 110 is a unit that captures an optical image of a subject or the like and outputs a digital image signal. The imaging unit 110 includes an imaging lens 111, a shutter 112, an imaging element 113, and an A / D converter 114. The photographing lens 111 is a photographing lens group included in the digital camera 100, and includes a focus lens, a zoom lens, a color filter, and the like. The photographing lens 111 forms an optical image of an object or the like on the image pickup element 113 through a shutter 112 having an aperture and an ND filter function. The imaging element 113 is an imaging element such as a CCD or a CMOS sensor, for example, converts an optical image formed through the imaging lens 111 into an analog image signal, and outputs the analog image signal to the A / D converter 114. The A / D conversion unit 114 performs A / D conversion processing on the analog image signal input from the imaging element 113 to generate a digital image signal (frame image signal).

  In each configuration of the digital camera 100 having the configuration as described above, the flow of imaging and image signal processing according to the present embodiment will be specifically described with reference to the flowchart of FIG. Note that the processing of the flowchart in FIG. 2 is started, for example, when an instruction for photographing is given to the digital camera 100 from the user.

  First, in step S200 in FIG. 2, the system control unit 101 obtains the result of the above-described arithmetic processing by the image processing unit 103, and based on the result of the arithmetic processing, for example, an appropriate exposure value and its appropriate exposure value. Determine different exposure values. The exposure value is a control amount related to exposure control, including the shutter speed, the aperture opening amount, and the ISO sensitivity setting in the imaging unit 110. Furthermore, the system control unit 101 sets the determined exposure value in the imaging unit 110, and controls the imaging unit 110 to continuously capture a plurality of images with different exposure values in time series to generate an image signal. To get The plurality of digital image signals acquired in step S200 and output from the imaging unit 110 are stored in the memory 104 via the image processing unit 103 under the control of the system control unit 101. The number of images to be captured is not particularly limited as long as it is two or more. After step S200, the process of the digital camera 100 proceeds to step S201.

  In step S201, the image processing unit 103 performs brightness adjustment processing as described below on the plurality of digital image signals acquired in step S200 and temporarily stored in the memory 104. For example, the image processing unit 103 performs processing to match the brightness of the image signal by applying gain or gamma according to the ratio of each exposure value when the image signal is acquired. Then, the image signal after the brightness adjustment is stored in the memory 104. Hereinafter, the brightness adjustment processing of the image signal according to the ratio of the exposure value will be described in detail with reference to FIG.

  In FIGS. 3A to 3C, the horizontal axis represents the illuminance of the subject, and the vertical axis represents the output level of the image sensor 113. In FIG. 3A, the input / output characteristic 301 is the input / output characteristic of the image signal (high-exposure image signal picked up with a long exposure time) when imaging is performed with the exposure amount of the image pickup device 113 increased. It shows. Further, an input / output characteristic 302 in FIG. 3A is an input / output characteristic of an image signal (a low exposure image signal picked up with a short exposure time) when imaging is performed with the exposure amount of the image pickup device 113 reduced. Is shown. In the case of the input / output characteristic 302 of the low exposure image signal shown in FIG. 3A, the output level of the imaging element 113 is substantially linear from low to high illuminance without being saturated until the illuminance of the object is high. It is changing. For this reason, it is considered that the low exposure image signal can be expressed by the output level of the image sensor 113 up to the brighter illuminance of the object than the high exposure image signal. On the other hand, in the case of the input / output characteristic 301 of the high exposure image signal shown in FIG. 3A, the output level of the imaging device 113 is saturated at a stage where the illuminance of the object is somewhat low. However, since the high exposure image signal has a large exposure amount of the imaging element 113 even when the subject has low illuminance, the SN ratio in the low illuminance region is better than that of the low exposure image signal.

  In consideration of such input / output characteristics 301 and 302, the digital camera 100 of the present embodiment adjusts the brightness of the high exposure image signal when the brightness of the high exposure image signal and the brightness of the low exposure image signal are matched. To match the brightness of the low-exposure image signal. Specifically, the image processing unit 103 applies a gain or a gamma curve to the image signals according to the ratio of the exposure value when the high exposure image signal and the low exposure image signal are acquired. Adjust the brightness of the image signal.

  Here, for example, in FIG. 3A, the illuminance of the subject serving as a certain reference is L. In the example of FIG. 3A, in the input / output characteristic 301, the illuminance of the object is 1⁄8 L and the output level of the image sensor 113 is saturated, and in the input / output characteristic 301, the illuminance of the object is 8 L and the image sensor 113 Output level is saturated. From this, it can be seen that the output levels of the image pickup device 113 become the same when the illuminance of the subject is 1/8 L in the input / output characteristic 301 and when the illuminance of the subject is 8 L in the input / output characteristic 302.

  Therefore, for example, when the brightness of the high exposure image signal and the low exposure image signal are matched by a gain corresponding to the ratio of the exposure value, the image processing unit 103 detects the signal level of the high exposure image signal of the input / output characteristic 301. Divide by 64. As a result, the input / output characteristic 301 of the high exposure image signal shown in FIG. 3A becomes the input / output characteristic 303 as shown in FIG. 3B. The input / output characteristic 303 of the high exposure image signal shown in FIG. 3B is substantially the same as the input / output characteristic 302 of the low exposure image signal in the low luminance region of 1⁄8 L or less. The brightness of the exposure image signal and the brightness of the low exposure image signal are substantially matched.

  Also, for example, when the brightness of the high exposure image signal and the low exposure image signal are matched by the gamma curve according to the ratio of the exposure value, the high exposure image signal and low in the low luminance region of 1/8 L or less A gamma curve is designed in which both input and output characteristics of the exposed image signal are substantially the same. Then, the image processing unit 103 performs gamma correction on each of the high exposure image signal of the input / output characteristic 301 of FIG. 3A and the low exposure image signal of the input / output characteristic 302 using the gamma curve. Adjust the brightness. As a result, the input / output characteristic 301 of FIG. 3 (a) becomes the input / output characteristic 305 as shown in FIG. 3 (c), and the input / output characteristic 302 of FIG. 3 (a) becomes as shown in FIG. 3 (c). The input / output characteristic 306 is obtained. The input / output characteristic 305 and the input / output characteristic 306 in FIG. 3C respectively have substantially the same characteristic in the low luminance region of 1⁄8 L or less, and accordingly, the high exposure image signal and the low exposure image signal in this case. The brightness is almost in a state of matching.

  According to the present embodiment, for the reason described above, the image processing unit 103 adjusts the brightness of the high exposure image signal to the brightness of the low exposure image signal in the range where the illuminance of the object is 1⁄8 L or less. I have to. In the present embodiment, since the low exposure image signal is used as the reference for adjusting the brightness, the low exposure image serving as the reference for adjusting the brightness is hereinafter referred to as a "reference image", while the reference image is the reference image. An image taken at different exposures (high-exposure image in this example) will be referred to as a "non-reference image". In the present embodiment, an example is given in which the brightness is adjusted to the low-exposure image signal. Conversely, when the brightness is adjusted to the high-exposure image signal, the high-exposure image is the standard. It becomes an image, and a low exposure image becomes a non-reference image. Besides, for example, when there is a proper exposure image captured with the proper exposure time described above together with a low exposure image and a high exposure image, and the proper exposure image is used as a reference image, a low exposure image and The high exposure image is a non-reference image. In the following description of the embodiment, only an example in which a low exposure image is a reference image and a high exposure image is a non-reference image will be described.

  Returning to the description of the flowchart in FIG. 2, after step S201, the process of the digital camera 100 proceeds to step S202. In step S202, the image processing unit 103 reduces the plurality of images (the reference image and the non-reference image after the brightness matching process) after the brightness matching process is performed in step S201, respectively. Generate a reduced image of The reduced image is subjected to HDR combining processing as described later and displayed on the display unit 105. Therefore, the image processing unit 103 performs reduction processing to an image size in accordance with the display resolution of the display unit 105. Hereinafter, the reference image and the non-reference image, which are respectively reduced according to the display resolution, will be referred to as "the reference image and the non-reference image for display". After step S202, the process of the digital camera 100 proceeds to step S203.

  In step S203, the image processing unit 103 performs HDR combining processing based on a combined evaluation value described later using the display reference image and the non-reference image generated in step S202, respectively, to match the display resolution. Generate an HDR image. At the time of the HDR combining process, the image processing unit 103 determines an image area in which a blackout and a whiteout occur, from the reference image for display and the non-reference image. Specifically, the image processing unit 103 determines the brightness of the image based on the output level of the imaging element 113 used in the above-described step S201, thereby causing the blackness of the reference image for display and the non-reference image. Determine the dark image area or the overexposed bright image area. It is to be noted that the black collapse is a state in which the gradation of the dark area of the image is lost and the black area is almost black, and the whiteout is the gray area of the bright area of the image. It is in a state of becoming For this reason, in the following description, an image area in which the gradation is lost due to black crushing or white out is referred to as a "disappearing area", while on the other hand, no black crushing or white out occurs. An image area in which is not lost is referred to as a "graded area".

  Then, the image processing unit 103 has a gray scale region in the reference image for display, and in the non-reference image, the image region corresponding to the gray scale region of the reference image is the gray scale region The HDR combining processing is performed such that the gray level area of the reference image is replaced with the gray level area of the non-reference image. Specifically, the image processing unit 103 sets the composition ratio of the reference image for display to the gradation loss area as 0%, while setting the composition ratio to the gradation area of the non-reference image for display as 100%. Synthesize That is, in this case, the HDR combining processing is performed such that each pixel of the gray scale region of the reference image for display is replaced by each pixel of the gray region of the non-reference image for display.

  In addition, when the gray level area of the reference image is replaced with the gray level area of the non-reference image, switching of the image by the replacement is performed at the boundary between the image area to be replaced and the image area not replaced. May be noticeable. For this reason, the image processing unit 103 makes the switching of the image in the boundary portion inconspicuous by changing the composition ratio according to the brightness of the image area related to the boundary portion in the above-mentioned boundary portion.

  As described above, the image processing unit 103 realizes the enlargement of the dynamic range by performing the HDR composition processing using the composition ratio according to the brightness of the reference image for display and the non-reference image as the composition evaluation value. To generate an HDR composite image. In the following description, as described above, an HDR image matching the display resolution of the display unit 105 is generated by HDR combining the reference image for display and the non-reference image with the combined evaluation value according to the brightness. In particular, it is written as "HDR image for display".

  Here, in the present embodiment, the composite evaluation value according to the brightness of the image area when generating the above-described HDR image for display is shown, for example, in FIGS. 4A and 4B. It is determined based on the brightness ratio map. The maps of FIG. 4A and FIG. 4B are for determining the composition ratio as a composition evaluation value representing the composition ratio of pixels based on the image signal of one of the reference image and the non-reference image. . In the present embodiment, an example in which the synthesis ratio is determined based on the reference image will be described.

  As described above, in the present embodiment, the brightness adjustment processing of the reference image and the non-reference image is performed in step S201 of FIG. Therefore, the image processing unit 103 uses the brightness ratio maps of FIG. 4A and FIG. 4B according to the brightness of the reference image and the non-reference image at the time of the brightness matching process in step S201. Generate it. Then, when generating the HDR image for display in step S203, the image processing unit 103 refers to the composition ratio for each pixel from the brightness ratio map, and based on the composition ratio, the reference image and the non-reference image Synthesize

  In the brightness ratio maps shown in FIGS. 4A and 4B, the horizontal axis represents the level of the image signal subjected to the HDR synthesis processing (the image signal level after the brightness adjustment processing), and the vertical axis represents the synthesis It shows the ratio. FIG. 4A shows a brightness ratio map with respect to the reference image signal, and the level area 407 in the horizontal axis direction represents the area of the signal level which is blacked out, and the level area 408 is the blacked out or blown out. Represents the area of the signal level which is not The brightness ratio map illustrated in FIG. 4A is a brightness ratio map when replacing a black gradation region in the reference image with a non-black gradation region of the non-reference image. It has become. On the other hand, FIG. 4B shows the brightness ratio map to the non-reference image signal, and the level area 409 in the horizontal axis direction represents an area of the signal level which is not crushed black or overexposed, and the level area 410 is It represents the area of the signal level which is overexposed. In the brightness ratio map illustrated in FIG. 4B, the brightness at the time of replacing the black-out blurred area of the reference image with the gray-scale area that is not black-out and white-out in the non-reference image. It is a ratio map.

  More specifically, as shown in FIG. 4A, the signal of the level area 407 in the reference image signal is an image signal of a gray level area which is blacked out. Therefore, in the brightness ratio map of FIG. 4A, the synthesis ratio 401 of the level area 407 is 0% in order to prevent the image signal of the level area 407 from being synthesized during the HDR synthesis process. It is done. On the other hand, as shown in FIG. 4B, the signal of the level region 409 in the non-reference image signal is an image signal of a gradation region which is neither crushed black nor blown out. Therefore, in the brightness ratio map of FIG. 4B, the synthesis ratio 404 of the level area 409 is set to 100% in order to use the image signal of the level area 409 for the synthesis in the HDR synthesis process. There is. As described above, the composition ratio 401 of the level area 407 in FIG. 4A and the composition ratio 404 of the level area 409 in FIG. 4B are the non-gradation areas which are blacked out in the standard image. Is a brightness ratio map for replacing with a gradation region which is neither black crushed nor overexposed.

  Further, the level area 410 in FIG. 4B is the level area of the image signal in the gray level area where the pixel value of the non-reference image signal is saturated and whiteout occurs. Therefore, in the brightness ratio map of FIG. 4B, the synthesis ratio 406 of the level area 410 is 0% in order to prevent the image signal of the level area 410 from being synthesized during the HDR synthesis process. It is done. On the other hand, the level area 408 of FIG. 4A is the level area of the image signal of the gradation area in which neither black crushing nor whiteout occurs in the reference image signal. Therefore, in order to use the image signal of the level area 408 at the time of the HDR combining process, the combining ratio 403 of the level area 408 is made 100% in the brightness ratio map of FIG. 4A. ing. As described above, the composition ratio 403 of the level area 408 in FIG. 4A and the composition ratio 406 of the level area 410 in FIG. 4B are gradation areas in which the black image is neither crushed nor white in the reference image. It is a brightness ratio map for preventing the image area of the non-reference image from being replaced.

  Further, the combination ratio 402 in FIG. 4A and the combination ratio 405 in FIG. 4B are combinations in the image area of the signal level where whether or not replacement of the reference image and the non-reference image is performed is switched. It is a ratio. For this reason, the image processing unit 103 sets the reference for the image area of the signal level of such switching according to the synthesis ratio by the synthesis ratio 402 in FIG. 4A and the synthesis ratio 405 in FIG. 4B. The combining process is performed by changing the ratio of combining the image and the non-reference image.

  As described above, according to the present embodiment, the image processing unit 103 generates the brightness ratio maps shown in FIGS. 4A and 4B, and uses the brightness ratio map as a reference. The HDR image is generated by combining the image and the non-reference image. Then, the HDR image for display generated by the HDR combining processing by the image processing unit 103 is stored in the memory 104 of FIG.

  After the HDR image for display is stored in the memory 104, in step S204, the system control unit 101 controls the reading of the memory 104 to output the HDR image signal for display to the display unit 105. As a result, the display unit 105 displays the HDR image for display. The HDR image for display at this time is displayed on the display unit 105 as live view display or thumbnail display. After step S204, the system control unit 101 advances the process to step S205.

  In step S205, the system control unit 101 determines, for example, whether or not the user has input an instruction for enlarged display from the HDR image for display displayed on the display unit 105 in FIG. For example, when the user performs a touch operation of instructing the touch panel 106 to perform enlarged display, the system control unit 101 determines from the output signal of the touch panel 106 the instruction of the magnified display and the center of the enlarged area according to the input position of the instruction An enlargement indication signal indicating a position is detected. The means for the user to input the enlargement display instruction is not particularly limited to the touch panel 106 as long as the system control unit 101 can detect the input of the enlargement instruction and the input position (center position of the enlargement area). . If it is determined in step S205 that the enlargement instruction is not input, the system control unit 101 returns the process to step S200 to perform the process of capturing a plurality of images having different exposure values as described above. . On the other hand, if the system control unit 101 determines in step S205 that the enlargement instruction input has been made, the system control unit 101 advances the process to step S206.

  In step S206, based on the center position of the enlarged area and the combined evaluation value (brightness ratio map) used when the image processing unit 103 generates the HDR image for display, the system control unit 101 The reference image signal and the non-reference image signal are switched and read out. Then, the read image signal is sent to the display unit 105 and displayed. At this time, the reference image signal or the non-reference image signal which is switched and selected from the memory 104 and read out is an image signal (a signal of a low exposure image or a high exposure image) after being subjected to the brightness adjustment processing in step S201 described above. . In the following description, the reference image and the non-reference image that have been subjected to the brightness matching process in the above-described step S201 will be particularly referred to as "the reference image and the non-reference image of the main photographing".

  As described above, in step S206, the reference image and the non-reference image for main imaging are selectively switched and displayed on the display unit 105. Since the reference image and the non-reference image of the main imaging are image signals of large size captured by the imaging unit 110, only a part of the image signals of large size is displayed on the display unit 105 at this time, that is, enlarged display It will be done. Specifically, in the display unit 105, the center position of the enlarged display is the screen center of the display unit 105, and an image is selected by switching the reference image or the non-reference image according to the brightness ratio map of the composite evaluation value described above. Only a part of the will be magnified. In the following description, the enlarged image for display, in which the reference image and the non-reference image for main shooting are switched according to the brightness ratio map of the combined evaluation value, is referred to as an “image for enlargement”.

  As described above, in the present embodiment, the enlargement image includes the reference image and the non-reference image of the main shooting based on the combined evaluation value which is the brightness ratio map used in generating the HDR image for display. It is made with the image selected switching. For this reason, it can be said that the HDR image for display and the image for enlargement are images generated based on the brightness ratio map having at least the same tendency. On the other hand, in the present embodiment, the HDR image to be recorded in the removable memory (not shown) or the like, externally output, etc. is actually photographed based on the brightness ratio map used at the time of generation of the HDR image for display. The reference image and the non-reference image are generated by the HDR synthesis processing. In the following description, an HDR image that is recorded in a removable memory or the like, output to the outside, or the like will be referred to as a “HDR image for real shooting”. From these things, the above-mentioned image for enlargement may be considered as an image having a dynamic range close to the HDR image of the main shooting and noise in the dark part. Therefore, for example, in the live view display, if the above-described enlargement image is displayed on the display unit 105, the user is likely to see what the dynamic range of the HDR image of the main shooting and the noise of the dark part look like Can be confirmed. In addition, since the enlargement image is an image obtained by switching the reference image and the non-reference image of the main imaging, the time required for the processing is very short. Therefore, the user does not have to wait for a long time before inputting the enlargement instruction and displaying the enlargement. As described above, according to the present embodiment, the user can view details of the image by viewing the magnified image having the dynamic range close to the HDR image of the main shooting to be generated later and the noise of the dark part by live view display. Confirmation is possible.

  Hereinafter, the brightness ratio map shown in FIG. 4A and FIG. 4B described above is used as a combined evaluation value, and a process of generating an enlargement image by switching between the reference image and the non-reference image for main imaging is illustrated. A description will be given with reference to 5 (a) to 5 (e). Although either of the brightness ratio map of FIG. 4 (a) and the brightness ratio map of FIG. 4 (b) is used as the composite evaluation value, the brightness map of FIG. 4 (a) is used here. An example used is described.

  FIG. 5A is an example of the display HDR image 511 described above, and shows a case where, for example, the image area where the user inputs an enlargement instruction to the display HDR image is the area 501, for example. FIG. Hereinafter, the image area 501 for which the input of the enlargement instruction has been made will be referred to as “the enlargement instruction area 501”. 5 (b) shows an example of an image in which the brightness ratio map of FIG. 4 (a) is applied to the HDR image 511 for display, and FIG. 5 (c) shows the HDR image 511 for display. It is the figure which expanded and showed only the expansion instruction area 501 of. In the example of FIG. 5C, center coordinates 503 which is the center position of the enlargement instruction area 501 are shown.

  In the present embodiment, when the enlargement instruction is input to the HDR image for display, the system control unit 101 performs the reference image of the main imaging based on the combination ratio corresponding to the center coordinates 503 of the enlargement instruction area 501. Alternatively, one of the non-reference images is selected as an enlargement image. Specifically, in the case where the combining ratio used when the display HDR image is generated at the center coordinates 503 of the enlargement instructing area 501 is, for example, the combining ratio 403 of 100% in FIG. 4A. The system control unit 101 selects a reference image for main imaging as an enlargement image. On the other hand, when the composite ratio used when the HDR image for display is generated is, for example, the composite ratio 401 of 0% of FIG. The unit 101 selects a non-reference image for main imaging as an enlargement image.

  In the case where both the composition ratio of 0% and 100% are mixed and applied to the enlargement instructing area 501, the system control unit 101 may select only the reference image of the main imaging, for example. Good. For example, both composite ratio 0% and 100% are mixed with respect to the enlargement designated area 501, and both the reference image and the non-reference image are mixed and displayed corresponding to the composite ratio of both of them. If this is done, overexposure will occur in the image area of the non-reference image. Therefore, when the enlargement instructing area 501 has a combination ratio of 0% and 100%, the system control unit 101 selects only the reference image for main imaging and displays it for enlargement. Thereby, the user is considered to be able to confirm the effect of the HDR that suppresses overexposure by viewing the display for enlargement.

  However, assuming that the reference image for main shooting is always selected when the composite ratio of 0% and 100% is mixed, the selection is not based on the brightness ratio map (composite evaluation value) described above. The noise of the magnified image is somewhat different from the noise of the HDR image of the actual shooting. Therefore, the system control unit 101 selects the reference image and the non-reference image for switching based on the combination ratio applied to the center coordinates 503 of the enlargement instruction area 501, specifically as follows. To do.

  FIG. 5D and FIG. 5E are diagrams showing the enlargement instruction area 501 further enlarged in the case where the combination ratio of 0% and 100% is mixed. In FIGS. 5D and 5E, a region 520 is a region to which a composition ratio for setting the reference image for display to 100% (0% for the non-reference image) is applied, and a region 521 is for display It is assumed that a region to which a composition ratio with 0% as the reference image (100% as the non-reference image) is applied. In the present embodiment, as shown in FIG. 5D, when the center coordinate 503 of the enlargement instruction area 501 is in the area 520, the system control unit 101 performs the main photographing after the above-described brightness adjustment processing. The reference image of is taken as the image for enlargement. On the other hand, as shown in FIG. 5E, when the center coordinate 503 of the enlargement instruction area 501 is in the area 521, the system control unit 101 determines that the non-reference image of the main shooting after the above-described brightness adjustment processing. As the image for enlargement. For example, in the case of FIG. 5E, whiteout may occur in the area 520, but the noise of the image around the center coordinates 503 of the enlargement image is the noise of the HDR image of the main shooting. It is considered to be something that almost matches the feeling. Therefore, the user can confirm noise by live view display of the enlargement image also in this example. The digital camera 100 according to the present embodiment switches between the noise confirmation function (noise preview display function) by such live view display and the normal live view display (normal preview display function) according to the user's instruction. It is made possible.

  After step S206 in FIG. 2 as described above, the system control unit 101 advances the process to step S207. In step S207, the system control unit 101 determines whether an instruction to end the display of the enlargement image is input from the user via the touch panel 106, and the instruction to end the display of the enlargement image is not input. Returns processing to output 206. On the other hand, if it is determined in step S207 that an instruction to end the display of the enlargement image has been input, the system control unit 101 advances the process to step S208.

  In step S208, the system control unit 101 determines whether an instruction to end the above-described processing for displaying the display HDR image as live view display or thumbnail display is input from the user via the touch panel 106. If the system control unit 101 determines that the end instruction is input in step S208, the process of the flowchart of FIG. 2 is ended. If the end instruction is not input, the process proceeds to step S200. return.

  As described above, in the digital camera 100 according to the first embodiment, the HDR image for display is displayed as live view display or thumbnail display, and when the enlargement instruction is further input, the present evaluation is performed based on the combined evaluation value. The reference image for shooting and the non-reference image are switched and displayed. Therefore, according to the first embodiment, even when it takes a long time to generate the HDR image for the main shooting, the HDR image for display and the image for enlargement can be displayed immediately without waiting time.

Second Embodiment
In the first embodiment, the HDR image for display and the enlargement image are generated using two images of the reference image for main shooting and the non-reference image, but the HDR image for display and the enlargement image are The reference image may be generated from three or more different exposure images. As an example, in the second embodiment, an example in which one reference image and two non-reference images of main imaging are switched to generate an enlargement image will be described. Here, in order to make the description easy to understand, an image captured with the shortest exposure time is used as a reference image, an image captured with an exposure time longer than the reference image is used as a first non-reference image, and the first An image captured with a longer exposure time than the non-reference image is taken as a second non-reference image.

  The flow of processing in the second embodiment will be described below with reference to the flowchart of FIG. 2 described above. In the flowchart of FIG. 2, the description of steps in which the same processing as that of the first embodiment is performed will be omitted.

  In the case of the second embodiment, when generating an HDR image for display in step S203 of the flowchart of FIG. 2, a brightness ratio map for combining three images is required. 6A to 6C show an example of the brightness ratio map used in the second embodiment. FIG. 6 (a) shows an example of a brightness ratio map for a reference image for display reduced according to the display resolution, and similarly FIG. 6 (b) shows a first non-reference image for display. FIG. 6C shows the brightness ratio map of the second non-reference image for display. Similar to the examples of FIG. 4A and FIG. 4B described above, the horizontal axes in FIG. 6A to FIG. The image signal level after the alignment processing is shown, and the vertical axis shows the composition ratio. The level area 611 in FIG. 6A represents the area of the signal level that is blacked out, the level area 612 represents the area of the signal level that is not blacked out or blown out, and the combination ratio 604 is 0%. The synthesis ratio 601 is 100%. The level area 613 in FIG. 6 (b) represents the area of the signal level which is blacked out, the level area 614 is the area of the signal level which is not blacked out and whitened out, and the level of the signal area which is leveled out 615 is whitened Region, and the synthesis ratio 602 is 100%. The level area 616 in FIG. 6C represents the area of the signal level not shattered and overexposed, the level area 617 represents the area of the signal level overexposed, and the synthesis ratio 603 is 100%. There is.

  Then, in the case of the second embodiment, in the case of the above-described first embodiment based on the brightness ratio maps of FIGS. 6A to 6C in step S203. In the same manner as in, an HDR image for display is generated.

  Further, in the case of the second embodiment, in step S206, the system control unit 101 uses the brightness ratio map as the synthetic evaluation value when generating the enlargement image, as in the first embodiment described above. , Switching between and selecting the reference image of the main imaging and the first and second non-reference images. That is, the system control unit 101 refers to the synthesis ratio of the brightness ratio map at the center coordinates of the enlargement instruction area among the reference image for main shooting after the brightness adjustment processing and the first and second non-reference images. And make a switching selection. Specifically, the system control unit 101 selects the reference image for main imaging based on the 100% composite ratio 601 of the level area 612, while 0% for the first and second non-reference images. Do not select based on the composition ratio of Similarly, the system control unit 101 selects the first non-reference image based on the 100% composite ratio 602 of the level area 614, and the second non-reference image uses the 100% composite ratio of the level area 616. Select based on 603.

  Further, in the second embodiment, even when both 0% and 100% of the composition ratio are mixed with the enlargement instruction area, the image is displayed in substantially the same manner as in the first embodiment. A switch selection is made. For example, level 607 in FIGS. 6A to 6C is an image signal level value at the center coordinates of a certain enlargement instruction area. In this case, in FIG. 6A, since the composition ratio 604 of the level 607 is 0%, the reference image is not selected. On the other hand, since the level 607 in FIG. 6B and the level 607 in FIG. 6C are level regions where black is not squashed and overexposed, respectively, either of the first and second non-reference images is selected. It will be done. In this case, the system control unit 101 selects one having a higher synthesis ratio of the brightness ratio map used to generate the HDR image for display. 6 (b) and 6 (c), the combination ratio 605 at the level 607 in FIG. 6 (b) and the combination ratio 606 at the level 607 in FIG. 6 (c) are as shown in FIG. 6 (c). The composition ratio 606 of is a higher value. Therefore, in this case, the system control unit 101 selects, of the first and second non-reference images, the second non-reference image corresponding to FIG.

  As described above, according to the second embodiment, even when the reference image and the first and second non-reference images are captured, it is possible to generate an HDR image for display and an image for enlargement. It is done.

  In the second embodiment, for example, when performing noise preview display for confirming noise by live view display, based on a value calculated as follows from an area including the center coordinates of the enlargement instruction area An image to be an enlargement image may be selected. For example, the average value of the composition ratio of the brightness ratio map is calculated in the range of the 10% image area including the center coordinates in the enlargement designated area, and the average value is calculated as the reference image and the first and second non-standard images. The one with the higher composition ratio is selected as the enlargement image in comparison with each composition ratio of the reference image.

Third Embodiment
Next, as the third embodiment, when the reference image and the non-reference image are captured, the position of the image area of the subject (hereinafter referred to as a moving body area) is the reference image when the subject is moved. The generation of the HDR image for display and the image for enlargement will be described in the case where the image and the non-reference image are different. In the third embodiment, a moving subject region is detected from the reference image and the non-reference image, and the composition ratio of pixels with respect to the moving subject region is determined. Then, in the third embodiment, generation of an HDR image for display and generation of an image for enlargement are performed using the combination ratio for the moving object region and the brightness ratio map described above as a combination evaluation value.

  The flow of processing in the third embodiment will be described below with reference to the flowchart of FIG. 2 described above. In the flowchart of FIG. 2, the description of steps in which the same processing as that of the first embodiment is performed will be omitted.

  In the case of the third embodiment, the process of generating an HDR image for display in step S203 of the flowchart of FIG. 2 is described with reference to FIGS. 4A and 4B and FIGS. 7A to 7. This will be described with reference to (c). FIG. 7A shows a reference image 710 for display created in the process up to step S202 in FIG. 2, and FIG. 7B shows a non-reference image 711 for display created in the same manner. In the examples shown in FIGS. 7A and 7B, the position of the subject 703 is different between the reference image 710 for display and the non-reference image 711 due to the movement of the subject 703.

  In the case of the third embodiment, in step S203, the image processing unit 103 performs an inter-frame interval between the display reference image 710 of FIG. 7A and the display non-reference image 711 of FIG. 7B. Find the difference. Further, the image processing unit 103 compares the inter-frame difference with a predetermined threshold, and obtains an area having a difference amount larger than the threshold as a detection result of the moving subject area. Note that the detection result of the moving body area is the feature amount of the image based on the inter-frame difference amount between the reference image and the non-reference image. In FIG. 7C, the difference between the frames of the reference image 710 for display and the non-reference image 711 is compared with a predetermined threshold, and a region of a difference amount larger than the threshold is extracted. An example is shown. In FIG. 7C, a moving object region 702 corresponding to the subject 703 in the reference image 710 for display in FIG. 7A and a moving object corresponding to the subject 703 in the non-reference image 711 for display in FIG. An area 701 is obtained as a detection result 712 of the moving body area.

  Furthermore, the image processing unit 103 uses the brightness ratio maps as shown in FIG. 4A and FIG. 4B and the detection result 712 of the moving object region in FIG. The combination ratio of the reference image 710 for display and the non-reference image 711 when generating the HDR image for the image is determined. Specifically, after the image processing unit 103 determines the composition ratio of the brightness ratio map for the display reference image 710, the moving object regions 701 and 702 as shown in FIG. Check if. Then, the image processing unit 103 does not perform combining with reference to the combining ratio of the brightness ratio map for the pixels included in the moving object regions 701 and 702 in FIG. The composition ratio of the reference image for the image is set to 100%. As for the pixels not included in the moving object regions 701 and 702, an HDR image for display based on the combination ratio of the brightness ratio map is generated as described above. Thus, the image processing unit 103 generates a display HDR image in which the moving object regions 701 and 702 are replaced with the display HDR image.

  In the third embodiment, when generating the enlargement image in step S206 of FIG. 2, the system control unit 101 uses the brightness ratio map and the detection result 712 of the moving subject region as a reference for the main shooting. It is determined which of the image and the non-reference image is to be selected as the enlargement image. Specifically, when the system control unit 101 generates an enlargement image, the pixel at the center coordinates of the enlargement instruction area (or the pixel within the range determined from the center coordinates of the enlargement instruction area) is shown in FIG. It is determined whether or not it is included in the moving object regions 701 and 702 of. When the system control unit 101 determines that the pixel at the center coordinate of the enlargement instruction area (the pixel within the range determined from the center coordinate) is included in the moving object area 701 and 702, the system control unit 101 determines the reference image for main imaging. Select as an image for enlargement. If the pixel at the center coordinate of the enlargement instruction area (or the pixel within the range including the center coordinate) is not in the moving object area 701 or 702, the system control unit 101 performs an image based on the brightness ratio map ( The reference image or the non-reference image of the actual shooting is selected as the enlargement image.

  By performing the above-described processing, in the third embodiment, it is possible to generate a display HDR image and an enlargement image in consideration of the detection result of the moving body region.

Fourth Embodiment
For example, in the case where camera shake or the like occurs in a HDR image for display or an image for enlargement due to so-called hand-held photography etc., as the fourth embodiment, image alignment processing as described below is also performed. It is possible. The alignment process is a process of correcting the positional deviation when the position of the non-reference image is deviated from the reference image due to camera shake or the like. Hereinafter, as an example, the case where the alignment process is performed after the process of step S202 of FIG. 2 will be described.

  The image processing unit 103 performs alignment processing on the reference image and the non-reference image after the process of step SS201 in FIG. At the time of this alignment processing, the image processing unit 103 obtains a motion vector between the reference image and the non-reference image. Specifically, the image processing unit 103 creates a template image of a predetermined size from the reference image, performs template matching for obtaining corresponding points between the template image and the non-reference image, and generates an orientation between the corresponding points. The magnitude (distance) is calculated as a motion vector. Then, the image processing unit 103 calculates a conversion coefficient based on the direction and magnitude of the motion vector, and deforms the non-reference image based on the conversion coefficient. As described above, the non-reference image deformed by the conversion coefficient based on the motion vector is a non-reference image in which the positional displacement is corrected with respect to the reference image, that is, the non-reference image Become.

  After that, the digital camera 100 performs the process after step S203 in FIG. 2 as described above, using the reference image and the non-reference image on which the alignment process has been performed on the reference image. As a result, the digital camera 100 according to the fourth embodiment can perform good live view display, thumbnail display, and enlargement image display in which shake is corrected even when camera shake or the like occurs due to hand-held shooting.

Fifth Embodiment
In each embodiment described above, generation of a display HDR image and an enlargement image is performed based on the combination ratio of the brightness ratio map created from the reference image and the non-reference image in step S201 of FIG. On the other hand, in the fifth embodiment, when generating the display HDR image and the enlargement image, the display HDR image and the enlargement are referred to by referring to the brightness ratio map generated in the past in the past. An example of generating an image for

  A process of generating an HDR image for display and an image for enlargement in the fifth embodiment will be described with reference to FIG. The horizontal axis of FIG. 8 represents the time sequence of each display frame N-3, N-2, N-1, N, N + 1, N + 2, and it should be updated in time as it goes to the right of the horizontal axis. Represents The vertical axis in FIG. 8 represents the composition ratio of the brightness ratio map of the reference image for display. Further, an input 808 in FIG. 8 represents the timing when the user inputs an enlargement instruction. Furthermore, each circle in FIG. 8 represents each combination ratio 801 to 807 of the reference image for display in each display frame N-3, N-2, N-1, N, N + 1, N + 2.

  Here, it is assumed that the digital camera 100 is performing live view display. For example, assuming that the input 808 of the enlargement instruction is made, the digital camera 100 according to the present embodiment performs the processing of steps S200 to S202 in FIG. 2 described above in the current display frame N immediately after the input 808 of the enlargement instruction. Do. At this time, the system control unit 101 causes the memory 104 to hold the synthesis ratio of the past brightness ratio map generated in the display frame that is a predetermined number of frames before the current display frame N. In the case of the fifth embodiment, in step S203, the image processing unit 103 determines whether or not there is a large difference between the composition ratio of the brightness ratio map of the display frame N and the composition ratio of the past brightness ratio map. Do. As an example, the image processing unit 103 determines whether the difference between the composition ratio of the brightness ratio map of the display frame N and the composition ratio of the past brightness ratio map is larger than, for example, a predetermined threshold. Then, when the difference between the combining ratios is larger than the threshold, the image processing unit 103 sets the combining ratio of the display frame N as the combining ratio of the display frame after the predetermined number of frames. In this case, the image processing unit 103 sets the composition ratio of the display frame before the predetermined number of frames as the composition ratio of the display frame N, and based on the composition ratio, from the reference image for display and the non-reference image in the display frame N Generate an HDR image for display.

  As an example, when the predetermined number of frames is two frames, the image processing unit 103 combines the combining ratio 804 in the display frame N and the combining ratio 802 generated and held in the display frame N-2 two frames before. Compare. In the case of this example, there is a difference between the combining ratio 804 and the combining ratio 802 that is larger than the threshold value. Therefore, the image processing unit 103 displays the combining ratio 804 obtained in the display frame N two display frames N + 2 later. It is set as the synthetic | combination ratio 807 in Further, the image processing unit 103 at this time sets the combining ratio 802 of the display frame N-2 two frames before as the combining ratio 805 of the display frame N. Similarly, the image processing unit 103 sets the combining ratio 803 of the display frame N-1 as the combining ratio 806 of the display frame N + 1. Thus, the image processing unit 103 uses the composition ratio 805 in the display frame N, uses the composition ratio 806 in the display frame N + 1, and uses the composition ratio 807 in the display frame N + 2 to display the reference image and the non-reference image for display Will generate an HDR image for

  In addition, since the system control unit 101 at this time receives the input of the enlargement instruction 808, the processing of steps S204 to S206 in FIG. 2 is performed. Then, in step S206 in FIG. 2, the system control unit 101 generates an enlargement image based on the combination ratio of the brightness ratio map before the predetermined number of frames referred to by the image processing unit 103. As described above, when the predetermined number of frames is two, the system control unit 101 switches between the reference image and the non-reference image of the main captured image for the display frame N based on the composition ratio 805 to select the enlargement image. Generate Similarly, the system control unit 101 switches and selects the reference image and the non-reference image of the main photographed image based on the combining ratio 806 in the display frame N + 1 and based on the combining ratio 807 in the display frame N + 2. It will generate an image.

  In the case of the fifth embodiment, even when, for example, the brightness of an object or the like changes rapidly when the input of the enlargement instruction is made, the brightness ratio map generated before the brightness changes rapidly. The HDR image for display and the image for enlargement will be displayed based on. Therefore, according to the fifth embodiment, even when there is an abrupt change in the brightness of the subject or the like, the HDR image for display or for enlargement is free from the discomfort that the occurrence of the abrupt change in the brightness is suppressed. An image can be displayed.

Sixth Embodiment
In the sixth embodiment, when the process of generating the HDR image of the main shooting is completed, the digital camera 100 uses the HDR image of the main shooting, for example, when displaying the thumbnail display or the enlargement image. For example, when the thumbnail display is performed on the display unit 105 after the generation of the HDR image for the main shooting is completed, the image processing unit 103 generates a thumbnail image from the HDR image for the main shooting, and the system control unit 101 Makes the display unit 105 display the thumbnail image. In addition, when the thumbnail image display generated from the HDR image of the main shooting is performed, when the enlargement instruction is input from the user, the system control unit 101 receives the HDR image of the main imaging according to the input of the enlargement instruction. Is directly used as an enlargement image.

  Further, for example, when the display of the HDR image for display or the display of the enlargement image is performed as described above, when the generation processing of the HDR image of the main shooting is completed, the image processing unit 103 In response to this, a notification of completion of generation of the HDR image of the main shooting is output. For example, when the system control unit 101 receives the generation completion notification while the display HDR image is being displayed, the system control unit 101 converts the display HDR image into an HDR image obtained by reducing the main shooting HDR image. Replace and display. Further, for example, if the system control unit 101 receives the generation completion notification while the display of the enlargement image described above is being performed, the system control unit 101 replaces the enlargement image with the HDR image of the main shooting and causes the display to be displayed. .

<Other Embodiments>
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. Can also be realized. It can also be implemented by a circuit (eg, an ASIC) that implements one or more functions.

  The above-described embodiments are merely examples of implementation for practicing the present invention, and the technical scope of the present invention should not be interpreted limitedly by these. That is, the present invention can be implemented in various forms without departing from the technical concept or the main features thereof.

  DESCRIPTION OF SYMBOLS 100 digital camera, 101 system control part, 102 non-volatile memory, 103 image processing part, 104 memory, 106 touch panel, 110 lens unit, 111 imaging lens, 112 shutter, 113 imaging element, 114 A / D conversion part

Claims (12)

Reducing means for reducing a plurality of photographed images photographed at different exposure values for high dynamic range synthesis to generate a plurality of reduced images;
Generation means for generating a synthesis evaluation value representing a synthesis ratio of pixels in high dynamic range synthesis of the plurality of reduced images based on at least the brightness of the plurality of photographed images;
Combining means for combining the plurality of reduced images based on the combined evaluation value to generate a combined image for display;
When an enlargement instruction for the composite image for display is input, a plurality of photographings before the reduction are performed based on the input position of the enlargement instruction and the composite evaluation value at the time of generating the composite image for display An image processing apparatus comprising: selection means for switching and selecting an image to form an enlarged image for display.   The selection means refers to the synthesis evaluation value used in the high dynamic range synthesis at the input position of the enlargement instruction to the synthesis image for display, and the reduced image synthesized based on the synthesis evaluation value is The image processing apparatus according to claim 1, wherein the plurality of photographed images before the reduction are switched and selected. And adjusting means for adjusting the brightness of another photographed image so as to match the brightness of the photographed image of the reference with one photographed image of the plurality of photographed images as a reference.
The reduction means reduces the plurality of photographed images after the brightness is adjusted to generate the plurality of reduced images.
3. The image processing apparatus according to claim 1, wherein the selection unit selects the plurality of photographed images after matching the brightness as the enlarged image for display by switching and selecting the plurality of photographed images. The combining unit detects an image area of a moving object based on a difference between the plurality of reduced images, determines the combined evaluation value for the image area of the moving object, and detects the image of the moving object in the reduced image. For the area, synthesis is performed based on the determined synthetic evaluation value,
When the input position of the enlargement instruction for the composite image for display is included in the image area of the moving body, the selection means performs the plurality of photographing based on the composite evaluation value determined for the image area of the moving body. The image processing apparatus according to any one of claims 1 to 3, wherein switching selection of an image is performed. The combining means determines, as the combined evaluation value for the image area of the moving body, a combining ratio using only a reduced image obtained by reducing one of the plurality of captured images which has been made a reference.
When the input position of the enlargement instruction for the composite image for display is included in the image area of the moving body, the selection means performs the plurality of photographing based on the composite evaluation value determined for the image area of the moving body. 5. The image processing apparatus according to claim 4, wherein one photographed image which is set as a reference among the images is selected. A correction unit configured to correct a positional deviation between the plurality of photographed images;
The reduction means generates the plurality of reduced images from the plurality of photographed images after the positional deviation is corrected,
The combining unit generates a combined image for display from the plurality of reduced images in which the positional deviation is corrected and reduced.
The image processing apparatus according to any one of claims 1 to 5, wherein the selection unit performs the switching selection using a plurality of photographed images after the positional deviation has been corrected. The combining means includes means for high dynamic range combining of the plurality of photographed images based on the combined evaluation value,
When the high dynamic range synthesis of the plurality of photographed images by the synthesizing unit is completed, the reduction unit generates a reduced image by reducing an image in which the plurality of photographed images are high dynamic range synthesized.
The combining means generates a composite image for display by combining the plurality of reduced images, and when high dynamic range combining of the plurality of photographed images by the combining means is completed, Replacing a reduced image of a high dynamic range composited image of a plurality of captured images with the composite image for display;
The selection unit, when switching between and selecting the photographed image before the reduction as the enlarged image for display, the high dynamic range synthesis of the plurality of photographed images by the synthesizing unit is completed, The image processing apparatus according to any one of claims 1 to 6, wherein an image obtained by high dynamic range synthesis of the photographed image is switched and selected as the enlarged image for display. A control means for holding a synthetic evaluation value generated from a captured image captured in the past by a predetermined number of frames before the current captured image;
In the case where there is a difference between the synthetic evaluation value generated from the current captured image and the held synthetic evaluation value when the enlargement instruction to the synthetic image for display is input, the generation means The synthetic evaluation value generated from a photographed image photographed in the past before the predetermined number of frames is set as a synthetic evaluation value of the current photographed image. Image processing apparatus as described.   In the case where there is a difference between the synthetic evaluation value generated from the current captured image and the held synthetic evaluation value when the enlargement instruction to the synthetic image for display is input, the generation means 9. The image processing apparatus according to claim 8, wherein a combined evaluation value generated from a current captured image is set as a combined evaluation value of a captured image captured after a predetermined number of frames. Imaging means for imaging a plurality of images with different exposure values for high dynamic range synthesis;
Display means for displaying an image;
Acquisition means for acquiring an input position of at least an enlargement instruction from a user;
The imaging device which has each means of the image processing apparatus of any one of Claims 1-9. The reduction means reduces a plurality of photographed images photographed at different exposure values for high dynamic range synthesis to generate a plurality of reduced images;
Generating, based on at least the brightness of the plurality of photographed images, a synthesis evaluation value representing a synthesis ratio of pixels in high dynamic range synthesis of the plurality of reduced images;
Combining means for combining the plurality of reduced images based on the combined evaluation value to generate a combined image for display;
When the selection means receives an enlargement instruction for the composite image for display, the reduction means is reduced based on the input position of the enlargement instruction and the composite evaluation value at the time of generating the composite image for display. And switching the selected image to select an enlarged image for display.   The program for functioning a computer as each means of the image processing apparatus of any one of Claims 1-9.

Images