JP2012090041A - Image processing device, method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device which, when generating an HDR image using plural images, even if a blur of a moving subject etc. occurs between the plural images, is capable of obtaining an HDR image closer to a user's intention.SOLUTION: A digital camera 100 analyses a difference between plural images having different exposure conditions including proper exposure and determines from an analysis result whether to perform image composition. Then, when it is determined, from a determination result, that the image composition is not possible, an image photographed by one of the exposure conditions is selected on the basis of a luminance histogram with respect to a proper exposure image and generates a high dynamic range image.

Description

  The present invention relates to an image processing apparatus and method, and a program, and more particularly to a technique for generating a single high dynamic range (HDR) image by combining a plurality of image data captured by a camera.

  In a consumer digital still camera, a solid-state image sensor such as a CCD or a CMOS is used as an image sensor. In general, the dynamic range of solid-state image sensors is narrow, so when shooting a high-contrast subject with proper exposure, the background of the main subject may be appropriate, but the background may be overexposed or blackout. May end up. In order to solve such a problem, in Patent Literature 1, the dark range and the bright portion of the subject are photographed twice so that each portion is appropriate, and the dynamic range is expanded by combining the two images. A camera has been proposed.

  In Patent Document 2, a standard image obtained by appropriate exposure is taken, it is determined whether or not dynamic range expansion is performed based on a histogram of the signal level of the image, and a composite of a plurality of images is output as necessary. An image composition device that selects whether to output a single image has been proposed.

JP-A-5-64075 Japanese Patent Application Laid-Open No. 8-214111

  However, since Patent Document 1 always synthesizes two images, it is likely to be affected by moving body blur and the like, and an expected result may not be obtained.

  Also, if the contrast of the shooting subject is not strong and there is no need to expand the dynamic range, it is necessary to take again an image with proper exposure for the entire subject, which is troublesome and misses a photo opportunity. There's a problem.

  In Patent Document 2, it is automatically determined whether or not the dynamic range needs to be expanded based on the histogram, so that shooting can be performed efficiently.

  However, if it is determined that the dynamic range needs to be expanded, a composite image is created from images captured a plurality of times as in Patent Document 1, so there is no problem with still object shooting. There is a problem that an image is generated.

  The present invention has been made in view of the above problems, and in the case of generating an HDR image using a plurality of images, an HDR image that is closer to the user's intention even when a moving body blur or the like occurs between the plurality of images. It is an object to provide a technique capable of obtaining the above.

  In order to achieve the above object, the image processing apparatus of the present invention is an image processing apparatus capable of generating one high dynamic range image from a plurality of images photographed with different exposure conditions on the same subject. Image analysis means for analyzing the difference between the images, image composition determination means for determining whether or not to perform image composition based on the analysis result by the image analysis means, and image composition from the determination result by the image composition determination means An image processing means for generating a high dynamic range image using any one of a plurality of images having different exposure conditions based on a luminance histogram for a properly exposed image when it is determined that the exposure cannot be performed. Features.

  According to the present invention, when an HDR image is generated using a plurality of images, an HDR image closer to the user's intention can be obtained even when a moving body blur or the like occurs between the plurality of images.

1 is a diagram illustrating a schematic configuration of a digital camera as an example of an image processing apparatus according to an embodiment of the present invention. It is a flowchart which shows the flow of imaging | photography and image processing in HDR imaging | photography mode. It is a figure which shows an example of the histogram of the image image | photographed by appropriate exposure. It is a figure which shows the relationship between an input pixel value and an output pixel value.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the present embodiment, a case where the present invention is applied to a digital camera will be described.

  FIG. 1 is a diagram illustrating a schematic configuration of a digital camera 100 as an example of an image processing apparatus according to an embodiment of the present invention.

  In the digital camera 100, a light beam from a subject is guided to an image sensor 14 through a photographing lens 10 and a mechanical shutter 12 having a diaphragm function, and is converted into an electric signal by the image sensor 14. The A / D converter 16 converts the analog signal output from the image sensor 14 into a digital signal.

  The timing generation circuit 18 supplies a clock signal and a control signal to the image sensor 14 and the A / D converter 16. The timing generation circuit 18 is controlled by the memory control circuit 22 and the system control circuit 50. In addition to the mechanical shutter 12, the accumulation time can be controlled as an electronic shutter by controlling the reset timing of the image sensor 14 of the timing generation circuit 18, and can be used for moving image shooting and the like.

  The image processing circuit 20 applies a TTL AWB (Auto White Balance) to the data from the memory control circuit 22 that stores the output signal from the A / D converter 16 or the output signal from the A / D converter 16. Apply processing. Thereafter, an image signal is generated by performing development processing including gamma processing, interpolation processing, and matrix conversion.

  In the image processing circuit 20, based on the obtained image signal, the system control circuit 50 calculates a control amount for the control performed on the exposure control unit 40 and the distance measurement control unit 42. As a result, TTL AF (Auto Focus) processing, AE (Auto Exposure) processing, and EF (Electronic Flash) processing are realized.

  The memory control circuit 22 controls the A / D converter 16, the timing generation circuit 18, the image processing circuit 20, the memory 30, and the compression / decompression circuit 32. Data from the A / D converter 16 is written into the memory 30 via the image processing circuit 20 and the memory control circuit 22, or data from the A / D converter 16 is directly written through the memory control circuit 22.

  The image display unit 28 is made of a TFT liquid crystal or the like, and the display image data written in the memory 30 is displayed via the memory control circuit 22. If the image data captured using the image display unit 28 is sequentially displayed, the electronic viewfinder function can be realized. Further, the image display unit 28 can arbitrarily turn on / off the display according to an instruction from the system control circuit 50. When the display is turned off, the power consumption of the digital camera 100 can be greatly reduced.

  The memory 30 is a storage device for storing captured still images and moving images, and has a sufficient storage capacity for storing a predetermined number of still images and moving images for a predetermined time. This makes it possible to write a large amount of images to the memory 30 at high speed even in continuous shooting or panoramic shooting in which a plurality of still images are continuously shot. The memory 30 can also be used as a work area for the system control circuit 50.

  The non-volatile memory 31 is configured by a flash ROM or the like, and a program executed by the system control circuit 50 is written into the non-volatile memory 31 and executed while being read sequentially. In addition, an area for storing system information and an area for storing user setting information are provided in the nonvolatile memory 31, and various information and settings are read and restored at the next startup.

  The compression / decompression circuit 32 is a compression / decompression circuit that compresses and decompresses image data by adaptive discrete cosine transform (ADCT) or the like, reads an image stored in the memory 30, performs compression processing or decompression processing, and finishes the processing. The stored data is written into the memory 30.

  The exposure control unit 40 controls the mechanical shutter 12 having an aperture function. Further, it also has a flash light control function in conjunction with the flash 48. The distance measurement control unit 42 controls focusing of the photographing lens 10, and the zoom control unit 44 controls zooming of the photographing lens 10.

  The flash 48 also has an AF auxiliary light projecting function and a flash light control function. The exposure control unit 40 and the distance measurement control unit 42 are controlled using the TTL method. Based on the calculation result obtained by calculating the captured image data by the image processing circuit 20, the system control circuit 50 performs the exposure control unit 40 and the distance measurement control unit 42. The distance controller 42 is controlled.

  The system control circuit 50 is a control unit that controls the entire digital camera 100. Reference numerals 60, 62, 64, 66, 70, and 72 denote operation units for inputting various operation instructions of the system control circuit 50. A single unit such as a switch, a dial, a touch panel, pointing by eye-gaze detection, a voice recognition device, or the like Consists of multiple combinations.

  The mode dial switch 60 is a switch that can switch and set each function mode such as power-off, automatic shooting mode, normal shooting mode, HDR shooting mode, panoramic shooting mode, moving image shooting mode, playback mode, and PC connection mode. .

  The shutter switch SW1_62 is turned ON during the operation of the shutter button, and instructs to start operations such as AF (autofocus) processing, AE (automatic exposure) processing, and AWB (auto white balance) processing. The shutter switch SW2_64 is turned on when the operation of the shutter button is completed. In the case of flash photography, the shutter switch SW2_64 serves as an instruction to start the process of exposing the image sensor 14 for the exposure time determined by the AE process after performing the EF (flash pre-emission) process. In the case of flash photography, the shutter switch SW2_64 emits light during the exposure period, and is shielded from light by the exposure control unit 40 simultaneously with the end of the exposure period, thereby giving an instruction to start the process of ending the exposure to the image sensor 14. .

  The signal read from the image sensor 14 is written as image data in the memory 30 via the A / D converter 16 and the memory control circuit 22. Further, the shutter switch SW2 performs a development process using the calculation in the image processing circuit 20 and the memory control circuit 22, and a recording process in which the image data is read from the memory 30, compressed by the compression / decompression circuit 32, and written to the recording medium 200. This is an instruction to start a series of processes including the above.

  The display changeover switch 66 is a switch for switching the display of the image display unit 28. With this function, it is possible to save power by cutting off the current supply to the image display unit 28 when shooting using the optical viewfinder 104.

  The operation unit 70 includes various buttons, a touch panel, a rotary dial, and the like, and includes a menu button, a set button, a macro button, a multi-screen playback page break button, a flash setting button, a single shooting / continuous shooting / self-timer switching button, and the like. . Menu move + (plus) button, menu move-(minus) button, playback image move + (plus) button, playback image move-(minus) button, shooting image quality selection button, exposure compensation button, date / time setting button Etc.

  The zoom switch 72 is a switch for the user to give an instruction to change the magnification of the captured image. The zoom switch 72 includes a tele switch that changes the imaging field angle to the telephoto side and a wide switch that changes the imaging angle of view to the wide angle side. By using the zoom switch 72, the zoom control unit 44 is instructed to change the imaging field angle of the photographing lens 10 and becomes a trigger for performing an optical zoom operation. In addition, it also serves as a trigger for electronic zooming change of the imaging angle of view by image cropping by the image processing circuit 20 or pixel interpolation processing.

  The power supply unit 86 includes a primary battery of an alkaline battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li ion battery, an AC adapter, or the like. The connector 90 is connected to a recording medium such as a memory card or hard disk (I / F), and the connector 92 is connected to a recording medium such as a memory card or hard disk. The optical viewfinder 104 can perform photographing using only the optical viewfinder without using the electronic viewfinder function of the image display unit 28.

  The communication unit 110 has various communication functions such as USB, IEEE 1394, LAN, and wireless communication. The connector 112 connects the digital camera 100 to other devices via the communication unit 110. In the case of wireless communication, the connector 112 is an antenna. The recording medium 200 is a memory card, a hard disk, or the like, and includes a recording unit 202 composed of a semiconductor memory, a magnetic disk, or the like, an interface (I / F) 204 with the digital camera 100, and a connector 206 for connecting with the digital camera 100. ing.

  Here, the HDR shooting mode is a high dynamic range shooting mode, and can obtain an image (hereinafter referred to as an HDR image) in which overexposure and blackout are suppressed even for a subject with a large contrast. It is a possible shooting mode. In the digital camera 100, the mode dial switch 60 is photographed in accordance with the HDR photographing mode to generate an appropriate HDR image according to the photographed subject using a plurality of images photographed under different exposure conditions. Can do.

  Next, shooting / image processing in the HDR shooting mode of the digital camera 100 will be described with reference to FIG.

  FIG. 2 is a flowchart showing a flow of shooting / image processing in the HDR shooting mode. This process is performed by the system control circuit 50 executing a program read from the memory.

  First, the system control circuit 50 performs photometry according to a preset photometry method, and performs AE shooting based on the AE result set in the exposure control unit 40 so that the subject is properly exposed (step S200). The image captured here is stored in the memory 30 as the image 1.

  Next, the system control circuit 50 takes an image with an exposure that is two steps below the appropriate exposure, and stores it in the memory 30 as the image 2 (step S201).

  Further, the system control circuit 50 takes an image with an exposure that is two steps over the appropriate exposure, and stores it in the memory 30 as an image 3 (step S202).

  The shooting from step S200 to step S202 is performed as a continuous shooting sequence with the shooting interval shortened to a minimum in the present embodiment in order to minimize the difference between images. These images are still images before development processing, and are so-called RAW images.

  Next, in accordance with an instruction from the system control circuit 50, the difference in temporal position between the images is analyzed in the image processing circuit 20 for the three images of appropriate exposure, under and over in the memory 30. (Step S203). Then, it is determined from the analysis result whether the difference between the images is large or small (step S204). Regarding the difference analysis, in the present embodiment, the absolute value of the difference between the same pixel addresses is obtained for the entire region, and the sum value is used as the difference value. Here, the image processing circuit 20 functions as an image analysis unit, and the system control circuit 50 functions as an image composition determination unit. In addition, the analysis method is not particularly limited as long as it can determine the difference between images.

  If it is determined from the determination result in step S204 that the difference between the images is small, it is determined that the images can be combined. Then, the image processing circuit 20 and the system control circuit 50 calculate the composition ratio of each image and perform image composition (step S205). The system control circuit 50 develops the obtained composite image in the image processing circuit 20, compresses it with the image compression / decompression circuit 32, records it on the recording medium 200 (step S206), and ends this processing.

  On the other hand, when it is determined from the determination result in step S204 that the difference between the images is large (for example, when the main subject is a moving object), it is determined that the image cannot be combined. At this time, a luminance histogram of the image 1 is created in the image processing circuit 20 (step S207). Here, since the image 1 is an image captured with appropriate exposure, there is a high probability that an image having both gradations of the bright part and the dark part remains. Therefore, in this embodiment, a luminance histogram of the image 1 is taken. Yes. However, a luminance histogram may be taken using image 2 to preferentially suppress overexposure as much as possible and image 3 to preferentially suppress blackout.

  Next, the system control circuit 50 determines whether or not the frequency Hh classified as the maximum luminance level on the luminance histogram is larger than the predetermined threshold value Kh on the high luminance side (step S208). As a result of this determination, if it is determined that the frequency Hh is large, the process proceeds to step S212. On the other hand, if it is determined in step S208 that the frequency Hh is smaller than or equal to the predetermined threshold value Kh on the high luminance side, the process proceeds to step S209.

  In step S212, the system control circuit 50 determines that there are many overexposed areas, and the image processing circuit 20 leaves as much gradation as possible on the high luminance side of the raw image of the image 2 that is an underexposed image. Performs key conversion processing. In the present embodiment, this gradation conversion processing is performed by gamma processing that is part of the development processing. Then, the image after the development processing is compressed by the image compression / decompression circuit 32 and recorded on the recording medium 200, and this processing ends. As a process for leaving the high luminance side gradation, for example, there is a gradation conversion process that raises the low luminance side tone curve as shown in FIG.

  In step S209, the system control circuit 50 determines whether or not the frequency Hl classified as the minimum luminance level on the luminance histogram is larger than the predetermined threshold Kl on the low luminance side. As a result of this determination, if it is determined that the frequency H1 is small or equal, the system control circuit 50 determines that the black-out area is small, and the image 1 that is a properly exposed image is the same as in normal shooting. Development processing with gradation conversion processing is performed. Thereafter, the compression process and the recording process are performed, and this process ends (step S210).

  On the other hand, if it is determined in step S209 that the frequency Hl is larger than the predetermined threshold Kl on the low luminance side, the process proceeds to step S211. In step S211, the system control circuit 50 determines that there are many underexposure areas, and the tone conversion process is performed on the image 3 that is an overexposed image so that the tone on the low luminance side is left as much as possible by the image processing circuit 20. I do. Then, the image after the development processing is compressed by the image compression / decompression circuit 32 and recorded on the recording medium 200, and this processing ends. As a process for leaving the low luminance side gradation, for example, there is a gradation conversion process in which the tone curve on the low luminance side as shown in FIG.

  Since the frequency in the digital camera of the present embodiment indicates the number of pixels constituting the image, the total frequency at each signal level is the number of output pixels (effective pixel number). The frequency Hh can be paraphrased as the number of pixels where overexposure occurs, and the frequency Hl can be paraphrased as the number of pixels where blackout occurs. Therefore, in determining the threshold values Kh and Kl, it is necessary to set the threshold values in consideration of how far overexposure and underexposure are allowed.

  FIG. 3 is a diagram illustrating an example of a histogram of an image captured by appropriate exposure.

  In FIG. 3, the horizontal axis represents the signal level, and the vertical axis represents the frequency (frequency). The signal levels Max and Min are the maximum value and the minimum value that can be expressed by the digital camera of FIG. 1, respectively, and this range is the dynamic range. In this example, since the frequency Hh is not zero, it can be seen that overexposure occurs. In this case, since Hh> Kh is established in the determination in step S208, an image with reduced overexposure can be generated from the underexposed image as performed in step S212.

  In this embodiment, a proper exposure image (image 1), an exposure image that is two steps under the proper exposure (image 2), and an exposure image that is two steps over (image 3) are captured. However, the present invention is not limited to this, and the number of steps and the number of shots may be increased. By increasing the number of steps and the number of shots, an image with less overexposure and underexposure can be obtained even for a subject with higher contrast.

  In addition, although the recording is one image, there may be a case where there is not much influence even if there is a moving body blur, so it was obtained by synthesizing each image of proper exposure, underexposure correction, and overexposure correction. An HDR image may be recorded at the same time.

  Moreover, although the position shift between images was taken as the difference between images in step S204, other than this, for example, a color tone change between images may be used. When the change in color tone between images is large, an HDR image can be created from one image by executing the processing from step S207 onward, so that failure can be prevented.

  In the process of FIG. 2, when the frequency belonging to the highest luminance level on the luminance histogram exceeds a predetermined ratio and the frequency belonging to the lowest luminance level exceeds a predetermined ratio, the following configuration may be used. In other words, an image captured by appropriate exposure may be selected, developed and recorded to deal with both overexposure and underexposure.

  Further, in this embodiment, in order to prevent overexposure over blackout, the comparison of Hh and Kh (step S208), which is a determination of overexposure, is performed before the comparison of Hl and Kl (step S209). Are in the steps. However, the present invention is not limited to this, and when blackout is preferentially prevented, the comparison between Hl and Kl may be brought to the previous step.

  According to the above-described embodiment, even in the HDR shooting mode in which the same subject is shot a plurality of times under different exposure conditions, even if a positional shift occurs between images due to moving body shake, camera shake, etc., a single image that has been shot is used. HDR images can be obtained, and shooting efficiency can be increased. As a result, an HDR image close to the user's intention can be obtained.

  In the above embodiment, a digital camera has been described as an image processing apparatus capable of generating a single high dynamic range image from a plurality of images photographed with different exposure conditions on the same subject. The present invention is not limited and can be applied to an image processing apparatus such as a PC.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

14 Image sensor 16 A / D converter 18 Timing generation circuit 20 Image processing circuit 50 System control circuit 60 Mode dial switch 70 Operation unit 100 Digital camera

Claims (9)

In an image processing apparatus capable of generating one high dynamic range image from a plurality of images taken under different exposure conditions for the same subject,
Image analysis means for analyzing a difference between the plurality of images;
Image composition determination means for determining whether to perform image composition based on the analysis result by the image analysis means;
When it is determined that the image cannot be combined from the determination result by the image combining determination unit, based on the luminance histogram for the properly exposed image, the high dynamic range image is selected using any one of the plurality of images having different exposure conditions. An image processing apparatus comprising: an image processing unit that performs generation. The image analysis means analyzes a temporal position difference between the plurality of images,
The image processing apparatus according to claim 1, wherein the image composition determination unit determines whether or not to perform image composition by determining whether or not a difference between images is large. The image analysis means analyzes a color tone change between the plurality of images,
The image processing apparatus according to claim 1, wherein the image composition determination unit determines whether or not to perform image composition by determining whether or not a change in color tone between images is large.   The image processing means selects an image taken with underexposure from a plurality of images with different exposure conditions when the frequency classified into the maximum luminance level exceeds a threshold among luminance histograms for a properly exposed image. The image processing apparatus according to claim 1, wherein a high dynamic range image is generated.   The image processing means selects an image shot with overexposure from a plurality of images with different exposure conditions when the frequency classified into the minimum luminance level exceeds a threshold in a luminance histogram for a properly exposed image. The image processing apparatus according to claim 1, wherein a high dynamic range image is generated.   When the frequency belonging to the highest luminance level exceeds a predetermined ratio and the frequency belonging to the lowest luminance level exceeds a predetermined ratio in the luminance histogram for the properly exposed image, the image processing means determines whether the exposure conditions differ from a plurality of images. The image processing apparatus according to claim 1, wherein an image captured by appropriate exposure is selected to generate a high dynamic range image.   2. The image according to claim 1, wherein the image processing means records one high dynamic range image obtained by combining the plurality of images together with a high dynamic range image generated without combining. Processing equipment. In an image processing method of an image processing apparatus capable of generating one high dynamic range image from a plurality of images taken under different exposure conditions for the same subject,
An image analysis step of analyzing a difference between the plurality of images;
An image synthesis determination step for determining whether to perform image synthesis based on the analysis result of the image analysis step;
If it is determined from the determination result in the image combination determination step that the image cannot be combined, a high dynamic range image is obtained using any one of a plurality of images with different exposure conditions based on a luminance histogram for a properly exposed image. An image processing method comprising: generating an image processing step.   A computer-readable program for causing an image processing apparatus to execute the image processing method according to claim 8.

Images