JP6810299B2 - Image processing equipment, methods, and programs as well as imaging equipment - Google Patents

Description

The present invention relates to an image processing device, a method, a program, and an imaging device, and particularly performs high dynamic range composition (HDR (High-dynamic-range) composition) in which a plurality of images having different exposures are combined to widen the dynamic range. Regarding technology.

When a subject with a wide dynamic range is imaged with a digital camera or the like, the output of the image sensor is saturated in the highlight area, so-called "blown out" occurs, and the gradation of the image sensor output is lost in the shadow area. Therefore, so-called "black crushing" may occur.

It is known that HDR processing is performed as one of the methods for expressing the dynamic range of a subject in a wider image. In HDR processing, for example, an image (underexposure image) captured with a reduced exposure amount and an image (overexposure image) captured with an increased exposure amount are acquired with respect to proper exposure, and a pixel in which a dark subject is captured is captured. By using the pixels of the overexposed image and the pixels of the underexposed image for the pixels in which a bright subject is captured (actually, not only one of them but also the pixels may be mixed), an HDR image with less overexposure and underexposure can be obtained. create.

Conventionally, there are devices described in Patent Documents 1 and 2 as an apparatus for generating one HDR image from a plurality of images taken with different exposures.

The imaging system described in Patent Document 1 exposes a plurality of images (long-exposure image and short-exposure image) captured under different exposure conditions on the same subject among the brightness signals related to the long-exposure image. The brightness signal of the area (appropriate exposure area) other than the overexposed area (inappropriate exposure area) is acquired, and in the case of the brightness signal related to the short exposure image, it corresponds to the improper exposure area in the long exposure image. Get the brightness signal. Then, a wide dynamic range image is generated by synthesizing the appropriate exposure areas of each image.

Further, the imaging system described in Patent Document 1 estimates a main subject area from a long-exposure image and a short-exposure image, and performs gradation correction so that the gradation width is mainly distributed to the estimated main subject area. By synthesizing the appropriate exposure range of each image after gradation correction, the gradation for the main subject in the appropriate exposure range is enriched.

On the other hand, the image pickup apparatus described in Patent Document 2 includes one image pickup element having a pixel group for long-time exposure and a pixel group for short-time exposure, and the image data of the first pixel group read from the image pickup element and the second image sensor. An HDR image is generated by synthesizing the image data of the pixel group.

Further, in Patent Document 2, the mixing ratio of the flash image data which is the long exposure data and the non-flash image data which is the short exposure data is added while adjusting according to the estimated value of the distance to the subject. There is a description of obtaining one composite image. As a result, an image having a high SN ratio (Signal to Noise Ratio) of the main subject is obtained while leaving the atmosphere of the background image.

JP-A-2002-232777 Japanese Unexamined Patent Publication No. 2009-20625

Patent Document 1 does not clearly describe the mixing ratio of the long-exposure image and the short-exposure image when generating the HDR image, but the luminance signal corresponding to the improper exposure range of the long-exposure image is not described. Since the luminance signal related to the short-exposure image is used, it is considered to be a general HDR composition in which the mixing ratio of the long-exposure image and the short-exposure image is determined by the magnitude of the luminance signal. Therefore, as will be described later, if there is a shadow on a part of the main subject, there is a problem that the shadow area remains dark.

On the other hand, in the invention described in Patent Document 2, the mixing ratio of the flash image data which is the long exposure data and the non-flash image data which is the short exposure data is adjusted according to the estimated value of the distance to the subject. While adding, one composite image is generated. Further, the invention described in Patent Document 2 synthesizes flash image data and non-flash image data, and does not synthesize non-emission images (non-flash images) having different exposure conditions.

As described above, in Patent Documents 1 and 2, when a plurality of images captured under different exposure conditions for the same subject are combined to generate an HDR image, the intensity of the light shining on the subject is determined. There is no description of adjusting the composition ratio of a plurality of images accordingly.

When deciding whether to use the pixel value of the over image or the pixel value of the under image based on the brightness value obtained from the image sensor, as in general HDR processing, the intensity of the light hitting the subject is used. Regardless, the same HDR processing is performed, and the following problems occur.

For example, as shown in FIG. 21, it is assumed that the left half of the face of a person is the sunlit area A and the right half is the shaded area B. In this case, if "black hair (1) exposed to strong light" and "shadowed skin (2)" have the same brightness value or substantially the same brightness value, the same applies to HDR synthesis. The over-image is used, and not only the shaded skin (2) becomes brighter, but also the black hair (1) becomes brighter.

The present invention has been made in view of such circumstances, and is an image processing device capable of acquiring an HDR image having excellent gradation characteristics, in which the difference in brightness is adjusted according to the brightness of the light hitting the subject. It is an object of the present invention to provide methods and programs as well as imaging devices.

In order to achieve the above object, the image processing apparatus according to one aspect of the present invention is an image obtained by capturing the same subject, and a plurality of non-emission images captured under different exposure conditions under non-emission of flash light. Distribution of distance between an image acquisition unit that acquires an image and a light emitting image captured under the same exposure conditions as the exposure condition of any of a plurality of non-light emitting images under the light emission of flash light, and a subject. The light intensity of the light shining on the subject based on the distance distribution information acquisition unit that acquires the distance distribution information indicating the above, the light emitting image, the non-light emitting image captured under the same exposure conditions as the light emitting image, and the distance distribution information. A light intensity distribution information calculation unit that calculates distribution information, a composition ratio determination unit that determines the composition ratio of a plurality of non-emission images based on the light intensity distribution information, and a composition ratio determination unit that synthesizes a plurality of non-emission images according to the composition ratio. It includes an image compositing unit that generates a compositing image with an expanded dynamic range.

According to one aspect of the present invention, the light intensity distribution information of the light shining on the subject is calculated based on the light emitting image, the non-light emitting image captured under the same exposure conditions as the light emitting image, and the distance distribution information. .. Then, the composite image (HDR) having an expanded dynamic range is determined by determining the composite ratio of the plurality of non-emission images based on the calculated light intensity distribution information and synthesizing the plurality of non-emission images according to the determined composite ratio. Image) is generated. The HDR image generated in this way is an HDR image having excellent gradation characteristics, in which the difference in brightness is adjusted according to the brightness of the light hitting the subject without impairing the brightness of the subject.

In the image processing apparatus according to another aspect of the present invention, the composition ratio is determined for each of the corresponding pixels or regions of the plurality of non-emission images, and the image compositing unit determines the corresponding pixels or regions of the plurality of non-emission images. It is preferable to combine a plurality of non-emission images by applying the composition ratio determined for each.

In the image processing apparatus according to still another aspect of the present invention, the plurality of exposure conditions corresponding to the plurality of non-emission images include an underexposure exposure condition and an overexposure exposure condition rather than the proper exposure exposure condition. Is preferable. This makes it possible to create an HDR image with less overexposure and underexposure.

In the image processing apparatus according to still another aspect of the present invention, when determining the composition ratio for each corresponding pixel or region of a plurality of non-emission images, the pixel or region having a large light intensity is based on the light intensity distribution information. Then, the mixing ratio of the first non-emissive image captured by the exposure condition having the smallest exposure value among the plurality of non-emission images is taken by the exposure condition having the largest exposure value among the plurality of non-emission images. 2 It is preferable that the mixing ratio of the non-emissive image is larger than that of the first non-emissive image in the pixel or region where the light intensity is low.

That is, in a pixel or region having a high light intensity (a pixel corresponding to a subject exposed to strong light), the mixing ratio of the first non-emission image is made larger than the mixing ratio of the second non-emission image to make white. For pixels or regions that generate HDR images with less skipping and have low light intensity (pixels corresponding to subjects exposed to weak light), the mixing ratio of the second non-emission image is higher than the mixing ratio of the first non-emission image. By increasing the size of the image, an HDR image with less blackout is generated.

In the image processing apparatus according to still another aspect of the present invention, the composite ratio determining unit determines the composite ratio for each corresponding pixel or region of a plurality of non-emission images, based on the light intensity distribution information. When the light intensity increases between the minimum light intensity and the maximum light intensity in the intensity distribution information, it is preferable to monotonically decrease the mixing ratio of the second non-emission image. When the light intensity increases between the minimum light intensity and the maximum light intensity in the light intensity distribution information, the mixing ratio of the second non-emission image is monotonically decreased (the mixing ratio of the first non-emission image is monotonically increased). By doing so, HDR synthesis in which the mixing ratio is continuously changed according to the increase or decrease in light intensity can be performed. The minimum light intensity is not limited to the minimum value in the light intensity distribution information, and may be, for example, a light intensity with a value offset by a certain value from the minimum value. Similarly, the maximum light intensity is in the light intensity distribution information. The light intensity is not limited to the maximum value of, and may be, for example, a value offset by a certain value from the maximum value. It goes without saying that the monotonous decrease includes the case of a linear decrease.

The image processing apparatus according to still another aspect of the present invention includes a distribution information creating unit that creates distribution information indicating the frequency distribution or frequency distribution of light intensity based on the light intensity distribution information, and the synthesis ratio determining unit is created. The first light intensity corresponding to the first luminosity or frequency and the second light intensity corresponding to the second luminosity or frequency based on the distributed distribution information are larger than the first light intensity. When the second light intensity is obtained and the composition ratio is determined for each corresponding pixel or region of a plurality of non-emission images, if the light intensity is equal to or less than the first light intensity, the mixing ratio of the second non-emission image is determined. Is set to the maximum value, the mixing ratio of the second non-emissive image is set to the minimum value when the light intensity is equal to or higher than the second light intensity, and when the light intensity is larger than the first light intensity and lower than the second light intensity, It is preferable to monotonically reduce the mixing ratio of the second non-emission image between the maximum value and the minimum value.

For example, outdoors during the daytime on a sunny day, subjects with a wide dynamic range, from dark subjects that are not exposed to sunlight (shade subjects) to bright subjects that are exposed to sunlight (subjects in the sun), are subject to imaging. In this case, the imaging target can be roughly divided into a shaded subject and a sunlit subject. When distribution information showing the frequency distribution or frequency distribution of light intensity is created based on the light intensity distribution information of such a scene, the distribution information includes two peaks (first vertex and second vertex which are two vertices). ) Exists. The first vertex is the vertex corresponding to the power or frequency of the light intensity distribution information of the subject area corresponding to the shaded subject, and the second vertex is the power of the light intensity distribution information of the subject area corresponding to the subject in the sun. It is the apex corresponding to the frequency.

Then, the light intensity corresponding to the first vertex (first light intensity) and the light intensity corresponding to the second vertex (second light intensity) are obtained, and for each of the corresponding pixels or regions of the plurality of non-emission images. When determining the composition ratio, the mixing ratio of the second non-emissive image is maximized when the light intensity is lower than the first light intensity, and the second non-emissive image is when the light intensity is higher than the second light intensity. Minimize the mixing ratio of. This is because the region having a light intensity equal to or lower than the first light intensity is considered to be a shaded region, and the region having a light intensity equal to or higher than the second light intensity is considered to be a region in the sun. Further, in the case of a light intensity between the first light intensity and the second light intensity, the mixing ratio of the second non-emissive image is monotonically decreased between the maximum value and the minimum value, and continuously as the light intensity increases or decreases. The mixing ratio is changed. It should be noted that the above is not limited to outdoor scenes during the daytime on a sunny day, but also in scenes where sunlight shines into a part of the room, and where there are areas illuminated by artificial light sources and areas that are shaded. A distribution curve showing a frequency distribution or a frequency distribution of light intensity, such as, may have two vertices.

The image processing apparatus according to still another aspect of the present invention includes a distribution information creating unit that creates distribution information indicating the frequency distribution or frequency distribution of light intensity based on the light intensity distribution information, and the synthesis ratio determining unit is created. The light corresponding to the bottom point is obtained when the light intensity corresponding to the low frequency or infrequent bottom point is obtained based on the distributed distribution information and the composition ratio is determined for each corresponding pixel or region of a plurality of non-emission images. The intensity is set as a threshold value, and when the light intensity is below the threshold value, the mixing ratio of the second non-emission image is maximized, and when the light intensity exceeds the threshold value, the mixing ratio of the second non-emission image is set to the minimum value. Is preferable.

Similar to the above, in the case of a daytime outdoor scene on a sunny day, there is a valley (bottom point) between two mountains in the distribution information indicating the frequency distribution or frequency distribution of light intensity. Therefore, the bottom point is obtained from the distribution information indicating the frequency distribution or frequency distribution of the light intensity, the light intensity corresponding to this bottom point is set as a threshold, and when the light intensity is below the threshold, the mixing ratio of the second non-emission image is obtained. Is the maximum value (the mixing ratio of the first non-emission image is the minimum value), and when the light intensity exceeds the threshold value, the mixing ratio of the second non-emission image is set to the minimum value (the mixing ratio of the second non-emission image is the maximum value). ) Is preferable. For light intensities in a certain range before and after the threshold value, the mixing ratio of the second non-emission image is continuously changed between the maximum value and the minimum value according to the magnitude of the light intensity. Also includes.

The image processing apparatus according to still another aspect of the present invention includes a luminance distribution information calculation unit that calculates luminance distribution information indicating the luminance distribution in an image of at least one non-luminous image among a plurality of non-luminous images. It is preferable that the composite ratio determining unit determines the composite ratio of a plurality of non-emission images based on the light intensity distribution information and the brightness distribution information.

Thereby, not only the light intensity distribution information but also the conventional luminance distribution information can be taken into consideration to determine the composition ratio of a plurality of non-emission images.

In the image processing apparatus according to still another aspect of the present invention, the image processing apparatus includes a detection unit that detects pixels or regions that the flash light has not reached in the emitted image, and the synthesis ratio determining unit is a pixel or a pixel that the flash light has not reached. When determining the composite ratio for a region, it is preferable to determine the composite ratio of a plurality of non-emission images based only on the luminance distribution information.

For pixels or regions that the flash light has not reached, the light intensity corresponding to the pixels or regions becomes unknown. Therefore, in this case, it is preferable to determine the composition ratio of the plurality of non-emission images based on the luminance distribution information as in the conventional case.

In the image processing apparatus according to still another aspect of the present invention, the composition ratio determining unit is a parameter indicating the position of a pixel or region when determining the composition ratio for each corresponding pixel or region of a plurality of non-emission images. I, the mixing ratio of the second non-light emitting image captured under the exposure condition having the largest exposure value among the plurality of non-light emitting images is αi, and the mixing ratio of the second non-light emitting image based on the luminance distribution information. Assuming that βi is the mixing ratio that becomes smaller as the brightness is higher, the mixing ratio γi of the second non-emission image based on the light intensity distribution information and the brightness distribution information is calculated by the following equation.

It is preferable to calculate by γi = αi × βi.

In the image processing apparatus according to still another aspect of the present invention, the exposure condition of the light emitting image is preferably the exposure condition having the smallest exposure value among the plurality of exposure conditions corresponding to the plurality of non-light emitting images. As a result, it is possible to increase the amount of flash light emitted when capturing a light emitting image, and to minimize the number of pixels that have not reached the flash light.

The imaging device according to still another aspect of the present invention includes an imaging unit capable of imaging a subject under different exposure conditions, a flash emitting unit that emits flash light, and a flash control that controls flash light emitted from the flash emitting unit. The image acquisition unit includes a unit and the image processing device described above, and the image acquisition unit includes a plurality of non-emission images captured under different exposure conditions by the imaging unit under the non-emission of the flash light emitted from the flash light emitting unit. Under the light emission of the flash light emitted from the flash light emitting unit, the imaging unit acquires a light emitting image captured under the same exposure conditions as the exposure condition of any one of the plurality of non-light emitting images.

In the image pickup apparatus according to still another aspect of the present invention, the flash control unit causes the flash light to be dimmed from the flash light emitting unit, and the flash light is not dimmed with the dimming image captured by the imaging unit under the dimming light emission. When a region where the dimmed flash light does not reach is detected based on the image captured by the imaging unit, the maximum amount of light emitted is calculated so that the region where the dimmed flash light reaches does not saturate. It is preferable that the flash light is mainly emitted from the flash light emitting unit at the maximum light emission amount calculated when the light emission image is acquired. By determining the amount of light emitted during the main light emission in this way, it is possible to minimize the pixels that have not reached the flash light.

In the image pickup apparatus according to still another aspect of the present invention, it is preferable that the flash control unit calculates the maximum amount of light emission based on the average reflectance of the subject and the distance distribution information acquired by the distance distribution information acquisition unit. As a result, it is possible to calculate the maximum amount of light emitted without overexposure in the area where the flash light reaches.

In the image pickup apparatus according to still another aspect of the present invention, the flash control unit causes the flash light to be dimmed from the flash light emitting unit, and the flash light is not dimmed with the dimming image captured by the imaging unit under the dimming light emission. When it is detected that the flash light reaches the entire region of the dimming image based on the image captured by the imaging unit, the image acquisition unit preferably acquires the dimming image as a light emitting image.

This is because when the flash light reaches the entire region of the dimming image, the light intensity distribution information of the entire region can be calculated, and it is not necessary to perform the main light emission again.

In the image pickup apparatus according to still another aspect of the present invention, the flash control unit causes the flash light to be dimmed from the flash light emitting unit, and the flash light is not dimmed with the dimming image captured by the imaging unit under the dimming light emission. A region where the flash light does not reach the dimming image is detected based on the image captured by the imaging unit, and the distance of the region where the flash light does not reach is emitted from the flash light emitting unit based on the distance distribution information. When it is farther than the reach of the maximum emission amount of the flash light to be generated, the image acquisition unit preferably acquires the dimming image as the emission image.

If the distance of the area where the flash light of the dimming light emission does not reach is farther than the reachable distance of the maximum light emission amount of the flash light, the flash light cannot reach the pixels which have not reached the flash light even if the main light emission is performed. This is because there is no need to make a new flash.

The image processing method according to still another aspect of the present invention includes a step of acquiring a plurality of non-emission images of the same subject captured under different exposure conditions under non-emission of flash light. , A step of acquiring a luminescent image captured under the same exposure conditions as the exposure condition of any of a plurality of non-luminous images under the emission of flash light, and distance distribution information indicating the distribution of the distance of the subject. And the step of calculating the light intensity distribution information of the light shining on the subject based on the light emitting image, the non-light emitting image captured under the same exposure conditions as the light emitting image, and the distance distribution information, and the light. It includes a step of determining a composite image of a plurality of non-emission images based on intensity distribution information, and a step of synthesizing a plurality of non-emission images according to the composite ratio to generate a composite image with an expanded dynamic range.

In the image processing method according to still another aspect of the present invention, the step of determining the composition ratio is based on the light intensity distribution information when determining the composition ratio for each corresponding pixel or region of the plurality of non-emission images. In a pixel or region with high light intensity, the mixing ratio of the first non-emission image captured under the exposure condition with the smallest exposure value among the plurality of non-emission images is determined by the most exposure value among the plurality of non-emission images. The mixing ratio of the second non-emissive image captured under a large exposure condition should be larger than the mixing ratio of the second non-emissive image, and the mixing ratio of the second non-emissive image should be larger than the mixing ratio of the first non-emissive image in a pixel or region where the light intensity is low. Is preferable.

The image processing program according to still another aspect of the present invention has a function of acquiring a plurality of non-emission images captured under different exposure conditions under non-emission of flash light, which are images of the same subject. , A function to acquire a luminescent image captured under the same exposure conditions as the exposure condition of any of a plurality of non-luminous images under the emission of flash light, and distance distribution information indicating the distribution of the distance of the subject. And a function to calculate the light intensity distribution information of the light shining on the subject based on the light emitting image, the non-light emitting image captured under the same exposure conditions as the light emitting image, and the distance distribution information, and the light. A computer has a function to determine the composition ratio of multiple non-emission images based on intensity distribution information and a function to synthesize multiple non-emission images according to the composition ratio to generate a composite image with an expanded dynamic range. make it happen.

In the image processing program according to still another aspect of the present invention, the function of determining the composition ratio is based on the light intensity distribution information when determining the composition ratio for each corresponding pixel or region of a plurality of non-emission images. In a pixel or region with high light intensity, the mixing ratio of the first non-emission image captured under the exposure condition with the smallest exposure value among the plurality of non-emission images is determined by the most exposure value among the plurality of non-emission images. The mixing ratio of the second non-emissive image captured under a large exposure condition should be larger than the mixing ratio of the second non-emissive image, and the mixing ratio of the second non-emissive image should be larger than the mixing ratio of the first non-emissive image in a pixel or region where the light intensity is low. Is preferable.

According to the present invention, since the composition ratio of a plurality of non-emission images is determined according to the brightness (light intensity) of the light hitting the subject and HDR composition is performed, the brightness of the subject is not impaired and the subject is not impaired. It is possible to acquire an HDR image with excellent gradation characteristics, in which the difference in brightness is adjusted according to the brightness of the light hitting the light.

FIG. 1 is a perspective view of the image pickup apparatus according to the present invention as viewed diagonally from the front. FIG. 2 is a rear view of the image pickup apparatus. FIG. 3 is a block diagram showing an embodiment of the internal configuration of the image pickup apparatus. FIG. 4 is a front view showing a configuration example of the image sensor. FIG. 5 is a functional block diagram showing a first embodiment of a main body side CPU 220 that functions as an image processing device according to the present invention. 6 is a graph showing the relationship between the mixing ratio αi of the light intensity distribution information S _i and a second non-emission image. FIG. 7 is a graph showing another relationship between the light intensity distribution information Si and the mixing ratio αi of the second non-emission image. FIG. 8 is a graph showing still another relationship between the light intensity distribution information Si and the mixing ratio αi of the second non-emission image. FIG. 9 is a functional block diagram showing a second embodiment of the main body side CPU 220 that functions as the image processing device according to the present invention. FIG. 10 is a frequency distribution diagram showing an example of the frequency distribution of light intensity. FIG. 11 is a graph showing still another relationship between the light intensity distribution information Si and the mixing ratio αi of the second non-emission image. FIG. 12 is a frequency distribution diagram similar to the frequency distribution of the light intensity shown in FIG. It is a frequency distribution diagram which shows an example of the frequency distribution of light intensity. FIG. 13 is a graph showing still another relationship between the light intensity distribution information Si and the mixing ratio αi of the second non-emission image. FIG. 14 is a functional block diagram showing a third embodiment of the main body side CPU 220 that functions as the image processing device according to the invention. FIG. 15 is a functional block diagram showing a fourth embodiment of the main body side CPU 220 that functions as the image processing device according to the invention. FIG. 16 is a functional block diagram showing a fifth embodiment of the main body side CPU 220 functioning as the image processing device according to the invention. FIG. 17 is a flowchart showing an embodiment of the image processing method according to the present invention. FIG. 18 is a flowchart showing a detailed operation in step S12 of FIG. FIG. 19 is an external view of a smartphone according to an embodiment of the imaging device according to the present invention. FIG. 20 is a block diagram showing the configuration of a smartphone. FIG. 21 is a diagram used to explain the problems of conventional HDR synthesis.

Hereinafter, preferred embodiments of the image processing apparatus, method, program, and imaging apparatus according to the present invention will be described with reference to the accompanying drawings.

<Appearance of imaging device>

FIG. 1 is a perspective view of the image pickup apparatus according to the present invention as viewed diagonally from the front, and FIG. 2 is a rear view of the image pickup apparatus.

As shown in FIG. 1, the image pickup apparatus 10 is a mirrorless digital single-lens camera composed of an interchangeable lens 100 and a camera body 200 to which the interchangeable lens 100 can be attached and detached.

In FIG. 1, a main body mount 260 on which an interchangeable lens 100 is mounted, a finder window 20 of an optical finder, and the like are provided on the front surface of the camera main body 200, and a shutter release switch 22 and a shutter are mainly provided on the upper surface of the camera main body 200. A speed dial 23, an exposure compensation dial 24, a power lever 25, and a built-in flash 30 are provided.

Further, as shown in FIG. 2, a liquid crystal monitor 216, an eyepiece 26 of an optical viewfinder, a MENU / OK key 27, a cross key 28, a play button 29, and the like are mainly provided on the back surface of the camera body 200.

The liquid crystal monitor 216 displays a live view image in the shooting mode, reproduces and displays the image captured in the playback mode, functions as a display unit for displaying various menu screens, and provides various information to the user. It functions as a notification unit to notify. The MENU / OK key 27 has both a function as a menu button for issuing a command to display a menu on the screen of the liquid crystal monitor 216 and a function as an OK button for instructing confirmation and execution of selected contents. The key. The cross key 28 is an operation unit for inputting instructions in four directions of up, down, left, and right, and functions as a multifunction key for selecting an item from the menu screen or instructing selection of various setting items from each menu. In addition, the up and down keys of the cross key 28 function as a zoom switch during imaging or a playback zoom switch during playback mode, and the left and right keys are frame advance (forward and reverse direction) buttons during playback mode. Functions as. It also functions as an operation unit for designating an arbitrary subject for which focus adjustment is performed from a plurality of subjects displayed on the liquid crystal monitor 216.

Further, by using the menu screen displayed on the MENU / OK key 27, the cross key 28, and the liquid crystal monitor 216, in addition to the still image imaging mode for capturing one still image, the still image is continuously photographed. Set various imaging modes including continuous shooting mode, which is a type of continuous shooting mode, HDR imaging mode that captures multiple images used for HDR composition that expands the dynamic range, and moving image imaging mode that captures moving images. Can be done.

The play button 29 is a button for switching to a play mode in which the recorded still image or moving image is displayed on the liquid crystal monitor 216.

<Internal configuration of imaging device>

[interchangeable lens]

FIG. 3 is a block diagram showing an embodiment of the internal configuration of the image pickup apparatus 10.

The interchangeable lens 100 that functions as an imaging optical system constituting the imaging device 10 is manufactured in accordance with the communication standard of the camera body 200, and can communicate with the camera body 200 as described later. It is an interchangeable lens. The interchangeable lens 100 includes an imaging optical system 102, a focus lens control unit 116, an aperture control unit 118, a lens side CPU (Central Processing Unit) 120, a flash ROM (Read Only Memory) 126, a lens side communication unit 150, and a lens mount. It is equipped with 160.

The imaging optical system 102 of the interchangeable lens 100 includes a lens group 104 including a focus lens and an aperture 108.

The focus lens control unit 116 moves the focus lens according to a command from the lens side CPU 120, and controls the position (focus position) of the focus lens. The aperture control unit 118 controls the aperture 108 according to a command from the lens-side CPU 120.

The lens-side CPU 120 controls the interchangeable lens 100 in an integrated manner, and incorporates a ROM 124 and a RAM (Random Access Memory) 122.

The flash ROM 126 is a non-volatile memory for storing programs and the like downloaded from the camera body 200.

The lens-side CPU 120 collectively controls each part of the interchangeable lens 100 with the RAM 122 as a work area according to a control program stored in the ROM 124 or the flash ROM 126.

The lens-side communication unit 150 is connected to the camera body 200 via a plurality of signal terminals (lens-side signal terminals) provided on the lens mount 160 in a state where the lens mount 160 is mounted on the body mount 260 of the camera body 200. To communicate. That is, the lens-side communication unit 150 receives a request signal and a response signal from the main body-side communication unit 250 of the camera body 200 connected via the lens mount 160 and the main body mount 260 in accordance with the command of the lens-side CPU 120. Transmission / reception (two-way communication) is performed, and lens information (position information of the focus lens, focal length information, aperture information, etc.) of each optical member of the imaging optical system 102 is notified to the camera body 200.

Further, the interchangeable lens 100 includes a detection unit (not shown) that detects the position information of the focus lens and the aperture information. Here, the diaphragm information is information indicating the diaphragm value (F value) of the diaphragm 108, the aperture diameter of the diaphragm 108, and the like.

It is preferable that the lens-side CPU 120 holds various lens information including the detected focus lens position information and aperture information in the RAM 122 in order to respond to the request for lens information from the camera body 200. Further, the lens information is detected when there is a request for lens information from the camera body 200, or is detected when the optical member is driven, or at a constant cycle (a cycle sufficiently shorter than the frame cycle of the moving image). It is detected and the detection result can be retained.

[Camera body]

The camera body 200 constituting the image pickup apparatus 10 shown in FIG. 3 includes an image sensor 201, an image sensor control unit 202, an analog signal processing unit 203, an A / D (Analog / Digital) converter 204, an image input controller 205, and a digital signal. Processing unit 206, RAM 207, compression / decompression processing unit 208, media control unit 210, memory card 212, display control unit 214, liquid crystal monitor 216, main unit CPU 220, operation unit 222, clock unit 224, flash ROM 226, ROM 228, AF (Autofocus) ) Control unit 230, AE (Auto Exposure) control unit 232, white balance correction unit 234, wireless communication unit 236, GPS (Global Positioning System) receiver unit 238, power supply control unit 240, battery 242, main unit side communication unit 250, main unit It includes a mount 260, a flash light emitting unit 270 constituting the built-in flash 30 (FIG. 1), a flash control unit 272, a focal-plane shutter (FPS) 280, and an FPS control unit 296.

<Configuration of image sensor>

The image sensor 201 is composed of a CMOS (Complementary Metal-Oxide Semiconductor) type color image sensor. The image sensor 201 is not limited to the CMOS type, but may be an XY address type or a CCD (Charge Coupled Device) type image sensor.

FIG. 4 is a front view showing the configuration of the image sensor 201.

As shown in FIG. 4, the image sensor 201 is an image sensor having a plurality of pixels composed of photoelectric conversion elements (light receiving cells) arranged two-dimensionally in the horizontal direction (x direction) and the vertical direction (y direction). a is, the light beam incident thereon through the interchangeable lens 100, and a first pixel S _a, include a second pixel S _B selectively received by pupil division by the pupil dividing means described below.

As shown in FIG. 4, the image sensor 201 is applied with a pupil image separation method using a pupil imaging lens (microlens) 201A as a pupil dividing means.

Pupil image separation method, pixels of a plurality for one microlens 201A (2 two in this example) (the first pixel _{S A,} second pixel _{S B)} is assigned, incident to one microlens 201A pupil image which is pupil division by the microlens 201A, the first pixel _{S a} two (a pair), and is received by the second pixel _{S B.} Thus, the pupil image is separated according to the incident angle of the light entering the microlens 201A, the corresponding pair of the first pixel S _A, is received by the second pixel S _B.

A pair of first pixel S _A, second pixel S _B corresponds to a pair of phase difference pixels available on the phase difference AF. The pair of first pixel S _A, pixel values obtained by adding the pixel values obtained from the pixels of the second pixel S _B are to become equal to the pixel value of normal pixels that are not pupil division, the pair of first pixel _{S A,} second pixel _{S B} can be treated as a normal pixel (1 pixel).

A pair of first pixel _{S A} of the image sensor 201 as shown in FIG. _4, the second pixel _{S B,} red

A color filter of any one of the three primary color filters (R filter, G filter, B filter) of (R), green (G), and blue (B) is arranged according to a predetermined color filter arrangement. ing. The color filter array shown in FIG. 4 is a general Bayer array, but the color filter array is not limited to this, and may be another color filter array such as a Trans (registered trademark) array.

Returning to FIG. 3, the optical image of the subject imaged on the light receiving surface of the image sensor 201 by the imaging optical system 102 of the interchangeable lens 100 is converted into an electric signal by the image sensor 201. (First pixel S _A shown in FIG. _4, the second pixel S _B) each pixel of the image sensor 201, the charge corresponding to the amount of light incident are accumulated, the charge accumulated in each pixel from the image sensor 201 An electric signal corresponding to the amount (signal charge) is read out as an image signal. That is, the first pixel S _A, a signal corresponding to the charge amount accumulated in the second pixel S _B (first pixel S _A, pixel values of the second pixel S _B) can be read independently.

The image sensor control unit 202 performs read control of an image signal from the image sensor 201 according to a command from the CPU 220 on the main body side. Further, when the image sensor control unit 202 captures a still image, the exposure time is controlled by opening and closing the FPS 280, and then the entire line of the image sensor 201 is read out with the FPS 280 closed. Further, the image sensor 201 and the image sensor control unit 202 of this example sequentially perform an exposure operation for at least one or more lines or pixels (that is, reset each line or pixel sequentially to accumulate charges. It can be driven by the so-called rolling shutter method (which is a method of starting and reading out the accumulated charge), and in particular, a function of capturing a moving image or a live view image by the rolling shutter method with the FPS280 open. Has.

The analog signal processing unit 203 performs various analog signal processing on the analog image signal obtained by imaging the subject with the image sensor 201. The analog signal processing unit 203 includes a sampling hold circuit, a color separation circuit, an AGC (Automatic Gain Control) circuit, and the like. The AGC circuit functions as a sensitivity adjustment unit that adjusts the sensitivity at the time of imaging (ISO (International Organization for Standardization)), adjusts the gain of the amplifier that amplifies the input image signal, and adjusts the signal level of the image signal. Make sure it is within the appropriate range. The A / D converter 204 converts the analog image signal output from the analog signal processing unit 203 into a digital image signal.

The image data (mosaic image data) for each pixel of RGB output via the image sensor 201, the analog signal processing unit 203, and the A / D converter 204 when capturing a still image or moving image is obtained from the image input controller 205 to the RAM 207. Is entered in and is temporarily stored.

When the image sensor 201 is a CMOS image sensor, the analog signal processing unit 203 and the A / D converter 204 are often built in the image sensor 201.

The digital signal processing unit 206 performs various digital signal processing on the image data stored in the RAM 207. The digital signal processing unit 206 appropriately reads out the image data stored in the RAM 207, and for the read image data, offset processing, gain control processing including sensitivity correction, gamma correction processing, demosaicing processing (demosaiking processing, simultaneous). Digital signal processing such as conversion processing) and RGB / YCrCb conversion processing is performed, and the image data after the digital signal processing is stored in the RAM 207 again. The demosaic process is, for example, in the case of an image sensor composed of color filters of three RGB colors, a process of calculating all RGB color information for each pixel from a mosaic image composed of RGB, and mosaic data (point-sequential RGB data). ) To generate the image data of three RGB planes simultaneously.

The RGB / YCrCb conversion process is a process of converting the simultaneous RGB data into luminance data (Y) and color difference data (Cr, Cb).

The compression / decompression processing unit 208 performs compression processing on the uncompressed luminance data Y and the color difference data Cb and Cr once stored in the RAM 207 when recording a still image or a moving image. In the case of a still image, for example, it is compressed in the JPEG (Joint Photographic coding Experts Group) format, and in the case of a moving image, for example, H. Compress in 264 format. The image data compressed by the compression / decompression processing unit 208 is recorded in the memory card 212 via the media control unit 210. Further, the compression / decompression processing unit 208 performs decompression processing on the compressed image data obtained from the memory card 212 via the media control unit 210 in the playback mode to generate uncompressed image data.

The media control unit 210 controls to record the image data compressed by the compression / decompression processing unit 208 on the memory card 212. Further, the media control unit 210 controls to read the compressed image data from the memory card 212.

The display control unit 214 controls the liquid crystal monitor 216 to display the uncompressed image data stored in the RAM 207. Although the liquid crystal monitor 216 is composed of a liquid crystal display device, it may be configured by a display device such as organic electroluminescence instead of the liquid crystal monitor 216.

When the live view image is displayed on the liquid crystal monitor 216, the digital image signal continuously generated by the digital signal processing unit 206 is temporarily stored in the RAM 207. The display control unit 214 converts the digital image signal temporarily stored in the RAM 207 into a signal format for display, and sequentially outputs the digital image signal to the liquid crystal monitor 216. As a result, the captured image is displayed on the liquid crystal monitor 216 in real time, and the liquid crystal monitor 216 can be used as an electronic viewfinder.

The shutter release switch 22 is an imaging instruction unit for inputting an imaging instruction for a still image or a moving image, and is composed of a two-stage stroke type switch composed of a so-called "half-press" and "full-press".

In the still image imaging mode, the shutter release switch 22 is half-pressed to output an S1 on signal, and half-pressed to a full-pressed output to output an S2 on signal and an S1 on signal. Then, the CPU 220 on the main body side executes shooting preparation processing such as AF control (automatic focus adjustment) and AE control (automatic exposure control), and when an S2 on signal is output, executes still image imaging processing and recording processing. To do.

The AF control and the AE control are automatically performed when the auto mode is set by the operation unit 222, respectively, and the AF control and the AE control are not performed when the manual mode is set. Needless to say.

Further, in the moving image imaging mode, when the S2 ON signal is output by fully pressing the shutter release switch 22, the camera body 200 enters the moving image recording mode for starting the moving image recording, and the moving image processing of the moving image is performed. And, when the recording process is executed and then the S2 ON signal is output by fully pressing the shutter release switch 22 again, the camera body 200 enters the standby state and pauses the moving image recording process.

The shutter release switch 22 is not limited to the form of a two-stage stroke type switch consisting of a half-press and a full-press, and may output an S1 on signal and an S2 on signal with a single operation, and each of them may be output individually. A switch may be provided to output an S1 on signal and an S2 on signal.

Further, in the form of giving an operation instruction by a touch panel or the like, the operation instruction may be output by touching the area corresponding to the operation instruction displayed on the screen of the touch panel as these operation means. The form of the operating means is not limited to these as long as it instructs the preparatory process and the imaging process.

The still image or moving image acquired by imaging is compressed by the compression / decompression processing unit 208, and the compressed image data is the necessary attachment of the imaging date / time, GPS information, and imaging conditions (F value, shutter speed, ISO sensitivity, etc.). The information is converted into an image file added to the header, and then stored in the memory card 212 via the media control unit 210.

The CPU 220 on the main body side comprehensively controls the operation of the entire camera main body 200 and the drive of the optical member of the interchangeable lens 100. Based on the input from the operation unit 222 including the shutter release switch 22, each part of the camera main body 200 and each part and the like. Controls the interchangeable lens 100.

As a timer, the clock unit 224 measures the time based on a command from the CPU 220 on the main body side. In addition, the clock unit 224 measures the current date and time as a calendar.

The flash ROM 226 is a non-volatile memory that can be read and written, and stores setting information.

The ROM 228 contains a camera control program executed by the CPU 220 on the main body side, an image alignment assist program for executing imaging in the composite image imaging mode according to the present invention, defect information of the image sensor 201, various types used for image processing, and the like. Parameters and tables are stored. The main body side CPU 220 controls each part of the camera main body 200 and the interchangeable lens 100 while using the RAM 207 as a work area according to the camera control program stored in the ROM 228 or the image alignment assist program.

The AF control unit 230, which functions as an automatic focus adjustment unit, calculates the defocus amount required for controlling the phase difference AF, and based on the calculated defocus amount, commands the position (focus position) at which the focus lens should move. Is notified to the interchangeable lens 100 via the main body side CPU 220 and the main body side communication unit 250.

Here, the AF control unit 230 includes a phase difference detection unit and a defocus amount calculation unit. The phase difference detection unit is within the AF area of the image sensor 201 (the area where the main subject specified by the user exists, the area of the main subject automatically detected by face detection, or the area set by default). a first pixel group including a first pixel S _a which color filters of the same color are disposed, respectively the pixel data from the second pixel group composed of the first pixel S _a (the first pixel value and second pixel value) It is acquired and the phase difference is detected based on these first pixel value and second pixel value. This phase difference is obtained when the correlation between the plurality of first pixel values of the first pixel group and the plurality of second pixel values of the second pixel group is maximized (the plurality of first pixel values and the plurality of second pixel values). It can be calculated from the shift amount in the pupil division direction between the first pixel value and the second pixel value (when the integrated value of the absolute difference value becomes the minimum).

The defocus amount calculation unit calculates the defocus amount by multiplying the phase difference detected by the phase difference detection unit with the coefficient corresponding to the current F value (light ray angle) of the interchangeable lens 100.

The position command of the focus lens corresponding to the defocus amount calculated by the AF control unit 230 is notified to the interchangeable lens 100, and the lens-side CPU 120 of the interchangeable lens 100 that receives the position command of the focus lens is the focus lens control unit 116. The focus lens is moved via the lens to control the position (focus position) of the focus lens.

The AE control unit 232 is a part that detects the brightness of the subject (subject brightness), and is a numerical value (exposure value) required for AE control and AWB (Auto White Balance) control corresponding to the subject brightness. )) Is calculated. The AE control unit 232 calculates the EV value from the brightness of the image acquired via the image sensor 201, the shutter speed at the time of acquiring the brightness of the image, and the F value.

The main body side CPU 220 can determine the F value, the shutter speed, and the ISO sensitivity from a predetermined program diagram based on the EV value obtained from the AE control unit 232, and perform AE control.

The white balance correction unit 234 calculates the white balance gain (WB (White Balance) gain) Gr, Gg, Gb for each color data of RGB data (R data, G data and B data), and calculates the R data, G data and Gb. White balance correction is performed by multiplying the B data by the calculated WB gains Gr, Gg, and Gb, respectively. Here, as a method of calculating the WB gains Gr, Gg, and Gb, the subject is illuminated based on the scene recognition (outdoor / indoor determination, etc.) based on the brightness (EV value) of the subject and the color temperature of the ambient light. A method is conceivable in which the WB gain corresponding to the specified light source type is read out from the storage unit in which the appropriate WB gain is stored in advance for each light source type, but at least the EV value is used to read the WB gain. Other known methods for determining Gr, Gg, Gb can be considered.

The wireless communication unit 236 is a part that performs short-range wireless communication of standards such as Wi-Fi (Wireless Fidelity) (registered trademark) and Bluetooth (registered trademark), and is connected to peripheral digital devices (mobile terminals such as smartphones). Send and receive necessary information between.

The GPS receiving unit 238 receives GPS signals transmitted from a plurality of GPS satellites according to instructions from the CPU 220 on the main body side, executes positioning calculation processing based on the received plurality of GPS signals, and executes positioning calculation processing based on the received GPS signals, and the latitude and longitude of the camera main body 200. , And GPS information consisting of altitude is acquired. The acquired GPS information can be recorded in the header of the image file as ancillary information indicating the imaging position of the captured image.

The power supply control unit 240 applies the power supply voltage supplied from the battery 242 to each part of the camera main body 200 according to the command of the main body side CPU 220. Further, the power supply control unit 240 applies the power supply voltage supplied from the battery 242 to each part of the interchangeable lens 100 via the main body mount 260 and the lens mount 160 in accordance with the command of the main body side CPU 220.

The lens power switch 244 switches the power supply voltage applied to the interchangeable lens 100 on and off and switches the level via the main body mount 260 and the lens mount 160 in accordance with the command of the main body side CPU 220.

The main body side communication unit 250 transmits and receives a request signal and a response signal between the main body side communication unit 250 and the lens side communication unit 150 of the interchangeable lens 100 connected via the main body mount 260 and the lens mount 160 according to a command from the main body side CPU 220. Two-way communication). As shown in FIG. 1, the main body mount 260 is provided with a plurality of terminals 260A, and when the interchangeable lens 100 is attached to the camera main body 200 (the lens mount 160 and the main body mount 260 are connected), the main body is mounted. A plurality of terminals 260A (FIG. 1) provided on the mount 260 and a plurality of terminals (not shown) provided on the lens mount 160 are electrically connected, and the main body side communication unit 250 and the lens side communication unit 150 are electrically connected. Two-way communication with and from is possible.

The built-in flash 30 (FIG. 1) is, for example, a TTL (Through The Lens) automatic dimming flash, and includes a flash light emitting unit 270 and a flash control unit 272.

The flash control unit 272 has a function of adjusting the amount of flash light (guide number) emitted from the flash light emitting unit 270. That is, the flash control unit 272 pre-emits (dimming) a small amount of flash light from the flash light emitting unit 270 in synchronization with the flash imaging instruction from the main body side CPU 220, and the imaging optical system 102 of the interchangeable lens 100. The amount of light emitted from the flash light to be actually emitted is determined based on the reflected light (including ambient light) incident on the flash, and the determined amount of flash light is emitted from the flash light emitting unit 270 (main emission). When the HDR imaging mode is selected, the flash control unit 272 performs emission control different from the imaging of a normal flash emission image, the details of which will be described later.

The FPS 280 constitutes the mechanical shutter of the image pickup apparatus 10 and is arranged immediately before the image sensor 201. The FPS control unit 296 controls the opening and closing of the front curtain and the rear curtain of the FPS 280 based on the input information (S2 on signal, shutter speed, etc.) from the CPU 220 on the main body side, and controls the exposure time (shutter speed) in the image sensor 201. To do.

Next, the control of the imaging device 10 that captures a plurality of images with different exposures used for HDR composition that widens the dynamic range by the HDR imaging mode and performs HDR composition of the captured plurality of images will be described.

[First Embodiment of HDR processing]

FIG. 5 is a functional block diagram showing a first embodiment of the main body side CPU 220 that functions as an image processing device according to the present invention, and is an HDR composition based mainly on a plurality of images for HDR composition captured in the HDR imaging mode. It is a functional block diagram at the time of executing the image processing (HDR processing) for.

When imaging in the HDR imaging mode, the CPU 220 on the main body side mainly controls the control for imaging in the HDR imaging mode, and as shown in FIG. 5, the distance distribution information acquisition unit 220A, the light intensity distribution information calculation unit 220B, It functions as a composition ratio determination unit 220C and an image composition unit 220D.

That is, when performing imaging in the HDR imaging mode, the CPU 220 on the main body side captures a plurality of (two in this example) non-emission images under different exposure conditions under the non-emission of the flash light, and under the emission of the flash light. Capturing a light emitting image under the same exposure conditions as the exposure condition of any of the non-light emitting images among the plurality of non-light emitting images is executed.

Here, the plurality of exposure conditions corresponding to the plurality of non-emission images include an underexposure exposure condition and an overexposure exposure condition rather than the proper exposure exposure condition. In this example, the CPU 220 on the main body side captures two non-emission images under two exposure conditions, underexposure and overexposure, with respect to the appropriate exposure determined according to the EV value calculated by the AE control unit 232. Let me do it. The exposure compensation value (for example, ± 3EV) that causes underexposure or overexposure with respect to the proper exposure may be arbitrarily set by the user, or is automatically set according to the brightness (EV value) of the field of view. It may be possible to set the target.

Further, the CPU 220 on the main body side displays the light emitting image under the same exposure condition (in this example, the underexposure exposure condition) as the exposure condition of any one of the plurality of non-light emitting images under the light emission of the flash light. Perform imaging.

The two non-emission images and one emissive image are images captured for the same subject, and are continuously captured images with the imaging interval as short as possible. preferable.

In FIG. 5, the image acquisition unit 221 is overexposed with a non-emission image (first non-emission image) captured by an imaging unit (interchangeable lens 100, image sensor 201) under non-emission of flash light. Under the light emission of the non-emission image (second non-emission image) captured by exposure and the flash light emitted from the flash light emitting unit 270, the image is captured by the imaging unit under the same exposure conditions as the exposure condition of the first non-emission image. This is the part to acquire the light emission image.

The image acquisition unit 221 may acquire the first non-emission image, the second non-emission image, and the emissive image temporarily held in the RAM 207 via, for example, the image input controller 205 from the RAM 207, or the image acquisition unit may acquire the image. The captured first non-emission image, second non-emission image, and emissive image may be directly acquired via the image input controller 205.

The image input controller 205 of this embodiment, a pair of first pixel _{S A} of the image sensor 201 shown in FIG. 4, by adding the pixel value of the second pixel _{S B,} 1 pixel per one microlens 201A In addition to the first non-emission image, the second non-emission image, and the emissive image, which are the pixel values, the first of the first non-emission image, the second non-emission image, and the emissive image. acquiring a second image comprising a first image and a second pixel S _B only pixel groups consisting of the pixel group of only the pixel S _a. The first image and the second image are phase difference images having a phase difference according to the subject distance.

The distance distribution information acquisition unit 220A acquires a first image and a second image, which are retardation images having a phase difference from each other, from the image acquisition unit 221, and based on these first and second images, the distance distribution information acquisition unit 220A in the image. Acquires distance distribution information indicating the distribution of the distance of the subject.

That is, the distance distribution information acquisition unit 220A calculates the amount of deviation (phase difference) between the feature points in the first image and the feature points in the second image corresponding to each feature point, and the calculated position. The defocus amount of each feature point is calculated based on the phase difference and the F value of the current aperture 108, and the defocus amount of each feature point calculated further and the lens of the current interchangeable lens 100 (focus lens of the imaging optical system 102) The distance of each feature point (subject) is calculated from the position. For example, a feature point with a defocus amount of zero is at the focusing distance at the lens position of the current focus lens, and as the defocus amount of each feature point shifts to the front pin side or the rear bin side. , The distance of each feature point is the distance moved to the far side or the near side from the focusing distance.

The light intensity distribution information calculation unit 220B acquires a light emitting image and a first non-light emitting image captured under the same exposure conditions from the image acquisition unit 221 and shows the distribution of the distance of the subject in the image from the distance distribution information acquisition unit 220A. Acquire distance distribution information. Then, based on the acquired light emission image, the first non-light emission image, and the distance distribution information, the light intensity distribution information which is the distribution information of the index indicating the intensity of the light (ambient light, not the flash light) shining on the subject. Is calculated.

Hereinafter, the method of calculating the light intensity distribution information by the light intensity distribution information calculation unit 220B will be described in detail.

A parameter indicating the coordinates of each pixel of the first non-emission image and i, the luminance value of the pixel i Y _i, the luminance value Y _Li pixel i of the same coordinates of the luminescent _image, the coordinate distance information d _{i (=} When the distance from the image pickup device 10 to the subject reflected therein), the light intensity distribution information Si is _expressed by the following equation.

[Number 1]

_{S i = Y i / {(} Y Li -Y i) × (d i 2)}

Defined by. The parameter i is not limited to the parameter indicating the coordinates of the pixels, and may be a parameter indicating the position of each region obtained by dividing the image into a plurality of regions. In this case, the luminance value _{Y i, Y Li,} distance information _{d i,} and the light intensity distribution information _{S i} be the luminance value of each region i (representative luminance value), distance information, and the light intensity distribution information.

The equation [Equation 1] will be described below.

Yi of the first non-emission image, the intensity of the B _i of the light hitting the object _there, the reflectance R _i of the _object, the photoelectric conversion coefficient by the image sensor 201 and K, Yi of the first non-emission image Is expressed by the following equation.

[Number 2]

Y _i = _Bi x R _i x K

Y _Li of the light-emitting image when the intensity of the flash light hitting the object and F _i, is expressed by the following equation.

[Number 3]

Y _Li = (B _i + _Fi ) x R _i x K

F _i, when the Fs strength of the flash light in a place unit distance away from the flash light emitting section 270, from the inverse-square law regarding attenuation of light is represented by the following equation.

[Number 4]

^{_{F i = Fs / (d i}} 2)

The equation [Equation 1] of the light intensity distribution information can be transformed into the equation [Equation 5] by the equations [Equation 2] to [Equation 4].

[Number 5]

_{S i = Y i / {(} Y Li -Y i) × (d i 2)} = (B i × R i × K) / [{(B i + F i) × R i × K- (B i × _{R i × K)} × (} d i 2)] = (B i / {F i × (d i 2)} = B i / Fs

Regardless of the pixel i, since Fs is constant, the light intensity distribution information S _i, the pixel i (= 1~N (N: it can be said that the index can be compared relatively light intensity B _i that is hitting the subject per pixel number) ..

The composition ratio determining unit 220C determines the composition ratio of a plurality of non-emission images (first non-emission image and second non-emission image) based on the light intensity distribution information. Since the part where the value of the light intensity distribution information S _i is large is exposed to strong light, it is preferable to use the first non-emission image which is an under image, and the part where the value of the light intensity distribution information S _i is small is too light. Since it does not hit, it is preferable to use a second non-emission image which is an over image.

Next, a method of determining the synthesis ratio by the synthesis ratio determination unit 220C will be described.

<First method for determining the synthesis ratio>

Synthesis ratio determining unit 220C, in determining the respective combination ratio for each pixel of the plurality of non-emission image, among the plurality of non-light emission image at the pixel light intensity is high on the basis of the light intensity distribution information S _i The mixing ratio of the first non-emission image captured under the exposure condition with the smallest exposure value is larger than the mixing ratio of the second non-emission image captured under the exposure condition with the largest exposure value among the plurality of non-emission images. For pixels that are large and have low light intensity, the mixing ratio of the second non-emission image is made larger than the mixing ratio of the first non-emission image. The composition ratio may be determined for each region in which the image is divided into a plurality of regions.

FIGS. 6 7 are each graphs showing the relationship between the mixing ratio αi of the light intensity distribution information S _i and a second non-emission image.

Synthesis ratio determining unit 220C as shown in FIG. 6, the mixing ratio of the second non-emission image and .alpha.i, minimum value S _min of the light intensity distribution information S _i, as shown in _FIG., The maximum value S _{max .} and when, most briefly, the minimum value S _min. ~ maximum value S _max. of the light intensity distribution information S _i, the mixing ratio αi a (= 0-1) by linear interpolation, the first non-emission image And the second non-emission image are determined. In this case, the mixing ratio of the first non-emission image is (1-αi), and the composite ratio of the first non-emission image and the second non-emission image is (1-αi): αi.

Further, as shown in FIG. 7, the synthesis ratio determining unit 220C sets a minimum value α _min. (> 0) and a maximum value α _max. (<1) in the mixing ratio αi, and the minimum value S _{min. To the} maximum value S. _{max by.} the light intensity distribution information S _i, the mixing ratio αi minimum value alpha _min. by linear interpolation the maximum value alpha _max. range between the synthesis ratio of the first non-emission image and the second non-emission image Can be determined.

Further, the synthesis ratio determining unit 220C sets a minimum value α _min. (> 0) and a maximum value α _max. (<1) in the mixing ratio αi as shown in FIG. 8, and the minimum value S _{min. To the} maximum value S. _{The first is} performed by non-linear interpolation (non-linear interpolation so that the mixing ratio αi decreases monotonically) in the range of the minimum value α _{min. To the} maximum value α _max. By the light intensity distribution information S _i of _max. The composite ratio of the non-emission image and the second non-emission image can be determined.

Returning to FIG. 5, the image compositing unit 220D synthesizes the underexposed first non-emission image and the overexposed second non-emission image according to the compositing ratio determined by the compositing ratio determining unit 220C, and has a dynamic range. Generates an enlarged composite image (HDR image). That is, the image synthesizing unit 220D synthesizes the first non-emission image and the second non-emission image by applying the composition ratio determined for each corresponding pixel of the first non-emission image and the second non-emission image. When the compositing ratio is determined for each region in which the image is divided into a plurality of regions, the compositing ratio determined for each corresponding region of the first non-emission image and the second non-emission image is applied to the first. The non-emission image and the second emission image are combined to generate an HDR image.

The HDR image generated in this way is an HDR image having excellent gradation characteristics, in which the difference in brightness is adjusted according to the brightness of the light hitting the subject without impairing the brightness of the subject.

[Second Embodiment of HDR Processing]

FIG. 9 is a functional block diagram showing a second embodiment of the main body side CPU 220 that functions as the image processing device according to the present invention. In FIG. 9, the same reference numerals are given to the parts common to those of the first embodiment shown in FIG. 5, and detailed description thereof will be omitted.

The main body side CPU 220 of the second embodiment shown in FIG. 9 is different from the first embodiment shown in FIG. 5 in that the function of the distribution information creating unit 220E is added.

Distribution information creating unit 220E calculates the distribution information indicating a frequency distribution of light intensity based on the light intensity distribution information calculated by the light intensity distribution information calculation unit 220B S _i (e.g., histogram or frequency distribution table) ..

FIG. 10 is a frequency distribution diagram showing an example of the frequency distribution of light intensity.

For example, a dark subject (shaded subject) that is not exposed to sunlight outdoors during the day on a sunny day.

A subject with a wide dynamic range is the subject to be imaged, from a subject to a bright subject exposed to sunlight (a subject in the sun), and the subject to be imaged in this case can be roughly divided into a subject in the shade and a subject in the sun. As shown in FIG. 10, the frequency distribution map of the light intensity generated based on the light intensity distribution information of such a scene has two peaks (the first vertex P1 and the second vertex P2 which are the two vertices). To do. The first vertex P1 is the vertex corresponding to the power of the light intensity distribution information of the subject area corresponding to the shaded subject, and the second vertex P2 is the vertex corresponding to the power of the light intensity distribution information of the subject area corresponding to the subject in the sun. The corresponding vertices.

In addition, not only the outdoor scene during the daytime on a sunny day, but also the scene where sunlight shines into a part of the room, the scene where there is an area illuminated by an artificial light source and the area where it is shaded, etc. There can be two vertices in the light intensity histogram.

In this example, the distribution information showing the frequency distribution of the light intensity is created, but the distribution information showing the frequency distribution of the light intensity may be created, or the frequency distribution or the distribution information obtained by approximating the frequency distribution by a curve is created. You may.

The composite ratio determining unit 220C shown in FIG. 9 has a first light intensity (S) corresponding to the first vertex P1 having a high luminous intensity as shown in FIG. 10 based on the distribution information created by the distribution information creating unit 220E. _dark ) and the second light intensity (S _blight ) corresponding to the second vertex P2 having a high light intensity frequency are obtained.

Then, when the composition ratio determining unit 220C determines the composition ratio for each corresponding pixel of the first non-emission image and the second non-emission image, as shown in FIG. 11, the composition ratio is equal to or less than the first light intensity (S _dark ). In the case of the light intensity of, the mixing ratio αi of the second non-emission image is set to the maximum value α _max. , And in the case of the light intensity of the second light intensity (S _blight ) or more, the mixing ratio αi of the second non-emission image is set. When the minimum value is α _{min. And the} light intensity is larger than the first light intensity (S _dark ) and smaller than the second light intensity (S _blight ), the mixing ratio αi of the second non-emission image is set to the maximum value α _max. It decreases linearly (monotonically decreases) between and the minimum value α _min .

A pixel or region having a light intensity equal to or lower than the first light intensity (S _dark ) is considered to be a shaded region, and it is preferable to set the mixing ratio αi of the overexposed second non-emission image to the maximum value α _max. On the other hand, a pixel or region having a light intensity equal to or higher than the second light intensity (S _blight ) is considered to be a region of the sun, and the mixing ratio αi of the overexposed second non-emission image is set to the minimum value α _min. (Underexposure) _. It is preferable to set the mixing ratio of the first non-emission image to the maximum value). As a result, the gradation of the highlight portion and the shadow portion can be enriched.

FIG. 12 is a frequency distribution diagram similar to the frequency distribution of the light intensity shown in FIG.

In the frequency distribution map shown in FIG. 12, since there are two peaks with high frequencies, there is a valley (bottom point V) between the two peaks with low frequencies.

In another embodiment of the synthesis ratio determining unit 220C shown in FIG. 9, the light intensity corresponding to the bottom point V having a low light intensity frequency is shown in FIG. 12 based on the distribution information created by the distribution information creating unit 220E. Find (St).

Then, when the composition ratio determination unit 220C determines the composition ratio for each corresponding pixel of the first non-emission image and the second non-emission image, the light intensity (St) is set as a threshold value as shown in FIG. When the light intensity is below the threshold value, the mixing ratio αi of the second non-emission image is set to the maximum value α _max. , And when the light intensity exceeds the threshold value, the mixing ratio αi of the second non-emission image is set to the minimum value α _{min. .} to (maximum mixing ratio of the first non-emission image of underexposure).

For the light intensity in a certain range ΔS before and after the light intensity (St) which is the threshold value, the mixing ratio of the second non-emission image is set to the maximum value α _max. And the minimum value according to the magnitude of the light intensity _. It also includes the case where the mixing ratio is continuously changed with α _min .

[Third Embodiment of HDR processing]

FIG. 14 is a functional block diagram showing a third embodiment of the main body side CPU 220 that functions as the image processing device according to the present invention. In FIG. 14, the same reference numerals are given to the parts common to those of the first embodiment shown in FIG. 5, and detailed description thereof will be omitted.

The main body side CPU 220 of the third embodiment shown in FIG. 14 is different from the first embodiment shown in FIG. 5 in that the function of the luminance distribution information calculation unit 220F is added.

The luminance distribution information calculation unit 220F calculates the luminance distribution information indicating the luminance distribution in the image of at least one of the plurality of non-emission images (first non-emission image, second non-emission image). It is a part. The luminance distribution information can be created, for example, as an image composed of the luminance signals of the first non-emission image, or as information indicating the luminance (representative luminance) for each region in which the image is divided.

The composite ratio determination unit 220C shown in FIG. 14 has a plurality of non-emission images (first non-emission image) based on the light intensity distribution information calculated by the light intensity distribution information calculation unit 220B and the brightness distribution information calculated by the brightness distribution information calculation unit 220F. The composite ratio of the luminescent image and the second non-luminous image) is determined.

Specifically, the composite ratio determination unit 220C sets a parameter indicating the position of the pixel or region when determining the composite ratio for each corresponding pixel or region of the first non-emission image and the second non-emission image. Let αi be the mixing ratio of the second non-emission image based on the light intensity distribution information, and βi be the mixing ratio of the second non-emission image based on the brightness distribution information, which becomes smaller as the brightness increases. The mixing ratio γi of the second non-emission image based on the light intensity distribution information and the brightness distribution information is calculated by the following equation.

[Number 6]

Calculated by γi = αi × βi. In this case, the mixing ratio of the first non-emission image is (1-γi), and the composite ratio of the first non-emission image and the second non-emission image is (1-γi): γi.

[Fourth Embodiment of HDR processing]

FIG. 15 is a functional block diagram showing a fourth embodiment of the main body side CPU 220 that functions as the image processing device according to the present invention. In FIG. 15, the same reference numerals are given to the parts common to the third embodiment shown in FIG. 14, and detailed description thereof will be omitted.

The main body side CPU 220 of the fourth embodiment shown in FIG. 15 is different from the third embodiment shown in FIG. 14 in that the function of the flash light unreachable pixel detection unit 220G is added.

The flash light unreachable pixel detection unit 220G calculates the difference (Y _Li − Y _i ) between the brightness value Y _i of the pixel i of the first non-emission image and the brightness value of the pixel i at the same coordinates of the light emission image with Y _Li. Then, the pixel i in which the difference (Y _Li − Y _i ) becomes 0 is detected as a pixel that has not reached the flash light.

The composite ratio determination unit 220C shown in FIG. 15 has a plurality of non-emission images (first non-emission image) based on the light intensity distribution information calculated by the light intensity distribution information calculation unit 220B and the brightness distribution information calculated by the brightness distribution information calculation unit 220F. The composite ratio of the light emitting image and the second non-light emitting image) is determined. When the flash light unreachable pixel detection unit 220G detects the flash light unreachable pixel, the composite ratio with respect to the flash light unreachable pixel is determined. In addition, the composite ratio of a plurality of non-emission images is determined based only on the luminance distribution information.

If the background of the subject is far away, the flash light does not reach and (Y _Li − Y _i ) becomes 0, and the light intensity distribution information S _i shown in the equation [Equation 1] cannot be calculated. result, since the mixing ratio αi of the second non-emission image based on the light intensity distribution information S _i also becomes impossible to obtain.

Therefore, in this case, by determining the composition ratio of the plurality of non-emission images based only on the luminance distribution information, it is possible to determine an appropriate composition ratio even for a subject to which the flash light does not reach.

Specifically, the synthesis ratio determining unit 220C calculates the mixing ratio γi of the second non-emission image based on the light intensity distribution information and the brightness distribution information by the equation [Equation 6], but the pixels that the flash light does not reach. When (flash light unreachable pixel) is detected by the flash light unreachable pixel detection unit 220G, it is based on the luminance distribution information by setting the mixing ratio αi of the second non-emission image of the flash light unreachable pixel to 1. The mixing ratio γi of the second non-emission image is obtained only from the mixing ratio βi of the second non-emission image.

The flash light unreachable pixel detection unit 220G is not limited to the case of detecting the flash light unreachable pixel, and may function as a detection unit for detecting the flash light unreachable region when the image is divided into a plurality of regions. Good.

Further, the flash light unreachable pixel detection unit 220G is also used for controlling the amount of flash light emitted by the flash control unit 272 from the flash light emitting unit 270. Regarding the details of the light emission amount control by the flash control unit 272. Will be described later.

[Fifth Embodiment of HDR processing]

FIG. 16 is a functional block diagram showing a fifth embodiment of the main body side CPU 220 that functions as the image processing device according to the present invention. In FIG. 16, the same reference numerals are given to the parts common to those of the first embodiment shown in FIG. 5, and detailed description thereof will be omitted.

The main body side CPU 220 of the fourth embodiment shown in FIG. 16 does not have the distance distribution information acquisition unit 220A shown in the first embodiment, and acquires the distance distribution information from the external distance distribution information acquisition unit 223. There is.

The distance distribution information acquisition unit 223 can be configured by, for example, a distance image sensor in which a plurality of light receiving elements are arranged two-dimensionally. As the distance image sensor, a sensor that acquires a distance image by a TOF (Time Of Flight) method can be considered. The TOF method is a method of irradiating a subject with light and measuring the time until the reflected light is received by a sensor to obtain the distance to the subject. The subject is irradiated with pulsed light and a plurality of reflected lights are emitted. A method of receiving light from a distance image sensor having the above pixels and acquiring a distance image of the subject from the amount of light received (light receiving intensity) for each pixel of the distance image sensor, and a method of irradiating the subject with high-frequency modulated light and reflecting it from the time of irradiation. A method of acquiring a distance image by detecting a phase shift until light is received is known.

Further, as the distance distribution information acquisition unit 223, a three-dimensional laser measuring instrument, a stereo camera, or the like can be applied. When the distance distribution information acquisition unit 223 is used, the image sensor 201 is not limited to the image sensor as in this example, but may be an image sensor that cannot measure the distance.

[Image processing method]

FIG. 17 is a flowchart showing an embodiment of the image processing method according to the present invention. Hereinafter, the operation of HDR processing by the main body side CPU 220 having the functions of each part shown in FIG. 5 and the like will be described.

In FIG. 17, the distance distribution information acquisition unit 220A of the main body side CPU 220 acquires the distance distribution information indicating the distance distribution of the subject in the image (step S10). In this example, the distance distribution information acquisition unit 220A acquires a first image and a second image, which are retardation images having a phase difference from each other, from the image acquisition unit 221 and is based on these first and second images. The distance distribution information indicating the distribution of the distance of the subject in the image is acquired.

Further, a plurality of non-emission images (first non-emission image) taken by the image acquisition unit 221 under different exposure conditions under the non-emission of the flash light, which are images of the same subject under different imaging conditions. And the second non-emission image), and the same exposure conditions as any one of the non-emission images (in this example, the exposure condition of the underexposed first non-emission image) under the light emission of the flash light. The light emission image captured by the above is acquired (step S12).

Next, the detailed operation in step S12 will be described with reference to the flowchart shown in FIG.

In FIG. 18, the flash control unit 272 determines the dimming light emission amount for the dimming image to be emitted from the flash light emitting unit 270 based on the distance distribution information acquired in step S10.

(Step S100). The dimming light emission amount is calculated as the maximum light emission amount that does not cause overexposure from the distance distribution, assuming that the average reflectance of the subject within the angle of view is 18%.

Next, the flash control unit 272 emits the flash light of the calculated dimming light emission amount from the flash light emitting unit 270, and the main body side CPU 220 emits the same exposure conditions (underexposure) as the first non-emission image under the dimming light emission. ), The dimming image captured in (step S102) is acquired.

Subsequently, the CPU 220 on the main body side determines whether there is a pixel or region where the flash light has not reached based on the detection output of the flash light unreachable pixel detection unit 220G (FIG. 15) (step S104). The flash light unreachable pixel detection unit 220G acquires a one-frame image (non-emission image) of the live view image captured under the same exposure conditions as the dimming image, and separates the dimming image and the non-emission image. The flash light unreachable pixel or area is detected by comparison.

When it is determined in step S104 that the flash light has reached the entire area (“No””.

In the case of), the main body side CPU 220 determines whether or not there is an overexposed area in the dimming image (step S106). When there is a region having a maximum pixel value (255 when the pixel value is represented by 8 bits) in the dimming image, that region can be determined as an overexposed region.

When it is determined in step S106 that there is no overexposed area in the dimming image (in the case of "No"), the dimming image is acquired as a light emitting image (step S108). In this case, it is not necessary to capture the luminescent image again, and the number of captured images can be reduced.

When it is determined in step S106 that there is an overexposed area in the dimming image (in the case of "Yes"), the flash control unit 272 calculates the maximum amount of light emission at which the overexposed area is not overexposed. Then, the calculated maximum light emission amount is used as the main light emission amount when the light emission image is acquired (step S110).

On the other hand, when it is determined in step S104 that there is an area where the flash light has not reached (in the case of "Yes"), the CPU 220 on the main body side determines that the area where the flash light does not reach is greater than the maximum reach of the flash light. It is determined whether or not the light is close (step S112). The distance of the region where the flash light has not reached can be obtained from the distance distribution information, and the maximum reachable distance of the flash light can be obtained from the guide number and the F value of the flash light emitting unit 270.

When it is determined in step S112 that the region where the flash light has not reached is farther than the maximum reach of the flash light (in the case of "No"), the composite ratio determining unit 220C shown in FIG. 15 receives the flash light. The mixing ratio αi of the unreached region (the mixing ratio αi of the second non-emission image shown in the equation [Equation 6]) is set to 1 (step S114). As a result, the mixing ratio γi of the second non-emission image in the region where the flash light does not reach is determined only by the mixing ratio βi of the second non-emission image based on the luminance distribution information.

On the other hand, when it is determined in step S112 that the area where the flash light has not reached is closer than the maximum reach of the flash light (in the case of "Yes"), the flash control unit 272 determines that the area where the flash light has reached is located. The maximum light emission amount that does not cause overexposure is calculated, and the calculated maximum light emission amount is used as the main light emission amount when acquiring a light emission image (step S116).

When the flash control unit 272 calculates the main light emission amount in step S110 or step S116, the flash light emission amount is mainly emitted from the flash light emission unit 270, and the image acquisition unit 221 is the first non-light emission under the main light emission. The light-emitting image captured by the imaging unit (interchangeable lens 100 and image sensor 201) is acquired under the same exposure conditions as the exposure condition of the light-emitting image (step S118).

When the light emitting image is acquired as described above, the non-light emitting image (first non-light emitting image) captured under the same exposure conditions (underexposure) as the light emitting image and the second non-light emitting image captured under overexposure are subsequently captured. And get.

Returning to FIG. 17, the light intensity distribution information calculation unit 220B is based on the non-emission image (the first non-emission image underexposed), the light emission image, and the distance distribution information acquired by the distance distribution information acquisition unit 220A. 1] The light intensity distribution information _Si is calculated by the equation (step S14).

Synthesis ratio determining unit 220C may determine the synthesis ratio for a plurality of non-emission image (first non-emission image, the second non-emission image) is HDR synthesized based on the light intensity distribution information S _i calculated in step S14 (Step S16). For example, the composite ratio determination unit 220C determines the composite ratio for each corresponding pixel or region of the plurality of non-emission images (first non-emission image and second non-emission image), and when the composite ratio is determined, the light intensity distribution information _Si In pixels or regions with high light intensity, the mixing ratio of the underexposed first non-emission image is made larger than the mixing ratio of the overexposed second non-emission image, and in pixels or regions with low light intensity, 2 The mixing ratio of the non-emission image is made larger than the mixing ratio of the first non-emission image.

The image compositing unit 220D performs HDR compositing of the first non-emission image and the second non-emission image based on the compositing ratio determined in step S16, and generates an HDR image (step S18).

The image pickup device 10 of the present embodiment is a mirrorless digital single-lens camera, but is not limited to this, and may be a single-lens reflex camera, a lens-integrated image pickup device, a digital video camera, or the like, and also takes an image in addition to the image pickup function. It is also applicable to mobile devices having other functions (call function, communication function, other computer functions) other than the above. Other aspects to which the present invention can be applied include, for example, mobile phones and smartphones having a camera function, PDAs (Personal Digital Assistants), and portable game machines. Hereinafter, an example of a smartphone to which the present invention can be applied will be described.

<Smartphone configuration>

FIG. 19 shows the appearance of the smartphone 500, which is an embodiment of the imaging device of the present invention. The smartphone 500 shown in FIG. 19 has a flat-plate housing 502, and a display input in which a display panel 521 as a display unit and an operation panel 522 as an input unit are integrated on one surface of the housing 502. The unit 520 is provided. Further, the housing 502 includes a speaker 531, a microphone 532, an operation unit 540, and a camera unit 541. The configuration of the housing 502 is not limited to this, and for example, a configuration in which the display unit and the input unit are independent can be adopted, or a configuration having a folding structure or a slide mechanism can be adopted.

FIG. 20 is a block diagram showing the configuration of the smartphone 500 shown in FIG. As shown in FIG. 20, as the main components of the smartphone, a wireless communication unit 510 that performs mobile wireless communication via a base station and a mobile communication network, a display input unit 520, a call unit 530, and an operation unit 540. A camera unit 541, a recording unit 550, an external input / output unit 560, a GPS (Global Positioning System) receiving unit 570, a motion sensor unit 580, a power supply unit 590, and a main control unit 501 are provided.

The wireless communication unit 510 performs wireless communication with the base station accommodated in the mobile communication network in accordance with the instruction of the main control unit 501. This wireless communication is used to send and receive various file data such as voice data and image data, e-mail data, and receive Web data and streaming data.

Under the control of the main control unit 501, the display input unit 520 displays images (still images and moving images), character information, and the like to visually convey the information to the user, and detects the user operation for the displayed information. It is a so-called touch panel, and includes a display panel 521 and an operation panel 522.

The display panel 521 uses an LCD (Liquid Crystal Display), an OLED (Organic Electro-Luminescence Display), or the like as a display device. The operation panel 522 is a device on which an image displayed on the display surface of the display panel 521 is visibly placed and detects one or a plurality of coordinates operated by a user's finger or a stylus. When such a device is operated with a user's finger or a stylus, a detection signal generated due to the operation is output to the main control unit 501. Next, the main control unit 501 detects the operation position (coordinates) on the display panel 521 based on the received detection signal.

As shown in FIG. 19, the display panel 521 and the operation panel 522 of the smartphone 500 illustrated as one embodiment of the image pickup apparatus of the present invention integrally constitute the display input unit 520, but the operation panel The arrangement is such that the 522 completely covers the display panel 521. When such an arrangement is adopted, the operation panel 522 may also have a function of detecting a user operation in an area outside the display panel 521. In other words, the operation panel 522 has a detection area (hereinafter, referred to as a display area) for the overlapping portion overlapping the display panel 521 and a detection area (hereinafter, non-display area) for the outer edge portion not overlapping the other display panel 521. ) And may be provided.

The size of the display area and the size of the display panel 521 may be completely matched, but it is not always necessary to match the two. Further, the operation panel 522 may include two sensitive regions, an outer edge portion and an inner portion other than the outer edge portion. Further, the width of the outer edge portion is appropriately designed according to the size of the housing 502 and the like. Further, examples of the position detection method adopted in the operation panel 522 include a matrix switch method, a resistance film method, a surface acoustic wave method, an infrared method, an electromagnetic induction method, a capacitance method, and the like, and any of these methods is adopted. You can also do it.

The call unit 530 includes a speaker 531 and a microphone 532, converts the user's voice input through the microphone 532 into voice data that can be processed by the main control unit 501, and outputs the data to the main control unit 501, or a wireless communication unit. The audio data received by the 510 or the external input / output unit 560 is decoded and output from the speaker 531. Further, as shown in FIG. 19, for example, the speaker 531 and the microphone 532 can be mounted on the same surface as the surface on which the display input unit 520 is provided.

The operation unit 540 is a hardware key using a key switch or the like, and receives an instruction from the user. For example, as shown in FIG. 19, the operation unit 540 is mounted on the side surface of the housing 502 of the smartphone 500, and is turned on when pressed with a finger or the like, and turned off by a restoring force such as a spring when the finger is released. It is a push button type switch.

The recording unit 550 includes the control program of the main control unit 501, control data, application software (including the image processing program according to the present invention), address data associated with the name and telephone number of the communication partner, and the transmitted / received e-mail. It stores data, Web data downloaded by Web browsing, downloaded content data, and temporarily stores streaming data and the like. Further, the recording unit 550 is composed of an internal storage unit 551 built in the smartphone and an external storage unit 562 having a detachable external memory slot. The internal storage unit 551 and the external storage unit 552 constituting the recording unit 550 are a flash memory type, a hard disk type, a multimedia card micro type, and a multimedia card micro type. It is realized by using a recording medium such as a card type memory (for example, Micro SD (registered trademark) memory), RAM (Random Access Memory), and ROM (Read Only Memory).

The external input / output unit 560 serves as an interface with all external devices connected to the smartphone 500, and communicates with other external devices (for example, universal serial bus (USB), IEEE1394, etc.) or Networks (eg, Internet, wireless LAN (Local Area Network), Bluetooth (Bluetooth) (registered trademark), RFID (Radio Frequency Identification), Infrared Data Association (IrDA) (registered trademark), UWB (Ultra Wideband) ( It is for direct or indirect connection by (registered trademark), ZigBee (registered trademark), etc.).

Examples of external devices connected to the smartphone 500 include a presence / wireless headset, a presence / wireless external charger, a presence / wireless data port, a memory card connected via a card socket, and a SIM (Subscriber). External audio / video device connected via Identity Module Card) / UIM (User Identity Module Card) card or audio / video I / O (Input / Output) terminal, external audio / video device connected wirelessly, Yes / Wireless There are connected smartphones, Yes / wirelessly connected personal computers, Yes / wirelessly connected PDAs, and earphones. The external input / output unit can transmit the data transmitted from such an external device to each component inside the smartphone 500, or can transmit the data inside the smartphone 500 to the external device.

The GPS receiving unit 570 receives GPS signals transmitted from the GPS satellites ST1 to STn according to the instruction of the main control unit 501, executes positioning calculation processing based on the received plurality of GPS signals, and determines the latitude of the smartphone 500. Detects the position consisting of longitude and altitude. When the GPS receiving unit 570 can acquire the position information from the wireless communication unit 510 or the external input / output unit 560 (for example, wireless LAN), the GPS receiving unit 570 can also detect the position using the position information.

The motion sensor unit 580 includes, for example, a three-axis acceleration sensor and a gyro sensor, and detects the physical movement of the smartphone 500 according to the instruction of the main control unit 501. By detecting the physical movement of the smartphone 500, the moving direction and acceleration of the smartphone 500 are detected. This detection result is output to the main control unit 501.

The power supply unit 590 supplies electric power stored in a battery (not shown) to each unit of the smartphone 500 according to the instruction of the main control unit 501.

The main control unit 501 includes a microprocessor, operates according to the control program and control data stored in the recording unit 550, and controls each unit of the smartphone 500 in an integrated manner. In addition, the main control unit 501 includes a mobile communication control function and an application processing function that control each unit of the communication system in order to perform voice communication and data communication through the wireless communication unit 510.

The application processing function is realized by operating the main control unit 501 according to the application software stored in the recording unit 550. Examples of the application processing function include an infrared communication function that controls an external input / output unit 560 to perform data communication with an opposite device, an e-mail function that transmits / receives e-mail, a Web browsing function that browses a Web page, and the present invention. There is an image processing function that performs HDR composition related to the above.

Further, the main control unit 501 is provided with an image processing function such as displaying an image on the display input unit 520 based on image data (still image or moving image data) such as received data or downloaded streaming data. The image processing function refers to a function in which the main control unit 501 decodes the image data, performs image processing on the decoding result, and displays the image on the display input unit 520.

Further, the main control unit 501 executes display control for the display panel 521 and operation detection control for detecting a user operation through the operation unit 540 and the operation panel 522.

By executing the display control, the main control unit 501 displays a software key such as an icon and a scroll bar for starting the application software, or displays a window for composing an e-mail. The scroll bar is a software key for receiving an instruction to move a display portion of an image for a large image or the like that cannot fit in the display area of the display panel 521.

Further, by executing the operation detection control, the main control unit 501 detects the user operation through the operation unit 540, operates the icon through the operation panel 522, and accepts the input of the character string in the input field of the window, or scrolls. Accepts scrolling requests for displayed images through the bar.

Further, by executing the operation detection control, the main control unit 501 has an overlapping portion (display area) whose operation position with respect to the operation panel 522 overlaps the display panel 521, or an outer edge portion (non-display area) which does not overlap the other display panel 521. ), And a touch panel control function that controls the sensitive area of the operation panel 522 and the display position of the software key.

The main control unit 501 can also detect a gesture operation on the operation panel 522 and execute a preset function according to the detected gesture operation. Gesture operation is not a conventional simple touch operation, but an operation of drawing a locus with a finger or the like, specifying a plurality of positions at the same time, or combining these to draw a locus of at least one from a plurality of positions. means.

The camera unit 541 is a digital camera that electronically photographs using an image pickup device such as a CMOS (Complementary Metal Oxide Semiconductor) or a CCD (Charge-Coupled Device), and corresponds to the image pickup device 10 shown in FIG. Further, the camera unit 541 converts the image data obtained by imaging into compressed image data such as JPEG (Joint Photographic coding Experts Group) under the control of the main control unit 501, and records it in the recording unit 550 or externally. It can be output through the input / output unit 560 and the wireless communication unit 510. As shown in FIG. 19, in the smartphone 500, the camera unit 541 is mounted on the same surface as the display input unit 520, but the mounting position of the camera unit 541 is not limited to this, and is mounted on the back surface of the display input unit 520. Alternatively, a plurality of camera units 541 may be mounted. When a plurality of camera units 541 are mounted, the camera units 541 used for shooting can be switched for shooting independently, or the plurality of camera units 541 can be used at the same time for shooting.

In addition, the camera unit 541 can be used for various functions of the smartphone 500. For example, the image acquired by the camera unit 541 can be displayed on the display panel 521, or the image of the camera unit 541 can be used as one of the operation inputs of the operation panel 522. Further, when the GPS receiving unit 570 detects the position, the position can be detected by referring to the image from the camera unit 541. Further, referring to the image from the camera unit 541, the optical axis direction of the camera unit 541 of the smartphone 500 is used without using the 3-axis acceleration sensor or in combination with the 3-axis acceleration sensor (gyro sensor). It is also possible to judge the current usage environment. Of course, the image from the camera unit 541 can also be used in the application software.

In addition, the position information acquired by the GPS receiving unit 570 and the voice information acquired by the microphone 532 (the voice text may be converted by the main control unit or the like to become the text information) in the image data of the still image or the moving image. It is also possible to add posture information or the like acquired by the motion sensor unit 580 and record it in the recording unit 550, or output it through the external input / output unit 560 or the wireless communication unit 510.

[Other]

In the present embodiment, the image processing device (main body side CPU 220) that performs HDR synthesis is built in the image pickup device, but the image processing device according to the present invention executes, for example, the image processing program according to the present invention. It may be a personal computer, a mobile terminal, or the like that is separate from the image pickup device. In this case, it is necessary to input a plurality of non-emission images, emissive images, distance distribution information, and the like as information for HDR composition.

Further, the plurality of non-emission images are not limited to two non-emission images, an underexposed first non-emission image and an overexposed second non-emission image, and three or more images of the same subject under different imaging conditions. It may be a non-emission image. In this case, the composition ratio determining unit determines the composition ratio of three or more non-emission images.

Further, in the present embodiment, for example, the hardware structure of the processing unit that executes various processes such as the main body side CPU 220 is various processors as shown below. For various processors, the circuit configuration can be changed after manufacturing the CPU (Central Processing Unit), FPGA (Field Programmable Gate Array), etc., which are general-purpose processors that execute software (programs) and function as various processing units. Programmable Logic Device (PLD), a dedicated electric circuit, which is a processor having a circuit configuration specially designed to execute a specific process such as an ASIC (Application Specific Integrated Circuit), etc. Is done.

One processing unit may be composed of one of these various processors, or may be composed of two or more processors of the same type or different types (for example, a plurality of FPGAs or a combination of a CPU and an FPGA). You may. Further, a plurality of processing units may be configured by one processor. As an example of configuring a plurality of processing units with one processor, first, one processor is configured by a combination of one or more CPUs and software, as represented by a computer such as a client or a server. There is a form in which a processor functions as a plurality of processing units. Secondly, as typified by System On Chip (SoC), there is a form in which a processor that realizes the functions of the entire system including a plurality of processing units with one IC (Integrated Circuit) chip is used. is there. As described above, the various processing units are configured by using one or more of the above-mentioned various processors as a hardware-like structure.

Further, the hardware structure of these various processors is, more specifically, an electric circuit (circuitry) in which circuit elements such as semiconductor elements are combined.

Furthermore, the present invention includes an image processing program that, when installed in an image pickup apparatus or a computer, functions as the image pickup apparatus or image processing apparatus according to the present invention, and a recording medium on which the image processing program is recorded.

Further, the present invention is not limited to the above-described embodiment, and it goes without saying that various modifications can be made without departing from the spirit of the present invention.

10 Imaging device

20 Finder window

22 Shutter release switch

23 Shutter speed dial

24 Exposure compensation dial

25 Power lever

26 Eyepiece

27 MENU / OK key

28 cross key

29 Play button

30 Built-in flash

100 interchangeable lens

102 Imaging optical system

104 lens group

108 aperture

116 Focus lens control unit

118 Aperture control unit

120 Lens side CPU

122, 207 RAM

124, 228 ROM

126 flash ROM

150 Lens side communication unit

160 lens mount

200 camera body

201 image sensor

201A micro lens

202 Image sensor control unit

203 Analog signal processing unit

204 A / D converter

205 image input controller

206 Digital signal processing unit

208 Compression / decompression processing unit

210 Media control unit

212 memory card

214 Display control unit

216 LCD monitor

220 CPU on the main unit

220A Distance distribution information acquisition unit

220B Light intensity distribution information calculation unit

220C synthesis ratio determination unit

220D image compositing unit

220E Distribution information creation unit

220F Luminance distribution information calculation unit

220G flash light unreachable pixel detector

221 Image acquisition section

222 Operation unit

223 Distance distribution information acquisition unit

224 Clock section

226 flash ROM

230 AF control unit

232 AE control unit

234 White balance correction unit

236 Wireless communication unit

238 GPS receiver

240 power control unit

242 battery

244 Lens power switch

250 Main unit communication unit

260 body mount

260A terminal

270 flash light emitting part

272 Flash control unit

296 FPS control unit

500 smartphone

501 main control unit

502 chassis

510 wireless communication unit

520 Display input section

521 display panel

522 operation panel

530 Call section

531 speaker

532 microphone

540 operation unit

541 camera unit

550 recording section

551 Internal storage

552 external storage unit

560 External input / output unit

562 External storage

570 receiver

570 GPS receiver

580 motion sensor unit

590 power supply

Steps S10 to S18 and S100 to S118

_Si light intensity distribution information

Y _Li brightness value

Y _i brightness value

d _i distance information

αi, βi, γi mixing ratio

Claims (20)

One of a plurality of non-emission images obtained by capturing the same subject under different exposure conditions under non-flash light emission and the plurality of non-emission images under flash light emission. An image acquisition unit that acquires a light-emitting image captured under the same exposure conditions as the non-light-emitting image of

A distance distribution information acquisition unit that acquires distance distribution information indicating the distance distribution of the subject,

Light intensity distribution information calculation that calculates the light intensity distribution information of the light shining on the subject based on the light emitting image, the non-light emitting image captured under the same exposure conditions as the light emitting image, and the distance distribution information. Department and

A composite ratio determining unit that determines the composite ratio of the plurality of non-emission images based on the light intensity distribution information,

An image compositing unit that synthesizes the plurality of non-emission images according to the compositing ratio and generates a composite image having an expanded dynamic range.

Image processing device equipped with.

The composite ratio determining unit determines the composite ratio for each corresponding pixel or region of the plurality of non-emission images.

The image processing apparatus according to claim 1, wherein the image synthesizing unit synthesizes the plurality of non-emission images by applying the determined synthesis ratio for each corresponding pixel or region of the plurality of non-emission images.

The image processing apparatus according to claim 1 or 2, wherein the plurality of exposure conditions corresponding to the plurality of non-emission images include an underexposure exposure condition and an overexposure exposure condition rather than a proper exposure exposure condition.

When determining the composite ratio for each of the corresponding pixels or regions of the plurality of non-emission images, the composite ratio determining unit may determine the plurality of pixels or regions having high light intensity based on the light intensity distribution information. The mixing ratio of the first non-emission image captured under the exposure condition having the smallest exposure value among the non-emission images is the second non-emission image captured under the exposure condition having the largest exposure value among the plurality of non-emission images. Any of claims 1 to 3, which is larger than the mixing ratio of the luminescent image and is larger than the mixing ratio of the first non-light emitting image in the pixel or region where the light intensity is low. The image processing apparatus according to item 1.

When determining the composite ratio for each pixel or region corresponding to each of the plurality of non-emission images, the composite ratio determining unit determines the minimum light intensity of the light intensity distribution information based on the light intensity distribution information. The image processing apparatus according to claim 4, wherein when the light intensity increases between the light intensity and the maximum light intensity, the mixing ratio of the second non-emission image is monotonically decreased.

It is provided with a distribution information creation unit that creates distribution information indicating the frequency distribution or frequency distribution of the light intensity based on the light intensity distribution information.

The composite ratio determining unit has a first light intensity corresponding to the first vertex having a high frequency or frequency and a second light intensity corresponding to the second peak having a high frequency or frequency based on the created distribution information. Therefore, when a second light intensity higher than the first light intensity is obtained and the synthesis ratio is determined for each corresponding pixel or region of the plurality of non-emission images, the light having the first light intensity or less is determined. In the case of intensity, the mixing ratio of the second non-emission image is maximized, and in the case of light intensity equal to or higher than the second light intensity, the mixing ratio of the second non-emission image is minimized, and the first light The image according to claim 4, wherein in the case of a light intensity larger than the intensity and lower than the second light intensity, the mixing ratio of the second non-emissive image is monotonically reduced between the maximum value and the minimum value. Processing equipment.

It is provided with a distribution information creation unit that creates distribution information indicating the frequency distribution or frequency distribution of the light intensity based on the light intensity distribution information.

The composite ratio determining unit obtains the light intensity corresponding to the low frequency or infrequent bottom point based on the created distribution information, and determines the composite ratio for each corresponding pixel or region of the plurality of non-emission images. When determining, the light intensity corresponding to the bottom point is set as a threshold value, the mixing ratio of the second non-emission image is maximized in the case of a light intensity below the threshold value, and the light intensity exceeds the threshold value. The image processing apparatus according to claim 5, wherein the mixing ratio of the second non-emission image is minimized.

A brightness distribution information calculation unit for calculating the luminance distribution information indicating the luminance distribution in the image of at least one non-luminance image among the plurality of non-emission images is provided.

The image processing apparatus according to any one of claims 1 to 7, wherein the composite ratio determining unit determines a composite ratio of the plurality of non-emission images based on the light intensity distribution information and the brightness distribution information.

A detection unit that detects pixels or regions that the flash light has not reached in the emitted image is provided.

The composite ratio determining unit determines the composite ratio of the plurality of non-emission images based only on the luminance distribution information when determining the composite ratio for pixels or regions not reached by the flash light. Item 8. The image processing apparatus according to item 8.

When determining the composite ratio for each of the corresponding pixels or regions of the plurality of non-emission images, the composite ratio determining unit sets a parameter indicating the position of the pixel or region i, and among the plurality of non-emission images. The mixing ratio of the second non-emissive image captured under the exposure condition having the largest exposure value is αi, which is the mixing ratio of the second non-emissive image based on the luminance distribution information, and the higher the luminance, the smaller the value. Assuming that the mixing ratio is βi, the mixing ratio γi of the second non-emission image based on the light intensity distribution information and the brightness distribution information is calculated by the following equation.

The image processing apparatus according to claim 8 or 9, which is calculated by γi = αi × βi.

The image processing apparatus according to any one of claims 1 to 10, wherein the exposure condition of the light emitting image is the exposure condition having the smallest exposure value among the plurality of exposure conditions corresponding to the plurality of non-light emitting images. ..

An image pickup unit that can image the subject under different exposure conditions,

A flash light emitting part that emits flash light and

A flash control unit that controls the flash light emitted from the flash light emitting unit,

The image processing apparatus according to any one of claims 1 to 11 is provided.

The image acquisition unit emits light from the flash light emitting unit and the plurality of non-light emitting images captured by the imaging unit under different exposure conditions under the non-emission of the flash light emitted from the flash light emitting unit. An imaging device that acquires the luminescent image captured by the imaging unit under the same exposure conditions as the exposure condition of any one of the plurality of non-emitting images under the light emission of the flash light.

The flash control unit dimmes the flash light from the flash light emitting unit, and the dimming image captured by the imaging unit under the dimming light emission and the image captured by the imaging unit without the dimming light emission. When a region not reached by the dimmed flash light is detected based on the above, the maximum amount of light emitted that the region reached by the dimmed flash light is not saturated is calculated, and the emitted image is displayed. The imaging device according to claim 12, wherein the flash light is mainly emitted from the flash light emitting unit according to the calculated maximum light emission amount at the time of acquisition.

The imaging device according to claim 13, wherein the flash control unit calculates the maximum amount of light emission based on the average reflectance of the subject and the distance distribution information acquired by the distance distribution information acquisition unit.

The flash control unit dimmes the flash light from the flash light emitting unit, and the dimming image captured by the imaging unit under the dimming light emission and the image captured by the imaging unit without the dimming light emission. The imaging according to claim 12, wherein when it is detected that the flash light has reached the entire region of the dimming image based on the above, the image acquisition unit acquires the dimming image as the light emitting image. apparatus.

The flash control unit dimmes the flash light from the flash light emitting unit, and the dimming image captured by the imaging unit under the dimming light emission and the image captured by the imaging unit without the dimming light emission. The area where the flash light does not reach the dimming image is detected based on the above, and the distance of the area where the flash light does not reach based on the distance distribution information is emitted from the flash light emitting unit. The imaging device according to claim 12, wherein when the distance is longer than the reach of the maximum amount of light emitted, the image acquisition unit acquires the dimming image as the emitted image.

A step of acquiring a plurality of non-emission images captured under different exposure conditions under non-emission of flash light, which are images of the same subject.

A step of acquiring a light emitting image captured under the same exposure conditions as the exposure condition of any of the plurality of non-light emitting images under the light emission of the flash light.

A step of acquiring distance distribution information indicating the distance distribution of the subject, and

A step of calculating the light intensity distribution information of the light irradiating the subject based on the light emitting image, the non-light emitting image captured under the same exposure conditions as the light emitting image, and the distance distribution information.

A step of determining the composition ratio of the plurality of non-emission images based on the light intensity distribution information, and

A step of synthesizing the plurality of non-emission images according to the synthesis ratio to generate a composite image having an expanded dynamic range.

Image processing method including.

The step of determining the composite ratio is that when the composite ratio is determined for each corresponding pixel or region of the plurality of non-emission images, the pixel or region having a large light intensity is described based on the light intensity distribution information. The mixing ratio of the first non-emission image captured under the exposure condition having the smallest exposure value among the plurality of non-emission images is the first imaged under the exposure condition having the largest exposure value among the plurality of non-emission images. 2. The 17th aspect of claim 17, wherein the mixing ratio of the second non-emission image is made larger than the mixing ratio of the first non-emissive image, and the mixing ratio of the second non-emissive image is made larger than the mixing ratio of the first non-emission image in a pixel or region where the light intensity is low. Image processing method.

A function to acquire a plurality of non-emission images captured under different exposure conditions under non-emission of flash light, which are images of the same subject.

A function of acquiring a light emitting image captured under the same exposure conditions as the exposure condition of any of the plurality of non-light emitting images under the light emission of the flash light.

A function to acquire distance distribution information indicating the distance distribution of the subject, and

A function of calculating the light intensity distribution information of the light irradiating the subject based on the light emitting image, the non-light emitting image captured under the same exposure conditions as the light emitting image, and the distance distribution information.

A function of determining the composition ratio of the plurality of non-emission images based on the light intensity distribution information, and

A function of synthesizing a plurality of non-emission images according to the composition ratio to generate a composite image with an expanded dynamic range.

An image processing program that makes a computer realize.

The function of determining the composite ratio is that when the composite ratio is determined for each corresponding pixel or region of the plurality of non-emission images, the pixel or region having a large light intensity based on the light intensity distribution information is described. The mixing ratio of the first non-emission image captured under the exposure condition having the smallest exposure value among the plurality of non-emission images is the first imaged under the exposure condition having the largest exposure value among the plurality of non-emission images. 2. The 19th aspect of claim 19, wherein the mixing ratio of the second non-light emitting image is made larger than the mixing ratio of the first non-light emitting image, and the mixing ratio of the second non-light emitting image is made larger than the mixing ratio of the first non-light emitting image in a pixel or region having a small light intensity. Image processing program.

Images