WO2019208155A1

WO2019208155A1 - Image processing device, method and program, and imaging device

Info

Publication number: WO2019208155A1
Application number: PCT/JP2019/015046
Authority: WO
Inventors: 祐也西尾; 智紀増田; 智大島田
Original assignee: 富士フイルム株式会社
Priority date: 2018-04-26
Filing date: 2019-04-05
Publication date: 2019-10-31
Also published as: JP6810299B2; JPWO2019208155A1

Abstract

Provided are an image processing device, method and program, and an imaging device which can acquire an HDR image in which the brightness/darkness difference has been adjusted according to the brightness of light hitting a subject. This image processing device comprises: an image acquisition unit 221 that acquires a first non-emission image captured by underexposure, a second non-emission image captured by over-exposure, and an emission image with the same exposure conditions as the first non-emission image; a distance distribution information acquisition unit 220A that acquires distance distribution information indicating the distribution of distances of the subject; a light intensity distribution information calculation unit 220B that, on the basis of the emission image, the first and the second non-emission images and the distance distribution information, calculates light intensity distribution information of the light being projected onto the subject; a synthesis ratio determination unit 220C that determines a synthesis ratio for the first and the second non-emission images on the basis of the light intensity distribution information; and an image synthesizing unit 220D that synthesizes the first and the second non-emission images according to the synthesis ratio.

Description

Image processing apparatus, method, program, and imaging apparatus

The present invention relates to an image processing apparatus, method, program, and imaging apparatus, and in particular, performs high dynamic range synthesis (HDR (High-dynamic-range) synthesis) that combines a plurality of images with different exposures to expand a dynamic range. Regarding technology.

When a subject with a wide dynamic range is imaged with a digital camera or the like, the output of the image sensor is saturated in the highlight area, so-called “whiteout” occurs, and the gradation of the output of the image sensor is lost in the shadow area. In other words, so-called “blackout” may occur.

It is known to perform HDR processing as one of methods for expressing a dynamic range of a subject with a wider image. In the HDR processing, for example, an image obtained by reducing the exposure amount with respect to the appropriate exposure (underexposure image) and an image obtained by increasing the exposure amount (overexposure image) are acquired, and a pixel on which a dark subject is imaged Uses an overexposed image pixel, and an underexposed image pixel is used for a pixel on which a bright subject is imaged (actually, not only one of them may be mixed), but an HDR image with little overexposure and underexposure is generated. create.

Conventionally, as an apparatus for generating one HDR image from a plurality of images taken at different exposures, there are devices described in

Patent Documents

1 and 2.

The imaging system described in Patent Literature 1 is an exposure of luminance signals related to a long-time exposure image from a plurality of images (long-time exposure image and short-time exposure image) captured on the same subject under different exposure conditions. A luminance signal in a region (appropriate exposure region) other than an over-exposed region (inappropriate exposure region) is acquired, and in the case of a luminance signal related to a short exposure image, it corresponds to the inappropriate exposure region in the long exposure image. Get the luminance signal. Then, a wide dynamic range image is generated by combining the appropriate exposure areas of the images.

In addition, the imaging system described in Patent Literature 1 estimates a main subject area from a long-exposure image and a short-exposure image, and performs tone correction so that a tone width is allocated to the estimated main subject area. In this way, the gradation for the main subject in the appropriate exposure area is enriched by synthesizing the appropriate exposure area for each image after gradation correction.

On the other hand, the imaging device described in Patent Document 2 includes one imaging device having a pixel group that is exposed for a long time and a pixel group that is exposed for a short time, and the image data of the first pixel group read from the imaging device and the second image data. The HDR image is generated by combining the image data of the pixel group.

Patent Document 2 adds a mixing ratio of flash image data that is long-time exposure data and non-flash image data that is short-time exposure data while adjusting the mixing ratio according to the estimated distance to the subject. There is a description of obtaining one composite image. As a result, an image with a high signal-to-noise ratio (Signal to Noise Ratio) of the main subject is obtained while leaving the atmosphere of the background image.

JP 2002-232777 A JP 2009-206925 A

In Patent Document 1, there is no clear description regarding the mixing ratio of the long-time exposure image and the short-time exposure image when generating the HDR image, but regarding the luminance signal corresponding to the improper exposure area of the long-time exposure image. Since the luminance signal related to the short-time exposure image is used, it is considered that this is a general HDR synthesis in which the mixing ratio between the long-time exposure image and the short-time exposure image is determined by the magnitude of the luminance signal. Therefore, as described later, when there is a shadow on a part of the main subject, there is a problem that the shadow area remains dark.

On the other hand, the invention described in Patent Document 2 adjusts the mixing ratio of flash image data that is long-time exposure data and non-flash image data that is short-time exposure data in accordance with the estimated distance to the subject. Adding together, one composite image is generated. The invention described in Patent Document 2 combines flash image data and non-flash image data, and does not combine non-light emitting images (non-flash images) with different exposure conditions.

As described above, in

Patent Documents

1 and 2, when a HDR image is generated by synthesizing a plurality of images captured with different exposure conditions for the same subject, the intensity of light applied to the subject is determined. There is no description to adjust the composition ratio of a plurality of images accordingly.

As in general HDR processing, when deciding whether to use the pixel value of the over image or the pixel value of the under image based on the luminance value obtained from the image sensor, it depends on the intensity of light hitting the subject. Regardless, the same HDR processing is performed and the following problems occur.

For example, as shown in FIG. 21, a case is assumed in which the left half of the person's face is the sunlit area A where the sunlight is exposed, and the right half is the shaded area B. In this case, if “the dark hair (1) exposed to strong light” and “skinned skin (2)” have the same luminance value or almost the same luminance value, the same applies to the HDR composition. The over image is used, and not only the shadowed skin (2) becomes bright, but also the black hair (1) becomes a bright color.

The present invention has been made in view of such circumstances, and an image processing apparatus capable of acquiring an HDR image with excellent gradation characteristics in which the contrast between light and dark is adjusted according to the brightness of light hitting a subject, It is an object to provide a method, a program, and an imaging apparatus.

In order to achieve the above object, an image processing apparatus according to an aspect of the present invention is an image obtained by imaging the same subject, and a plurality of non-light emitting devices that are imaged under different exposure conditions under non-flash light emission. An image acquisition unit that acquires an image and a light-emitting image captured under the same exposure condition as that of any one of the plurality of non-light-emitting images under flash light emission; A distance distribution information acquisition unit that acquires distance distribution information indicating the light intensity of light emitted to the subject based on the light emission image, the non-light emission image captured under the same exposure conditions as the light emission image, and the distance distribution information A light intensity distribution information calculation unit that calculates distribution information, a combination ratio determination unit that determines a combination ratio of a plurality of non-light-emitting images based on the light intensity distribution information, and a plurality of units according to the combination ratio The luminescent image synthesizing comprises an image combining unit for generating a composite image whose dynamic range is expanded, the.

According to one aspect of the present invention, light intensity distribution information of light irradiated on a subject is calculated based on a light emission image, a non-light emission image captured under the same exposure conditions as the light emission image, and distance distribution information. . Then, a composite ratio of the plurality of non-light-emitting images is determined based on the calculated light intensity distribution information, and the plurality of non-light-emitting images are combined according to the determined composite ratio, thereby generating a composite image (HDR) with an expanded dynamic range. Image). The HDR image generated in this way is an HDR image with excellent gradation characteristics in which the contrast of light and darkness is adjusted according to the brightness of the light hitting the subject without losing the contrast of the subject.

In the image processing device according to another aspect of the present invention, a composition ratio is determined for each pixel or region corresponding to each of the plurality of non-light-emitting images, and the image composition unit is configured to correspond to each pixel or region corresponding to the plurality of non-light-emitting images. It is preferable to synthesize a plurality of non-light emitting images by applying a synthesis ratio determined for each.

In the image processing apparatus according to still another aspect of the present invention, the plurality of exposure conditions corresponding to the plurality of non-light-emitting images include an under-exposure exposure condition and an over-exposure exposure condition rather than an appropriate exposure condition. Is preferred. This makes it possible to create an HDR image with less whiteout and blackout.

In the image processing apparatus according to still another aspect of the present invention, when determining a composition ratio for each pixel or region corresponding to each of a plurality of non-light-emitting images, a pixel or region having a high light intensity based on the light intensity distribution information Then, the mixing ratio of the first non-light-emitting images captured under the exposure condition with the smallest exposure value among the plurality of non-light-emitting images is represented by the mixing ratio of the first non-light-emitting images captured under the exposure condition with the largest exposure value among the plurality of non-light-emitting images. It is preferable that the mixing ratio of the second non-light-emitting image is larger than the mixing ratio of the first non-light-emitting image in a pixel or region having a low light intensity.

That is, in a pixel (region corresponding to a subject that is exposed to strong light) or a region having a high light intensity, the mixing ratio of the first non-light emitting image is set larger than the mixing ratio of the second non-light emitting image. In a pixel or region that generates an HDR image with less skipping and has low light intensity (a pixel corresponding to a subject that is exposed to weak light) or a region, the mixing ratio of the second non-emitting image is greater than the mixing ratio of the first non-emitting image. Is increased to generate an HDR image with less blackout.

In the image processing apparatus according to still another aspect of the present invention, the composition ratio determination unit determines the composition ratio based on the light intensity distribution information when determining the composition ratio for each corresponding pixel or region of the plurality of non-light-emitting images. When the light intensity increases between the minimum light intensity and the maximum light intensity in the intensity distribution information, it is preferable to monotonously decrease the mixing ratio of the second non-light emitting image. When the light intensity increases between the minimum light intensity and the maximum light intensity in the light intensity distribution information, the mixing ratio of the second non-emitting image is monotonously decreased (the mixing ratio of the first non-emitting image is monotonously increased). By doing so, it is possible to perform HDR synthesis in which the mixing ratio is continuously changed according to the increase or decrease of the light intensity. The minimum light intensity is not limited to the minimum value in the light intensity distribution information. For example, the minimum light intensity may be a light intensity offset by a certain value from the minimum value. Similarly, the maximum light intensity is the light intensity distribution information. For example, the light intensity may be a value offset from the maximum value by a fixed value. It goes without saying that the monotonic decrease includes a linear decrease.

In the image processing apparatus according to still another aspect of the present invention, the image processing apparatus includes a distribution information creating unit that creates distribution information indicating a frequency distribution or a frequency distribution of the light intensity based on the light intensity distribution information, and the composition ratio determining unit is created The first light intensity corresponding to the first vertex having a high frequency or frequency and the second light intensity corresponding to the second vertex having a high frequency or frequency based on the distribution information that is greater than the first light intensity When the second light intensity is obtained and the composition ratio is determined for each corresponding pixel or region of the plurality of non-light-emitting images, if the light intensity is equal to or lower than the first light intensity, the mixing ratio of the second non-light-emitting images Is set to the maximum value, the light intensity equal to or higher than the second light intensity is set to the minimum mixing ratio of the second non-emission image, and the light intensity is greater than the first light intensity and smaller than the second light intensity. Monotonic mixing ratio of second non-luminous image between maximum and minimum values It is preferable to lack.

For example, outdoors in the daytime on sunny days, subjects with a wide dynamic range, from dark subjects that are not exposed to sunlight (shaded subjects) to bright subjects that are exposed to sunlight (subjects to the sun), are subject to imaging. The imaging target in this case can be roughly divided into a shaded subject and a sunny subject. When the distribution information indicating the frequency distribution or frequency distribution of the light intensity is created based on the light intensity distribution information of such a scene, the distribution information includes two peaks (first vertex and second vertex which are two vertices). ) Exists. The first vertex is a vertex corresponding to the frequency or frequency of the light intensity distribution information of the subject region corresponding to the shaded subject, and the second vertex is the frequency of the light intensity distribution information of the subject region corresponding to the sunny subject or It is a vertex corresponding to the frequency.

Then, a light intensity corresponding to the first vertex (first light intensity) and a light intensity corresponding to the second vertex (second light intensity) are obtained, and for each corresponding pixel or region of the plurality of non-luminous images. When determining the composition ratio, the mixture ratio of the second non-light-emitting image is maximized when the light intensity is equal to or lower than the first light intensity, and the second non-light-emitting image when the light intensity is equal to or higher than the second light intensity. The mixing ratio of is minimized. This is because the light intensity area below the first light intensity is considered to be a shaded area, and the area above the second light intensity is considered to be a sunny area. In the case of the light intensity between the first light intensity and the second light intensity, the mixing ratio of the second non-light-emitting image is monotonously decreased between the maximum value and the minimum value, and continuously according to the increase or decrease of the light intensity. The mixing ratio is changed. Note that this is not limited to a daytime outdoor scene on a sunny day, but a scene in which sunlight is inserted into a part of the room, a scene in which an area illuminated by an artificial light source and a shadowed area exist, etc. In the distribution curve showing the frequency distribution or frequency distribution of the light intensity, there may be two vertices.

In the image processing apparatus according to still another aspect of the present invention, the image processing apparatus includes a distribution information creating unit that creates distribution information indicating a frequency distribution or a frequency distribution of the light intensity based on the light intensity distribution information, and the composition ratio determining unit is created The light corresponding to the bottom point is obtained when the light intensity corresponding to the bottom point of the frequency or the low frequency is obtained based on the distribution information obtained and the composition ratio is determined for each corresponding pixel or region of the plurality of non-light-emitting images. When the intensity is a threshold value, the mixing ratio of the second non-light-emitting image is maximized when the light intensity is less than or equal to the threshold value, and the mixing ratio of the second non-light-emitting image is minimized when the light intensity exceeds the threshold value. It is preferable.

Similarly to the above, in the case of a daytime outdoor scene on a sunny day, there is a valley (bottom point) between two peaks in the distribution information indicating the frequency distribution or frequency distribution of the light intensity. Therefore, the bottom point is obtained from the distribution information indicating the frequency distribution or frequency distribution of the light intensity, and the light intensity corresponding to the bottom point is set as a threshold value. Is set to the maximum value (the mixing ratio of the first non-light emitting image is the minimum value), and when the light intensity exceeds the threshold, the mixing ratio of the second non-light emitting image is set to the minimum value (the mixing ratio of the second non-light emitting image is the maximum value). ) Is preferable. For a certain range of light intensity before and after the threshold, the mixing ratio of the second non-light-emitting image is continuously changed between the maximum value and the minimum value according to the magnitude of the light intensity. Including.

An image processing apparatus according to still another aspect of the present invention includes a luminance distribution information calculation unit that calculates luminance distribution information indicating a luminance distribution in an image of at least one non-luminescent image among a plurality of non-luminescent images. The composition ratio determining unit preferably determines the composition ratio of the plurality of non-light-emitting images based on the light intensity distribution information and the luminance distribution information.

Thereby, not only the light intensity distribution information but also the conventional luminance distribution information can be taken into consideration to determine the composite ratio of a plurality of non-luminous images.

In the image processing device according to still another aspect of the present invention, the image processing apparatus includes a detection unit that detects a pixel or a region where the flash light does not reach in the light emission image, and the combination ratio determination unit includes the pixel or the region where the flash light does not reach When determining a composition ratio for a region, it is preferable to determine a composition ratio of a plurality of non-light-emitting images based only on luminance distribution information.

For a pixel or region where the flash light has not reached, the light intensity corresponding to that pixel or region is unknown. Therefore, in this case, it is preferable to determine the composite ratio of a plurality of non-light-emitting images based on the luminance distribution information as in the conventional case.

In the image processing device according to still another aspect of the present invention, the composition ratio determination unit determines a composition ratio for each pixel or region corresponding to each of the plurality of non-light-emitting images, and indicates a parameter indicating the position of the pixel or region. I, the mixing ratio of the second non-light-emitting image captured by the exposure condition having the largest exposure value among the plurality of non-light-emitting images, αi, the mixing ratio of the second non-light-emitting image based on the luminance distribution information, Assuming that the mixing ratio that is smaller as the luminance is higher is βi, the mixing ratio γi of the second non-luminescent image based on the light intensity distribution information and the luminance distribution information

It is preferable to calculate by γi = αi × βi.

In the image processing apparatus according to still another aspect of the present invention, the light emission image exposure condition is preferably the exposure condition having the smallest exposure value among the plurality of exposure conditions corresponding to the plurality of non-light emission images. As a result, it is possible to increase the amount of flash light emitted when capturing a light-emitting image, and to minimize the pixels that have not reached the flash light.

An imaging apparatus according to still another aspect of the present invention includes an imaging unit that can image a subject under different exposure conditions, a flash light emitting unit that emits flash light, and a flash control that controls flash light emitted from the flash light emitting unit. And an image processing device as described above, and the image acquisition unit includes a plurality of non-light-emitting images captured by the imaging unit under different exposure conditions under non-light emission of the flash light emitted from the flash light-emitting unit. Then, under the light emission of the flash light emitted from the flash light emitting unit, the light emitting image captured by the imaging unit under the same exposure condition as the exposure condition of any one of the plurality of non-light emitting images is acquired.

In the imaging device according to still another aspect of the present invention, the flash control unit dims the flash light from the flash light emitting unit, and does not perform the dimming light emission with the dimming image captured by the imaging unit under the dimming light emission. Based on the image captured by the imaging unit, if a region where the dimmed flash light has not reached is detected, the maximum amount of light emission that does not saturate the region where the dimmed flash light has reached is calculated. The flash light is preferably emitted from the flash light emitting unit at the maximum light emission amount calculated when the light emission image is acquired. By determining the light emission amount during the main light emission in this way, it is possible to minimize the pixels that have not reached the flash light.

In the imaging device according to still another aspect of the present invention, it is preferable that the flash control unit calculates the maximum light emission amount based on the average reflectance of the subject and the distance distribution information acquired by the distance distribution information acquisition unit. As a result, it is possible to calculate the maximum light emission amount in which the area where the flash light reaches does not blow out.

In the imaging device according to still another aspect of the present invention, the flash control unit dims the flash light from the flash light emitting unit, and does not perform the dimming light emission with the dimming image captured by the imaging unit under the dimming light emission. When it is detected that the flash light reaches the entire region of the light control image based on the image captured by the image capturing unit, the image acquisition unit preferably acquires the light control image as a light emission image.

This is because when the flash light reaches the entire area of the light control image, the light intensity distribution information of the entire area can be calculated, and it is not necessary to perform the main light emission again.

In the imaging device according to still another aspect of the present invention, the flash control unit dims the flash light from the flash light emitting unit, and does not perform the dimming light emission with the dimming image captured by the imaging unit under the dimming light emission. An area where the flash light has not reached the light control image is detected based on the image captured by the imaging section, and the distance of the area where the flash light has not reached is emitted from the flash light emission section based on the distance distribution information. In the case where the flash light is farther than the reach distance of the maximum light emission amount, the image acquisition unit preferably acquires the light control image as the light emission image.

If the distance of the area where the flashlight of the dimming light emission has not reached is longer than the reachable distance of the maximum light emission amount of the flashlight, the flashlight cannot reach the pixels that have not reached the flashlight even if the main light is emitted. This is because there is no need to make a new light emission.

According to still another aspect of the present invention, there is provided an image processing method for acquiring a plurality of non-light-emitting images that are images of the same subject and that are captured under different exposure conditions under non-flash light emission. , A step of obtaining a light-emitting image captured under the same exposure condition as the exposure condition of any one of the plurality of non-light-emitting images under flash light emission, and distance distribution information indicating a distribution of the distance of the subject The light intensity distribution information of the light applied to the subject based on the light emission image, the non-light emission image captured under the same exposure conditions as the light emission image, and the distance distribution information; Determining a composite ratio of the plurality of non-light-emitting images based on the intensity distribution information; combining the plurality of non-light-emitting images according to the composite ratio; And generating a composite image di is enlarged, the.

In the image processing method according to still another aspect of the present invention, the step of determining the composition ratio is based on the light intensity distribution information when determining the composition ratio for each corresponding pixel or region of the plurality of non-light-emitting images. In a pixel or region having a high light intensity, the mixing ratio of the first non-light-emitting images captured under the exposure condition with the smallest exposure value among the plurality of non-light-emitting images is represented by the highest exposure value among the plurality of non-light-emitting images. The mixing ratio of the second non-light emitting image is set larger than the mixing ratio of the first non-light emitting image in a pixel or region having a low light intensity. Is preferred.

An image processing program according to still another aspect of the present invention has a function of acquiring a plurality of non-light-emitting images that are images of the same subject and that are captured under different exposure conditions under non-flash light emission. , A function of acquiring a light-emitting image captured under the same exposure condition as the exposure condition of any one of the plurality of non-light-emitting images under flash light emission, and distance distribution information indicating the distance distribution of the subject A light emission image, a non-light emission image captured under the same exposure conditions as the light emission image, and a distance distribution information based on the light intensity distribution information of the light applied to the subject, A function that determines the composite ratio of multiple non-light-emitting images based on intensity distribution information, and combines multiple non-light-emitting images according to the composite ratio, expanding the dynamic range. And generating a composite image, to realize the computer.

In the image processing program according to still another aspect of the present invention, the function of determining the composition ratio is based on the light intensity distribution information when determining the composition ratio for each corresponding pixel or region of the plurality of non-light-emitting images. In a pixel or region having a high light intensity, the mixing ratio of the first non-light-emitting images captured under the exposure condition with the smallest exposure value among the plurality of non-light-emitting images is represented by the highest exposure value among the plurality of non-light-emitting images. The mixing ratio of the second non-light emitting image is set larger than the mixing ratio of the first non-light emitting image in a pixel or region having a low light intensity. Is preferred.

According to the present invention, the composition ratio of the plurality of non-light-emitting images is determined according to the brightness (light intensity) of the light hitting the subject, and the HDR composition is performed. It is possible to obtain an HDR image with excellent gradation characteristics in which the difference in brightness is adjusted according to the brightness of the light hitting

FIG. 1 is a perspective view of an imaging apparatus according to the present invention as viewed obliquely from the front. FIG. 2 is a rear view of the imaging apparatus. FIG. 3 is a block diagram illustrating an embodiment of the internal configuration of the imaging apparatus. FIG. 4 is a front view showing a configuration example of the image sensor. FIG. 5 is a functional block diagram showing the first embodiment of the main body side CPU 220 functioning as the image processing apparatus according to the present invention. FIG. 6 is a graph showing the relationship between the light intensity distribution information S _i and the mixing ratio αi of the second non-light-emitting image. FIG. 7 is a graph showing another relationship between the light intensity distribution information Si and the mixing ratio αi of the second non-luminescent image. FIG. 8 is a graph showing still another relationship between the light intensity distribution information Si and the mixing ratio αi of the second non-emission image. FIG. 9 is a functional block diagram showing a second embodiment of the main body side CPU 220 functioning as an image processing apparatus according to the present invention. FIG. 10 is a frequency distribution diagram showing an example of the frequency distribution of light intensity. FIG. 11 is a graph showing still another relationship between the light intensity distribution information Si and the mixing ratio αi of the second non-luminescent image. FIG. 12 is a frequency distribution diagram similar to the frequency distribution of light intensity shown in FIG. It is a frequency distribution figure which shows an example of frequency distribution of light intensity. FIG. 13 is a graph showing still another relationship between the light intensity distribution information Si and the mixing ratio αi of the second non-luminescent image. FIG. 14 is a functional block diagram showing a third embodiment of the main body side CPU 220 functioning as the image processing apparatus according to the invention. FIG. 15 is a functional block diagram showing a fourth embodiment of the main body side CPU 220 functioning as an image processing apparatus according to the invention. FIG. 16 is a functional block diagram showing a fifth embodiment of the main body side CPU 220 functioning as an image processing apparatus according to the invention. FIG. 17 is a flowchart showing an embodiment of an image processing method according to the present invention. FIG. 18 is a flowchart showing detailed operations in step S12 of FIG. FIG. 19 is an external view of a smartphone which is an embodiment of an imaging apparatus according to the present invention. FIG. 20 is a block diagram illustrating a configuration of the smartphone. FIG. 21 is a diagram used for explaining a problem of the conventional HDR synthesis.

Hereinafter, preferred embodiments of an image processing device, a method, a program, and an imaging device according to the present invention will be described with reference to the accompanying drawings.

<Appearance of imaging device>

FIG. 1 is a perspective view of an imaging apparatus according to the present invention as viewed obliquely from the front, and FIG. 2 is a rear view of the imaging apparatus.

As shown in FIG. 1, the imaging apparatus 10 is a mirrorless digital single-lens camera including an interchangeable lens 100 and a camera body 200 to which the interchangeable lens 100 can be attached and detached.

In FIG. 1, a main body mount 260 to which the interchangeable lens 100 is attached, a finder window 20 of an optical finder, and the like are provided on the front surface of the camera main body 200, and a shutter release switch 22 and a shutter are mainly provided on the upper surface of the camera main body 200. A speed dial 23, an exposure correction dial 24, a power lever 25, and a built-in flash 30 are provided.

As shown in FIG. 2, a liquid crystal monitor 216, an optical viewfinder eyepiece 26, a MENU / OK key 27, a cross key 28, a playback button 29, and the like are mainly provided on the back of the camera body 200.

The liquid crystal monitor 216 functions as a display unit for displaying various menu screens in addition to displaying a live view image in the shooting mode, reproducing and displaying an image captured in the playback mode, and displaying various information to the user. Functions as a notification unit for notification. The MENU / OK key 27 is an operation having both a function as a menu button for instructing to display a menu on the screen of the liquid crystal monitor 216 and a function as an OK button for instructing confirmation and execution of selection contents. Key. The cross key 28 is an operation unit for inputting instructions in four directions, up, down, left, and right, and functions as a multi-function key for selecting an item from the menu screen and instructing selection of various setting items from each menu. The up and down keys of the cross key 28 function as a zoom switch at the time of imaging or a playback zoom switch in the playback mode, and the left key and the right key are frame advance (forward and reverse) buttons in the playback mode. Function as. In addition, it also functions as an operation unit that designates an arbitrary subject for focus adjustment from a plurality of subjects displayed on the liquid crystal monitor 216.

Further, by using the menu screen displayed on the MENU / OK key 27, the cross key 28, and the liquid crystal monitor 216, in addition to the still image capturing mode for capturing one still image, still images are continuously captured. Various shooting modes including HDR shooting mode that captures a plurality of images used for HDR composition that expands the dynamic range, and moving image capturing mode that captures moving images, which are a kind of continuous shooting mode and continuous shooting mode Can do.

The playback button 29 is a button for switching to a playback mode in which the recorded still image or moving image is displayed on the liquid crystal monitor 216.

<Internal configuration of imaging device>

[interchangeable lens]

FIG. 3 is a block diagram illustrating an embodiment of the internal configuration of the imaging apparatus 10.

The interchangeable lens 100 that functions as an imaging optical system constituting the imaging apparatus 10 is manufactured in accordance with the communication standard of the camera body 200, and can communicate with the camera body 200 as described later. It is an interchangeable lens. The interchangeable lens 100 includes an imaging optical system 102, a focus lens control unit 116, an aperture control unit 118, a lens side CPU (Central Processing Unit) 120, a flash ROM (Read Only Memory) 126, a lens side communication unit 150, and a lens mount. 160.

The imaging optical system 102 of the interchangeable lens 100 includes a lens group 104 including a focus lens and a diaphragm 108.

The focus lens control unit 116 moves the focus lens in accordance with a command from the lens side CPU 120 and controls the position (focus position) of the focus lens. The diaphragm control unit 118 controls the diaphragm 108 in accordance with a command from the lens side CPU 120.

The lens-side CPU 120 controls the interchangeable lens 100 as a whole, and includes a ROM 124 and a RAM (Random Access Memory) 122.

The flash ROM 126 is a nonvolatile memory that stores programs downloaded from the camera body 200.

The lens-side CPU 120 performs overall control of each part of the interchangeable lens 100 using the RAM 122 as a work area according to a control program stored in the ROM 124 or the flash ROM 126.

The lens side communication unit 150 is connected to the camera body 200 via a plurality of signal terminals (lens side signal terminals) provided on the lens mount 160 in a state where the lens mount 160 is attached to the body mount 260 of the camera body 200. Communication. That is, the lens side communication unit 150 sends a request signal and a response signal to and from the main body side communication unit 250 of the camera body 200 connected via the lens mount 160 and the main body mount 260 in accordance with a command from the lens side CPU 120. Transmission / reception (bidirectional communication) is performed to notify the camera body 200 of lens information (focus lens position information, focal length information, aperture information, etc.) of each optical member of the imaging optical system 102.

The interchangeable lens 100 also includes a detection unit (not shown) that detects focus lens position information and aperture information. Here, the aperture information is information indicating the aperture value (F value) of the aperture 108, the aperture diameter of the aperture 108, and the like.

In order to respond to the lens information request from the camera body 200, the lens side CPU 120 preferably stores various lens information including the detected focus lens position information and aperture information in the RAM 122. Further, the lens information is detected when there is a request for lens information from the camera body 200, or is detected when the optical member is driven, or at a fixed period (a period sufficiently shorter than the frame period of the moving image). It is detected and the detection result can be held.

[Camera body]

3 includes an image sensor 201, an image sensor control unit 202, an analog signal processing unit 203, an A / D (Analog / Digital) converter 204, an image input controller 205, a digital signal. Processing unit 206, RAM 207, compression / decompression processing unit 208, media control unit 210, memory card 212, display control unit 214, liquid crystal monitor 216, main body CPU 220, operation unit 222, clock unit 224, flash ROM 226, ROM 228, AF (Autofocus ) Control unit 230, AE (Auto Exposure) control unit 232, white balance correction unit 234, wireless communication unit 236, GPS (Global Positioning System) reception unit 238, power supply control unit 240, battery 242, main unit side communication unit 250, main unit The mount 260 and the built-in flash 30 (FIG. 1) are configured. It comprises: (focal-plane shutter FPS) 280 and the FPS control unit 296, Rush emitting unit 270, the flash control unit 272, a focal plane shutter.

<Configuration of image sensor>

The image sensor 201 is composed of a complementary metal-oxide semiconductor (CMOS) type color image sensor. The image sensor 201 is not limited to the CMOS type, but may be an XY address type or a CCD (Charge Coupled Device) type image sensor.

FIG. 4 is a front view showing the configuration of the image sensor 201.

As shown in FIG. 4, the image sensor 201 includes an image sensor having a plurality of pixels configured by photoelectric conversion elements (light receiving cells) arranged two-dimensionally in the horizontal direction (x direction) and the vertical direction (y direction). In this case, the first and second pixels S _{A and} S _B are configured to selectively receive light by dividing the light beam incident through the interchangeable lens 100 by the pupil dividing means shown below.

As shown in FIG. 4, the image sensor 201 employs a pupil image separation method using a pupil imaging lens (microlens) 201A as pupil dividing means.

In the pupil image separation method, a plurality of (two in this example) pixels (first pixel S _A and second pixel S _B ) are assigned to one microlens 201A, and are incident on one microlens 201A. pupil image which is pupil division by the microlens 201A, the first pixel _{S a} two (a pair), and is received by the second pixel _{S B.} Thus, the pupil image is separated according to the incident angle of the light entering the microlens 201A, the corresponding pair of the first pixel S _A, is received by the second pixel S _B.

A pair of first pixel S _A, second pixel S _B corresponds to a pair of phase difference pixels available on the phase difference AF. The pair of first pixel S _A, pixel values obtained by adding the pixel values obtained from the pixels of the second pixel S _B are to become equal to the pixel value of normal pixels that are not pupil division, the pair of first The pixel S _A and the second pixel S _B can be handled as normal pixels (one pixel).

As shown in FIG. 4, the pair of first pixel S _{A and} second pixel S _B of the image sensor 201 includes red

A color filter of any one of the three primary color filters (R filter, G filter, B filter) of (R), green (G), and blue (B) is arranged according to a predetermined color filter array. ing. The color filter array shown in FIG. 4 is a general Bayer array, but the color filter array is not limited to this, and may be another color filter array such as a Trans (registered trademark) array, for example.

Returning to FIG. 3, the optical image of the subject formed on the light receiving surface of the image sensor 201 by the imaging optical system 102 of the interchangeable lens 100 is converted into an electrical signal by the image sensor 201. Charges corresponding to the amount of incident light are accumulated in each pixel of the image sensor 201 (the first pixel S _{A and} the second pixel S _B shown in FIG. 4), and the charge accumulated in each pixel from the image sensor 201. An electric signal corresponding to the amount (signal charge) is read out as an image signal. That is, the first pixel S _A, a signal corresponding to the charge amount accumulated in the second pixel S _B (first pixel S _A, pixel values of the second pixel S _B) can be read independently.

The image sensor control unit 202 performs reading control of an image signal from the image sensor 201 in accordance with a command from the main body side CPU 220. In addition, when a still image is captured, the image sensor control unit 202 reads all lines of the image sensor 201 with the FPS 280 closed after the exposure time is controlled by opening and closing the FPS 280. In addition, the image sensor 201 and the image sensor control unit 202 of this example sequentially perform an exposure operation for each of at least one line or each pixel (that is, sequentially reset for each line or each pixel to accumulate charges. This is a method of starting and reading out the accumulated electric charge), and can be driven by a so-called rolling shutter method, and in particular, a function of capturing a moving image or a live view image by a rolling shutter method with the FPS 280 opened. Have

The analog signal processing unit 203 performs various types of analog signal processing on an analog image signal obtained by imaging the subject with the image sensor 201. The analog signal processing unit 203 includes a sampling hold circuit, a color separation circuit, an AGC (Automatic Gain Control) circuit, and the like. The AGC circuit functions as a sensitivity adjustment unit that adjusts the sensitivity at the time of imaging (ISO: International Organization for Standardization), adjusts the gain of an amplifier that amplifies the input image signal, and the signal level of the image signal is Try to be in the proper range. The A / D converter 204 converts the analog image signal output from the analog signal processing unit 203 into a digital image signal.

Image data (mosaic image data) for each pixel of RGB output via the image sensor 201, the analog signal processing unit 203, and the A / D converter 204 when capturing a still image or moving image is transferred from the image input controller 205 to the RAM 207. Is temporarily stored.

When the image sensor 201 is a CMOS type image sensor, the analog signal processing unit 203 and the A / D converter 204 are often built in the image sensor 201.

The digital signal processing unit 206 performs various types of digital signal processing on the image data stored in the RAM 207. The digital signal processing unit 206 appropriately reads out image data stored in the RAM 207, and performs offset processing, gain control processing including sensitivity correction, gamma correction processing, demosaicing processing (demosaicing processing, simultaneous processing) on the read image data. Digital signal processing such as RGB / YCrCb conversion processing or the like, and image data after the digital signal processing is stored in the RAM 207 again. Note that the demosaic process is a process of calculating color information of all RGB for each pixel from a mosaic image composed of RGB, for example, in the case of an image sensor composed of RGB color filters, and mosaic data (dot sequential RGB data). ) To generate the RGB 3 plane image data.

The RGB / YCrCb conversion process is a process of converting the synchronized RGB data into luminance data (Y) and color difference data (Cr, Cb).

The compression / decompression processing unit 208 performs compression processing on the uncompressed luminance data Y and color difference data Cb, Cr once stored in the RAM 207 when recording a still image or a moving image. In the case of a still image, the image is compressed in, for example, JPEG (Joint Photographic coding Experts Group) format. Compress in H.264 format. The image data compressed by the compression / decompression processing unit 208 is recorded on the memory card 212 via the media control unit 210. In addition, the compression / decompression processing unit 208 performs decompression processing on the compressed image data obtained from the memory card 212 via the media control unit 210 in the playback mode, and generates uncompressed image data.

The media control unit 210 performs control to record the image data compressed by the compression / decompression processing unit 208 in the memory card 212. In addition, the media control unit 210 performs control for reading compressed image data from the memory card 212.

The display control unit 214 performs control to display uncompressed image data stored in the RAM 207 on the liquid crystal monitor 216. The liquid crystal monitor 216 is configured by a liquid crystal display device, but may be configured by a display device such as organic electroluminescence instead of the liquid crystal monitor 216.

When a live view image is displayed on the liquid crystal monitor 216, digital image signals continuously generated by the digital signal processing unit 206 are temporarily stored in the RAM 207. The display control unit 214 converts the digital image signal temporarily stored in the RAM 207 into a display signal format and sequentially outputs it to the liquid crystal monitor 216. Thereby, the captured image is displayed on the liquid crystal monitor 216 in real time, and the liquid crystal monitor 216 can be used as an electronic viewfinder.

The shutter release switch 22 is an imaging instruction unit for inputting an imaging instruction of a still image or a moving image, and is configured by a two-stage stroke type switch composed of so-called “half press” and “full press”.

In the still image capturing mode, an S1 ON signal is output when the shutter release switch 22 is half-pressed, an S2 ON signal is output when the shutter release switch 22 is further pressed halfway down, and an S1 ON signal is output. Then, the main body side CPU 220 executes shooting preparation processing such as AF control (automatic focus adjustment) and AE control (automatic exposure control), and when the S2 ON signal is output, executes still image shooting processing and recording processing. To do.

Note that AF control and AE control are automatically performed when the auto mode is set by the operation unit 222, respectively, and AF control and AE control are not performed when the manual mode is set. Needless to say.

In the moving image capturing mode, when the S2 ON signal is output when the shutter release switch 22 is fully pressed, the camera body 200 enters a moving image recording mode in which moving image recording is started, and image processing of the moving image is performed. When the shutter release switch 22 is fully pressed again and an S2 ON signal is output, the camera body 200 enters a standby state and temporarily stops the moving image recording process.

The shutter release switch 22 is not limited to a two-stroke type switch consisting of half-pressing and full-pressing, and may output an S1 ON signal and an S2 ON signal in a single operation. May be provided to output an S1 on signal and an S2 on signal.

Further, in the form in which the operation instruction is performed by the touch panel or the like, the operation instruction may be output by touching an area corresponding to the operation instruction displayed on the screen of the touch panel as these operation means. The form of the operation means is not limited to these as long as the preparation process or the imaging process is instructed.

The still image or moving image acquired by the imaging is compressed by the compression / decompression processing unit 208, and the compressed image data includes the required attachment of the imaging date and time, GPS information, and imaging conditions (F value, shutter speed, ISO sensitivity, etc.). The information is converted into an image file added to the header, and then stored in the memory card 212 via the media control unit 210.

The main body side CPU 220 controls the overall operation of the camera main body 200 and the driving of the optical members of the interchangeable lens 100, etc. Based on the input from the operation unit 222 including the shutter release switch 22, the main body side CPU 220 The interchangeable lens 100 is controlled.

The clock unit 224 measures time based on a command from the main body CPU 220 as a timer. The clock unit 224 measures the current date and time as a calendar.

The flash ROM 226 is a non-volatile memory that can be read and written, and stores setting information.

The ROM 228 stores a camera control program executed by the main body side CPU 220, an image alignment auxiliary program for executing imaging in the image capturing mode for composition according to the present invention, defect information of the image sensor 201, various types used for image processing, and the like. Parameters and tables are stored. The main body side CPU 220 controls each part of the camera main body 200 and the interchangeable lens 100 using the RAM 207 as a work area in accordance with a camera control program stored in the ROM 228 or an image alignment auxiliary program.

The AF control unit 230 that functions as an automatic focus adjustment unit calculates a defocus amount necessary for controlling the phase difference AF, and based on the calculated defocus amount, a position (focus position) command to which the focus lens should move is calculated. Is notified to the interchangeable lens 100 via the main body side CPU 220 and the main body side communication unit 250.

Here, the AF control unit 230 includes a phase difference detection unit and a defocus amount calculation unit. The phase difference detection unit is in an AF area of the image sensor 201 (an area where the main subject specified by the user exists, an area of the main subject automatically detected by face detection, or an area set by default). a first pixel group including a first pixel S _a which color filters of the same color are disposed, respectively the pixel data from the second pixel group composed of the first pixel S _a (the first pixel value and second pixel value) The phase difference is detected based on the first pixel value and the second pixel value. This phase difference is obtained when the correlation between the plurality of first pixel values of the first pixel group and the plurality of second pixel values of the second pixel group becomes maximum (the plurality of first pixel values and the plurality of second pixel values It can be calculated from the amount of shift in the pupil division direction between the first pixel value and the second pixel value (when the integrated absolute difference value is minimized).

The defocus amount calculation unit calculates the defocus amount by multiplying the phase difference detected by the phase difference detection unit by a coefficient corresponding to the current F value (ray angle) of the interchangeable lens 100.

The focus lens position command corresponding to the defocus amount calculated by the AF control unit 230 is notified to the interchangeable lens 100, and the lens side CPU 120 of the interchangeable lens 100 that receives the focus lens position command receives the focus lens control unit 116. The focus lens is moved via, and the position (focus position) of the focus lens is controlled.

The AE control unit 232 is a part that detects the brightness of the subject (subject brightness), and is a numerical value (exposure value (EV value)) necessary for AE control and AWB (Auto White Balance) control corresponding to the subject brightness. )) Is calculated. The AE control unit 232 calculates an EV value based on the brightness of the image acquired via the image sensor 201, the shutter speed when the brightness of the image is acquired, and the F value.

The main body side CPU 220 can determine the F value, shutter speed, and ISO sensitivity from a predetermined program diagram based on the EV value obtained from the AE control unit 232, and perform AE control.

The white balance correction unit 234 calculates white balance gains (WB (White Balance) gains) Gr, Gg, Gb for each color data of RGB data (R data, G data, and B data). White balance correction is performed by multiplying the B data by the calculated WB gains Gr, Gg, and Gb, respectively. Here, as a method of calculating the WB gains Gr, Gg, and Gb, the subject is illuminated based on scene recognition (outdoor / indoor determination, etc.) based on the brightness (EV value) of the subject, the color temperature of ambient light, and the like. A method of reading out a WB gain corresponding to a specified light source type from a storage unit in which an appropriate WB gain is stored in advance for each light source type is conceivable, but at least using an EV value Other known methods for determining Gr, Gg, Gb are conceivable.

The wireless communication unit 236 is a part that performs short-range wireless communication of a standard such as Wi-Fi (Wireless Fidelity) (registered trademark), Bluetooth (registered trademark), etc., and with a peripheral digital device (a mobile terminal such as a smartphone). Necessary information is sent and received between the two.

The GPS receiving unit 238 receives GPS signals transmitted from a plurality of GPS satellites according to instructions from the main body side CPU 220, executes positioning calculation processing based on the received plurality of GPS signals, and the latitude and longitude of the camera main body 200 And GPS information consisting of altitude. The acquired GPS information can be recorded in the header of the image file as attached information indicating the imaging position of the captured image.

The power supply control unit 240 applies power supply voltage supplied from the battery 242 to each unit of the camera main body 200 in accordance with a command from the main body side CPU 220. In addition, the power supply control unit 240 supplies the power supply voltage supplied from the battery 242 to each unit of the interchangeable lens 100 via the main body mount 260 and the lens mount 160 in accordance with a command from the main body side CPU 220.

The lens power switch 244 switches on and off the power supply voltage applied to the interchangeable lens 100 via the main body mount 260 and the lens mount 160 and switches the level in accordance with a command from the main body side CPU 220.

The main body side communication unit 250 transmits / receives a request signal and a response signal to / from the lens side communication unit 150 of the interchangeable lens 100 connected via the main body mount 260 and the lens mount 160 in accordance with a command from the main body side CPU 220 ( (Bidirectional communication). The main body mount 260 is provided with a plurality of terminals 260A as shown in FIG. 1, and when the interchangeable lens 100 is attached to the camera main body 200 (the lens mount 160 and the main body mount 260 are connected), the main body A plurality of terminals 260A (FIG. 1) provided on the mount 260 and a plurality of terminals (not shown) provided on the lens mount 160 are electrically connected, and the main body side communication unit 250 and the lens side communication unit 150 are connected. Bi-directional communication is possible.

The built-in flash 30 (FIG. 1) is, for example, a TTL (Through The Lens) automatic dimming flash, and includes a flash light emitting unit 270 and a flash control unit 272.

The flash control unit 272 has a function of adjusting the light emission amount (guide number) of flash light emitted from the flash light emitting unit 270. That is, the flash control unit 272 pre-flashes (dimming light emission) flash light with a small light emission amount from the flash light emitting unit 270 in synchronization with the flash imaging instruction from the main body side CPU 220, and the imaging optical system 102 of the interchangeable lens 100. The amount of flash light that is actually emitted is determined based on reflected light (including ambient light) incident via the flash light, and the flash light emission unit 270 emits the flash light of the determined emission amount (main emission). Note that, when the HDR imaging mode is selected, the flash control unit 272 performs light emission control different from the normal flash emission image imaging, and details thereof will be described later.

The FPS 280 constitutes a mechanical shutter of the imaging apparatus 10 and is disposed immediately before the image sensor 201. The FPS control unit 296 controls the opening and closing of the front and rear curtains of the FPS 280 based on input information (S2 ON signal, shutter speed, etc.) from the main body CPU 220, and controls the exposure time (shutter speed) in the image sensor 201. To do.

Next, control of the imaging apparatus 10 that captures a plurality of images with different exposures used for HDR composition that expands the dynamic range in the HDR imaging mode and performs HDR composition of the captured images will be described.

[First Embodiment of HDR Processing]

FIG. 5 is a functional block diagram showing a first embodiment of the main body side CPU 220 functioning as an image processing apparatus according to the present invention, and HDR synthesis is based mainly on a plurality of images for HDR synthesis captured in the HDR imaging mode. FIG. 6 is a functional block diagram when image processing (HDR processing) is executed.

When performing imaging in the HDR imaging mode, the main body side CPU 220 mainly controls imaging control in the HDR imaging mode, and as shown in FIG. 5, a distance distribution information acquisition unit 220A, a light intensity distribution information calculation unit 220B, It functions as a composition ratio determination unit 220C and an image composition unit 220D.

That is, when performing imaging in the HDR imaging mode, the main body side CPU 220 captures a plurality (two in this example) of non-light-emitting images under different exposure conditions under non-flash light emission and under flash light emission. Imaging of a luminescent image is performed under the same exposure condition as that of any one of the plurality of non-luminescent images.

Here, the plurality of exposure conditions corresponding to the plurality of non-light-emitting images include an under-exposure exposure condition and an over-exposure exposure condition rather than an appropriate exposure condition. In this example, the main body side CPU 220 captures two non-light-emitting images under two exposure conditions of underexposure and overexposure with respect to the appropriate exposure determined according to the EV value calculated by the AE control unit 232. Let it be done. The exposure correction value (for example, ± 3 EV) for underexposure and overexposure with respect to proper exposure may be arbitrarily set by the user, or automatically according to the brightness (EV value) of the object scene. It may be possible to set it automatically.

Further, the main body side CPU 220 reads the light-emitting image under the same exposure condition (in this example, the under-exposure exposure condition) as any non-light-emitting image of the plurality of non-light-emitting images under flash light emission. Perform imaging.

Note that the two non-light-emitting images and the one light-emitting image are images captured with respect to the same subject, and are continuously shot images that are continuously captured with the imaging interval as short as possible. preferable.

In FIG. 5, the image acquisition unit 221 includes a non-light-emitting image (first non-light-emitting image) captured by the image pickup unit (the interchangeable lens 100 and the image sensor 201) under underexposure under the non-flash light, and an overshoot. Under the light emission of the non-light-emitting image (second non-light-emitting image) captured by exposure and the flash light emitted from the flash light-emitting unit 270, the image is captured by the image-capturing unit under the same exposure condition as the exposure condition of the first non-light-emitting image. This is a part for acquiring the emitted light image.

For example, the image acquisition unit 221 may acquire the first non-light-emitting image, the second non-light-emitting image, and the light-emitting image temporarily stored in the RAM 207 via the image input controller 205 from the RAM 207, or by the imaging unit. The captured first non-emission image, second non-emission image, and emission image may be directly acquired via the image input controller 205.

The image input controller 205 of this embodiment, a pair of first pixel _{S A} of the image sensor 201 shown in FIG. 4, by adding the pixel value of the second pixel _{S B,} 1 pixel per one microlens 201A In addition to the first non-light-emitting image, the second non-light-emitting image, and the light-emitting image, the first non-light-emitting image, the second non-light-emitting image, and the light-emitting image of the first non-light-emitting image. _A first image including a pixel group including only the pixel S _A and a second image including a pixel group including only the second pixel S _B are acquired. The first image and the second image are phase difference images having a phase difference according to the subject distance.

The distance distribution information acquisition unit 220A acquires a first image and a second image that are phase difference images having a phase difference from the image acquisition unit 221, and based on the first image and the second image, Distance distribution information indicating the distance distribution of the subject is acquired.

That is, the distance distribution information acquisition unit 220A calculates a deviation amount (phase difference) between a feature point that is a feature in the first image and a feature point in the second image corresponding to each feature point. The defocus amount of each feature point is calculated based on the phase difference and the current F value of the diaphragm 108, and the calculated defocus amount of each feature point and the lens of the current interchangeable lens 100 (the focus lens of the imaging optical system 102). The distance between each feature point (subject) is calculated from the position. For example, a feature point with a defocus amount of zero is at the in-focus distance at which the lens position of the current focus lens is focused, and as the defocus amount of each feature point shifts to the front pin side or the rear bin side The distance between each feature point is the distance moved to the far side or the near side from the in-focus distance.

The light intensity distribution information calculation unit 220B acquires the light emission image and the first non-light emission image captured under the same exposure conditions from the image acquisition unit 221, and indicates the distribution of the distance of the subject in the image from the distance distribution information acquisition unit 220A. Get distance distribution information. Then, based on the acquired light emission image, first non-light emission image, and distance distribution information, light intensity distribution information that is distribution information of an index indicating the intensity of light (not flash light but ambient light) applied to the subject Is calculated.

Hereinafter, the light intensity distribution information calculation method by the light intensity distribution information calculation unit 220B will be described in detail.

The parameter indicating the coordinates of each pixel of the first non-light-emitting image is i, the luminance value of the pixel i is Y _i , the luminance value of the pixel i at the same coordinate of the light-emitting image is Y _Li , and distance information d _i (= The distance from the imaging device 10 to the subject reflected there), the light intensity distribution information S _i is _expressed by the following equation:

[Equation 1]

S _i = Y _i / {(Y _Li −Y _i ) × (d _i ² )}

Defined by The parameter i is not limited to the parameter indicating the pixel coordinates, and may be a parameter indicating the position of each region obtained by dividing the image into a plurality of regions. In this case, the luminance values Y _i , Y _Li , distance information d _i , and light intensity distribution information S _i are also the luminance value (representative luminance value), distance information, and light intensity distribution information for each region i.

Hereinafter, the formula 1 will be described.

Yi of the first non-light-emitting image is Yi of the first non-light-emitting image, where B _i is the intensity of light striking the subject, R _{i is} the reflectance of the subject, and K is the photoelectric conversion coefficient by the image sensor 201. Is represented by the following equation.

[Equation 2]

Y _i = B _i × R _i × K

Y _Li of the luminescent image is expressed by the following equation where F _i is the intensity of flash light hitting the subject.

[Equation 3]

Y _Li = (B _i + F _i ) × R _i × K

F _i is expressed by the following equation based on the inverse square law regarding light attenuation, where Fs is the intensity of flash light at a unit distance from the flash light emitting unit 270.

[Equation 4]

F _i = Fs / (d _i ² )

[Equation 1] of the light intensity distribution information can be transformed into [Equation 5] by [Equation 2] to [Equation 4].

[Equation 5]

S _i = Y _i / {(Y _Li −Y _i ) × (d _i ² )} = (B _i × R _i × K) / [{(B _i + F _i ) × R _i × K− (B _i × R _i × K)} × (d _i ² )] = (B _i / {F _i × (d _i ² )} = B _i / Fs

Since Fs is constant regardless of the pixel i, the light intensity distribution information S _i can be said to be an index by which the light intensity B _i hitting the subject for each pixel i (= 1 to N (N: number of pixels) can be relatively compared. .

The combination ratio determination unit 220C determines a combination ratio of a plurality of non-light-emitting images (first non-light-emitting image and second non-light-emitting image) based on the light intensity distribution information. Since a portion where the value of the light intensity distribution information S _i is large is strongly irradiated, it is preferable to use the first non-light-emitting image that is an under image, and light is not so much at a portion where the value of the light intensity distribution information S _i is small. Since it is not hit, it is preferable to use the second non-light emitting image which is an over image.

Next, a method for determining a composite ratio by the composite ratio determining unit 220C will be described.

<First determination method of composition ratio>

When the composition ratio determining unit 220C determines the composition ratio for each pixel corresponding to each of the plurality of non-light-emitting images, a pixel having a high light intensity based on the light intensity distribution information S _i includes a plurality of non-light-emitting images. The mixing ratio of the first non-light emitting image captured under the exposure condition with the smallest exposure value is greater than the mixing ratio of the second non-light emitting image captured under the exposure condition with the largest exposure value among the plurality of non-light emitting images. For a pixel having a small light intensity, the mixing ratio of the second non-light emitting image is made larger than the mixing ratio of the first non-light emitting image. Note that the composition ratio may be determined for each region obtained by dividing the image into a plurality of regions.

6 to 7 are graphs showing the relationship between the light intensity distribution information S _i and the mixing ratio α _i of the second non-light-emitting image, respectively.

As shown in FIG. 6, the composition ratio determining unit 220C sets the mixing ratio of the second non-light-emitting image as αi, and sets the minimum value of the light intensity distribution information S _i as S _{min. And} the maximum value as S _max as shown in FIG _. _. and when, simplest, the minimum value S _min. by ~ maximum value S _max. of the light intensity distribution information S _i, the mixing ratio αi (= 0 ~ 1) to by linear interpolation, the first non-emission image And the second non-luminescent image are determined. In this case, the mixing ratio of the first non-luminescent image is (1-αi), and the synthesis ratio of the first non-luminescent image and the second non-luminescent image is (1-αi): αi.

Further, as shown in FIG. 7, the composition ratio determining unit 220C provides a minimum value α _min. (> 0) and a maximum value α _max. (<1) for the mixing rate α i, and the minimum value S _min. _{max by.} the light intensity distribution information S _i, the mixing ratio αi minimum value alpha _min. by linear interpolation the maximum value alpha _max. range between the synthesis ratio of the first non-emission image and the second non-emission image Can be determined.

Further, as shown in FIG. 8, the composition ratio determination unit 220C provides the minimum value α _min. (> 0) and the maximum value α _max. (<1) for the mixing rate α i, and the minimum value S _min. First, by performing non-linear interpolation (non-linear interpolation so that the mixing rate αi monotonously decreases) in the range of the minimum value α _min. to the maximum value α _{max. based on} the light intensity distribution information S _i of _max. A composite ratio between the non-light emitting image and the second non-light emitting image can be determined.

Returning to FIG. 5, the image composition unit 220D composes the under-exposed first non-emission image and the over-exposed second non-emission image in accordance with the composition ratio determined by the composition ratio determination unit 220C, and the dynamic range. A composite image (HDR image) in which is enlarged is generated. That is, the image combining unit 220D combines the first non-light-emitting image and the second light-emitting image by applying the combination ratio determined for each corresponding pixel of the first non-light-emitting image and the second non-light-emitting image. When the composition ratio is determined for each area obtained by dividing the image into a plurality of areas, the composition ratio determined for each area corresponding to each of the first non-light-emitting image and the second non-light-emitting image is applied. The non-light emitting image and the second light emitting image are combined to generate an HDR image.

The HDR image generated in this way is an HDR image with excellent gradation characteristics in which the contrast of light and darkness is adjusted according to the brightness of the light hitting the subject without losing the contrast of the subject.

[Second Embodiment of HDR Processing]

FIG. 9 is a functional block diagram showing a second embodiment of the main body side CPU 220 functioning as an image processing apparatus according to the present invention. In FIG. 9, parts that are the same as those in the first embodiment shown in FIG. 5 are given the same reference numerals, and detailed descriptions thereof are omitted.

The main body side CPU 220 of the second embodiment shown in FIG. 9 is different from the first embodiment shown in FIG. 5 in that the function of the distribution information creation unit 220E is added.

The distribution information creation unit 220E calculates distribution information (for example, a frequency distribution chart or a frequency distribution table) indicating the frequency distribution of the light intensity based on the light intensity distribution information S _i calculated by the light intensity distribution information calculation unit 220B. .

FIG. 10 is a frequency distribution diagram showing an example of the frequency distribution of light intensity.

For example, a dark subject that is not exposed to sunlight (a shaded subject) outdoors on a sunny day

A subject having a wide dynamic range is a subject to be imaged, from a bright subject to sunlight (a subject in the sun), and in this case, the subject to be imaged can be roughly divided into a shaded subject and a sunny subject. In the light intensity frequency distribution diagram generated based on the light intensity distribution information of such a scene, there are two peaks (first vertex P1 and second vertex P2 which are two vertices) as shown in FIG. To do. The first vertex P1 is a vertex corresponding to the frequency of the light intensity distribution information of the subject region corresponding to the shaded subject, and the second vertex P2 is the frequency of the light intensity distribution information of the subject region corresponding to the sunny subject. The corresponding vertex.

In addition, not only in the daytime and outdoor scenes on a sunny day, but also in the case where there is a scene where sunlight is inserted into a part of the indoor, a scene illuminated by an artificial light source and a shadowed area, etc. In the frequency distribution diagram of light intensity, there can be two vertices.

In this example, distribution information indicating the frequency distribution of the light intensity is created. However, distribution information indicating the frequency distribution of the light intensity may be created, or distribution information that approximates the frequency distribution or the frequency distribution to a curve is created. May be.

Based on the distribution information created by the distribution information creation unit 220E, the composition ratio determination unit 220C shown in FIG. 9 has the first light intensity (S) corresponding to the first vertex P1 having a high light intensity frequency as shown in FIG. _dark ) and a second light intensity (S _blight ) corresponding to the second vertex P2 having a high light intensity.

Then, when the composition ratio determination unit 220C determines the composition ratio for each pixel corresponding to each of the first non-light-emitting image and the second non-light-emitting image, as shown in FIG. 11, the first light intensity (S _dark ) or less. If the light intensity is less than the second light intensity (S _blight ), the second non-light-emitting image mixing ratio αi is set to the maximum value α _max. When the light intensity is set to the minimum value α _{min. And} is larger than the first light intensity (S _dark ) and smaller than the second light intensity (S _blight ), the mixing ratio αi of the second non-light emitting image is set to the maximum value α _max. And the minimum value α _min. Are linearly decreased (monotonically decreased).

A pixel or region having a light intensity equal to or lower than the first light intensity (S _dark ) is considered to be a shaded area, and it is preferable that the mixture rate αi of the overexposed second non-emission image is set to the maximum value α _max. On the other hand, a pixel or region having a light intensity equal to or higher than the second light intensity (S _blight ) is considered to be a sunny area, and the mixture ratio αi of the overexposed second non-emission image is set to the minimum value α _min. It is preferable to set the mixing ratio of the first non-light-emitting image to a maximum value). Thereby, the gradation of the highlight part and the shadow part can be enriched.

FIG. 12 is a frequency distribution diagram similar to the frequency distribution of the light intensity shown in FIG.

In the frequency distribution diagram shown in FIG. 12, there are two peaks where the frequency is high, and therefore there is a valley (bottom point V) where the frequency is low between the two peaks.

In another embodiment of the composition ratio determination unit 220C shown in FIG. 9, based on the distribution information created by the distribution information creation unit 220E, the light intensity corresponding to the bottom point V having a low light intensity frequency as shown in FIG. (St) is obtained.

When the composition ratio determining unit 220C determines the composition ratio for each pixel corresponding to each of the first non-light-emitting image and the second non-light-emitting image, the light intensity (St) is set as a threshold value as shown in FIG. When the light intensity is less than or equal to the threshold value, the mixture ratio αi of the second non-light-emitting image is set to the maximum value α _{max. When the} light intensity exceeds the threshold value, the mixture ratio αi of the second non-light-emitting image is set to the minimum value α _{min. .} to (maximum mixing ratio of the first non-emission image of underexposure).

For the light intensity in a certain range ΔS before and after the threshold light intensity (St), the mixing ratio of the second non-luminescent image is set to the maximum value α _max. And the minimum value according to the light intensity _. This includes the case where the mixing ratio is continuously changed between α _min .

[Third Embodiment of HDR Processing]

FIG. 14 is a functional block diagram showing a third embodiment of the main body side CPU 220 functioning as an image processing apparatus according to the present invention. In FIG. 14, the same reference numerals are given to the portions common to the first embodiment shown in FIG. 5, and the detailed description thereof is omitted.

The main body side CPU 220 of the third embodiment shown in FIG. 14 is different from the first embodiment shown in FIG. 5 in that the function of the luminance distribution information calculation unit 220F is added.

The luminance distribution information calculation unit 220F calculates luminance distribution information indicating the luminance distribution in the image of at least one non-luminescent image among the plurality of non-luminescent images (first non-luminescent image and second non-luminescent image). Part. The luminance distribution information can be created, for example, as an image made up of luminance signals of the first non-emission image, or as information indicating the luminance (representative luminance) for each area into which the image is divided.

The combination ratio determination unit 220C illustrated in FIG. 14 includes a plurality of non-light-emitting images (first non-light-emitting images) based on the light intensity distribution information calculated by the light intensity distribution information calculation unit 220B and the luminance distribution information calculated by the luminance distribution information calculation unit 220F. The composite ratio of the light emission image and the second non-light emission image) is determined.

Specifically, the composition ratio determination unit 220C determines a parameter indicating the position of a pixel or a region when determining a composition ratio for each pixel or region corresponding to each of the first non-light-emitting image and the second non-light-emitting image. If the mixing ratio of the second non-light-emitting image based on the light intensity distribution information is αi, the mixing ratio of the second non-light-emitting image based on the luminance distribution information, and the mixing ratio that is smaller as the luminance is higher is βi, A mixing ratio γi of the second non-light-emitting image based on the light intensity distribution information and the luminance distribution information is expressed by the following equation:

[Equation 6]

γi = αi × βi is calculated. In this case, the mixing ratio of the first non-light-emitting image is (1-γi), and the combination ratio of the first non-light-emitting image and the second non-light-emitting image is (1-γi): γi.

[Fourth Embodiment of HDR Processing]

FIG. 15 is a functional block diagram showing a fourth embodiment of the main body side CPU 220 functioning as an image processing apparatus according to the present invention. In FIG. 15, the same reference numerals are given to the portions common to the third embodiment shown in FIG. 14, and the detailed description thereof is omitted.

The main body side CPU 220 of the fourth embodiment shown in FIG. 15 is different from the third embodiment shown in FIG. 14 in that the function of the flash light non-reaching pixel detection unit 220G is added.

The flash light non-reaching pixel detection unit 220G calculates a difference (Y _Li −Y _i ) between the luminance value Y _i of the pixel i of the first non-emission image and the luminance value of the pixel i at the same coordinates of the emission image as Y _Li. Then, the pixel i for which the difference (Y _Li -Y _i ) is 0 is detected as a flash light non-reaching pixel.

The combination ratio determination unit 220C illustrated in FIG. 15 has a plurality of non-light-emitting images (first non-light-emitting images) based on the light intensity distribution information calculated by the light intensity distribution information calculation unit 220B and the luminance distribution information calculated by the luminance distribution information calculation unit 220F. (The light emission image, the second non-light emission image) is determined. When the flash light non-reaching pixel detection unit 220G detects the flash light non-reaching pixel, the composite ratio for the flash light non-reaching pixel is determined. In addition, the composition ratio of the plurality of non-light emitting images is determined based only on the luminance distribution information.

If the background of the subject is far away, the flash light does not reach and (Y _Li -Y _i ) becomes 0, and the light intensity distribution information S _i shown in the equation 1 cannot be calculated. As a result, the mixing ratio αi of the second non-light-emitting image based on the light intensity distribution information S _i cannot be obtained.

Therefore, in this case, by determining the combination ratio of a plurality of non-light-emitting images based only on the luminance distribution information, an appropriate combination ratio can be determined even for a subject that does not reach the flash light.

Specifically, the composition ratio determination unit 220C calculates the mixing ratio γi of the second non-light-emitting image based on the light intensity distribution information and the luminance distribution information using the formula [6], but the pixel to which the flash light does not reach is calculated. When (flash light non-reaching pixel) is detected by the flash light non-reaching pixel detection unit 220G, the mixture ratio αi of the second non-light-emitting image of the flash light non-reaching pixel is set to 1, and based on the luminance distribution information The mixture ratio γi of the second non-light-emitting image is obtained only from the mixing ratio βi of the second non-light-emitting image.

The flash light non-reaching pixel detection unit 220G is not limited to detecting a flash light non-reaching pixel, but may function as a detection unit that detects a flash light non-reaching region when an image is divided into a plurality of regions. Good.

The flash light non-reaching pixel detection unit 220G is also used to control the light emission amount of the flash light emitted from the flash light emission unit 270 by the flash control unit 272. Details of the light emission amount control by the flash control unit 272 are as follows. Will be described later.

[Fifth Embodiment of HDR Processing]

FIG. 16 is a functional block diagram showing a fifth embodiment of the main body side CPU 220 functioning as an image processing apparatus according to the present invention. In FIG. 16, the same reference numerals are given to the portions common to the first embodiment shown in FIG. 5, and detailed description thereof is omitted.

The main body side CPU 220 of the fourth embodiment shown in FIG. 16 does not have the distance distribution information acquisition unit 220A shown in the first embodiment, and acquires distance distribution information from the external distance distribution information acquisition unit 223. Yes.

The distance distribution information acquisition unit 223 can be configured by, for example, a distance image sensor in which a plurality of light receiving elements are arranged two-dimensionally. As the distance image sensor, one that obtains a distance image by a TOF (Time Of Flight) method is conceivable. The TOF method is a method for obtaining the distance to the subject by irradiating the subject with light and measuring the time until the reflected light is received by the sensor. A distance image sensor that has two pixels, and obtains a distance image of the subject from the received light amount (light reception intensity) for each pixel of the distance image sensor, and irradiates the subject with light modulated at high frequency and reflects it from the time of irradiation There is known a method of acquiring a distance image by detecting a phase shift until light is received.

As the distance distribution information acquisition unit 223, a three-dimensional laser measuring instrument, a stereo camera, or the like can be applied. When the distance distribution information acquisition unit 223 is used, the image sensor 201 is not limited to the image sensor as in this example, and may be an image sensor that cannot measure the distance.

[Image processing method]

FIG. 17 is a flowchart showing an embodiment of an image processing method according to the present invention. Hereinafter, the operation of the HDR process performed by the main body side CPU 220 having the functions of the units illustrated in FIG. 5 and the like will be mainly described.

In FIG. 17, the distance distribution information acquisition unit 220 </ b> A of the main body side CPU 220 acquires distance distribution information indicating the distribution of the distance of the subject in the image (step S <b> 10). In this example, the distance distribution information acquisition unit 220A acquires a first image and a second image that are phase difference images having a phase difference from each other from the image acquisition unit 221, and based on the first image and the second image. Then, distance distribution information indicating the distribution of the distance of the subject in the image is acquired.

In addition, the image acquisition unit 221 captures the same subject under different imaging conditions, and includes a plurality of non-emission images (first non-emission images) captured under different exposure conditions under non-flash light emission. And the second non-emission image), and the non-emission image (in this example, the exposure condition of the first non-emission image under exposure) of any of the plurality of non-emission images under flash light emission. The light emission image imaged by (1) is acquired (step S12).

Next, the detailed operation in step S12 will be described with reference to the flowchart shown in FIG.

In FIG. 18, the flash control unit 272 determines the dimming light emission amount for the dimming image to be emitted from the flash light emitting unit 270 based on the distance distribution information acquired in step S10.

(Step S100). The dimming light emission amount is calculated as the maximum light emission amount that does not blow out from the distance distribution, assuming that the average reflectance of the subject within the angle of view is 18%.

Next, the flash control unit 272 causes the flash light emitting unit 270 to emit flash light having the calculated light control light emission amount, and the main body CPU 220 performs the same exposure conditions (under exposure) as the first non-light emitting image under the light control light emission. ) Is acquired (step S102).

Subsequently, the main body side CPU 220 determines whether there is a pixel or a region where the flash light has not reached, based on the detection output of the flash light non-reaching pixel detection unit 220G (FIG. 15) (step S104). Note that the flash light non-reaching pixel detection unit 220G acquires a frame image (non-light-emitting image) of the live view image captured under the same exposure condition as that of the light-control image, and uses the light-control image and the non-light-emitting image. In comparison, a flash light non-reaching pixel or region is detected.

If it is determined in step S104 that the flash light has reached the entire area ("No")

In this case, the main body side CPU 220 determines whether or not there is a whiteout region in the dimming image (step S106). If there is an area of the maximum pixel value (255 when the pixel value is represented by 8 bits) in the dimming image, the area can be determined as a whiteout area.

If it is determined in step S106 that there is no whiteout region in the light control image (in the case of “No”), the light control image is acquired as a light emission image (step S108). In this case, there is no need to capture a light-emitting image again, and the number of captured images can be reduced.

If it is determined in step S106 that there is a whiteout area in the light control image (in the case of “Yes”), the flash control unit 272 calculates the maximum light emission amount that the whiteout area does not white out. Then, the calculated maximum light emission amount is set as the main light emission amount when the light emission image is acquired (step S110).

On the other hand, if it is determined in step S104 that there is an area where the flash light has not reached (in the case of “Yes”), the main body CPU 220 determines that the area where the flash light has not reached is greater than the maximum reach distance of the flash light. Is also determined (step S112). The distance of the area where the flash light has not reached can be obtained from the distance distribution information, and the maximum reach distance of the flash light can be obtained from the guide number and F value of the flash light emitting unit 270.

If it is determined in step S112 that the area where the flash light has not reached is longer than the maximum reach distance of the flash light (in the case of “No”), the composition ratio determination unit 220C illustrated in FIG. The mixing ratio αi of the area that has not reached (the mixing ratio αi of the second non-light-emitting image shown in the equation [6]) is set to 1 (step S114). As a result, the mixing ratio γi of the second non-light emitting image in the region where the flash light has not reached is determined only by the mixing ratio βi of the second non-light emitting image based on the luminance distribution information.

On the other hand, when it is determined in step S112 that the area where the flash light has not reached is closer than the maximum reach distance of the flash light (in the case of “Yes”), the flash control unit 272 determines that the area where the flash light has reached A maximum light emission amount that does not cause whiteout is calculated, and the calculated maximum light emission amount is set as a main light emission amount when a light emission image is acquired (step S116).

When the flash control unit 272 calculates the main light emission amount in step S110 or step S116, the flash control unit 272 performs main flash light emission from the flash light emission unit 270 with the calculated main light emission amount, and the image acquisition unit 221 performs the first non-light emission under the main light emission. A light emission image captured by the imaging unit (the interchangeable lens 100 and the image sensor 201) under the same exposure condition as that of the light emission image is acquired (step S118).

When a luminescent image is acquired as described above, a non-luminescent image (first non-luminescent image) that is subsequently captured under the same exposure conditions (underexposure) as the luminescent image and a second non-luminescent image that is captured with overexposure And get.

Returning to FIG. 17, the light intensity distribution information calculation unit 220 </ b> B is based on the non-luminous image (under-exposed first non-luminous image), the luminescent image, and the distance distribution information acquired by the distance distribution information acquisition unit 220 </ b> A. 1] The light intensity distribution information S _i is calculated from the equation (step S14).

The combination ratio determination unit 220C determines a combination ratio for HDR combining the plurality of non-light-emitting images (first non-light-emitting image and second non-light-emitting image) based on the light intensity distribution information S _i calculated in step S14. (Step S16). For example, when the composition ratio determining unit 220C determines the composition ratio for each pixel or region corresponding to each of the plurality of non-light-emitting images (the first non-light-emitting image and the second non-light-emitting image), the light intensity distribution information S _{i is used.} Based on the above, in the pixel or region having a high light intensity, the mixing ratio of the first non-emission image that is under-exposed is made larger than the mixing ratio of the second non-emission image that is over-exposed. The mixing ratio of the two non-light emitting images is set to be larger than the mixing ratio of the first non-light emitting image.

The image composition unit 220D performs HDR composition of the first non-light-emitting image and the second non-light-emitting image based on the composition ratio determined in Step S16, and generates an HDR image (Step S18).

The imaging device 10 according to the present embodiment is a mirrorless digital single-lens camera, but is not limited thereto, and may be a single-lens reflex camera, a lens-integrated imaging device, a digital video camera, or the like. The present invention is also applicable to mobile devices having other functions (call function, communication function, and other computer functions). Other modes to which the present invention can be applied include, for example, mobile phones and smartphones having a camera function, PDAs (Personal Digital Assistants), and portable game machines. Hereinafter, an example of a smartphone to which the present invention can be applied will be described.

<Configuration of smartphone>

FIG. 19 shows the appearance of a smartphone 500 that is an embodiment of the imaging apparatus of the present invention. A smartphone 500 illustrated in FIG. 19 includes a flat housing 502, and a display input in which a display panel 521 as a display unit and an operation panel 522 as an input unit are integrated on one surface of the housing 502. Part 520. The casing 502 includes a speaker 531, a microphone 532, an operation unit 540, and a camera unit 541. Note that the configuration of the housing 502 is not limited to this, and, for example, a configuration in which the display unit and the input unit are independent, or a configuration having a folding structure or a slide mechanism may be employed.

20 is a block diagram showing a configuration of the smartphone 500 shown in FIG. As shown in FIG. 20, as a main component of a smartphone, a wireless communication unit 510 that performs mobile wireless communication via a base station and a mobile communication network, a display input unit 520, a call unit 530, and an operation unit 540 , A camera unit 541, a recording unit 550, an external input / output unit 560, a GPS (Global Positioning System) receiving unit 570, a motion sensor unit 580, a power supply unit 590, and a main control unit 501.

The wireless communication unit 510 performs wireless communication with a base station accommodated in the mobile communication network in accordance with an instruction from the main control unit 501. Using this wireless communication, transmission / reception of various file data such as audio data and image data, e-mail data, and reception of Web data and streaming data are performed.

The display input unit 520 displays images (still images and moving images), character information, and the like, and visually transmits information to the user under the control of the main control unit 501, and detects a user operation on the displayed information. This is a so-called touch panel, and includes a display panel 521 and an operation panel 522.

The display panel 521 uses an LCD (Liquid Crystal Display), an OELD (Organic Electro-Luminescence Display), or the like as a display device. The operation panel 522 is a device that is placed so that an image displayed on the display surface of the display panel 521 is visible and detects one or a plurality of coordinates operated by a user's finger or stylus. When such a device is operated by a user's finger or stylus, a detection signal generated due to the operation is output to the main control unit 501. Next, the main control unit 501 detects an operation position (coordinates) on the display panel 521 based on the received detection signal.

As shown in FIG. 19, the display panel 521 and the operation panel 522 of the smartphone 500 exemplified as an embodiment of the imaging apparatus of the present invention integrally constitute a display input unit 520. The arrangement is such that 522 completely covers the display panel 521. When such an arrangement is adopted, the operation panel 522 may have a function of detecting a user operation even in an area outside the display panel 521. In other words, the operation panel 522 includes a detection area (hereinafter referred to as a display area) for an overlapping portion that overlaps the display panel 521 and a detection area (hereinafter, a non-display area) for an outer edge portion that does not overlap the other display panel 521. May be included).

Although the size of the display area and the size of the display panel 521 may be completely matched, it is not always necessary to match them. In addition, the operation panel 522 may include two sensitive regions of the outer edge portion and the other inner portion. Furthermore, the width of the outer edge portion is appropriately designed according to the size of the housing 502 and the like. Further, examples of the position detection method employed in the operation panel 522 include a matrix switch method, a resistance film method, a surface acoustic wave method, an infrared method, an electromagnetic induction method, and a capacitance method, and any method is adopted. You can also.

The call unit 530 includes a speaker 531 and a microphone 532, and converts a user's voice input through the microphone 532 into voice data that can be processed by the main control unit 501, and outputs the voice data to the main control unit 501, or a wireless communication unit 510 or the audio data received by the external input / output unit 560 is decoded and output from the speaker 531. As shown in FIG. 19, for example, the speaker 531 and the microphone 532 can be mounted on the same surface as the surface on which the display input unit 520 is provided.

The operation unit 540 is a hardware key using a key switch or the like, and receives an instruction from the user. For example, as illustrated in FIG. 19, the operation unit 540 is mounted on the side surface of the housing 502 of the smartphone 500 and is turned on when pressed with a finger or the like, and is turned off by a restoring force such as a spring when the finger is released. It is a push button type switch.

The recording unit 550 includes a control program of the main control unit 501, control data, application software (including the image processing program according to the present invention), address data that associates the name and telephone number of the communication partner, and the sent and received e-mails. Data, web data downloaded by web browsing, and downloaded content data are stored, and streaming data and the like are temporarily stored. The recording unit 550 includes an internal storage unit 551 built in the smartphone and an external storage unit 562 having a removable external memory slot. Each of the internal storage unit 551 and the external storage unit 552 constituting the recording unit 550 includes a flash memory type, a hard disk type, a multimedia card micro type, This is realized by using a recording medium such as a card type memory (for example, Micro SD (registered trademark) memory), a RAM (Random Access Memory), a ROM (Read Only Memory) or the like.

The external input / output unit 560 serves as an interface with all external devices connected to the smartphone 500, and communicates with other external devices (for example, universal serial bus (USB), IEEE 1394, etc.) or Network (for example, Internet, wireless LAN (Local Area Network), Bluetooth (registered trademark), RFID (Radio Frequency Identification), infrared communication (Infrared Data Association: IrDA) (registered trademark), UWB (Ultra Wideband) ( (Registered trademark), ZigBee (registered trademark), etc.) for direct or indirect connection.

Examples of external devices connected to the smartphone 500 include a wired / wireless headset, wired / wireless external charger, wired / wireless data port, a memory card connected via a card socket, and a SIM (Subscriber). Identity Module Card (UIM) / User Identity Module Card (UIM) card, or external audio / video equipment connected via audio / video I / O (Input / Output) terminal, external audio / video equipment connected wirelessly, wired / wireless There are smartphones to be connected, personal computers to be connected to / from wireless, PDAs to be connected to wireless and wireless, and earphones. The external input / output unit can transmit the data transmitted from such an external device to each component inside the smartphone 500, or can transmit the data inside the smartphone 500 to the external device.

The GPS receiving unit 570 receives GPS signals transmitted from the GPS satellites ST1 to STn in accordance with instructions from the main control unit 501, performs positioning calculation processing based on the received plurality of GPS signals, A position consisting of longitude and altitude is detected. When the GPS reception unit 570 can acquire position information from the wireless communication unit 510 or the external input / output unit 560 (for example, a wireless LAN), the GPS reception unit 570 can also detect the position using the position information.

The motion sensor unit 580 includes, for example, a triaxial acceleration sensor and a gyro sensor, and detects the physical movement of the smartphone 500 in accordance with an instruction from the main control unit 501. By detecting the physical movement of the smartphone 500, the moving direction and acceleration of the smartphone 500 are detected. This detection result is output to the main control unit 501.

The power supply unit 590 supplies power stored in a battery (not shown) to each unit of the smartphone 500 in accordance with an instruction from the main control unit 501.

The main control unit 501 includes a microprocessor, operates according to a control program and control data stored in the recording unit 550, and controls each unit of the smartphone 500 in an integrated manner. Further, the main control unit 501 includes a mobile communication control function and an application processing function for controlling each unit of the communication system in order to perform voice communication and data communication through the wireless communication unit 510.

The application processing function is realized by the main control unit 501 operating in accordance with application software stored in the recording unit 550. Examples of the application processing function include an infrared communication function for controlling the external input / output unit 560 to perform data communication with the opposite device, an e-mail function for sending and receiving e-mails, a web browsing function for browsing web pages, and the present invention. There is an image processing function for performing HDR composition.

Further, the main control unit 501 has an image processing function such as displaying video on the display input unit 520 based on image data (still image or moving image data) such as received data or downloaded streaming data. The image processing function refers to a function in which the main control unit 501 decodes the image data, performs image processing on the decoding result, and displays an image on the display input unit 520.

Further, the main control unit 501 executes display control for the display panel 521 and operation detection control for detecting a user operation through the operation unit 540 and the operation panel 522.

By executing the display control, the main control unit 501 displays an icon for starting application software and a software key such as a scroll bar, or displays a window for creating an e-mail. Note that the scroll bar refers to a software key for accepting an instruction to move the display portion of a large image that does not fit in the display area of the display panel 521.

Further, by executing the operation detection control, the main control unit 501 detects a user operation through the operation unit 540, receives an operation on the icon and an input of a character string in the input field of the window through the operation panel 522, or scrolls. A request to scroll the display image through the bar is accepted.

Further, by executing the operation detection control, the main control unit 501 causes the operation position with respect to the operation panel 522 to overlap with the display panel 521 (display area) or other outer edge part (non-display area) that does not overlap with the display panel 521. And a touch panel control function for controlling the sensitive area of the operation panel 522 and the display position of the software key.

The main control unit 501 can also detect a gesture operation on the operation panel 522 and execute a preset function in accordance with the detected gesture operation. Gesture operation is not a conventional simple touch operation, but an operation of drawing a trajectory with at least one of a plurality of positions by drawing a trajectory with a finger or the like, or simultaneously specifying a plurality of positions. means.

The camera unit 541 is a digital camera that performs electronic photography using an imaging device such as a complementary metal oxide semiconductor (CMOS) or a charge-coupled device (CCD), and corresponds to the imaging device 10 illustrated in FIG. In addition, the camera unit 541 converts image data obtained by imaging into compressed image data such as JPEG (Joint Photographic coding Experts Group) under the control of the main control unit 501, and records the data in the recording unit 550. The data can be output through the input / output unit 560 and the wireless communication unit 510. As shown in FIG. 19, in the smartphone 500, the camera unit 541 is mounted on the same surface as the display input unit 520, but the mounting position of the camera unit 541 is not limited to this, and is mounted on the back surface of the display input unit 520. Alternatively, a plurality of camera units 541 may be mounted. Note that when a plurality of camera units 541 are mounted, the camera unit 541 used for shooting can be switched for shooting alone, or a plurality of camera units 541 can be used for shooting simultaneously.

The camera unit 541 can be used for various functions of the smartphone 500. For example, an image acquired by the camera unit 541 can be displayed on the display panel 521, or the image of the camera unit 541 can be used as one of operation inputs of the operation panel 522. Further, when the GPS receiving unit 570 detects the position, the position can also be detected with reference to an image from the camera unit 541. Furthermore, referring to an image from the camera unit 541, the optical axis direction of the camera unit 541 of the smartphone 500 without using the triaxial acceleration sensor or in combination with the triaxial acceleration sensor (gyro sensor). It is also possible to determine the current usage environment. Of course, the image from the camera unit 541 can be used in the application software.

In addition, the position information acquired by the GPS receiver 570 to the image data of the still image or the moving image, the voice information acquired by the microphone 532 (the text information may be converted into voice information by the main control unit or the like), Posture information and the like acquired by the motion sensor unit 580 can be added and recorded in the recording unit 550 or output through the external input / output unit 560 and the wireless communication unit 510.

[Others]

In the present embodiment, the image processing apparatus (main body side CPU 220) that performs HDR composition is built in the imaging apparatus, but the image processing apparatus according to the present invention executes, for example, the image processing program according to the present invention. It may be a personal computer, a portable terminal, or the like separate from the imaging device. In this case, it is necessary to input a plurality of non-light-emitting images, light-emitting images, distance distribution information, and the like as information for HDR synthesis.

The plurality of non-light-emitting images are not limited to the two non-light-emitting images of the under-exposed first non-light-emitting image and the over-exposed second non-light-emitting image. A non-light emitting image may be used. In this case, the combination ratio determination unit determines a combination ratio of three or more non-light emitting images.

In the present embodiment, for example, the hardware structure of a processing unit (processing unit) that executes various processes such as the main body CPU 220 is various processors as shown below. For various processors, the circuit configuration can be changed after manufacturing a CPU (Central Processing Unit), FPGA (Field Programmable Gate Array), which is a general-purpose processor that functions as various processing units by executing software (programs). Includes a dedicated electric circuit that is a processor having a circuit configuration specifically designed to execute a specific process such as a programmable logic device (PLD) or an application specific integrated circuit (ASIC). It is.

One processing unit may be configured by one of these various processors, or may be configured by two or more processors of the same type or different types (for example, a plurality of FPGAs or a combination of CPUs and FPGAs). May be. Further, the plurality of processing units may be configured by one processor. As an example of configuring a plurality of processing units with one processor, first, as represented by a computer such as a client or server, one processor is configured with a combination of one or more CPUs and software. There is a form in which the processor functions as a plurality of processing units. Secondly, as represented by System On Chip (SoC), etc., there is a form of using a processor that realizes the functions of the entire system including a plurality of processing units with one IC (Integrated Circuit) chip. is there. As described above, various processing units are configured using one or more of the various processors as a hardware structure.

Furthermore, the hardware structure of these various processors is more specifically an electric circuit (circuitry) in which circuit elements such as semiconductor elements are combined.

Furthermore, the present invention includes an image processing program that functions as an imaging apparatus or an image processing apparatus according to the present invention by being installed in the imaging apparatus or computer, and a recording medium on which the image processing program is recorded.

Moreover, it goes without saying that the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the spirit of the present invention.

10 Imaging device

20 Finder window

22 Shutter release switch

23 Shutter speed dial

24 Exposure compensation dial

25 Power lever

26 Eyepiece

27 MENU / OK key

28 Four-way controller

29 Play button

30 Built-in flash

100 interchangeable lenses

102 Imaging optical system

104 Lens group

108 aperture

116 Focus lens control unit

118 Aperture control unit

120 Lens side CPU

122, 207 RAM

124, 228 ROM

126 Flash ROM

150 Lens side communication unit

160 Lens mount

200 Camera body

201 Image sensor

201A micro lens

202 Image sensor controller

203 Analog signal processor

204 A / D converter

205 Image input controller

206 Digital signal processor

208 Compression / decompression processor

210 Media control unit

212 memory card

214 Display control unit

216 LCD monitor

220 Main CPU

220A Distance distribution information acquisition unit

220B Light intensity distribution information calculation unit

220C Composite ratio determination unit

220D image composition unit

220E Distribution information creation unit

220F Luminance distribution information calculation unit

220G Flash light non-reaching pixel detector

221 Image acquisition unit

222 Operation unit

223 Distance distribution information acquisition unit

224 clock part

226 flash ROM

230 AF control unit

232 AE control unit

234 White balance correction section

236 Wireless communication unit

238 GPS receiver

240 Power control unit

242 battery

244 Lens power switch

250 Communication unit on the main unit side

260 Body mount

260A terminal

270 Flash light emitting unit

272 Flash control unit

296 FPS controller

500 Smartphone

501 Main control unit

502 housing

510 wireless communication unit

520 Display input section

521 Display panel

522 Control panel

530 Call section

531 Speaker

532 microphone

540 operation unit

541 Camera section

550 recording section

551 internal storage

552 External storage unit

560 External input / output unit

562 External storage unit

570 receiver

570 GPS receiver

580 Motion sensor

590 power supply

Steps S10 to S18, S100 to S118

_Si light intensity distribution information

Y _Li luminance value

Y _i luminance value

d _i distance information

αi, βi, γi mixing ratio

Claims

One of the plurality of non-light emitting images captured under different exposure conditions under non-flash light emission and the plurality of non-light emission images under flash light emission, which are images of the same subject. An image acquisition unit that acquires a luminescent image captured under the same exposure conditions as the non-luminous image exposure conditions;

A distance distribution information acquisition unit for acquiring distance distribution information indicating a distribution of the distance of the subject;

Light intensity distribution information calculation for calculating light intensity distribution information of light irradiated on the subject based on the light emission image, the non-light emission image captured under the same exposure conditions as the light emission image, and the distance distribution information And

A composition ratio determining unit that determines a composition ratio of the plurality of non-light-emitting images based on the light intensity distribution information;

An image combining unit that combines the plurality of non-light-emitting images according to the combining ratio and generates a combined image with an expanded dynamic range;

An image processing apparatus.
The composition ratio determining unit determines the composition ratio for each pixel or region corresponding to each of the plurality of non-light-emitting images,

2. The image processing apparatus according to claim 1, wherein the image combining unit combines the plurality of non-light-emitting images by applying the determined combining ratio for each pixel or region corresponding to each of the plurality of non-light-emitting images.
The image processing apparatus according to claim 1, wherein the plurality of exposure conditions corresponding to the plurality of non-light-emitting images include an under-exposure exposure condition and an over-exposure exposure condition rather than an appropriate exposure condition.
When determining the combination ratio for each pixel or region corresponding to each of the plurality of non-light-emitting images, the combination ratio determination unit determines whether the plurality of pixels or regions have a high light intensity based on the light intensity distribution information. The mixing ratio of the first non-light-emitting images captured under the exposure condition with the smallest exposure value among the non-light-emitting images is represented as the second non-light-image ratio captured with the exposure conditions with the largest exposure value among the plurality of non-light-emitting images. 4. The method according to claim 1, wherein the mixture ratio of the second non-light-emitting image is made larger than the mixture ratio of the first non-light-emitting image in a pixel or region having a light intensity that is larger than the mixing ratio of the light-emitting image. The image processing apparatus according to item 1.
The combination ratio determination unit determines the minimum light intensity of the light intensity distribution information based on the light intensity distribution information when determining the combination ratio for each corresponding pixel or region of the plurality of non-light-emitting images. 5. The image processing apparatus according to claim 4, wherein when the light intensity increases between the maximum light intensity and the maximum light intensity, the mixing ratio of the second non-light-emitting image is monotonously decreased.
A distribution information creation unit that creates distribution information indicating the frequency distribution or frequency distribution of the light intensity based on the light intensity distribution information;

The combination ratio determining unit is configured to determine a first light intensity corresponding to a first vertex having a high frequency or frequency and a second light intensity corresponding to a second vertex having a high frequency or frequency based on the created distribution information. A second light intensity greater than the first light intensity, and determining the combination ratio for each corresponding pixel or region of the plurality of non-light-emitting images, the light less than the first light intensity In the case of intensity, the mixing ratio of the second non-light-emitting image is maximized, and in the case of light intensity equal to or higher than the second light intensity, the mixing ratio of the second non-light-emitting image is minimized. 5. The image according to claim 4, wherein when the light intensity is larger than the intensity and smaller than the second light intensity, the mixing ratio of the second non-light-emitting image is monotonously decreased between the maximum value and the minimum value. Processing equipment.
A distribution information creation unit that creates distribution information indicating the frequency distribution or frequency distribution of the light intensity based on the light intensity distribution information;

The composition ratio determination unit obtains light intensity corresponding to a frequency or low frequency base point based on the created distribution information, and calculates the composition ratio for each corresponding pixel or region of the plurality of non-light-emitting images. When determining, the light intensity corresponding to the bottom point is set as a threshold value, the light intensity below the threshold value is set to the maximum mixing ratio of the second non-light-emitting image, and the light intensity exceeds the threshold value The image processing apparatus according to claim 5, wherein the mixing ratio of the second non-light-emitting image is minimized.
A luminance distribution information calculation unit that calculates luminance distribution information indicating a luminance distribution in an image of at least one non-light-emitting image among the plurality of non-light-emitting images;

The image processing apparatus according to claim 1, wherein the synthesis ratio determination unit determines a synthesis ratio of the plurality of non-light-emitting images based on the light intensity distribution information and the luminance distribution information.
A detection unit for detecting a pixel or a region where the flash light does not reach in the emission image;

The composition ratio determining unit determines a composition ratio of the plurality of non-light-emitting images based only on the luminance distribution information when determining the composition ratio for a pixel or region where the flash light has not reached. Item 9. The image processing apparatus according to Item 8.
The composite ratio determining unit determines a parameter indicating a position of a pixel or a region when determining the composite ratio for each pixel or region corresponding to each of the plurality of non-light-emitting images, and among the plurality of non-light-emitting images Αi is the mixing ratio of the second non-light-emitting image captured under the exposure condition having the largest exposure value, and the mixing ratio of the second non-light-emitting image based on the luminance distribution information is smaller as the luminance is higher. When the mixing rate is βi, the mixing rate γi of the second non-light-emitting image based on the light intensity distribution information and the luminance distribution information is expressed by the following equation:

The image processing apparatus according to claim 8, wherein calculation is performed using γi = αi × βi.
The image processing apparatus according to claim 1, wherein the exposure condition of the light-emitting image is an exposure condition having a smallest exposure value among a plurality of exposure conditions corresponding to the plurality of non-light-emitting images. .
An imaging unit capable of imaging a subject under different exposure conditions;

A flash unit that emits flash light;

A flash control unit for controlling the flash light emitted from the flash light emitting unit;

An image processing apparatus according to any one of claims 1 to 11,

The image acquisition unit emits light from the flash light emitting unit and the plurality of non-light emitting images captured under different exposure conditions by the imaging unit under non-light emission of flash light emitted from the flash light emitting unit. An imaging apparatus that acquires the light-emitting image captured by the imaging unit under the same exposure condition as that of any one of the plurality of non-light-emitting images under the emission of flash light.
The flash control unit dimmes flash light from the flash light emitting unit, and a light control image captured by the image capturing unit under the light control light emission and an image captured by the image capturing unit without the light control light emission When the area where the dimmed flash light does not reach is detected based on the above, the maximum light emission amount that does not saturate the area where the dimmed flash light arrives is calculated, and The imaging apparatus according to claim 12, wherein when acquiring the flash light, the flash light is emitted from the flash light emitting unit according to the calculated maximum light emission amount.
The imaging device according to claim 13, wherein the flash control unit calculates the maximum light emission amount based on an average reflectance of a subject and the distance distribution information acquired by the distance distribution information acquisition unit.
The flash control unit dimmes flash light from the flash light emitting unit, and a light control image captured by the image capturing unit under the light control light emission and an image captured by the image capturing unit without the light control light emission The imaging according to claim 12, wherein when it is detected that flash light has reached the entire area of the light control image based on the image, the image acquisition unit acquires the light control image as the light emission image. apparatus.
The flash control unit dimmes flash light from the flash light emitting unit, and a light control image captured by the image capturing unit under the light control light emission and an image captured by the image capturing unit without the light control light emission A flash where the flash light does not reach the light control image is detected based on the distance distribution information, and the distance of the area where the flash light does not reach is detected based on the distance distribution information. The imaging device according to claim 12, wherein the image acquisition unit acquires the light control image as the light emission image when the distance is longer than a reach distance of the maximum light emission amount.
Obtaining a plurality of non-light-emitting images that are images of the same subject and that are imaged under different exposure conditions under non-light emission of flash light;

Obtaining a light-emitting image captured under the same exposure condition as that of any one of the plurality of non-light-emitting images under flash light emission; and

Obtaining distance distribution information indicating a distribution of the distance of the subject;

Calculating light intensity distribution information of light irradiated on the subject based on the light emission image, the non-light emission image captured under the same exposure conditions as the light emission image, and the distance distribution information;

Determining a synthesis ratio of the plurality of non-light-emitting images based on the light intensity distribution information;

Combining the plurality of non-light emitting images according to the combining ratio, and generating a combined image with an expanded dynamic range;

An image processing method including:
The step of determining the composition ratio includes determining the composition ratio for each pixel or region corresponding to each of the plurality of non-light-emitting images when the pixel or region has a high light intensity based on the light intensity distribution information. The mixing ratio of the first non-light-emitting images captured under the exposure condition with the smallest exposure value among the plurality of non-light-emitting images is represented by the first mixing ratio of the first non-light-emitting images captured under the exposure condition with the largest exposure value among the plurality of non-light-emitting images. The mixture ratio of the second non-light-emitting image is set to be larger than the mixing ratio of the first non-light-emitting image in a pixel or region having a low light intensity. Image processing method.
A function of acquiring a plurality of non-light-emitting images that are images of the same subject and that are captured under different exposure conditions under non-flash light;

A function of acquiring a light-emitting image captured under the same exposure condition as that of any one of the plurality of non-light-emitting images under flash light emission;

A function of acquiring distance distribution information indicating the distribution of the distance of the subject;

A function of calculating light intensity distribution information of light applied to the subject based on the light emission image, the non-light emission image captured under the same exposure conditions as the light emission image, and the distance distribution information;

A function of determining a composite ratio of the plurality of non-light-emitting images based on the light intensity distribution information;

A function of combining the plurality of non-light emitting images according to the combining ratio and generating a combined image with an expanded dynamic range;

An image processing program for realizing a computer.
The function of determining the composition ratio is such that when determining the composition ratio for each pixel or region corresponding to each of the plurality of non-light-emitting images, the pixel or region having a high light intensity based on the light intensity distribution information. The mixing ratio of the first non-light-emitting images captured under the exposure condition with the smallest exposure value among the plurality of non-light-emitting images is represented by the first mixing ratio of the first non-light-emitting images captured under the exposure condition with the largest exposure value among the plurality of non-light-emitting images. The mixture ratio of the second non-light-emitting image is made larger than the mixing ratio of the first non-light-emitting image in a pixel or region having a light intensity smaller than that of the two non-light-emitting images. Image processing program.