JP6909669B2 - Image processing device and image processing method - Google Patents

Description

The present invention is an image processing apparatus and an image processing method capable of acquiring image data of a plurality of frames at the time of shooting, subjecting the acquired image data to a compositing process, and recording the image data subjected to the compositing process. Regarding.

Various conventional proposals have been made for acquiring image data of a plurality of frames at the time of shooting, and subjecting the acquired plurality of image data to a compositing process for recording. For example, in Patent Document 1, the imaging direction is deviated. However, there is described an image processing apparatus that acquires a plurality of frame image data, performs panoramic image processing on the acquired image data, and generates panoramic image data. In this image processing device, when it is determined that the panoramic image is an all-around panorama image, the panoramic image data and the information indicating that the panorama image is an all-around panorama image are recorded in association with each other.

In addition, an exchangeable image file format (Exif) is used as an image file format containing metadata for photographs. As this Exif information, information such as shutter speed, aperture value (F value), and ISO sensitivity is recorded together with image data.

JP-A-2015-156523

In addition to the panoramic composition process, the composition process for the image data of a plurality of frames includes various processes such as a depth composition process, an ultra-high resolution composition process, and a high dynamic range composition process (HDR). The exposure conditions of the image data of the shooting frames used in these compositing processes may be the same or different. The above-mentioned Patent Document 1 does not describe any record of exposure conditions. Depending on how the shooting conditions such as these exposure conditions are recorded, the recording may be inappropriate.

The present invention has been made in view of such circumstances, and provides an image processing apparatus and an image processing method that create appropriate information according to image composition processing and associate it with image data. With the goal.

In order to achieve the above object, the image processing apparatus according to the first invention uses an image data acquisition unit capable of acquiring a plurality of captured image data by switching shooting conditions for the same object, and the image data acquisition unit. , The same exposure condition is obtained by the first image synthesizing unit that acquires a plurality of captured image data under different exposure conditions and performs the first image composition using the acquired plurality of captured image data, and the image data acquisition unit. A second image synthesizing unit that acquires a plurality of captured image data under different conditions other than exposure and performs a second image synthesizing using the acquired plurality of captured image data, and the first image synthesizing unit. Alternatively, it has a metadata creation unit that creates metadata representing the composite image data synthesized by the second image composition unit, and the metadata creation unit performs the first image composition and the first image composition. When performing image composition of 2, create a single metadata showing different exposure conditions.

In the image processing apparatus according to the second invention, in the first invention, the metadata creation unit creates information for distinguishing between the first image composition and the second image composition.
In the second invention, the image processing apparatus according to the third invention is a reference metadata based on the exposure conditions in the image data acquisition unit when the metadata creation unit performs the first image composition. When the data is created and the second image composition is performed, the actual exposure condition in the image data acquisition unit is used as the implementation metadata.

In the image processing apparatus according to the fourth invention, in the first invention, when the metadata creation unit performs the first image composition by the first image composition unit, the first image Create metadata that corresponds to the exposure effect of the composite image generated by the composite.

In the image processing apparatus according to the fifth invention, in the first invention, when the metadata creation unit performs the second image composition by the second image composition unit, the image data acquisition unit Create metadata based on the exposure conditions when acquiring the image data of one frame in.

In the image processing method according to the sixth invention, for the same object, a plurality of captured image data are acquired by switching the imaging conditions, and the first image is used by using the plurality of captured image data acquired under different exposure conditions. The composition was performed, and the second image composition was performed using a plurality of captured image data acquired under the same exposure conditions and different conditions other than the exposure, and the composition was performed by the first image composition or the second image composition. When creating the metadata representing the composite image data, a single metadata showing different exposure conditions is created in the case of performing the first image composition and the case of performing the second image composition.

According to the present invention, it is possible to provide an image processing apparatus and an image processing method that create appropriate information according to an image composition process and associate it with image data.

It is a block diagram which mainly shows the electrical structure of the camera which concerns on one Embodiment of this invention. It is a figure explaining the depth synthesis processing in the camera which concerns on one Embodiment of this invention. It is a figure explaining the ultra-high resolution synthesis processing in the camera which concerns on one Embodiment of this invention. It is a figure explaining the high dynamic range composition processing (HDR) in the camera which concerns on one Embodiment of this invention. It is a figure explaining the noise reduction (NR) synthesis processing in the camera which concerns on one Embodiment of this invention. It is a figure explaining the electronic camera shake prevention processing in the camera which concerns on one Embodiment of this invention. It is a flowchart which shows the camera control operation of the camera in one Embodiment of this invention. It is a flowchart which shows the camera control operation of the camera in one Embodiment of this invention. It is a figure which shows the example of the display of metadata in the camera in one Embodiment of this invention.

Hereinafter, an example applied to a digital camera (hereinafter referred to as a “camera”) as an embodiment of the present invention will be described. This camera has an imaging unit, the subject image is converted into image data by the imaging unit, and the subject image is displayed as a through image on a display unit arranged on the back surface of the main body based on the converted image data. The photographer determines the composition and shutter timing by observing the through image display. During the release operation, the image data is recorded on the recording medium. The image data recorded on the recording medium can be reproduced and displayed on the display unit when the reproduction mode is selected.

Further, the camera according to the present embodiment can acquire image data of a plurality of frames and perform a compositing process using the image data of the plurality of frames. In this case, different exposure conditions (shooting conditions) are used in the first image composition process for obtaining image data under a plurality of different exposure conditions and the second image composition process for obtaining a plurality of image data under the same exposure conditions. ) Is generated and recorded in association with the image data.

FIG. 1 is a block diagram showing a mainly electrical configuration of the camera according to the present embodiment. This camera has a control unit 101, an imaging unit 102, a recording unit 105, a display unit 106, an operation unit 107, and a communication unit 108.

The imaging unit 102 includes an optical system 102a, an image sensor 102b, a focus changing unit 102c, an exposure changing unit 102d, and an imaging position changing unit 102d. The focus changing unit 102c, the exposure changing unit 102d, and the imaging position changing unit 102d may be omitted as appropriate depending on the mode that can be set.

The optical system 102a has an optical lens such as a focus lens and can form a subject image. The focus position of the optical 102a is moved and controlled by the focus changing unit 102c described later.

The image sensor 102b has an image sensor such as a CCD image sensor or a CMOS image sensor. The image pickup device 102b is arranged near a position where the subject image is formed by the optical system 102a, and converts the subject image into image data and outputs the subject image according to the control from the image pickup control unit 103b. The image sensor 102b functions as an image data acquisition unit capable of acquiring a plurality of captured image data by switching the imaging conditions (first imaging conditions) for the same object.

The focus changing unit 102c has a drive mechanism and a drive circuit for moving the focus lens in the optical system 102a in the optical axis direction. The focus changing unit 102c moves the focus lens to the focusing position based on the focusing signal from the control unit 103. Further, when the depth of field synthesis mode is set, the focus changing unit 102c sequentially moves the focus lens to a predetermined different focus position, and image data is acquired at each focus position.

The exposure changing unit 102d changes the exposure conditions at the time of shooting. The exposure change unit 102d has at least one of an aperture and aperture control circuit, a shutter and a shutter control circuit, an ISO sensitivity change circuit, and the like. That is, the exposure changing unit 102d changes the exposure condition by changing at least one of the aperture value, the shutter speed value, and the ISO sensitivity. Further, when the high dynamic range composition (HDR) mode is set, the exposure changing unit 102d sequentially shoots under predetermined different exposure conditions, and image data is acquired under each exposure condition.

The image pickup position changing unit 102e has a moving mechanism and a moving circuit for moving the image pickup element 102b in a plane orthogonal to the optical axis. When the ultra-high resolution synthesis mode is set, the image pickup position changing unit 102e sequentially moves the image sensor 102b to a predetermined different image pickup position in a plane orthogonal to the optical axis, and each image pickup position Image data is acquired with.

The control unit 101 is composed of an ASIC (Application Specific Integrated Circuit) having a CPU (Central Processing Unit), peripheral circuits thereof, a memory, and the like. The CPU executes the entire camera by controlling each part in the camera 100 according to a program stored in the memory. The control unit 101 includes a mode control unit 101b, a display control unit 101c, a communication control unit 101d, and an image processing unit 103. Some functions of each of these parts are realized by a CPU and a program, and some other functions are realized by peripheral circuits.

The mode control unit 101b causes the image control unit 103b, the shooting control change unit 103c, the image composition unit 103d, and the like to execute control according to the mode set by the user.

The display control unit 101c controls the display on the display unit 106. The display includes various displays such as a through image display based on the image data acquired by the image sensor 102b, a reproduced image display of the image recorded in the recording unit 105, and a menu screen display. Further, when displaying the image when it is recorded in the recording unit 105, the metadata recorded in relation to the image data may be displayed (see FIG. 9 described later).

The communication control unit 101d controls wireless (including infrared rays) communication or wired communication with the outside via the communication unit 108. In the present embodiment, as will be described later, the image data or the like recorded in the recording unit 105 can communicate with the external device (see S45 and S47 in FIG. 8).

The image processing unit 103 includes an image pickup control unit 103b, a shooting control change unit 103c, an image composition unit 103d, a recording control unit 103e, and a metadata creation unit 104.

The image pickup control unit 103b controls the acquisition of image data from the image pickup element 102b. That is, the image pickup control unit 103b controls the photoelectric conversion time of the image pickup element 102b and the reading of the image data, and outputs the read image data to the control unit 101.

The shooting control changing unit 103c has the focus changing unit 102b, the exposure changing unit 102c, and the imaging according to the set modes such as the depth composition processing mode, the high dynamic range composition processing (HDR) mode, and the ultra-high resolution composition processing mode. The position changing unit 102d is controlled, and shooting control according to the set mode is executed.

The image compositing unit 103d has an image processing circuit and the like, and corresponds to a depth compositing processing mode, a high dynamic range compositing processing (HDR) mode, an ultra-high resolution compositing processing mode, a noise reduction mode, an electronic camera shake prevention mode, and the like. Image processing is performed on the acquired image data of a plurality of frames. Image processing in the depth composition processing mode, the high dynamic range composition processing (HDR) mode, and the ultra-high resolution composition processing mode will be described later with reference to FIGS. 2 to 4.

The image composition unit 103d functions as a first image composition unit that acquires a plurality of captured image data under different exposure conditions by the image data acquisition unit and performs the first image composition using the acquired plurality of captured image data. do. The image synthesizing unit 103d acquires a plurality of captured image data with the same exposure conditions and different conditions other than exposure by the image data acquisition unit, and uses the acquired plurality of captured image data to obtain a second image. It functions as a second image compositing unit for compositing.

The recording control unit 103e controls recording when the image data processed by the image processing unit 103 is recorded in the recording unit 105. As will be described later, the image data is associated with the metadata generated by the metadata creation unit 104 and recorded in the recording unit 105. Depending on the image composition mode, either implementation data or reference data is associated (see S13, S19, S25 in FIG. 7).

The metadata creation unit 104 generates information associated with the image data. In the present embodiment, the metadata creation unit 104 has an implementation data creation unit 104a and a reference data creation unit 104b. The implementation data creation unit 104a generates metadata based on the actual exposure conditions when the image data of a plurality of frames is acquired. The reference data creation unit 104b calculates a reference exposure condition from the actual plurality of exposure conditions when the image data of a plurality of frames is acquired, and generates metadata based on the exposure condition.

The metadata creation unit 104 functions as a metadata creation unit that creates metadata representing composite image data synthesized by the first image synthesis unit or the second image synthesis unit. In addition, this metadata creation unit creates a single metadata showing different exposure conditions (shooting conditions) in the case of performing the first image composition and the case of performing the second image composition (for example, the figure). 2 (d), FIG. 3 (d), FIG. 4 (d), and S19 and S25 of FIG. 7).

The metadata creation unit creates information that distinguishes the first image composition from the second image composition (see, for example, FIG. 9C). Further, when the first image composition is performed, the metadata creation unit creates reference metadata based on the exposure conditions in the image data acquisition unit (see, for example, S19 in FIG. 7), and the second image composition. When the above is performed, the actual exposure condition in the image data acquisition unit is used as the implementation metadata (see, for example, S25 in FIG. 7).

When the first image composition is performed by the first image composition unit, the metadata creation unit creates metadata corresponding to the exposure effect of the composite image generated by the first image composition (for example,). See S19 in FIGS. 4 and 7). When the second image composition is performed by the second image composition unit, the metadata creation unit creates metadata based on the exposure conditions when the image data acquisition unit acquires one frame of image data. (See, for example, S25 in FIGS. 2, 3, and 7).

The recording unit 105 is a recording medium having an electrically rewritable non-volatile memory. The image data 105b and the metadata 105c associated with the image data 105b can be recorded in the recording unit 105. The image data, metadata, and the like recorded in the recording unit 105 are recorded according to a standard such as Exif, for example.

The display unit 106 has a display display arranged on the back surface of the camera or the like, and various images such as a through image display, a playback display, and a menu screen are displayed on the display display.

The operation unit 107 has various operation members such as a power button, a release button, a play button, an OK button, and a touch panel. It has a switch that turns on according to the operation of the power button, release button, play button, OK button, etc., and the operation unit 107 detects the operation state of the switch and outputs it to the control unit 101.

The communication unit 108 has a communication circuit for communicating with an external device, the Internet, or the like. The image data with metadata acquired by the image sensor 102b and recorded in the recording unit 105 is transmitted to the outside via the communication unit 108.

Next, with reference to FIG. 2, a shooting method and an image composition method when the depth composition mode is set will be described. FIG. 2A shows how the user 10 uses the camera 20 to shoot the subject 30 in the depth composition mode. Since the upper part 30a and the lower part 30b of the subject 30 have different distances from the camera 20, it may be difficult to focus on both the upper part 30a and the lower part 30b. In such a situation, when the depth-of-field composition mode is set, multiple frames are shot while changing the focus position, the parts in the shot image are compared, and the in-focus parts are combined to focus. Create an image.

FIG. 2B shows a portion that is in focus with the shooting timing. When the release button is operated at time t0 and shooting starts, the first shooting is performed between times t1 and t2. The shutter speed SS at this time is 1/100 second, and the black-painted portion F1 (corresponding to the upper portion 30a) of the captured image is in focus. When the shooting is finished, the focus position of the optical system 102a is moved by the focus changing unit 102c between the times t2 to t3.

Subsequently, a second shooting is performed between the times t4 and t4. The shutter speed SS at this time is also 1/100 second, and the black-painted portion F2 (corresponding to the lower portion 30b) of the captured image is in focus. When the depth composition processing is performed on the two captured images taken the first time and the second time, it is possible to obtain an image in which the black-painted portions F1 and F2 (upper part 30a and lower part 30b) are in focus.

In FIG. 2C, the image 41 shows the first captured image, and the image data focused on the upper portion 30a of the subject 30 (see F1 in FIG. 2B) is acquired by this imaging. Next, the image 42 shows a second captured image, and the image data focused on the lower portion 30b of the subject 30 (see F2 in FIG. 2B) is acquired by this imaging.

After acquiring the image data of the image 41 and the image 42, the image synthesizing unit 103d next performs depth synthesizing of both image data. In this depth composition process, a region having a high sharpness (contrast) of the image is extracted for each image, and image data in the region having a high sharpness is synthesized. As a result, the image data of the image 43 in which both the upper portion 30a and the lower portion 30b are in focus is generated. The depth composition process can generate image data having a depth of field deeper than that of a single shot.

Further, when the image synthesizing unit 103d generates the image data by the depth synthesizing process, the metadata generation unit 104 also generates the metadata. The first shooting, the second shooting, and the exposure for performing depth composition are performed twice. In both the first and second exposure conditions, the shutter speed SS was 1/100 second, and the aperture was FNo. Since it is the same at 16, the exposure data becomes the implementation data. Therefore, as the metadata 44, as shown in FIG. 2D, the implementation data which is the common exposure condition is recorded, and the reference data is not recorded. By using this execution data, other users can obtain an image having the same brightness as the depth composite image.

Next, with reference to FIG. 3, a shooting method and an image composition method when the super-resolution composition mode is set will be described. FIG. 3A shows how the user 10 uses the camera 20 to shoot the subject 31 in the super-resolution synthesis mode. The subject 31 is located far away, and even if the subject 31 is photographed at a normal focal length, the subject 31 appears small, and even if it is enlarged and displayed (enlarged print), the display does not have a sufficient resolution.

In such a situation shown in FIG. 3A, when the super-resolution synthesis mode is set, the image pickup position changing unit 102e takes a plurality of frames while changing the position of the image pickup element 102b. Then, by interpolating the intermediate position between the pixels in the image acquired in the first shooting by using the image taken by changing the position of the image sensor 102b in the second to fourth shootings, the resolution is high. Combine images. When the image 45 is generated by the super-resolution composition, as shown in FIG. 3 (b), the resolution of the enlarged image 46 is high, and the enlarged viewing is possible.

FIG. 3C shows the timing chart and the movement of the image sensor 102b at the time of super-resolution shooting. When the release button is operated at time t0 and shooting starts, the first shooting is performed between times t11 to t12. The shutter speed SS at this time is 1/100 second, and the image sensor 102b is at the position P1. When the photographing is completed at the time t12, the image pickup position changing unit 102e moves the image pickup element 102b to the position P2 by the time t13.

Next, a second shooting is performed between the times t13 and t14. The shutter speed SS at this time is 1/100 second, and the image sensor 102b is at the position P2. At this timing, the position P1 is indicated by a broken line, and the position P2 of the image sensor 102b is moved to the lower right by a predetermined amount as shown in FIG. 3C. This predetermined amount is a value smaller than the pixel spacing (pixel pitch) (for example, 1/2 of the pixel pitch). When the photographing is completed at the time t14, the image pickup position changing unit 102e moves the image pickup element 102b to the position P3 by the time t15.

Next, a third shooting is performed between the times t15 and t16. The shutter speed SS at this time is 1/100 second, and the image sensor 102b is at the position P3. At this timing, the position P1 is indicated by a broken line, and the position P3 of the image sensor 102b is moved to the upper right by a predetermined amount as shown in FIG. 3C.

After that, the fourth shooting is performed at times t17 to t18, and the fifth shooting is performed at times t19 to t20. Even at these timings, the shutter speed SS is 1/100 second, and the image sensor 102b moves to positions P4 and P5 by a predetermined amount to the lower left and upper left of the position P1.

When a plurality of image data are acquired, the image synthesizing unit 103d performs a super-resolution synthesizing process. That is, the pixel values between the pixels are interpolated using the image data when the pixels move by a predetermined amount with reference to the position P1 of the image sensor 102b. For example, if the amount of movement of the image sensor 102b is 1/2 of the pixel pitch, the pixel values at positions shifted by 1/2 pitch from the front, back, left, and right of the pixel can be interpolated, and the number of pixels is equivalent to 4 times higher resolution. It becomes the image data of.

When the image synthesizing unit 103d generates the image data by the super-resolution synthesizing process, the metadata generation unit 104 also generates the metadata. In order to perform super-resolution synthesis, the exposure is performed 5 times. Under the first to fifth exposure conditions, the shutter speed SS was 1/100 second, and the aperture was FNo. Since it is the same at 16, the exposure data becomes the implementation data. Therefore, as the metadata 47, as shown in FIG. 3D, the implementation data which is the common exposure condition is recorded, and the reference data is not recorded.

That is, when a plurality of image data are generated, the image synthesizing unit 103d performs a super-resolution synthesizing process to generate super-resolution image data. In the example shown in FIG. 3, the metadata creation unit 104 uses the shutter speed SS = 1/100 second, which is the exposure condition for one shooting, and the aperture value FNo. 16 is generated. The metadata (execution data 104a) generated here is recorded in the recording unit 105 as metadata 105c in association with the image data. By using this implementation data, other users can obtain an image having the same brightness as the super-resolution composite image.

Next, with reference to FIG. 4, a shooting method and an image composition method when the high dynamic range composition processing (HDR) mode is set will be described. FIG. 4A shows a user 10 using the camera 20 to photograph a person 32 and a cherry tree 33 as subjects. These subjects are under clear skies, the cherry tree 33 is very bright, while the person is very dark due to backlight. That is, unless the range in which the image sensor 102b can reproduce light and darkness is very wide, the light and darkness of both subjects cannot be properly expressed. In such a case, the HDR mode is set for shooting.

FIG. 4C is a timing chart of imaging when the HDR mode is set. The standard exposure (appropriate exposure) when shooting in the state of FIG. 4A, that is, the standard exposure considering both the subjects 32 and 33, has a shutter speed of 1/100 second and an aperture value of FNo. It is set to 16. When the HDR mode is set, the first shooting is performed under an exposure condition that is overexposed from this standard exposure, and then the second shooting is performed under an exposure condition that is an underexposure.

In the present embodiment, as shown in FIG. 4C, when the release button is operated at time t0 and shooting starts, the first shooting is performed between times t21 to t23. The shutter speed SS at this time is twice the exposure time of 1/100 second (SS is 2/100 second). Subsequently, a second shooting is performed between the times t24 and t25. The shutter speed SS at this time is 1/2 times the exposure time of 1/100 (SS is 1/200 second). The focus position of the optical system 102a is the same in the first and second shootings.

Using the overexposed image data acquired in the first shooting and the exposure under image data acquired in the second shooting, the image compositing unit 103d performs HDR compositing to reproduce the light and dark from the dark part to the bright part. Generate excellent image data. In this HDR composition, image data without overexposure or underexposure is combined over the entire screen by joining the parts of each image (subject) that are properly exposed.

FIG. 4B shows an image 48 that has not been subjected to the HDR composition processing and an image 49 that has been subjected to the HDR composition processing. In the image 48 that has not been subjected to the HDR composition processing, the cherry tree 33 is overexposed, while the person 32 is underexposed. On the other hand, in the image 49 subjected to the HDR composition processing, the light and darkness of the cherry tree 33 and the person 32 is properly expressed.

As shown in FIG. 4B, the metadata 50 of the image 49 subjected to the HDR composition processing has a shutter speed SS = 1/100 second and an aperture FNo. = 16. As described above, in the HDR composition mode, the exposure is performed twice, but since the exposure amounts are different for each, the implementation data (SS: 1/50, 1/200) 104a is not recorded, and the reference data. (SS: 1/100) 104b is recorded (see FIG. 4D).

That is, when a plurality of image data are generated, the image synthesizing unit 103d performs the HDR synthesizing process to generate the HDR synthesizing image data. As reference data 104b, the metadata creation unit 104 has a shutter speed SS = 1/100 second, which corresponds to the exposure condition of the image generated by HDR composition, and an aperture value FNo. 16 is generated. The metadata (reference data 104b) generated here is recorded in the recording unit 105 as metadata 105c in association with the image data. If you take a picture with this reference data, you can get an image with the same brightness except for the color expression.

Next, the shooting timing when the noise reduction mode (NR mode) is set will be described with reference to FIG. When the ISO sensitivity is set to high sensitivity, noise increases in the image and it becomes unsightly. Therefore, it is possible to synthesize an image with reduced noise by shooting a plurality of frames under the same exposure condition and averaging the output values of the corresponding pixels of each image.

When the release button is operated and shooting starts with the NR mode set, shooting is performed between the times t31 to t32, t33 to t34, t35 to t36, t37 to t38, and t39 to t40. The exposure conditions at this time are that the shutter speed SS is 1/100 second and the ISO sensitivity is 6400. As for the number of times of shooting in the NR mode, the optimum number of times may be appropriately selected.

When a plurality of image data are generated, the image synthesizing unit 103d calculates the average value of the pixel values of the corresponding pixels and generates the NR composite image data. In the example shown in FIG. 5, the metadata creation unit 104 generates the ISO sensitivity 6400 as the execution data 104a, that is, the exposure condition at the time of each shooting, that is, the shutter speed SS is 1/100 second. The metadata (execution data 104a) generated here is recorded in the recording unit 105 as metadata 105c in association with the image data.

Next, with reference to FIG. 6, the shooting timing when the electronic camera shake prevention mode is set will be described. If the user holding the camera is shaking, the image will be blurred, resulting in an unsightly image. Therefore, by shooting a plurality of frames under the same exposure conditions, shifting the pixel positions of each image according to the amount of camera shake, and averaging the output values of the corresponding pixels, image data with reduced camera shake can be obtained. Can be done.

When the release button is operated and shooting starts with the electronic camera shake prevention mode set, shooting is performed between the times t41 to t42, t43 to t44, t45 to t46, t47 to t48, and t49 to t50. The exposure conditions at this time are that the shutter speed SS is 1/100 second and the ISO sensitivity is 6400. When the amount of camera shake is equal to or more than a predetermined amount (see the shooting at times t43 to t44 in FIG. 6), it is often difficult to correct the camera shake, so the image is excluded from the target image of the camera shake prevention composition process. As for the number of times of shooting in the electronic camera shake prevention mode, the optimum number of times may be appropriately selected.

When a plurality of image data are generated, the image synthesizing unit 103d detects the amount of camera shake, shifts the pixel positions according to the amount of camera shake, and calculates the average value of the pixel values of the corresponding pixels. Generates electronic camera shake prevention image data. As the camera shake amount, a sensor output such as a gyro may be used, or the camera shake amount may be calculated by a correlation calculation of two image data.

In the example shown in FIG. 6, the metadata creation unit 104 generates the ISO sensitivity 6400 as the execution data 104a, that is, the exposure condition at the time of each shooting, that is, the shutter speed SS is 1/100 second. The metadata (execution data 104a) generated here is recorded in the recording unit 105 as metadata 105c in association with the image data.

As described with reference to FIGS. 2 to 6, in the present embodiment, a plurality of frames are exposed, and the image synthesizing unit 103d performs image synthesizing using the image data acquired by the exposure. The metadata generation unit 104 creates either the execution data 104a or the reference data 104d according to the type of image composition, and records the metadata 105c in the recording unit 105.

In creating the metadata, when the exposure conditions in each of the plurality of frames are the same, the actually controlled exposure conditions are created as the implementation data 104a. On the other hand, when the exposure conditions in each of the plurality of frames are different, the exposure conditions corresponding to the images generated by the compositing process are calculated, and these exposure conditions are created as reference data 104b.

Further, as the metadata, the metadata creation unit 104 may also create information for distinguishing the synthesis processing, and record the information for distinguishing the synthesis processing in the recording unit 105. In this case, the data created by the metadata creation unit 104 may be information that can distinguish whether it is implementation data or reference data.

The image composition processing is not limited to the above-mentioned depth composition processing, super-resolution composition processing, HDR composition processing, NR composition processing, and electronic camera shake prevention composition processing, and is not limited to, for example, comparative bright composition processing, comparative dark composition processing, and averaging. Anything that performs compositing processing using image data of a plurality of frames, such as additive compositing processing and live bulb compositing processing, may be used. In this case, for example, in the case of comparative bright composition processing, the total exposure time may be recorded as implementation data as metadata.

Next, the camera control in the present embodiment will be described with reference to the flowcharts shown in FIGS. 7 and 8. This flowchart is executed by the CPU in the control unit 101 controlling each unit in the camera according to a program stored in the memory.

When entering the camera control flow, first, it is determined whether or not the shooting mode is set (S1). The camera mode is set to the shooting mode as the default unless the user sets it to another mode such as the playback mode. In this step, the mode setting state is detected, and the determination is made based on the detection result.

If the result of the determination in step S1 is the shooting mode, then the through image imaging is performed (S3). Here, the image data acquired by the image sensor 102b is acquired. This image data is repeatedly acquired at a predetermined frame rate.

After imaging for a through image, the through image display, focusing, and standard exposure are set (S5). The image processing unit 103 performs image processing for displaying a through image on the image data acquired in step S3, and based on the processed image data, the display unit 106 displays a through image. The through image display is updated at a predetermined frame rate. Further, the contrast value of the acquired image data is calculated, and the position of the focus lens of the optical system 102a is adjusted so that the contrast value becomes a peak. In addition to the contrast AF described above, focusing may be performed by other methods such as phase difference AF. In addition, brightness information and the like are calculated from the image data, and standard exposures such as shutter speed, aperture value, and ISO sensitivity are set based on the brightness information.

Subsequently, it is determined whether or not the mode has been switched (S7). The mode can be switched by the operation member of the operation unit 107. The mode here includes a mode in which a plurality of frames are exposed and a compositing process is performed, such as a depth compositing mode, a super-resolution compositing mode, and an HDR compositing mode, in addition to the above-mentioned shooting mode and playback mode.

If the mode is switched as a result of the determination in step S7, the mode is set (S9). Here, the mode in the mode control unit 101b is set according to the mode switching by the operation unit 107.

If the mode is set in step S9, or if there is no mode switching as a result of the determination in step S7, then it is determined whether or not the release switch has been operated (S11). When the release button is operated, the state of the release switch changes. Here, the determination is made based on the state change of the release switch. As a result of this determination, if the release switch is not operated, the process returns to step S1.

If the release switch is operated as a result of the determination in step S11, then it is determined whether or not the mode involves an exposure change (S13). Here, the determination is made based on the mode set first or the mode set in step S9. As a mode that involves changing the exposure, for example, there is an HDR composition mode. Further, as a mode that does not involve an exposure change, for example, there are a depth composition mode, a super-resolution composition mode, a noise reduction (NR) mode, an electronic camera shake prevention mode, and the like.

As a result of the determination in step S13, in the case of the mode accompanied by the exposure change, the standard exposure is changed to the long time and the short time, and a plurality of shots are taken (S15). In this case, when shooting a plurality of frames, the exposure conditions are different for each frame. For example, in the HDR composition mode described with reference to FIG. 4, the shutter speed is set to a long time so that the exposure is overexposed in the first frame (set in step S5), and the standard exposure is set in the second frame. The shutter speed is set to a short time so that the exposure is underexposed.

Subsequently, image composition is performed (S17). Here, the image compositing process is performed on the plurality of image data acquired in step S15 according to the mode in which the image compositing unit 103d is set.

After image composition, image data and metadata (reference exposure) are then recorded (S19). Here, the metadata creation unit 104 creates the reference data 104b. As described above, the reference data 104b is data indicating exposure conditions corresponding to the effect of the image generated by the image composition. The created reference data 104b is associated with the image data synthesized in step S17 and recorded as image data 105b and metadata 105c of the recording unit 105. When the data is recorded, the process returns to step S1.

Returning to step S13, as a result of the determination in this step, if the mode does not involve an exposure change, the focus and the position of the image sensor are changed with the standard exposure to take a plurality of images (S21). Here, processing is performed according to the set mode. For example, when the depth-of-field composition mode is set, the focus position is changed, and shooting is performed at each of the changed plurality of focus positions. When the super-resolution synthesis mode is set, the position of the image sensor 102b is changed, and shooting is performed at each of the changed positions. The exposure condition at this time is the standard exposure calculated in step S5.

Subsequently, image composition is performed (S23). Here, the image compositing process is performed on the plurality of image data acquired in step S21 according to the mode in which the image compositing unit 103d is set.

After image composition, image data and metadata (implemented exposure) are recorded (S25). Here, the metadata creation unit 104 creates the implementation data 104a. As described above, the implementation data 104a is data indicating exposure conditions in which exposure is actually controlled in each frame when a plurality of frames are photographed. The created execution data 104a is associated with the image data synthesized in step S23 and recorded as image data 105b and metadata 105c of the recording unit 105. When the data is recorded, the process returns to step S1.

Returning to step S1, if the result of this determination is not the shooting mode, it is determined whether or not the mode is the reproduction mode (S31). The playback mode can be set by the user operating an operation member such as a play button of the operation unit 107. As a result of the determination in this step, if the mode is not the playback mode, the mode shifts to another mode.

As a result of the determination in step S31, in the case of the playback mode, the icon list is displayed next (S33). Here, the icon of the recorded image is displayed on the display unit 106 based on the thumbnail data of the image data recorded in the recording unit 105.

Next, it is determined whether or not the image is selected (S35). The user selects an icon to be enlarged and displayed from the icons displayed in the list displayed on the display unit 106 by touching the icon or by using a cross button or the like. In this step, it is determined whether or not the icon has been selected by the user.

If the image is not selected as a result of the determination in step S35, it is determined whether or not to return (S37). When the user wants to exit the playback mode, he / she operates the back button in the operation unit 107 or clicks the back icon (collectively, the back operation). Here, the determination is made based on whether or not the return operation has been performed. As a result of this determination, if the return operation has not been performed, the process returns to step S33. On the other hand, when the return operation is performed, the process returns to step S1.

If an image is selected as a result of the determination in step S35, an enlarged display is performed (S39). Here, the image data of the selected icon is read and enlarged and displayed on the display unit 106.

When the enlarged display is performed, it is next determined whether or not the detailed display is performed (S41). When it is desired to display the metadata recorded in association with the image selected by the user, the detail button in the operation unit 107 is operated or the detail icon is clicked (collectively, the detail display operation). Here, the determination is made based on whether or not the detailed display operation has been performed.

As a result of the determination in step S41, if the detailed display operation has been performed, the metadata is displayed (S43). Here, the metadata 105c recorded in association with the image data is read from the recording unit 105 and displayed on the display unit 106. A display example will be described later with reference to FIG.

When the metadata is displayed in step S43, or as a result of the determination in step S41, if the detailed display operation has not been performed, it is determined whether or not the transmission is external (S45). When the user transmits the image data recorded in the recording unit 105 via the Internet or the like by wireless or wired communication, the transmission button of the operation unit 107 is operated or the transmission icon is touch-operated (generic name). And external transmission operation). In this step, a determination is made based on whether or not an external transmission operation has been performed.

If an external transmission operation is performed as a result of the determination in step S45, the transmission is performed to the setting destination (S47). Here, the image data of the image selected in step S35 and the associated metadata are transmitted to the set destination. At the destination, an image and metadata based on the received image data can be displayed. A display example at this time will be described later with reference to FIG.

If the transmission is performed to the setting destination in step S47, or as a result of the determination in step S45, if the external transmission operation has not been performed, it is determined whether or not to return (S49). If the user wants to exit the playback mode, perform a return operation. Here, the determination is made based on whether or not this return operation has been performed. As a result of this determination, if the return operation has not been performed, the process returns to step S39. On the other hand, when the return operation is performed, the process returns to step S1.

As described above, in the camera control flow in the present embodiment, when the release operation is performed and the mode accompanied by the exposure change is set (S13Yes), reference data is created and recorded as metadata. (S19). On the other hand, when the mode not accompanied by the exposure change is set (S13No), the execution data is created and recorded as metadata (S25). That is, when the exposure is not changed, the actual exposure-controlled control value is recorded, while when the exposure is changed, the exposure control value according to the effect of the composite image is recorded. There is an effect that it is easy to understand what kind of control was made when looking at the metadata.

In steps S19 and S25, only one of the reference data and the implementation data was created and recorded as the metadata. By recording only one of them, the capacity of the metadata 105c can be reduced, and by simplifying the processing, the processing becomes easy. However, in steps S19 and S25, both reference data and implementation data may be created and recorded as metadata 105c. In this case, for example, among the metadata recording areas, an area that can be recorded by the manufacturer, such as an Exif standard maker notebook, may be used.

Next, the display of metadata will be described with reference to FIG. This display is performed at the destinations transmitted in steps S43 and S47 of FIG.

FIG. 9A is a display example when the depth synthesis mode as shown in FIG. 2 is set. In this case, the depth-combined image 43 and the metadata showing the shutter speed and the aperture value recorded as the execution data are displayed. FIG. 9B is a display example when the super-resolution synthesis mode as shown in FIG. 3 is set. In this case, the depth-combined image 46 and the metadata showing the shutter speed and the aperture value recorded as the execution data are displayed.

FIG. 9C is a display example when the HDR composition mode as shown in FIG. 4 is set. In this case, the HDR-combined image 49 and the metadata showing the shutter speed and the aperture value recorded as reference data are displayed. In FIG. 9C, "reference" is displayed to clearly indicate that the data is reference data. For this purpose, information indicating that the data is reference data is also recorded in the metadata 105c.

In this way, by displaying the metadata in addition to the recorded image, it is possible to easily confirm the exposure condition of the captured image. In addition, in this display, the actual exposure value at the time of shooting each frame is displayed when the exposure is not changed, and the exposure value corresponding to the effect of the composite image is displayed when the exposure is changed. Because it is done, the display will be easy to understand. Users other than the photographer can easily take a similar photograph by referring to the exposure value based on this metadata.

In FIG. 9, "reference" is displayed only in the case of reference data, but this display may be deleted. Further, in the case of the implementation data, a display that clearly indicates that the implementation data may be added. Further, in FIG. 9, although the metadata is displayed outside the image, it may be superimposed on the image and displayed.

As described above, in one embodiment of the present invention, a plurality of captured image data are acquired by switching the imaging conditions for the same object (see, for example, S3 in FIG. 7), and acquired under different exposure conditions. The first image composition was performed using the plurality of captured image data obtained (see, for example, S17 in FIG. 7), and the plurality of captured image data acquired under the same exposure conditions and different conditions other than exposure were used. The second image composition is performed (for example, see S23 in FIG. 7), and the first image composition is performed when creating metadata representing the composite image data synthesized by the first image composition or the second image composition. In the case and in the case of performing the second image composition, a single metadata showing different exposure conditions is created (for example, FIGS. 2 (d), 3 (d), 4 (d), and 7). See S19 and S25). Therefore, appropriate information can be created according to the image composition process and associated with the image data. That is, when acquiring a plurality of captured image data, different exposure conditions are shown depending on whether the exposure is changed or not, and information can be displayed according to the image composition process. It becomes.

In one embodiment of the present invention, it is determined whether or not the mode is accompanied by an exposure change (see S13 in FIG. 7), and reference data or implementation data is recorded as metadata according to this result. ing. However, the mode is not limited to whether or not the exposure is changed, and the type of metadata may be appropriately determined according to the shooting conditions such as whether or not the shooting distance is changed. In this case, the image data acquisition unit can acquire a plurality of captured image data by switching the first shooting conditions for the same object, and the first image synthesizing unit can acquire a plurality of captured image data by the image data acquisition unit. A plurality of captured image data are acquired with different shooting conditions of 2, the first image composition is performed using the acquired plurality of captured image data, and the second image composition unit is second by the image data acquisition unit. A plurality of captured image data are acquired with different shooting conditions of the third image without changing the photographing conditions of the above, and the second image composition is performed using the acquired plurality of photographed image data. When creating the metadata representing the composite image data synthesized by the first image synthesizing unit or the second image synthesizing unit, the case where the first image synthesizing is performed and the case where the second image synthesizing is performed are different. A single metadata showing the shooting conditions may be created.

Further, in one embodiment of the present invention, the focus changing unit 102c, the exposure changing unit 102d, the imaging position changing unit 102e, etc. may realize their functions by software by a program stored in the CPU and the memory. Of course, the function may be realized by the program stored in the CPU and the memory in the control unit 101.

Further, in one embodiment of the present invention, all or part of the functions of the imaging control unit 103b, the imaging control changing unit 103c, and the like are realized by a CPU (Central Processing Unit), peripheral circuits, and a program. It may be realized by a circuit executed by a program code such as a DSP (Digital Signal Processor), or may be a hardware configuration such as a gate circuit generated based on a programming language described by Verilog. Of course, it does not matter if it is executed by a hardware circuit.

Further, in one embodiment of the present invention, a digital camera has been described as a device for photographing, but the camera may be a digital single-lens reflex camera, a mirrorless camera, a compact digital camera, a video camera, or a movie. It may be a video camera such as a camera, and further, a camera built into a mobile phone, a smartphone, a mobile information terminal, a personal computer (PC), a tablet computer, a game device, etc., a medical camera, a scientific device such as a microscope, etc. It may be a camera for a vehicle, a camera mounted on a car, or a surveillance camera. In any case, the present invention can be applied to any device for photographing that records metadata together with image data.

Further, among the techniques described in the present specification, the controls mainly described in the flowchart can be set by a program, and may be stored in a recording medium or a recording unit. The recording method to the recording medium and the recording unit may be recorded at the time of product shipment, the distributed recording medium may be used, or may be downloaded via the Internet.

Further, in one embodiment of the present invention, the operation in the present embodiment has been described using a flowchart, but the order of the processing procedures may be changed, or any step may be omitted. Steps may be added, and specific processing contents in each step may be changed.

In addition, even if the scope of claims, the specification, and the operation flow in the drawings are explained using words expressing the order such as "first" and "next" for convenience, in the parts not particularly explained, It does not mean that it is essential to carry out in this order.

The present invention is not limited to the above embodiment as it is, and at the implementation stage, the components can be modified and embodied within a range that does not deviate from the gist thereof. In addition, various inventions can be formed by an appropriate combination of the plurality of components disclosed in the above-described embodiment. For example, some components of all the components shown in the embodiment may be deleted. In addition, components across different embodiments may be combined as appropriate.

10 ... User, 20 ... Camera, 101 ... Control unit, 101b ... Mode control unit, 101c ... Display control unit, 101d ... Communication control unit, 102 ... Imaging unit, 102a ... Optical system, 102b ... Imaging element, 102c ... Focus changing unit, 102d ... Exposure changing unit, 102e ... Imaging position changing unit, 103 ... Image processing unit, 103b ...・ Imaging control unit, 103c ・ ・ ・ Shooting control change unit, 103d ・ ・ ・ Image composition unit, 103e ・ ・ ・ Recording control unit, 104 ・ ・ ・ Metadata creation unit, 104a ・ ・ ・ Implementation data, 104b ... Reference data, 105 ... Recording unit, 105a ... Image data, 105b ... Metadata, 106 ... Display unit, 107 ... Operation unit, 108 ... Communication unit

Claims (6)

An image data acquisition unit that can acquire multiple captured image data by switching the shooting conditions for the same object,
A first image synthesizing unit that acquires a plurality of captured image data under different exposure conditions by the image data acquiring unit and performs a first image synthesizing using the acquired plurality of captured image data.
A second image data acquisition unit acquires a plurality of captured image data under the same exposure conditions and different conditions other than exposure, and performs a second image composition using the acquired plurality of captured image data. Image composition part and
A metadata creation unit that creates metadata representing composite image data synthesized by the first image composition unit or the second image composition unit, and a metadata creation unit.
Have,
The metadata creation unit is an image processing apparatus characterized in that it creates a single metadata showing different exposure conditions in the case of performing the first image composition and the case of performing the second image composition. .. The image processing apparatus according to claim 1, wherein the metadata creation unit creates information that distinguishes the first image composition from the second image composition. The metadata creation unit creates reference metadata based on the exposure conditions in the image data acquisition unit when the first image composition is performed, and the metadata creation unit creates the reference metadata when the second image composition is performed. The image processing apparatus according to claim 2, wherein the actual exposure condition in the image data acquisition unit is used as the implementation metadata. When the first image composition is performed by the first image composition unit, the metadata creation unit creates metadata corresponding to the exposure effect of the composite image generated by the first image composition. The image processing apparatus according to claim 1, wherein the image processing apparatus is characterized by the above. When the second image composition is performed by the second image composition unit, the metadata creation unit performs metadata based on the exposure conditions when the image data acquisition unit acquires one frame of image data. The image processing apparatus according to claim 1, wherein the image processing apparatus is created. For the same object, switch the shooting conditions to acquire multiple shot image data,
The first image composition is performed using a plurality of captured image data acquired under different exposure conditions.
A second image composition is performed using a plurality of captured image data acquired under the same exposure conditions and different conditions other than exposure.
When creating the metadata representing the composite image data synthesized by the first image composition or the second image composition, the case where the first image composition is performed and the case where the second image composition is performed are different. Create a single metadata showing the exposure conditions,
An image processing method characterized by that.

Images