JP2019029954A - Image processing system and image processing method - Google Patents

Abstract

To provide an image processing systems and an image processing method associating the image data, by creating appropriate information according to the image composition processing.SOLUTION: For the same object, multiple captured image data are acquired by switching the photography conditions (S3), first image composition is performed by using multiple captured image data acquired under different exposure conditions (S17), second image composition is performed under the same exposure conditions, while differentiating the conditions other than exposure (S23), and when creating metadata representative of the composite image data composited by first image composition or second image composition, single metadata indicating different exposure conditions, when first image composition is performed and when second image composition is performed, is created (S19, S25).SELECTED DRAWING: Figure 7

Description

  The present invention provides an image processing apparatus and an image processing method capable of acquiring image data of a plurality of frames at the time of shooting, performing a synthesis process on the acquired image data, and recording the image data subjected to the synthesis process About.

  Various proposals have been made in the past to acquire image data of a plurality of frames at the time of shooting, and to perform synthesis processing on the acquired plurality of image data. For example, Patent Document 1 displaces the imaging direction. However, there is described an image processing apparatus that acquires a plurality of frame image data, performs panorama image processing on the acquired image data, and generates panorama image data. In this image processing apparatus, when it is determined that the image is an all-round panorama image, the panorama image data is recorded in association with information indicating that the image is an all-round panorama image.

  Further, the exchangeable image file format (Exif) is used as an image file format including metadata for photographs. As this Exif information, information such as shutter speed, aperture value (F value), ISO sensitivity and the like is recorded together with the image data.

JP-A-2015-156523

  In addition to panorama synthesis processing, there are various types of processing such as depth synthesis processing, ultra-high resolution synthesis processing, and high dynamic range synthesis processing (HDR) in addition to panorama synthesis processing. The exposure conditions of the image data of the shooting frames used in these combining processes may be the same or different. Patent Document 1 described above does not describe any recording of exposure conditions. Depending on how the shooting conditions such as the exposure conditions are recorded, the recording is inappropriate.

  The present invention has been made in view of such circumstances, and provides an image processing apparatus and an image processing method that create appropriate information in accordance with image composition processing and associate it with image data. With the goal.

  In order to achieve the above object, an image processing apparatus according to a first aspect of the present invention includes an image data acquisition unit capable of acquiring a plurality of captured image data by switching shooting conditions for the same object, and the image data acquisition unit. A first image composition unit that acquires a plurality of captured image data under different exposure conditions and performs a first image composition using the acquired plurality of captured image data, and the same exposure condition by the image data acquisition unit. A second image composition unit that obtains a plurality of captured image data under different conditions other than exposure, and performs a second image composition using the obtained plurality of photographed image data; and the first image composition unit Or a metadata creating unit that creates metadata representing the synthesized image data synthesized by the second image synthesizing unit, wherein the metadata creating unit performs the first image synthesis, In the case of performing the second image synthesis, to create a single metadata indicating the different exposure conditions.

In the image processing apparatus according to a second invention, in the first invention, the metadata creating unit creates information for distinguishing the first image composition from the second image composition.
An image processing apparatus according to a third invention is the image processing apparatus according to the second invention, wherein, when the first image composition is performed, the metadata generation unit performs a reference meta-data based on an exposure condition in the image data acquisition unit. When data is created and the second image composition is performed, actual exposure conditions in the image data acquisition unit are set as execution metadata.

  An image processing apparatus according to a fourth invention is the image processing apparatus according to the first invention, wherein the metadata creation unit performs the first image composition when the first image composition unit performs the first image composition. Metadata corresponding to the exposure effect of the composite image generated by the composition is created.

  The image processing apparatus according to a fifth aspect of the present invention is the image processing apparatus according to the first aspect, wherein the metadata generation unit performs the second image composition by the second image composition unit. The metadata is created based on the exposure condition when acquiring the image data of one frame.

  An image processing apparatus according to a sixth aspect of the present invention includes an image data acquisition unit capable of acquiring a plurality of captured image data by switching the first imaging condition for the same object, and the second image data acquisition unit. A plurality of captured image data is acquired under different imaging conditions, and a first image composition unit that performs first image composition using the acquired plurality of captured image data, and the image data acquisition unit A second image composition unit that obtains a plurality of photographed image data by changing the third photography condition without changing the photographing conditions, and performs a second image composition using the obtained plurality of photographed image data; A metadata creating unit that creates metadata representing the synthesized image data synthesized by the first image synthesizing unit or the second image synthesizing unit, and the metadata creating unit performs the first image synthesis. If you do and above In case of the second image synthesis, to create a single metadata indicating the different imaging conditions.

  In the image processing method according to the seventh aspect of the present invention, a plurality of photographed image data is acquired by switching photographing conditions for the same object, and the first image is obtained using the plurality of photographed image data obtained under different exposure conditions. The second image composition is performed using a plurality of photographed image data obtained by combining and obtaining the same exposure condition and different conditions other than the exposure, and the first image composition or the second image composition is performed. When creating metadata representing composite image data, single metadata indicating different exposure conditions is created when the first image composition is performed and when the second image composition is performed.

  According to the present invention, it is possible to provide an image processing apparatus and an image processing method in which appropriate information is created in accordance with image synthesis processing and associated with image data.

1 is a block diagram mainly showing an electrical configuration of a camera according to an embodiment of the present invention. It is a figure explaining the depth synthetic | combination process in the camera which concerns on one Embodiment of this invention. It is a figure explaining the super high resolution synthetic | combination process in the camera which concerns on one Embodiment of this invention. It is a figure explaining the high dynamic range synthetic | combination process (HDR) in the camera which concerns on one Embodiment of this invention. It is a figure explaining the noise reduction (NR) synthetic | combination process in the camera which concerns on one Embodiment of this invention. It is a figure explaining the electronic camera-shake prevention process in the camera which concerns on one Embodiment of this invention. It is a flowchart which shows the camera control operation | movement of the camera in one Embodiment of this invention. It is a flowchart which shows the camera control operation | movement of the camera in one Embodiment of this invention. It is a figure which shows the example of a metadata display in the camera in one Embodiment of this invention.

  Hereinafter, an example applied to a digital camera (hereinafter referred to as “camera”) as an embodiment of the present invention will be described. This camera has an imaging unit, and the subject image is converted into image data by the imaging unit, and based on the converted image data, the subject image is displayed as a through image on a display unit disposed on the back of the main body. The photographer determines the composition and shutter timing by observing the through image display. During the release operation, image data is recorded on the recording medium. The image data recorded on the recording medium can be reproduced and displayed on the display unit when the reproduction mode is selected.

  In addition, the camera according to the present embodiment can acquire image data of a plurality of frames and perform a synthesis process performed using the image data of the plurality of frames. In this case, different exposure conditions (imaging conditions) are used in the first image composition processing performed by obtaining image data under a plurality of different exposure conditions and the second image composition processing performed by obtaining a plurality of image data under the same exposure conditions. ) Is generated and recorded in association with the image data.

  FIG. 1 is a block diagram mainly showing an electrical configuration of the camera according to the present embodiment. This camera includes a control unit 101, an imaging unit 102, a recording unit 105, a display unit 106, an operation unit 107, and a communication unit 108.

  The imaging unit 102 includes an optical system 102a, an imaging element 102b, a focus changing unit 102c, an exposure changing unit 102d, and an imaging position changing unit 102d. Note that the focus change unit 102c, the exposure change unit 102d, and the imaging position change unit 102d may be omitted as appropriate according to a settable mode.

  The optical system 102a has an optical lens such as a focus lens, and can form a subject image. The movement of the focus position of the optical 102a is controlled by a focus changing unit 102c described later.

  The image sensor 102b includes an image sensor such as a CCD image sensor or a CMOS image sensor. The image sensor 102b is disposed in the vicinity of a position where a subject image is formed by the optical system 102a, and converts the subject image into image data and outputs it according to control from the imaging control unit 103b. The imaging element 102b functions as an image data acquisition unit that can acquire a plurality of captured image data by switching the imaging condition (first imaging condition) for the same object.

  The focus changing unit 102c includes a drive mechanism and a drive circuit that move the focus lens in the optical system 102a in the optical axis direction. The focus changing unit 102 c moves the focus lens to the in-focus position based on the focus signal from the control unit 103. In addition, when the depth synthesis mode is set, the focus changing unit 102c sequentially moves the focus lens to a predetermined different focus position, and image data is acquired at each focus position.

  The exposure changing unit 102d changes the exposure condition at the time of shooting. The exposure changing unit 102d has at least one of an aperture and aperture control circuit, a shutter and shutter control circuit, an ISO sensitivity change circuit, and the like. That is, the exposure changing unit 102d changes the exposure condition by changing at least one of the aperture value, the shutter speed value, and the ISO sensitivity. In addition, when the high dynamic range composition (HDR) mode is set, the exposure changing unit 102d sequentially performs shooting under different exposure conditions, and acquires image data under each exposure condition.

  The imaging position changing unit 102e includes a moving mechanism and a moving circuit that move the imaging element 102b in a plane orthogonal to the optical axis. The imaging position changing unit 102e sequentially moves the imaging element 102b to a predetermined different imaging position in a plane orthogonal to the optical axis when the super-high resolution synthesis mode is set, and each imaging position In this way, image data is acquired.

  The control unit 101 is configured by an ASIC (Application Specific Integrated Circuit) having a CPU (Central Processing Unit), its peripheral circuits, a memory, and the like. The CPU executes the entire camera by controlling each part in the camera 100 according to a program stored in the memory. The control unit 101 includes a mode control unit 101b, a display control unit 101c, a communication control unit 101d, and an image processing unit 103. Some functions of these units are realized by a CPU and a program, and some other functions are realized by a peripheral circuit.

  The mode control unit 101b executes control by the imaging control unit 103b, the imaging control change unit 103c, the image composition unit 103d, and the like according to the mode set by the user.

  The display control unit 101 c controls display on the display unit 106. The display includes various displays such as a through image display based on image data acquired by the image sensor 102b, a reproduced image display of an image recorded in the recording unit 105, and a menu screen display. Further, when displaying an image when recorded in the recording unit 105, metadata recorded in association with the image data may be displayed (see FIG. 9 described later).

  The communication control unit 101d controls wireless (including infrared rays) communication or wired communication with the outside through the communication unit 108. In this embodiment, as will be described later, image data and the like recorded in the recording unit 105 can communicate with an external device (see S45 and S47 in FIG. 8).

  The image processing unit 103 includes an imaging control unit 103b, a shooting control change unit 103c, an image composition unit 103d, a recording control unit 103e, and a metadata creation unit 104.

  The imaging control unit 103b controls acquisition of image data from the imaging element 102b. That is, the imaging control unit 103b controls the photoelectric conversion time of the imaging element 102b and reading of image data, and causes the control unit 101 to output the read image data.

  The imaging control changing unit 103c is configured to change the focus changing unit 102b, the exposure changing unit 102c, and the imaging according to a set mode such as a depth synthesis processing mode, a high dynamic range synthesis processing (HDR) mode, and an ultra-high resolution synthesis processing mode. The position changing unit 102d is controlled to perform shooting control according to the set mode.

  The image composition unit 103d has an image processing circuit and the like, and according to a depth composition processing mode, a high dynamic range composition processing (HDR) mode, an ultra-high resolution composition processing mode, a noise reduction mode, an electronic camera shake prevention mode, and the like. Image processing is performed on the acquired image data of a plurality of frames. Image processing in the depth synthesis processing mode, high dynamic range synthesis processing (HDR) mode, and ultra-high resolution synthesis processing mode will be described later with reference to FIGS.

  The image composition unit 103d functions as a first image composition unit that acquires a plurality of captured image data under different exposure conditions by the image data acquisition unit and performs a first image composition using the acquired plurality of captured image data. To do. The image composition unit 103d uses the image data acquisition unit to acquire a plurality of captured image data under different conditions other than the same exposure condition and exposure, and uses the acquired plurality of captured image data to generate the second image. It functions as a second image composition unit that performs composition.

  The recording control unit 103 e performs recording control when image data subjected to image processing by the image processing unit 103 is recorded in the recording unit 105. As will be described later, the image data is associated with the metadata generated by the metadata creation unit 104 and recorded in the recording unit 105. Depending on the image composition mode, either execution data or reference data is associated (see S13, S19, and S25 in FIG. 7).

  The metadata creation unit 104 generates information attached to the image data. In the present embodiment, the metadata creation unit 104 includes an implementation data creation unit 104a and a reference data creation unit 104b. The execution data creation unit 104a generates metadata based on actual exposure conditions when multiple frames of image data are acquired. The reference data creation unit 104b calculates a reference exposure condition from a plurality of actual exposure conditions when acquiring image data of a plurality of frames, and generates metadata based on the exposure conditions.

  The metadata creation unit 104 functions as a metadata creation unit that creates metadata representing the composite image data synthesized by the first image synthesis unit or the second image synthesis unit. Further, the metadata creation unit creates single metadata indicating different exposure conditions (imaging conditions) when the first image composition is performed and when the second image composition is performed (for example, FIG. 2 (d), FIG. 3 (d), FIG. 4 (d), and S19 and S25 in FIG. 7).

  The metadata creation unit creates information for distinguishing between the first image composition and the second image composition (see, for example, FIG. 9C). Further, when the first image composition is performed, the metadata creation unit creates reference metadata based on the exposure condition in the image data acquisition unit (see, for example, S19 in FIG. 7), and the second image composition. When performing the above, the actual exposure condition in the image data acquisition unit is set as the implementation metadata (see, for example, S25 in FIG. 7).

  When the first image composition unit performs the first image composition, the metadata creation unit creates metadata corresponding to the exposure effect of the composite image generated by the first image composition (for example, (See S19 in FIGS. 4 and 7). When the second image composition unit performs the second image composition, the metadata creation unit creates the metadata based on the exposure condition when the image data acquisition unit obtains one frame of image data. (For example, see S25 in FIGS. 2, 3, and 7).

  The recording unit 105 is a recording medium having an electrically rewritable nonvolatile memory. The recording unit 105 can record image data 105b and metadata 105c associated with the image data 105b. Image data, metadata, and the like recorded in the recording unit 105 are recorded in accordance with a standard such as Exif, for example.

  The display unit 106 has a display display disposed on the back surface of the camera, and various images such as a through image display, a reproduction display, and a menu screen are displayed on the display display.

  The operation unit 107 includes various operation members such as a power button, a release button, a playback button, an OK button, and a touch panel. A switch that is turned on in response to an operation of a power button, a release button, a playback button, an OK button, or the like is provided. The operation unit 107 detects an operation state of the switch and outputs the detected switch state to the control unit 101.

  The communication unit 108 includes a communication circuit for communicating with an external device, the Internet, or the like. The image data with metadata acquired by the image sensor 102 b and recorded in the recording unit 105 is transmitted to the outside via the communication unit 108.

  Next, a photographing method and an image composition method when the depth composition mode is set will be described with reference to FIG. FIG. 2A shows a state in which the user 10 uses the camera 20 to photograph the subject 30 in the depth synthesis mode. Since the upper portion 30a and the lower portion 30b of the subject 30 have different distances from the camera 20, it may be difficult to focus on both the upper portion 30a and the lower portion 30b. In this situation, when the depth compositing mode is set, multiple frames are shot while changing the focus position, the parts in the shot image are compared, and the in-focus parts are combined to focus. Create an image.

  FIG. 2B shows a portion where the shooting timing is in focus. At time t0, when the release button is operated and shooting is started, the first shooting is performed between times t1 and t2. The shutter speed SS at this time is 1/100 second, and it is focused on the black portion F1 (corresponding to the upper portion 30a) in the photographed image. When shooting is completed, the focus position of the optical system 102a is moved by the focus changing unit 102c between times t2 and t3.

  Subsequently, the second shooting is performed between times t4 and t4. The shutter speed SS at this time is also 1/100 second, and is focused on a black portion F2 (corresponding to the lower portion 30b) in the photographed image. When depth synthesis processing is performed on the two captured images captured for the first time and the second time, it is possible to obtain images in which the black portions F1 and F2 (the upper portion 30a and the lower portion 30b) are in focus.

  In FIG. 2C, an image 41 indicates the first photographed image, and image data obtained by focusing on the upper portion 30a of the subject 30 (see F1 in FIG. 2B) is obtained. Next, the image 42 shows a second photographed image, and image data obtained by focusing on the lower portion 30b of the subject 30 (see F2 in FIG. 2B) is obtained.

  When the image data of the image 41 and the image 42 are acquired, the image composition unit 103d then performs depth composition on both image data. In this depth synthesis process, for each image, a region having a high sharpness (contrast) of the image is extracted, and image data of the region having a high sharpness is synthesized. Thereby, the image data of the image 43 in which both the upper part 30a and the lower part 30b are in focus are generated. The depth synthesis process can generate image data having a depth of field deeper than that of a single shot.

  Further, when the image composition unit 103d generates image data by depth composition processing, the metadata generation unit 104 also generates metadata. The first shooting for depth synthesis, the second shooting, and the exposure are performed twice. In both the first and second exposure conditions, the shutter speed SS is 1/100 second, and the aperture is FNo. 16 is the same, the exposure data is the execution data. Therefore, as the metadata 44, as shown in FIG. 2D, the execution data as the common exposure condition is recorded, and the reference data is not recorded. Other users can obtain an image having the same brightness as the depth composite image by using this implementation data.

  Next, with reference to FIG. 3, a photographing method and an image composition method when the super-resolution composition mode is set will be described. FIG. 3A shows a state in which the user 10 photographs the subject 31 using the camera 20 in the super-resolution synthesis mode. The subject 31 is far away, and even if the subject 31 is photographed at a normal focal length, the subject 31 appears small, and even if enlarged display (enlarged print) is performed, a display with sufficient resolution is not obtained.

  In the situation shown in FIG. 3A, when the super-resolution synthesis mode is set, the imaging position changing unit 102e captures a plurality of frames while changing the position of the imaging element 102b. Then, by interpolating the pixel-to-pixel intermediate position in the image acquired by the first shooting using the image shot by changing the position of the image sensor 102b in the second to fourth shooting, the resolution is high. Composite the images. When the image 45 is generated by super-resolution composition, as shown in FIG. 3B, the resolution of the enlarged image 46 is high and enlargement viewing is possible.

  FIG. 3C shows a timing chart at the time of super-resolution imaging and movement of the image sensor 102b. At time t0, when the release button is operated and shooting is started, the first shooting is performed between times t11 and t12. The shutter speed SS at this time is 1/100 second, and the image sensor 102b is at the position P1. When shooting is completed at time t12, the imaging position changing unit 102e moves the imaging element 102b to the position P2 until time t13.

  Next, the second shooting is performed between times t13 and t14. The shutter speed SS at this time is 1/100 second, and the image sensor 102b is at the position P2. At this timing, the position P1 is indicated by a broken line, and the position P2 of the image sensor 102b is moved to the lower right by a predetermined amount as shown in FIG. This predetermined amount is a value (for example, 1/2 of the pixel pitch) smaller than the pixel interval (pixel pitch). When shooting is completed at time t14, the imaging position changing unit 102e moves the imaging element 102b to the position P3 until time t15.

  Next, the third shooting is performed between times t15 and t16. The shutter speed SS at this time is 1/100 second, and the image sensor 102b is at the position P3. At this timing, the position P1 is indicated by a broken line, and the position P3 of the image sensor 102b is moved to the upper right by a predetermined amount as shown in FIG.

  Thereafter, the fourth shooting is performed from time t17 to t18, and the fifth shooting is performed from time t19 to t20. Even at these timings, the shutter speed SS is 1/100 second, and the image sensor 102b is moved to positions P4 and P5 by a predetermined amount from the position P1 to the lower left and the upper left.

  When a plurality of image data are acquired, the image composition unit 103d performs super-resolution composition processing. That is, the pixel value between the pixels is interpolated using the image data when the pixel moves by a predetermined amount with reference to the position P1 of the image sensor 102b. For example, if the moving amount of the image sensor 102b is ½ of the pixel pitch, the pixel value at a position shifted by ½ pitch of the front, back, left, and right of the pixel can be interpolated, and the number of pixels is four times as high. Image data.

  When the image composition unit 103d generates image data by super-resolution composition processing, the metadata generation unit 104 also generates metadata. In order to perform super-resolution synthesis, exposure is performed five times. In the first to fifth exposure conditions, the shutter speed SS is 1/100 second, and the aperture is FNo. 16 is the same, the exposure data is the execution data. Therefore, as the metadata 47, as shown in FIG. 3D, the execution data which is the common exposure condition is recorded, and the reference data is not recorded.

  That is, when a plurality of pieces of image data are generated, the image composition unit 103d performs super-resolution composition processing to generate super-resolution image data. In the example shown in FIG. 3, the metadata creation unit 104 uses the shutter speed SS = 1/100 seconds, which is the exposure condition for one shooting, and the aperture value FNo. 16 is generated. The metadata (execution data 104a) generated here is recorded in the recording unit 105 as metadata 105c in association with the image data. Other users can obtain an image having the same brightness as the super-resolution composite image by using this implementation data.

  Next, a shooting method and an image composition method when the high dynamic range composition processing (HDR) mode is set will be described with reference to FIG. FIG. 4A shows a situation where the user 10 uses the camera 20 to photograph a person 32 and a cherry tree 33 that are subjects. These subjects are under clear sky, the cherry tree 33 is very bright, while the person is very dark due to backlight. That is, unless the range in which the image sensor 102b can reproduce light and dark is very wide, the light and dark of both subjects cannot be expressed properly. In such a case, shooting is performed in the HDR mode.

  FIG. 4C is a timing chart of imaging when the HDR mode is set. The standard exposure (appropriate exposure) at the time of shooting in the state of FIG. 4A, that is, the standard exposure considering both the subjects 32 and 33, the shutter speed is 1/100 second, the aperture value FNo. 16 When the HDR mode is set, the first shooting is performed under an exposure condition that is overexposed from the standard exposure, and then the second shooting is performed under an exposure condition that is underexposure.

  In the present embodiment, as shown in FIG. 4C, when the release button is operated and shooting starts at time t0, the first shooting is performed between times t21 and t23. The shutter speed SS at this time is an exposure time twice as long as 1/100 second (SS is 2/100 second). Subsequently, the second shooting is performed between times t24 and t25. The shutter speed SS at this time is an exposure time (SS is 1/200 second) that is ½ times 1/100. The focus position of the optical system 102a is the same in the first and second shooting.

  Using the overexposure image data acquired in the first shooting and the underexposure image data acquired in the second shooting, the image composition unit 103d performs HDR composition to reproduce the light and darkness from the dark part to the bright part. Generate superior image data. In this HDR composition, image data that does not have overexposure or underexposure over the entire screen is composed by connecting portions that are properly exposed in each image (subject).

  FIG. 4B shows an image 48 that is not subjected to HDR synthesis processing and an image 49 that is subjected to HDR synthesis processing. In the image 48 that has not been subjected to the HDR synthesizing process, the cherry tree 33 is overexposure while the person 32 is underexposure. On the other hand, in the image 49 that has been subjected to the HDR synthesizing process, the cherry tree 33 and the person 32 appropriately express light and dark.

  As shown in FIG. 4B, the metadata 50 of the image 49 that has been subjected to the HDR synthesizing process includes a shutter speed SS = 1/100 second and an aperture FNo. = 16. As described above, in the HDR synthesis mode, the exposure is performed twice, but since the exposure amounts are different from each other, the execution data (SS: 1/50, 1/200) 104a is not recorded, and the reference data (SS: 1/100) 104b is recorded (see FIG. 4D).

  That is, when a plurality of pieces of image data are generated, the image composition unit 103d performs HDR composition processing to generate HDR composite image data. In the example shown in FIG. 4, the metadata creation unit 104 uses the shutter speed SS = 1/100 seconds corresponding to the exposure condition of the image generated by the HDR composition, and the aperture value FNo. 16 is generated. The metadata (reference data 104b) generated here is recorded in the recording unit 105 as metadata 105c in association with the image data. By photographing with this reference data, an image with the same brightness can be obtained except for color expression.

  Next, the imaging timing when the noise reduction mode (NR mode) is set will be described with reference to FIG. When the ISO sensitivity is set to a high sensitivity, noise increases in the image and it becomes difficult to see. Therefore, by shooting a plurality of frames under the same exposure condition and averaging the output values of the corresponding pixels of each image, it is possible to synthesize an image with reduced noise.

  When the release button is operated and shooting is started in the state where the NR mode is set, shooting is performed between times t31 to t32, t33 to t34, t35 to t36, t37 to t38, and t39 to t40. As for the exposure conditions at this time, the shutter speed SS is 1/100 second and the ISO sensitivity is 6400. It should be noted that the optimum number of times of photographing in the NR mode may be selected as appropriate.

  When a plurality of pieces of image data are generated, the image composition unit 103d calculates an average value of the pixel values of the corresponding pixels, and generates NR composite image data. In the example shown in FIG. 5, the metadata creation unit 104 generates the ISO condition 6400 as the execution data 104a in the example shown in FIG. The metadata (execution data 104a) generated here is recorded in the recording unit 105 as metadata 105c in association with the image data.

  Next, with reference to FIG. 6, the shooting timing when the electronic camera shake prevention mode is set will be described. If the hand of the user holding the camera is shaking, the image will be blurred, resulting in an unsightly image. Therefore, by taking multiple frames under the same exposure conditions, shifting the pixel position of each image according to the amount of camera shake, and averaging the output values of the corresponding pixels to obtain image data with reduced camera shake Can do.

  When the release button is operated and shooting is started in a state where the electronic camera shake prevention mode is set, shooting is performed between times t41 to t42, t43 to t44, t45 to t46, t47 to t48, and t49 to t50. As for the exposure conditions at this time, the shutter speed SS is 1/100 second and the ISO sensitivity is 6400. If the amount of camera shake is equal to or greater than the predetermined amount (see the shooting at times t43 to t44 in FIG. 6), camera shake correction is often difficult, and is excluded from the target image for camera shake prevention synthesis processing. What is necessary is just to select the optimal frequency | count suitably as the frequency | count of imaging | photography in the electronic camera shake prevention mode.

  When a plurality of pieces of image data are generated, the image composition unit 103d detects the amount of camera shake, shifts the pixel position according to the amount of camera shake, calculates the average value of the pixel values of the corresponding pixels, Electronic image stabilization image data is generated. As the camera shake amount, the output of a sensor such as a gyro may be used, or the camera shake amount may be calculated by a correlation calculation of two image data.

  In the example shown in FIG. 6, the metadata creation unit 104 generates an ISO sensitivity 6400 of 1/100 seconds as an exposure condition for each photographing, that is, a shutter speed SS in the example illustrated in FIG. 6. The metadata (execution data 104a) generated here is recorded in the recording unit 105 as metadata 105c in association with the image data.

  As described with reference to FIGS. 2 to 6, in this embodiment, a plurality of frames are exposed, and the image composition unit 103d performs image composition using image data acquired by the exposure. The metadata generation unit 104 creates either the execution data 104 a or the reference data 104 d according to the type of image composition, and records the metadata 105 c in the recording unit 105.

  When the metadata is created, if the exposure conditions in each of the plurality of frames are the same, the actually controlled exposure condition is created as the execution data 104a. On the other hand, when the exposure conditions of each frame of the plurality of frames are different, an exposure condition corresponding to the image generated by the composition process is calculated, and this exposure condition is created as reference data 104b.

  As metadata, the metadata creation unit 104 may also create information for distinguishing the composition processing, and record the information for distinguishing the composition processing in the recording unit 105. In this case, the data created by the metadata creation unit 104 may be information that can distinguish whether the data is implementation data or reference data.

  The image composition process is not limited to the above-described depth composition process, super-resolution composition process, HDR composition process, NR composition process, and electronic image stabilization composition process. For example, the comparative light composition process, the comparative dark composition process, the average What is necessary is just to perform composition processing using image data of a plurality of frames, such as addition composition processing and live valve composition processing. In this case, for example, in the case of comparatively bright combination processing, the total exposure time may be recorded as metadata as implementation data.

  Next, camera control in the present embodiment will be described using the flowcharts shown in FIGS. This flowchart is executed by the CPU in the control unit 101 controlling each unit in the camera according to a program stored in the memory.

  If the camera control flow is entered, it is first determined whether or not the camera is in shooting mode (S1). The camera mode is set to the shooting mode as a default unless the user sets another mode such as a playback mode. In this step, the mode setting state is detected and a determination is made based on the detection result.

  If the result of determination in step S <b> 1 is shooting mode, next, through-image imaging is performed (S <b> 3). Here, the image data acquired by the image sensor 102b is acquired. This image data is repeatedly acquired at a predetermined frame rate.

  Once the through image has been picked up, next, through image display, focusing, and standard exposure are set (S5). The image processing unit 103 performs image processing for through image display on the image data acquired in step S3, and displays the through image on the display unit 106 based on the processed image data. The through image display is updated at a predetermined frame rate. Further, the contrast value of the acquired image data is calculated, and the position of the focus lens of the optical system 102a is adjusted so that the contrast value reaches a peak. The focusing may be performed by other methods such as phase difference AF in addition to the above-described contrast AF. Also, luminance information and the like are calculated from the image data, and standard exposures such as a shutter speed, an aperture value, and ISO sensitivity are set based on the luminance information.

  Subsequently, it is determined whether or not the mode has been switched (S7). The mode can be switched by the operation member of the operation unit 107. The modes here include modes in which a plurality of frames are exposed and combined, such as a depth combining mode, a super-resolution combining mode, and an HDR combining mode, in addition to the above-described shooting mode and playback mode.

  If the result of determination in step S7 is that there is mode switching, mode setting is performed (S9). Here, the mode in the mode control unit 101b is set according to switching to the mode by the operation unit 107.

  If mode setting is performed in step S9 or if the result of determination in step S7 is that there is no mode switching, it is next determined whether or not the release switch has been operated (S11). When the release button is operated, the state of the release switch changes. Here, the determination is made based on the state change of the release switch. If the result of this determination is that the release switch has not been operated, processing returns to step S1.

  If the result of determination in step S11 is that the release switch has been operated, it is next determined whether or not the mode involves exposure change (S13). Here, the determination is made based on the first set mode or the mode set in step S9. As a mode involving exposure change, for example, there is an HDR synthesis mode. Examples of modes that do not involve exposure change include a depth synthesis mode, a super-resolution synthesis mode, a noise reduction (NR) mode, and an electronic camera shake prevention mode.

  As a result of the determination in step S13, in the case of a mode involving exposure change, a plurality of images are taken after changing from standard exposure to long time and short time (S15). In this case, when shooting a plurality of frames, the exposure conditions are different for each frame. For example, in the HDR composition mode described with reference to FIG. 4, the shutter speed is set to a long time so that the first frame is overexposed than the standard exposure (set in step S5), and the standard exposure is set for the second frame. In this case, the shutter speed is set to a short time so that the exposure undershoots.

  Subsequently, image composition is performed (S17). Here, image composition processing is performed on the plurality of image data acquired in step S15 according to the mode in which the image composition unit 103d is set.

  Once image composition has been carried out, next, image data and metadata (reference exposure) are recorded (S19). Here, the metadata creation unit 104 creates the reference data 104b. As described above, the reference data 104b is data indicating an exposure condition corresponding to the effect of an image generated by image synthesis. The created reference data 104b is recorded as image data 105b and metadata 105c of the recording unit 105 in association with the image data synthesized in step S17. When data is recorded, the process returns to step S1.

  Returning to step S13, if the result of determination in this step is that the mode is not accompanied by exposure change, a plurality of images are taken with the standard exposure changed in focus and image sensor position (S21). Here, processing according to the set mode is performed. For example, when the depth synthesis mode is set, the focus position is changed, and shooting is performed at each of the changed focus positions. Further, when the super-resolution synthesis mode is set, the position of the image sensor 102b is changed, and shooting is performed at each of the changed positions. The exposure condition at this time is the standard exposure calculated in step S5.

  Subsequently, image composition is performed (S23). Here, the image composition processing is performed on the plurality of image data acquired in step S21 according to the mode in which the image composition unit 103d is set.

  Once image composition has been performed, image data and metadata (implemented exposure) are recorded (S25). Here, the metadata creation unit 104 creates the implementation data 104a. As described above, the execution data 104a is data indicating an exposure condition in which exposure control is actually performed in each frame when shooting a plurality of frames. The created implementation data 104a is recorded as image data 105b and metadata 105c of the recording unit 105 in association with the image data synthesized in step S23. When data is recorded, the process returns to step S1.

  Returning to step S1, if the result of this determination is that it is not in shooting mode, it is determined whether or not it is in playback mode (S31). When the user operates an operation member such as a playback button of the operation unit 107, the playback mode can be set. If the result of determination in this step is not playback mode, transition to other modes is made.

  If the result of determination in step S31 is playback mode, an icon list is then displayed (S33). Here, based on the thumbnail data of the image data recorded in the recording unit 105, the icon of the recorded image is displayed on the display unit 106.

  Next, it is determined whether an image has been selected (S35). The user touches an icon desired to be enlarged from the icons displayed in the list displayed on the display unit 106, or selects the icon with a cross button or the like. In this step, it is determined whether or not an icon has been selected by the user.

  If the result of determination in step S35 is that no image has been selected, it is determined whether or not to return (S37). When the user wants to end the reproduction mode, the user operates the return button in the operation unit 107 or clicks the return icon (collectively returning operation). Here, the determination is made based on whether or not a return operation has been performed. If the result of this determination is that no return operation has been performed, processing returns to step S33. On the other hand, if a return operation has been performed, the process returns to step S1.

  If the result of determination in step S35 is that an image has been selected, enlargement display is performed (S39). Here, the image data of the selected icon is read out and enlarged on the display unit 106.

  Once the enlarged display is performed, it is next determined whether or not the detailed display is performed (S41). When it is desired to display metadata recorded in association with the image selected by the user, a detail button in the operation unit 107 is operated or a detail icon is clicked (collectively, a detailed display operation). Here, the determination is made based on whether or not a detailed display operation has been performed.

  If the result of determination in step S41 is that a detail display operation has been performed, metadata is displayed (S43). Here, the metadata 105 c recorded in association with the image data is read from the recording unit 105 and displayed on the display unit 106. A display example will be described later with reference to FIG.

  If metadata is displayed in step S43 or if the result of determination in step S41 is that a detailed display operation has not been performed, it is determined whether or not external transmission is performed (S45). When the user transmits the image data recorded in the recording unit 105 via the Internet or the like by wireless or wired communication, the user operates the transmission button of the operation unit 107 or touches the transmission icon (generic name). And external send operation). In this step, the determination is made based on whether or not an external transmission operation has been performed.

  If the result of determination in step S45 is that an external transmission operation has been made, transmission is made to the setting destination (S47). Here, the image data of the image selected in step S35 and the associated metadata are transmitted to the set transmission destination. At the transmission destination, an image and metadata based on the received image data can be displayed. A display example at this time will be described later with reference to FIG.

  If transmission is made to the setting destination in step S47, or if the result of determination in step S45 is that there is no external transmission operation, it is determined whether or not to return (S49). When the user wants to end the playback mode, a return operation is performed. Here, the determination is made based on whether or not this return operation has been performed. If the result of this determination is that no return operation has been performed, processing returns to step S39. On the other hand, if a return operation has been performed, the process returns to step S1.

  As described above, in the camera control flow in the present embodiment, when a mode involving exposure change is set when a release operation is performed (Yes in S13), reference data is created and recorded as metadata. (S19). On the other hand, when a mode without exposure change is set (No in S13), the execution data is created and recorded as metadata (S25). That is, when the exposure change is not accompanied, the control value that is actually controlled for exposure is recorded, whereas when the exposure change is accompanied, the exposure control value according to the effect of the composite image is recorded. When the metadata is viewed, there is an effect that it is easy to understand what control is performed.

  In steps S19 and S25, only one of reference data and implementation data is created and recorded as metadata. By recording only one of them, the capacity of the metadata 105c can be reduced, and simplification simplifies the processing. However, in steps S19 and S25, both reference data and implementation data may be created and recorded as metadata 105c. In this case, for example, an area that can be recorded by the manufacturer, such as an Exif standard manufacturer note, may be used in the metadata recording area.

  Next, display of metadata will be described with reference to FIG. This display is performed at the transmission destination transmitted in step S43 and step S47 of FIG.

  FIG. 9A shows a display example when the depth synthesis mode as shown in FIG. 2 is set. In this case, the depth-combined image 43 and metadata indicating the shutter speed and aperture value recorded as the execution data are displayed. FIG. 9B is a display example when the super-resolution synthesis mode as shown in FIG. 3 is set. In this case, the depth-combined image 46 and metadata indicating the shutter speed and aperture value recorded as the execution data are displayed.

  FIG. 9C is a display example when the HDR synthesis mode as shown in FIG. 4 is set. In this case, HDR combined image 49 and metadata indicating shutter speed and aperture value recorded as reference data are displayed. In FIG. 9C, “reference” is displayed to clearly indicate that the data is reference data. For this purpose, information indicating reference data is also recorded in the metadata 105c.

  Thus, by displaying metadata in addition to the recorded image, it is possible to easily check the exposure condition of the captured image. In this display, if there is no exposure change, the actual exposure value when each frame was shot is displayed. If there is an exposure change, an exposure value corresponding to the effect of the composite image is displayed. As a result, the display is easy to understand. Other users other than the photographer can easily take a similar picture by referring to the exposure value based on the metadata.

  In FIG. 9, “reference” is displayed only in the case of reference data, but this display may be deleted. Also, in the case of implementation data, a display that clearly indicates the implementation data may be added. In FIG. 9, the metadata is displayed outside the image, but may be displayed superimposed on the image.

  As described above, in one embodiment of the present invention, a plurality of captured image data are acquired by switching the imaging conditions for the same object (for example, see S3 in FIG. 7) and acquired under different exposure conditions. The first image composition is performed using the plurality of photographed image data (see, for example, S17 in FIG. 7), and the plurality of photographed image data acquired by changing the same exposure conditions and conditions other than the exposure are used. The second image composition is performed (see, for example, S23 in FIG. 7), and the first image composition is performed when creating the metadata representing the first image composition or the composite image data synthesized by the second image composition. And a case where the second image composition is performed and single metadata indicating different exposure conditions is created (for example, FIG. 2D, FIG. 3D, FIG. 4D, and FIG. 7). (See S19 and S25). For this reason, it is possible to create appropriate information according to the image composition processing and associate it with the image data. In other words, when acquiring multiple shot image data, different exposure conditions are shown depending on whether exposure is changed or not, and information can be displayed according to the image composition process. It becomes.

  In one embodiment of the present invention, it is determined whether or not the mode involves exposure change (see S13 in FIG. 7), and reference data or implementation data is recorded as metadata according to the result. ing. However, the type of metadata may be appropriately determined according to the shooting conditions, for example, whether or not the mode involves exposure change, for example, whether or not the shooting distance is changed. In this case, the image data acquisition unit can acquire a plurality of captured image data by switching the first imaging condition with respect to the same object, and the first image composition unit can be obtained by the image data acquisition unit. A plurality of photographed image data is obtained by changing the photographing conditions of 2 and the first image composition is performed using the obtained plurality of photographed image data. Without changing the shooting conditions, the third shooting conditions are changed to obtain a plurality of shot image data, and the second image composition is performed using the acquired shot image data. When creating metadata representing the synthesized image data synthesized by the first image synthesizing unit or the second image synthesizing unit, the case where the first image synthesis is performed is different from the case where the second image synthesis is performed. Single showing shooting conditions Metadata can be created.

  In one embodiment of the present invention, the focus changing unit 102c, the exposure changing unit 102d, the imaging position changing unit 102e, and the like may realize their functions by software using programs stored in the CPU and the memory. Of course, the function may be realized by the CPU stored in the control unit 101 and the program stored in the memory.

  In one embodiment of the present invention, all or part of the functions of the respective units such as the imaging control unit 103b and the imaging control change unit 103c are realized by a CPU (Central Processing Unit), a peripheral circuit, and a program. It may be realized by a circuit executed by a program code such as a DSP (Digital Signal Processor), or may be a hardware configuration such as a gate circuit generated based on a program language described by Verilog, Of course, it may be executed by a hardware circuit.

  In the embodiment of the present invention, the digital camera is used as the photographing device. However, the camera may be a digital single lens reflex camera, a mirrorless camera, a compact digital camera, a video camera, a movie A camera for moving images such as a camera may be used, and a scientific device such as a mobile phone, a smartphone, a personal digital assistant (PC), a personal computer (PC), a tablet computer, a game machine, etc., a medical camera, a microscope, etc. It may be a camera for a car, a camera mounted on a car or a camera for monitoring. In any case, the present invention can be applied to any device for photographing that records metadata together with image data.

  Of the techniques described in this specification, the control mainly described in the flowchart is often settable by a program and may be stored in a recording medium or a recording unit. The recording method for the recording medium and the recording unit may be recorded at the time of product shipment, may be a distributed recording medium, or may be downloaded via the Internet.

  Further, in one embodiment of the present invention, the operation in the present embodiment has been described using a flowchart. However, the processing procedure may be changed in order, or any step may be omitted. Steps may be added, and specific processing contents in each step may be changed.

  In addition, regarding the operation flow in the claims, the specification, and the drawings, even if it is described using words expressing the order such as “first”, “next”, etc. It does not mean that it is essential to implement in this order.

  The present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, you may delete some components of all the components shown by embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

DESCRIPTION OF SYMBOLS 10 ... User, 20 ... Camera, 101 ... Control part, 101b ... Mode control part, 101c ... Display control part, 101d ... Communication control part, 102 ... Imaging part, 102a ... Optical system, 102b ... Image sensor, 102c ... Focus change unit, 102d ... Exposure change unit, 102e ... Imaging position change unit, 103 ... Image processing unit, 103b ... Image capturing control unit, 103c ... Shooting control changing unit, 103d ... Image composition unit, 103e ... Recording control unit, 104 ... Metadata creation unit, 104a ... Execution data, 104b ... Reference data, 105 ... recording unit, 105a ... image data, 105b ... metadata, 106 ... display unit, 107 ... operation unit, 108 ... communication unit

Claims (7)

An image data acquisition unit capable of acquiring a plurality of captured image data by switching the imaging conditions for the same object;
A first image composition unit that obtains a plurality of captured image data under different exposure conditions by the image data acquisition unit, and performs a first image composition using the obtained plurality of captured image data;
The image data obtaining unit obtains a plurality of photographed image data by changing the same exposure condition and a condition other than the exposure, and performs a second image composition using the obtained plurality of photographed image data. An image composition unit;
A metadata creating unit that creates metadata representing the synthesized image data synthesized by the first image synthesizing unit or the second image synthesizing unit;
Have
The metadata creation unit creates single metadata indicating different exposure conditions when performing the first image composition and when performing the second image composition. .   The image processing apparatus according to claim 1, wherein the metadata creating unit creates information for distinguishing the first image composition from the second image composition.   The metadata creation unit creates reference metadata based on the exposure condition in the image data acquisition unit when the first image composition is performed, and the metadata creation unit when the second image composition is performed. The image processing apparatus according to claim 2, wherein actual exposure conditions in the image data acquisition unit are set as implementation metadata.   The metadata creation unit creates metadata corresponding to the exposure effect of the composite image generated by the first image composition when the first image composition unit performs the first image composition. The image processing apparatus according to claim 1.   In the case where the second image composition unit performs the second image composition, the metadata generation unit performs metadata based on an exposure condition when the image data acquisition unit obtains one frame of image data. The image processing apparatus according to claim 1, wherein: An image data acquisition unit capable of acquiring a plurality of captured image data by switching the first imaging condition for the same object;
A first image composition unit that obtains a plurality of photographed image data by changing the second photographing condition by the image data obtaining unit, and performs a first image composition using the obtained plurality of photographed image data;
The image data acquisition unit acquires a plurality of captured image data by changing the third imaging condition without changing the second imaging condition, and uses the acquired plurality of captured image data to generate a second image composition. A second image composition unit for performing
A metadata creating unit that creates metadata representing the synthesized image data synthesized by the first image synthesizing unit or the second image synthesizing unit;
Have
The metadata creation unit creates single metadata indicating different shooting conditions when performing the first image composition and when performing the second image composition. . For the same object, switch the shooting conditions to obtain multiple shot image data,
A first image composition is performed using a plurality of photographed image data acquired under different exposure conditions,
A second image composition is performed using a plurality of photographed image data acquired with the same exposure condition and different conditions other than exposure,
When creating metadata representing the synthesized image data synthesized by the first image synthesis or the second image synthesis, there is a difference between the case of performing the first image synthesis and the case of performing the second image synthesis. Create a single piece of metadata describing exposure conditions,
An image processing method.

Images