JP6732572B2 - Imaging device and imaging method - Google Patents

Description

The present invention relates to an imaging device and an imaging method.

Various types of image processing are known that combine images obtained as a result of multiple times of imaging. For example, in a high dynamic range (HDR) composition process, the dynamic range of an image is expanded by compositing images obtained as a result of multiple times of imaging with different exposure amounts (for example, Patent Document 1). Further, in the depth stacking process, the focus state of the entire image is improved by synthesizing the images obtained as a result of multiple times of imaging with different focus states.

JP-A-8-37628

When synthesizing images obtained as a result of a plurality of times of imaging, depending on the situation, for example, it may be difficult to perform synthesizing when an object moves during each imaging.

The present invention has been made in view of the above circumstances, and provides an image pickup apparatus and an image pickup method capable of improving the image quality of images after combination when images obtained as a result of a plurality of times of image pickup are combined. The purpose is to do.

In order to achieve the above-mentioned object, an imaging device according to a first aspect of the present invention includes an imaging unit that captures an image of an object to obtain image data of the object, and a movement of the object in the image data. As a result of the motion detection unit to detect and the motion detection of the object by the motion detection unit, when the object is considered to be stationary, a plurality of application processes involving image combination processing of the plurality of image data. Whether or not to implement is determined, the total number of frames is calculated based on the number of frames required for each of the application processes determined to be performed, and a plurality of frames corresponding to the total number of frames are changed while changing the imaging condition of the imaging unit. And a control unit for performing a process of synthesizing image data obtained by a plurality of times of imaging according to each of the application processes , the control unit including a plurality of units determined to be implemented. When there is a frame that can be duplicated in the applying process, the number of frames that can be duplicated is reduced to calculate the total number of frames .

In order to achieve the above-mentioned object, an imaging method according to a second aspect of the present invention provides an image of an object by an imaging unit to obtain image data of the object, and the object of the image data. As a result of detecting the motion and detecting the motion of the target object, when it is considered that the target object is stopped, it is possible to determine whether or not to perform a plurality of application processes involving an image synthesizing process of the plurality of image data. and Rukoto to determine, and calculating the total number of frames based on the number of frames required for the process of applying each determined to be performed, depending on the total number of frames while changing the image pickup condition for pre-Symbol imaging unit Performing the imaging a plurality of times, and synthesizing the image data obtained by the imaging a plurality of times according to each of the application processes, and calculating the total number of frames is determined to be performed by a plurality of When there is a frame that can be duplicated in the application process of, the number of frames that can be duplicated is reduced to calculate the total number of frames .

According to the present invention, it is possible to provide an image pickup apparatus and an image pickup method capable of improving the image quality of images after combination when images obtained as a result of a plurality of times of image pickup are combined.

It is a block diagram which shows the structure of an example of the imaging device which concerns on one Embodiment of this invention. 7 is a flowchart showing an operation of the image pickup apparatus in a still image shooting mode. It is a flow chart which shows scene discriminating processing. It is a flow chart which shows image compositing execution judging processing. It is a flowchart which shows a frame number determination process. It is a figure for explaining an example of image pickup control for composition. 7 is a flowchart showing an operation of the image pickup apparatus in a moving image shooting mode. It is a figure for demonstrating the modification of imaging control for composition.

Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram showing a configuration of an example of an image pickup apparatus according to an embodiment of the present invention. The imaging device shown in FIG. 1 is various mobile terminal devices such as a digital camera, a mobile phone with a camera function, and a smartphone. The imaging device shown in FIG. 1 includes an imaging unit 102, a display unit 104, a temporary recording unit 106, a recording unit 108, an operation unit 110, and a control unit 112.

The image pickup unit 102 has an image pickup optical system 102a, an image pickup element 102b, and a drive unit 102c. The imaging optical system 102a has a diaphragm, a lens, and the like, and causes a light flux from an object (not shown) to enter the imaging element 102b. The image pickup optical system 102a includes a focus lens for adjusting the in-focus state. The image sensor 102b includes, for example, a CMOS image sensor or a CCD image sensor, images an object, and acquires image data of the object. The image sensor 102b may include a phase difference detection pixel so that the distance to the object to be photographed can be detected. Further, the image pickup element 102b in the present embodiment may be configured to be movable in the image pickup optical system 102a in a plane orthogonal to the optical axis. The drive unit 102c drives the focus lens of the image pickup optical system 102a in the optical axis direction and drives the image pickup device 102b under the control of the control unit 112.

The display unit 104 is, for example, a liquid crystal display or an organic EL display, and displays various images such as an image based on the image data acquired by the imaging unit 102.

The temporary recording unit 106 is a volatile memory such as a RAM, and is a recording medium for temporarily recording various data. In the temporary recording unit 106, for example, image data used for image combination processing is temporarily recorded.

The recording unit 108 is a non-volatile memory such as a flash memory, and is a recording medium for recording various data. An image file obtained as a result of the shooting operation is recorded in the recording unit 108, for example. In addition, the recording unit 108 records a program used for controlling the imaging device.

The operation unit 110 is an operation member for a user to operate the imaging device. The operation unit 110 has a power switch, a release button, etc. as operation members. The power switch is an operation member for giving an instruction to turn on/off the power of the image pickup apparatus. The release button is an operation member for giving an instruction to start shooting. Further, the operation unit 110 may have another operation member such as a touch panel.

The control unit 112 is a control circuit such as a CPU and an ASIC, and controls the operation of the image pickup apparatus as a whole. The control unit 112 has a function as the image pickup control unit 1121, a function as the display control unit 1122, a function as the image processing unit 1123, a function as the scene determination unit 1124, and a function as the motion detection unit 1125. Have Here, each function of the control unit 112 may be realized by a single piece of hardware or software, or may be realized by a plurality of pieces of hardware or software. In addition, some functions may be provided separately from the control unit 112.

The imaging control unit 1121 controls the imaging operation of the object by the imaging unit 102 and the operation of reading image data from the imaging unit 102.

The display control unit 1122 controls when displaying various images on the display unit 104.

The image processing unit 1123 performs various types of image processing on image data. This image processing includes white balance processing, gradation correction processing, and the like. Further, this image processing includes a combining process for combining a plurality of image data.

The scene determination unit 1124 determines the image data and the scene in which the image data is captured. For example, the scene discrimination unit 1124 discriminates the brightness of the image data, the in-focus state with respect to the target object, and the like. Further, the scene determination unit 1124 determines the image capturing condition when the image data is captured, for example, the image capturing sensitivity. The ones listed here are only examples. The scene determination unit 1124 may perform various types of determination such as color determination of image data. Further, the scene determination unit 1124 may determine various imaging conditions such as the imaging distance. Further, here, the scene discrimination unit 11 2 4 determines may be performed in combination with information of various sensors. Alternatively, it may be set manually instead of being automatically determined. Furthermore, this may be replaced by a technique that can be expressed in addition to the word "scene determination". For example, the brightness determination, the image capturing determination, the image determination, the distance determination, and the object determination may be referred to again. Further, since the image processing and the composition method are changed based on this result, the image processing determination and the composition determination may be performed.

The motion detection unit 1125 detects the motion of the target object during a plurality of times of imaging. The motion detection unit 1125 detects the motion of the target object by detecting a motion vector, for example.

Next, a shooting operation of the image pickup apparatus according to the present embodiment will be described. The imaging device according to the present embodiment has a still image shooting mode and a moving image shooting mode as shooting modes. The still image shooting mode is a shooting mode for shooting a still image according to a user's shooting instruction. The moving image shooting mode is a shooting mode for shooting a moving image in accordance with a user's shooting instruction. The shooting mode is set, for example, by the user operating the operation unit 110.

First, the processing in the still image shooting mode will be described. FIG. 2 is a flowchart showing the operation of the image pickup apparatus in the still image shooting mode. The process of FIG. 2 is started, for example, when the shooting mode of the imaging device is switched to the still image shooting mode.

In step S1, the control unit 112 determines whether or not the still image shooting mode is the scene determination mode. In the present embodiment, the still image shooting mode has a normal mode and a scene determination mode. The normal mode is a mode in which an application process involving image combination processing of a plurality of image data obtained by a plurality of times of imaging such as HDR combination processing, depth combination processing, and noise reduction (NR) combination processing is not performed. The scene determination mode is a mode in which an application process involving image combination processing of a plurality of image data obtained by a plurality of image pickups is performed. In the scene determination mode in the present embodiment, an application process is appropriately selected from application processes such as HDR synthesis process, depth synthesis process, and noise reduction (NR) process according to the scene determination result. The setting of whether the still image shooting mode is the normal mode or the scene determination mode is performed, for example, by the operation of the operation unit 110 by the user. When it is determined in step S1 that the still image shooting mode is not the scene determination mode, that is, the normal mode, the process proceeds to step S2. When it is determined in step S1 that the still image shooting mode is the scene determination mode, the process proceeds to step S7. In the scene determination, as will be described later, whether or not the image is to be combined is determined. Therefore, step S1 may be a branch of an expression such as whether or not to allow composition, in addition to the description of the scene determination mode. .. In addition, there is always a specification for permitting composition, and it is possible to always branch to step S7 without this branch.

In step S2, which is processing in the normal mode, the control unit 112 determines whether or not there is a still image shooting request. Still image capturing request is made by, for example, the operation of the operation or the touch panel of the release button by the user. If it is determined in step S2 that there is a still image shooting request, the process proceeds to step S3. When it is determined in step S2 that there is no still image capturing request, the process proceeds to step S4.

In step S3, the control unit 112 starts still image shooting. For example, the control unit 112 sets the aperture setting of the image pickup optical system 102a, the exposure time of the image pickup element 102b, and the like according to the brightness of the object. Then, the control unit 112 causes the image pickup unit 102 to perform an image pickup operation. The control unit 112 performs necessary image processing on the image data obtained by the imaging operation to create a still image file. Then, the control unit 112 records the created still image file in the recording unit 108. After such a still image shooting operation, the process proceeds to step S4.

In step S4, the control unit 112 performs display imaging control. For example, the control unit 112 causes the image capturing unit 102 to perform the image capturing operation by setting the aperture for displaying the through image and setting the exposure time.

In step S5, the control unit 112 causes the display unit 104 to display an image based on the image data obtained as a result of the display imaging control in step S4. As a result, a through image display is performed.

In step S6, the control unit 112 determines whether to end the operation in the still image shooting mode. For example, when the shooting mode is switched to the moving image shooting mode or the power of the image pickup apparatus is turned off, it is determined that the operation in the still image shooting mode ends. When it is determined in step S6 that the operation in the still image shooting mode is to be ended, the processing in FIG. 2 is ended. When it is determined in step S6 that the operation in the still image shooting mode is not ended, the process returns to step S1.

In step S7, which is the process of the scene determination mode, the control unit 112 performs the scene determination process. The scene determination process will be described below. FIG. 3 is a flowchart showing the scene discrimination processing. This may be replaced by a technique that can be expressed other than the word "scene determination". For example, the brightness determination, the image capturing determination, the image determination, the distance determination, and the object determination may be referred to again. In addition, even if the word “judgment” is used, the user may manually set the value instead of automatically making the judgment.

In step S21, the control unit 112 determines whether the ratio of low-level pixels and saturation-level pixels to all pixels in the image data obtained via the imaging unit 102 is a predetermined value or more (for example, 10%), that is, It is determined whether or not the difference in brightness in the image data is large. If data of one pixel is represented by 256 gradations, a low-level pixel is a pixel having data of 16 gradations or less, and a saturation level pixel is a pixel having data of 246 gradations or more, for example. Is. When it is determined in step S21 that the ratio of the low level pixel to the saturation level pixel is equal to or greater than the predetermined value, the process proceeds to step S22. When it is determined in step S21 that the ratio of the low level pixel to the saturation level pixel is not equal to or more than the predetermined value, the process proceeds to step S23.

In step S22, the control unit 112 determines to execute the HDR combining process as the applying process. After that, the process proceeds to step S23. The HDR synthesizing process is a process of expanding the dynamic range of image data by synthesizing image data obtained by a plurality of times of imaging with different exposure amounts. Such an HDR synthesizing process is effective in a scene having a large difference in light and shade, that is, a scene requiring a high dynamic range.

In step S23, the control unit 112 determines whether or not the image pickup sensitivity set in the image pickup element 102b is a certain value (for example, ISO8000) or more. The imaging sensitivity is set, for example, by the user operating the operation unit 110. When it is determined in step S23 that the imaging sensitivity set in the image sensor 102b is equal to or higher than a certain value, the process proceeds to step S24. When it is determined in step S23 that the imaging sensitivity set in the image sensor 102b is not equal to or higher than the certain value, the process proceeds to step S25.

In step S24, the control unit 112 determines to perform the NR combination process as the application process. After that, the process proceeds to step S25. The NR synthesizing process is a process of reducing noise of image data by averaging and synthesizing image data obtained by a plurality of times of image pickup. Such an NR combination process is effective in a scene with a lot of noise. For example, as the imaging sensitivity increases, the noise increases, so that the NR synthesis process is effective. In reality, the effect of noise is not so large unless the brightness of the object is low, so in step S23, when the brightness of the object is equal to or lower than a predetermined value and the imaging sensitivity is equal to or higher than the predetermined value, the step is performed. In S24, it may be decided to perform the NR combination process.

In step S25, the control unit 112 determines whether the target object is out of the depth of focus, that is, whether the target object is out of focus. The target object is, for example, the face of a person. For example, when the hyperfocal depth is H, the focal length is f, the aperture value is N, and the permissible circle of confusion is c, when the defocus amount for the object is H=f ² /Nc or more, the object is focused. It is determined to be out of depth. When it is determined in step S25 that the object is out of the depth of focus, the process proceeds to step S26. When it is determined that the object is not out of the depth of focus, the process of FIG. 3 ends.

In step S26, the control unit 112 determines to perform the depth stacking process as the applying process. After that, the process of FIG. 3 ends. The depth combination process is a process for improving the focus state of image data by combining image data obtained by a plurality of times of image pickup in which the focus position of the image pickup optical system 102a is different. The depth stacking process is effective in a scene in which only a part of the object is in focus.

As described above, since the image processing and the composition method are changed according to the result of the flowchart shown in FIG. 3, this may be written as the image processing judgment and the composition judgment instead of the “scene judgment”. When manually set by the user, it may be expressed as a combination switching process. It may be expressed as a combination process selection step. In some cases, there is only one combination method, and in this case, there is no switching, so the scene determination step can be omitted.

Here, returning to the description of FIG. In step S8 after the scene determination process, the control unit 112 performs an image synthesis execution determination process. The image compositing execution determination process will be described below. FIG. 4 is a flowchart showing the image compositing execution determination process. In step S31, the control unit 112 calculates a motion vector of each target object from image data of a plurality of frames obtained by imaging for displaying a through image.

In step S32, the control unit 112 can determine whether or not the magnitude of the maximum motion vector is equal to or smaller than a fixed value (for example, about 5 pixels on the image data), that is, both the imaging device and the object are stopped. It is determined whether or not the state. When it is determined in step S32 that the maximum motion vector magnitude is equal to or smaller than the fixed value, the process proceeds to step S33. When it is determined in step S32 that the maximum motion vector magnitude is not equal to or less than the fixed value, the process of FIG. 4 ends.

In step S33, the control unit 112 determines to perform the image combining process. After that, the processing of FIG. 4 ends. That is, in the present embodiment, the image combining process is performed in a state where the object can be regarded as stopped. As a result, the image combining process can be performed with high image quality.

Here, returning to the description of FIG. In step S9 after the image synthesis execution determination process, the control unit 112 determines whether to perform the image synthesis process based on the result of the image synthesis execution determination process. When it is determined in step S9 that the image combining process is performed, the process proceeds to step S10. When it is determined in step S9 that the image synthesizing process is not performed, the process proceeds to step S17.

In step S10, the control unit 112 determines the image capture frame rate. The imaging frame rate is set according to, for example, a still image recording mode. That is, in the recording mode in which high image quality (high number of pixels) is required even if some responsiveness is sacrificed, the control unit 112 sets the imaging frame rate to a low value (for example, about 60 fps). On the other hand, in the recording mode in which the responsiveness is prioritized, the control unit 112 sets the imaging frame rate to a high value (for example, to about 240 fps). The recording mode is set, for example, by the user operating the operation unit 110.

In step S11, the control unit 112 performs a frame number determination process. The frame number determination process will be described below. FIG. 5 is a flowchart showing the frame number determination process. In step S41, the control unit 112 determines whether to execute the HDR combining process. If it is determined in step S41 that the HDR combining process is to be performed, the process proceeds to step S42. When it is determined in step S41 that the HDR combining process is not performed, the process proceeds to step S43.

In step S42, the control unit 112 sets a frame for HDR combination processing. Then, a process transfers to step S43. The HDR combination processing requires at least two pieces of image data having different exposure amounts. Therefore, the control unit 112 adds, for example, two frames as a frame for HDR combination processing. Of course, the number of frames for HDR combination processing may be three or more.

In step S43, the control unit 112 determines whether to perform the NR synthesis process. If it is determined in step S43 that the NR synthesis process is to be performed, the process proceeds to step S44. When it is determined in step S43 that the NR synthesis process is not performed, the process proceeds to step S45.

In step S44, the control unit 112 sets a frame for NR synthesis processing. Then, a process transfers to step S45. The number of frames for the NR synthesis processing is set, for example, according to the imaging sensitivity. For example, when the imaging sensitivity is ISO 16000 or less, the control unit 112 adds one frame as a frame for NR synthesis processing. Further, when the imaging sensitivity is higher than ISO16000 and ISO32000 or lower, the control unit 112 adds two frames as a frame for NR synthesis processing. Further, when the imaging sensitivity is higher than ISO32000 and ISO64000 or less, the control unit 112 adds four frames as a frame for NR synthesis processing. In general, the noise amount is reduced to 1/n of the route by synthesizing the image data of n frames in the NR synthesizing process. The method of calculating the number of frames for the NR synthesis processing is not limited to the method described here.

In step S45, the control unit 112 determines whether to perform the depth stacking process. If it is determined in step S45 that the depth stacking process is to be performed, the process proceeds to step S46. When it is determined in step S45 that the depth stacking process is not performed, the process proceeds to step S47.

In step S46, the control unit 112 sets a frame for depth stacking processing. After that, the process proceeds to step S47. For example, the control unit 112 calculates the drive time of the focus lens. The drive time of the focus lens is calculated by (drive amount of focus lens required for depth stacking)/(drive amount of focus lens per unit time). After calculating the drive time of the focus lens, the control unit 112 calculates the number of frames to be added for the depth stacking process. The number of frames to be added is calculated, for example, by (driving time of the focus lens)/(time for one imaging frame). The method of calculating the number of frames for the depth stacking process is not limited to the one described here.

In step S47, the control unit 112 calculates the total number of frames. Then, the process of FIG. 5 ends. The total number of frames is basically calculated by (the number of frames for HDR combination processing)+(the number of frames for NR combination processing)+(the number of frames for depth combination processing). However, if there is a frame that can be overlapped with a frame for HDR synthesis processing, a frame for NR synthesis processing, and a frame for depth synthesis processing, the total number of frames is reduced accordingly. obtain. An upper limit may be set for the total number of frames. For example, when it is assumed that the combining process is performed in the live view display, the combining process needs to be completed in a time shorter than the display frame rate of the display unit 104. In this case, the total number of frames is limited by the display frame rate of the display unit 104.

Here, returning to the description of FIG. In step S12 after the frame number determination process, the control unit 112 executes the composite imaging control. Thereafter, the processing shifts to step S13. Hereinafter, an example of the image capturing control for composition will be described with reference to FIG. In the composite image capturing control, image capturing is performed while changing one or more image capturing conditions for each image capturing. The imaging conditions here are the exposure amount required for HDR composition and the focus lens position required for depth composition. It should be noted that the squares in FIG. 6 indicate the timing of each image pickup in the image pickup control for synthesis. “Exposure” in FIG. 6 indicates the amount of change in exposure at each imaging timing. -1 indicates that the exposure is one step below the proper exposure, and +1 indicates that the exposure is one step above the proper exposure. “HDR composition” in FIG. 6 indicates whether or not the image data obtained in each imaging is used for HDR composition processing. When a circle is attached, the image data obtained by the corresponding image capturing is used for the HDR synthesizing process. “NR combination 1” and “NR combination 2” in FIG. 6 indicate whether or not the image data obtained in each imaging is used for the NR combination process. It should be noted that “NR synthesis 1” indicates NR synthesis processing by synthesizing image data with exposure amounts of +1 steps. “NR combination 2” indicates that the NR combination process is performed by combining the image data having the exposure amount of −1 stage. When a circle is attached, the image data obtained by the corresponding imaging is used for the NR synthesis process. “Depth synthesis” indicates whether or not the image data obtained in each image pickup is used for the depth synthesis processing. When a circle is attached, the image data obtained by the corresponding image capturing is used for the depth stacking process. “Focus lens position” indicates the position of the focus lens in each image pickup.

In the example of FIG. 6, 2 frames are used for the HDR synthesizing process, 3 frames are used for the NR synthesizing process, and 3 frames are used for the depth synthesizing process. However, the image data obtained in a part of the frames is used for a plurality of processes of the HDR synthesizing process, the NR synthesizing process, and the depth synthesizing process. Therefore, the total number of frames may be 6 frames.

In the first frame shown in FIG. 6, the control unit 112 sets the imaging unit 102 so that it is underexposed by one stage. Then, the control unit 112 causes the image capturing unit 102 to capture an image. After the imaging, the control unit 112 temporarily records the image data of the first frame in the temporary recording unit 106.

In the second frame, the control unit 112 causes the image capturing unit 102 to capture an image under the same conditions as in the first frame. After imaging, the control unit 112 temporarily records the image data of the second frame in the temporary recording unit 106 separately from the image data of the first frame.

Also in the third frame, the control unit 112 causes the image capturing unit 102 to capture an image under the same conditions as in the first frame. After the imaging, the control unit 112 temporarily records the image data of the third frame in the temporary recording unit 106 separately from the image data of the first frame and the image data of the second frame.

In the fourth frame, the control unit 112 sets the image pickup unit 102 so that it is overexposed by one stage. Further, the control unit 112 starts driving the focus lens. Then, the control unit 112 causes the image capturing unit 102 to capture an image. After the image capturing, the control unit 112 temporarily records the image data of the fourth frame in the temporary recording unit 106 separately from the image data of the first frame to the third frame.

In the fifth frame, the control unit 112 takes an image with the imaging unit 102 under the same conditions as in the fourth frame (however, the focus lens is moving). After the image capturing, the control unit 112 temporarily records the image data of the fifth frame in the temporary recording unit 106 separately from the image data of the first frame to the fourth frame.

Even in the sixth frame, the control unit 112 causes the image capturing unit 102 to capture an image under the same conditions as in the fourth frame. After the image capturing, the control unit 112 temporarily records the image data of the sixth frame in the temporary recording unit 106 separately from the image data of the first frame to the fifth frame.

Here, returning to the description of FIG. In step S13 after the image capturing control for composition, the control unit 112 performs composition processing. After that, the process proceeds to step S14. The synthesis process will be described below.

As shown in FIG. 6, the control unit 112 performs HDR combination processing using the image data of the first frame and the image data of the fourth frame. In the HDR combining process, the control unit 112 replaces, for example, the whiteout area of the image data of the fourth frame with the image data of the first frame corresponding to the whiteout area. Thereby, high dynamic range image data is generated.

Further, as shown in FIG. 6, the control unit 112 performs the NR synthesis processing using the image data of the first frame, the second frame, and the third frame, and at the same time, the fourth frame, the fifth frame, and the sixth frame. NR synthesis processing is performed using the image data. In the NR synthesis process, the control unit 112 averages the image data of the first frame, the second frame, and the third frame, and averages the image data of the fourth frame, the fifth frame, and the sixth frame. As a result, noise-reduced bright and dark images are generated.

Further, as illustrated in FIG. 6, the control unit 112 performs the depth stacking process using the image data of the fourth frame, the fifth frame, and the sixth frame. In the depth composition processing, the control unit 112 performs composition processing for replacing the low-contrast area in the image data of the sixth frame with the image data of the high-contrast area in the fourth frame or the fifth frame corresponding to this area. To do. In this way, image data focused on each area is generated.

In the above-described example, the HDR synthesizing process, the NR synthesizing process, and the depth synthesizing process are individually performed to generate image data of 4 frames. Further, image data of four frames obtained as a result of the HDR synthesizing process, the NR synthesizing process, and the depth synthesizing process may be synthesized. In this case, the control unit 112 further performs NR synthesis processing using the noise-reduced bright image and dark image. That is, the control unit 112 combines the noise-reduced bright image and dark image by averaging. After that, the control unit 112 synthesizes the image data obtained by the HDR synthesis process with the black crushed or overexposed areas in the image data after the NR synthesis process, and reduces the contrast in the image data after the NR synthesis process. The image data obtained by the depth combination processing is combined.

Here, returning to the description of FIG. In step S14 after the combining process, the control unit 112 causes the display unit 104 to display an image based on the image data obtained as a result of the combining process. In step S14, the control unit 112 may display images based on the image data of four frames obtained as a result of the HDR synthesizing process, the NR synthesizing process, and the depth synthesizing process side by side, or display the image data of these four frames. An image based on the image data obtained by recombining may be displayed as a through image.

In step S15, the control unit 112 determines whether or not there is a still image shooting request. Still image capturing request, as in step S2, made by for example the operation of the operation or the touch panel of the release button by the user. If it is determined in step S15 that there is a still image shooting request, the process proceeds to step S16. If it is determined in step S15 that there is no still image capturing request, the process proceeds to step S6.

In step S16, the control unit 112 starts still image shooting. The image capturing operation of the image capturing unit 102 in still image capturing at this time is performed with the same setting as that in the composite image capturing control in step S12. Then, the control unit 112 synthesizes the image data obtained as a result of the image pickup operation in the same manner as in step S13, and then performs the necessary image processing to create a still image file. Then, the control unit 112 records the created still image file in the recording unit 108. After such a still image shooting operation, the process proceeds to step S6.

In step S17 after it is determined that the image combining process is not performed, the control unit 112 performs display imaging control. In step S18, the control unit 112 causes the display unit 104 to display an image based on the image data obtained as a result of the display imaging control in step S17. The processing of steps S17 and S18 is the same as the processing of steps S4 and S5.

In step S19, the control unit 112 determines whether or not the setting of the scene determination mode is the finish priority setting. The setting of the scene determination mode is set, for example, by the user operating the operation unit 110. If it is determined in step S19 that the setting of the scene determination mode is not the finish priority setting, the process proceeds to step S15. When it is determined in step S19 that the setting of the scene determination mode is the finish priority setting, the process proceeds to step S6. That is, when it is determined that the image synthesizing process is not performed, and it is determined that the setting of the scene determination mode is the finish priority setting, determination of whether or not there is a still image shooting request is not performed. Therefore, even if the user makes a still image shooting request, the request becomes invalid. Therefore, in the finish priority mode, shooting is not performed until the image combining process is performed.

Next, processing in the moving image shooting mode will be described. FIG. 7 is a flowchart showing the operation of the image pickup apparatus in the moving image shooting mode. The process of FIG. 7 is started, for example, when the shooting mode of the image pickup apparatus is switched to the moving image shooting mode and an instruction to start moving image recording is given. Note that in FIG. 7, description of processes similar to those in FIG. 2 will be appropriately omitted.

In step S51, the control unit 112 determines whether the moving image shooting mode is the scene determination mode. Similar to the still image shooting mode, the moving image shooting mode also has a normal mode and a scene determination mode. The setting as to whether the moving image shooting mode is the normal mode or the scene determination mode is performed, for example, by the user operating the operation unit 110. When it is determined in step S51 that the moving image shooting mode is not the scene determination mode, that is, the normal mode, the process proceeds to step S52. When it is determined in step S51 that the moving image shooting mode is the scene determination mode, the process proceeds to step S56.

In step S52, which is the process in the normal mode, the control unit 112 performs imaging control. For example, the control unit 112 causes the image capturing unit 102 to perform an image capturing operation by setting a diaphragm for movie shooting and setting an exposure time.

In step S53, the control unit 112 causes the display unit 104 to display an image based on the image data obtained as a result of the imaging control in step S52. As a result, a through image display is performed.

In step S54, the control unit 112 records a moving image. For example, the control unit 112 creates a moving image file after performing necessary image processing on the image data obtained as a result of the imaging control in step S52. Then, the control unit 112 records the created moving image file in the recording unit 108. When the moving image file has already been created, the control unit 112 adds the image-processed image data to the created moving image file. After recording such a moving image, the process proceeds to step S55.

In step S55, the control unit 112 determines whether to end the moving image recording. For example, when the user operates the operation unit 110 to instruct the end of the moving image recording, the shooting mode is switched to the still image shooting mode, or the power of the imaging device is turned off, it is determined that the moving image recording is ended. .. When it is determined in step S55 that the moving image recording is to be ended, the processing of FIG. 7 is ended. When it is determined in step S55 that the moving image recording is not to be ended, the process returns to step S51.

In step S56 which is the process of the scene determination mode, the control unit 112 performs the scene determination process. After the scene determination process, the process proceeds to step S57. The scene determination process in the moving image shooting mode may be basically the same as the scene determination process in the still image shooting mode. However, various thresholds used in the scene determination process may be different between the still image shooting mode and the moving image shooting mode. Further, in the scene determination processing in the moving image shooting mode, all the determinations of the scene determination processing in the still image shooting mode may not be performed. For example, the process of step S25 (determination process for performing the depth stacking process) may be omitted.

In step S57, the control unit 112 performs an image composition execution determination process. After the image compositing execution determination process, the process proceeds to step S58. The image combination execution determination process in the moving image shooting mode may be basically the same as the image combination execution determination process in the still image shooting mode.

In step S58, the control unit 112 determines from the result of the image synthesis execution determination process whether or not to perform the image synthesis process. If it is determined in step S58 that the image synthesizing process is to be performed, the process proceeds to step S59. When it is determined in step S58 that the image synthesizing process is not performed, the process proceeds to step S52.

In step S59, the control unit 112 determines the image capture frame rate. The imaging frame rate is set according to, for example, the frame rate of a moving image. The image capturing frame rate of a moving image may be set in consideration of image quality as in the case of a still image.

In step S60, the control unit 112 performs a frame number determination process. After the frame number determination process, the process proceeds to step S61. The frame number determination process in the moving image shooting mode may be basically the same as the frame number determination process in the still image shooting mode.

In step S61, the control unit 112 executes image capturing control for composition. Then, a process transfers to step S62. The image capturing control for combination in the moving image shooting mode may be performed as shown in FIG. 6 as in the still image shooting mode. However, the image capturing control for composition may be performed as shown in FIG. 8 below.

A modified example of the image capturing control for composition will be described with reference to FIG. Here, the quadrangle in FIG. 8 indicates the timing of each image pickup in the image pickup control for synthesis. “Exposure” in FIG. 8 indicates the amount of change in exposure at each imaging timing. -1 indicates that the exposure is one step below the proper exposure, and +1 indicates that the exposure is one step above the proper exposure. “Focus lens position” in FIG. 8 indicates the position of the focus lens in each image pickup. “NR” in FIG. 8 indicates the number of frames subjected to NR synthesis processing.

In the example of FIG. 6, the exposure amount changes discontinuously from -1 stage to +1 stage. On the other hand, in the example of FIG. 8, the exposure amount changes substantially continuously between −1 stage and +1 stage. For example, in FIG. 8, in imaging the first frame, the exposure amount is set to be under by -1 step. On the other hand, the exposure amount is set to 0 stage (appropriate exposure) in the imaging of the second frame. Then, the exposure amount is set to be over by +1 step in the imaging of the third frame. By continuously changing the exposure amount in this way, abrupt fluctuation of the exposure between frames can be suppressed. Therefore, it is possible to reduce the uncomfortable feeling of an image when recording a moving image or displaying a through image.

Further, in the example of FIG. 6, the NR combination processing is performed after the imaging of 6 frames. On the other hand, in the example of FIG. 8, the NR combination process is performed in parallel with the image pickup control. In this case, the control unit 112 synthesizes the image data obtained by the image pickup each time the image pickup of each frame is performed. Specifically, the control unit 112 temporarily records the image data of the second frame in the temporary recording unit 106, and then combines the image data of the second frame and the image data of the first frame. Further, the control unit 112 temporarily records the image data of the third frame in the temporary recording unit 106, and then combines the image data of the third frame and the combined image data obtained in the second frame. Further, the control unit 112 temporarily records the image data of the fourth frame in the temporary recording unit 106, and then combines the image data of the fourth frame and the combined image data obtained in the third frame. The same applies hereafter. The NR synthesizing process has a smaller effect on an image recorded as a moving image or displayed as a through image than other processes. Therefore, the NR synthesis process can be performed in parallel with the imaging control. Note that, in FIG. 6, the NR combination processing and the imaging control may be performed in parallel as in FIG.

Further, in the example of FIG. 8, the drive of the focus lens is started after the change of the exposure amount is completed. With such control, the exposure amount does not change while the focus lens is being driven. Therefore, it is possible to reduce the influence of the change of the imaging condition on the image recorded in the moving image or displayed as the through image. Further, in the example of FIG. 8, the number of frames from the start of the change of the exposure amount to the completion of the change is two, and the number of frames from the start of the drive of the focus lens to the completion of the drive is three. It is a frame. By thus reducing the driving amount of the focus lens for each frame, it is possible to reduce the influence on the image recorded as a moving image or displayed as a through image.

Here, in FIG. 8, the exposure amount is changed from -1 step to +1 step. Conversely, the exposure amount may be changed from +1 step to -1 step. This also applies to the case of FIG. Even in this case, it is desirable that the order of changing the imaging conditions is the order of the number of NR combination processes, the exposure amount, and the focus lens position. By setting this order, the influence on the image recorded in the moving image or displayed as the through image can be minimized.

Here, the description returns to FIG. 7. In step S62 after the image capturing control for composition, the control unit 112 performs composition processing. After that, the process proceeds to step S63. The combining process in the moving image shooting mode may be performed in the same manner as in the still image shooting mode.

In step S63, the control unit 112 causes the display unit 104 to display an image based on the image data obtained as a result of the combining process. After that, the process proceeds to step S54. This display may be performed similarly to the display of the result of the combining process in the still image shooting mode.

As described above, in the present embodiment, the combining process involving a plurality of times of imaging is performed when it can be considered that the target object is stopped. As a result, the composition is performed properly, and the image quality of the composite image is improved.

In addition, in the present embodiment, when the finish priority setting is set in the scene determination mode of the still image shooting mode, the user is not in a state in which it can be considered that the target object is still, that is, the user does not perform the synthesizing process. Still images are not recorded even if a shooting request is made. In this way, in the finish priority setting, the finish of the image is prioritized over the user's instruction. This also makes it possible to improve the image quality.

Although the present invention has been described based on the above-described embodiments, the present invention is not limited to the above-described embodiments, and it goes without saying that various modifications and applications are possible within the scope of the gist of the present invention. For example, in the above-described embodiment, three types of HDR combination processing, NR combination processing, and depth combination processing are exemplified as the combination processing. However, the combining process is not limited to these three processes. That is, the technique of the present embodiment can be applied to various combining processes involving a plurality of imaging operations.

Further, in the above-described embodiment, it is determined whether or not the target object is stopped only before the start of the image capturing control for synthesis. In practice, the object may move during the image capturing control for composition, and thus it may be determined whether the object is stopped during image capturing control for composition. In this case, when the target object cannot be regarded as stopped, the image capturing may be performed again under the same condition.

Further, in the above-described embodiment, the image pickup apparatus is exemplified by an image pickup apparatus for recording an image such as a digital camera. On the other hand, the technique of the present embodiment is applied to various image pickup apparatuses, and can also be applied to an image pickup apparatus that does not record an image. In this respect, the technique of this embodiment can be applied to an imaging device such as an endoscope device, a microscope device, or a monitoring device. For example, improvement in image quality is expected to improve visibility. For this reason, the technique of the present embodiment is also useful for those image pickup apparatuses that often check images. In particular, when performing close-up photography or microscope observation when the observation target has various reflectances and perspectives, image synthesis is performed according to the situation such as focus difference, exposure difference, and enlargement. Since the difference and the effect are easily obtained, the invention of the present application is often effective.

Further, in the above description of each operation flowchart, the operation is described by using “first”, “next”, etc. for convenience, but it does not mean that it is essential to perform the operation in this order. Absent.

Further, each processing according to the above-described embodiment can be stored as a program that can be executed by the control unit 112 that is a computer. In addition, it can be stored and distributed in a recording medium of an external storage device such as a magnetic disk, an optical disk, or a semiconductor memory. Then, the control unit 112 reads the program stored in the recording medium of the external storage device, and the operation is controlled by the read program, whereby the above-described processing can be executed.

Further, the embodiments described above include inventions at various stages, and various inventions can be extracted by appropriately combining a plurality of disclosed constituent elements. For example, even if some constituents are deleted from all constituents shown in the embodiment, the constituents are deleted when the above-mentioned problems can be solved and the above-described effects can be obtained. The configuration can also be extracted as an invention.

102 image pickup unit, 102a image pickup optical system, 102b image pickup device, 102c drive unit, 104 display unit, 106 temporary recording unit, 108 recording unit, 110 operation unit, 112 control unit, 1121 image pickup control unit, 1122 display control unit, 1123 image Processing unit, 1124 scene determination unit, 1125 motion detection unit.

Claims (5)

An image capturing unit that captures an image of an object to obtain image data of the object;
A motion detector that detects the motion of the object in the image data,
As a result of the detection of the movement of the object by the movement detection unit, when it is considered that the object has stopped, it is determined whether or not to perform a plurality of application processes involving an image synthesizing process of the plurality of image data. The total number of frames is calculated based on the number of frames required for each of the application processes determined to be performed, and the imaging is performed a plurality of times according to the total number of frames while changing the imaging conditions of the imaging unit. And a control unit that performs a process of synthesizing image data obtained by a plurality of times of imaging according to each of the applying processes ,
An imaging device , wherein the control unit calculates the total number of frames by reducing the number of frames that can be overlapped when there is a frame that can be overlapped among the plurality of application processes that are determined to be performed. .. The image pickup apparatus according to claim 1, wherein the control unit calculates the total number of frames so as to be limited to an upper limit according to a display frame rate. Further comprising an operation unit for receiving a photographing request from a user,
Wherein, when not considered when the object has stopped, the image pickup apparatus according to claim 1 or 2, characterized in that no processing is performed for the synthesized as the imaging request is received. Wherein the control unit, the imaging apparatus according to multiple any one of claims 1 to 3 the imaging condition is characterized by continuously changing during the imaging. Capturing an image of an object with an image capturing unit to obtain image data relating to the object,
Detecting movement of the object in the image data,
Result of the detection of movement of the object, when it is considered the object is stopped, and Rukoto to determine the presence or absence of implementation of the plurality of application processing involving image synthesis processing of the plurality of the image data,
Calculating the total number of frames based on the number of frames required for each of the application processes determined to be implemented;
Before SL performed a plurality of times of the image pickup in accordance with the total number of frames while changing the imaging condition of the imaging unit, and synthesizing the image data obtained by the imaging of multiple in accordance with the respective said application process,
Equipped with,
Calculating the total number of frames includes calculating the total number of frames by reducing the number of frames that can be overlapped when there is a frame that can be overlapped among the plurality of application processes that are determined to be performed. An image pickup method comprising :