JP2021083018A - Imaging device, imaging method, program, and recording medium - Google Patents

Abstract

To provide an imaging device that allows a user to confirm a composite image before main imaging when the composite image having a pseudo long exposure time is created.SOLUTION: An imaging device includes imaging means that performs provisional imaging and main imaging, compositing means, and display means, and the imaging means repeatedly captures images in the provisional imaging, and the compositing means generates provisional composite image by synthesizing the first number of images or the less number of images than the first number of the images captured by the provisional imaging by the imaging means, and the display means displays the provisional composite image when a predetermined condition is satisfied, and the imaging means displays the image captured by the provisional imaging when the predetermined condition is not satisfied, and the provisional composite image has a pseudo-exposure time longer than the exposure time of the image used to create the provisional composite image.SELECTED DRAWING: Figure 3

Description

The present invention relates to an imaging device, and more particularly to an imaging device that synthesizes images captured a plurality of times.

Conventionally, there is known an imaging method called long-second imaging that captures an image in a state where a moving subject has motion blur by setting the exposure time longer than the exposure time of the imaging device and expresses the movement of the subject. ing. In long-second imaging, since the exposure time is long, the exposure amount of the sensor may reach the saturation level and overexposure may occur.

In Patent Document 1, exposure and output of an image signal from a sensor are repeatedly imaged at regular time intervals, and the captured images are combined to generate a composite image that becomes an image equivalent to a predetermined exposure time. .. By using the technique described in Patent Document 1, it is possible to realize long-second imaging while suppressing overexposure.

Japanese Unexamined Patent Publication No. 2019-92018

When the technique described in Patent Document 1 is used, before the main imaging, a temporary image is taken on a live view screen or the like and composited, and when the composite image is displayed, the user confirms the composite image before the main imaging while checking the composite image. The exposure time and the like can be set.

However, when the composite image is displayed before the main imaging, if the user adjusts the composition or the like, blurring due to the user's action is reflected in the composite image. Therefore, when the user adjusts the imaging device, it becomes difficult for the user to check the composite image, and it becomes difficult to further adjust the composition and parameters.

The present invention has been made in view of the above problems, and the images are repeatedly imaged at regular time intervals, the captured images are combined, and when the user captures the images, the composition of the composite image can be easily confirmed. The purpose is to provide the device.

In order to solve the above problems, the present invention includes an imaging means for performing temporary imaging and main imaging, a synthesizing means, and a display means, and the imaging means repeatedly captures an image in the temporary imaging. , The synthesizing means synthesizes a first number of images captured by the tentative imaging means or a number less than the first number to generate a tentatively composited image. When the display means satisfies the predetermined conditions, the provisional composite image is displayed, and when the predetermined conditions are not satisfied, the imaging means displays the image captured by the provisional imaging, and the provisional image is displayed. The composite image provides an imaging apparatus characterized in that the pseudo-exposure time is longer than the exposure time of the image used for creating the tentative composite image.

According to the configuration of the present invention, when images are repeatedly imaged at regular time intervals and the captured images are combined, the user can easily check the degree of blurring and set the composition and parameters when performing the imaging. ..

It is a block diagram for demonstrating the structure of the image pickup apparatus in 1st Embodiment. It is a flowchart for demonstrating the operation of image pickup in 1st Embodiment. It is a figure for demonstrating the display in the display part in 1st Embodiment. It is a block diagram for demonstrating the structure of the image pickup apparatus in 2nd Embodiment. It is a flowchart for demonstrating the operation of image pickup in 2nd Embodiment. It is a figure for demonstrating the display in the display part in 2nd Embodiment.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

(First Embodiment)
FIG. 1 is a block diagram for explaining the structure of an imaging device as an imaging device according to the present embodiment.

The control unit 101 is, for example, a CPU, and reads a control program for each block included in the image pickup apparatus 100 from the ROM 102 described later, expands the control program into the RAM 103 described later, and executes the control program. As a result, the control unit 101 controls the operation of each block included in the image pickup apparatus 100. The ROM 102 is a non-volatile memory that can be electrically erased and recorded, and stores, in addition to the operation program of each block included in the image pickup apparatus 100, parameters and the like necessary for the operation of each block. The RAM 103 is a rewritable volatile memory, and is used for developing a program executed by the control unit 101 or the like, temporarily storing data generated by the operation of each block included in the image pickup apparatus 100, or the like. Further, as will be described later, the image signal of the read divided exposure image is temporarily recorded and used when generating a composite image.

The optical system 104 is composed of a lens group including a zoom lens and a focus lens, and forms a subject image on an imaging surface of an imaging unit 105 described later. The optical system 104 is provided with an operation means (not shown) for directly operating and controlling the zoom position and focus position of the optical system 104.

The image pickup unit 105 is, for example, an image pickup element such as a CCD or a CMOS sensor. The optical image formed on the image pickup surface of the image pickup unit 105 by the optical system 104 is photoelectrically converted, and the obtained analog image signal is A / D. Output to the conversion unit 106. The A / D conversion unit 106 converts the input analog image signal into digital image data. The digital image data output from the A / D conversion unit 106 is temporarily stored in the RAM 103. The image processing unit 107 applies various image processing such as white balance adjustment, color interpolation, and gamma processing to the image data stored in the RAM 103. Further, the image processing unit 107 also performs a process of reading the divided exposure image temporarily recorded in the RAM 103 and combining (for example, addition averaging) as an operation for long-time imaging using the divided exposure.

The recording unit 108 is a removable memory card or the like. The recording unit 108 records the image data processed by the image processing unit 107 as a recorded image via the RAM 103. The display unit 109 is a display device such as an LCD, and displays an image stored in the RAM 103 or an image recorded in the recording unit 108, displays an operation user interface for receiving an instruction from the user, and the like. The instruction input unit 110 includes the switch SW1 and the switch SW2, and receives instructions for starting and ending the imaging operation. Here, the switch SW1 may correspond to the half-pressing operation of the shutter button, and the switch SW2 may correspond to the full-pressing operation of the same shutter button. Further, the instruction input unit 110 includes a touch panel, a mouse, and the like, and inputs an instruction by the user.

FIG. 2 is a flowchart for explaining the operation of imaging in the present embodiment.

In step S201, the imaging unit 105 performs provisional imaging. The provisional imaging in step S201 is for adjusting the composition before the user performs the main imaging, and is temporarily recorded in the recording unit 108 or the like.

In step S202, the control unit 101 detects whether or not the switch SW1 is pressed. If the switch SW1 is pressed, the process proceeds to step S204, and if the switch SW1 is not pressed, the process proceeds to step S203. Unless the switch SW1 is pressed, the provisional imaging in step S201 and the live view display in step S203 are repeated.

In step S203, the display unit 109 displays the image captured by the imaging unit 105 in live view. When the image pickup unit 105 continuously captures images of a plurality of frames, in step S203, the frames captured by the image pickup unit 105 recorded immediately before the image pickup unit 105 displays are displayed.

In step S204, under the command of the control unit 101, the image processing unit 107 creates a tentative composite image using the image captured by the imaging unit 105 in the tentative imaging in step S201. The tentative composite image in step S204 is a composite image in which the exposure time is artificially lengthened from the image captured by the imaging unit 105 in tentative imaging in step S201, and the user composes the image in the main imaging in step S207 described later. It was tentatively created for the adjustment of.

When there are many images captured by the imaging unit 105 in the provisional imaging in step S201, it is not necessary for the control unit 101 to use all the images obtained by the provisional imaging in step S204 for the provisional composition. A predetermined number of tentatively captured images may be used for tentative composition according to the user's setting or the processing capacity of the control unit 101.

In step S205, the display unit 109 displays the composite image created in step S201. The user can make various settings for the main imaging while observing the image displayed in step S205.

In step S206, the control unit 101 detects whether or not the switch SW2 is pressed. If the switch SW2 is pressed, the process proceeds to step S207, and if the switch SW2 is not pressed, the process returns to step S201. When the switch SW2 is not pressed and the switch SW1 is being pressed, the imaging unit 105 continues to perform temporary imaging, and the control unit 101 continues to recreate the temporary composite image.

In step S207, the imaging unit 105 performs the main imaging. In this imaging, a plurality of images exposed at regular time intervals are captured under the control unit 101.

In step S208, under the command of the control unit 101, the image processing unit 107 creates a composite image using the image captured in the main imaging in step S207. As a method of creating a composite image, for example, addition averaging can be mentioned. The composite image created in step S208 has a pseudo exposure time equal to the sum of the exposure times of the images obtained in the main imaging in step S207, and an image having a sufficiently long exposure time is obtained without the sensor being saturated. Be done.

In the flow shown above, the user can confirm the brightness, focus position, composition, etc. of the composite image in advance by performing the provisional imaging and the generation of the provisional composite image before the main imaging. Further, when the user determines that the composite image is not a desired image, the orientation of the imaging device, imaging parameters, and the like can be adjusted before the main imaging starts.

However, if the user adjusts the imaging device before the main imaging, the following problems arise. If the switch SW2 is not pressed and the switch SW1 is being pressed before the main imaging in step S207, the imaging device repeats the steps from step S201 to step S205. Unless the control unit 101 selects an image to be used for the tentative composite image, the images newly obtained by the tentative imaging in step S201 are used one after another in step S204 to create the tentative composite image. For example, when the user changes the orientation of the imaging device, the angle of view of the image captured by the temporary imaging in step S201 immediately after the change is different from the image captured by the temporary imaging by the time when the orientation of the imaging device is changed. Come on. If images having different angles of view are combined in step S204 as they are, the quality of the combined image is significantly deteriorated, and the user who originally expected the combined image cannot confirm the combined image in advance.

In the present embodiment, in order to solve the above-mentioned problems, the control unit 101 controls the display of the display unit 109 as follows.

FIG. 3 is a diagram for explaining the display on the display unit 109 in the present embodiment. Since the image used for synthesizing the composite image and the tentative composite image was originally captured with the exposure time obtained by dividing the desired exposure time, it will be referred to as a "split exposure image" below. In FIG. 3, the case where the desired exposure time (exposure time of the composite image) set by the user is T and the exposure time is divided into four to capture the divided exposure image is shown as an example.

FIG. 3A is a diagram showing an operation of generating a divided exposure image. The control unit 101 repeatedly performs a read operation of the electric charge accumulated in the sensor of the image pickup unit 105 and a reset operation at a predetermined time interval (for example, 1/30 second). In the present embodiment, the divided exposure is performed at shorter time intervals than the exposure time T specified by the user. As shown by the solid line in FIG. 3A, the charge is read and reset before the charge accumulated in the imaging unit 105 reaches the saturation level. As described above, even when the exposure time T specified by the user is long, it is possible to prevent the exposure amount of the sensor from reaching the saturation level and causing overexposure.

R1, R2 ... Shown in FIG. 3B represent the divided exposure images captured and read out at times t1, t2 ... The captured divided exposure images shall be temporarily recorded in the RAM 103 for the number of images required to generate the composite image, and when a new divided exposure image is generated, the images shall be sequentially updated and replaced.

FIG. 3C shows a signal of a tentatively composited image obtained by synthesizing (adding and averaging) a predetermined number of divided exposure images. In FIG. 3, since the case where the exposure time T specified by the user is divided into four and the divided exposure is performed is described, the composite image is generated by synthesizing the four divided exposure images. For example, the image processing unit 107 generates a tentative composite image Rave4 by synthesizing (adding and averaging) four divided exposure images R1 to R4 at time t4. Further, the image processing unit 107 synthesizes (additionally averages) four divided exposure images R2 to R5 at time t5 to generate a composite image Rave5. The newest composite image of the generated composite image shall be temporarily recorded in the RAM 103, and when a new composite RAW image is generated, it shall be sequentially updated and replaced.

FIG. 3D shows a live view image displayed on the display unit 109. As described above, from the time t1 when the first divided exposure image is captured to the time t5 when the user presses the switch SW1, the divided exposure images R1, R2 ... Are sequentially displayed as live view images. .. When the user presses the switch SW1 (time t5), the provisional composite images Rave5, Rave6 ... Are sequentially displayed as live view images. In the live view before pressing SW1, the user can check the composition while looking at the divided exposure image with small motion blur, and when the SW1 is pressed after the composition is decided, the degree of motion blur in this imaging is confirmed. Because you can do things, the convenience of the user is improved.

Actually, the divided exposure image or the tentative composite image is read out from the RAM 103, various image processing is performed by the control unit 101 and the image processing unit 107, and then the image is displayed on the display unit 109. There will be a time delay before doing so. However, here, it is assumed that the delay time due to the processing of reading and displaying is small compared to the exposure time and can be ignored.

The method of generating the divided exposure image and the tentative composite image in the present embodiment has been described above.

In the present embodiment, the case of controlling the selection of the image to be displayed as the live view image according to the operation of the switch SW1 has been described. However, any method can be used as long as it is a method of controlling based on the information of the user's imaging operation. Such a method may be used. For example, the control unit 101 uses information on whether or not an operation for instructing an AF operation other than the shutter button or a manual focus (MF) adjustment operation is being performed, and when the operation is performed, the user May determine that is trying to confirm the in-focus state. At this time, the control unit 101 may control to display a divided exposure image having a small motion blur as a live view image. In addition, the user adjusts the color and brightness of the image when the operation is performed by using the information on whether or not the user has changed the imaging parameters other than the exposure time such as WB and exposure compensation. You may judge that you are trying to confirm.

Further, the control may be performed based on the information on whether or not the user is performing an operation for adjusting the exposure time for the main imaging. In this case, the temporary composite image is controlled to be displayed as a live view image while the exposure time adjustment operation is being performed. This enables the user to take an image while checking the degree of blurring in the main image pickup.

Further, in the present embodiment, the case where the divided exposure image is displayed without being combined before the user presses the switch SW1 has been described, but the present invention does not limit the display method of the image to this. .. For example, when updating the live view image at 1/30 second intervals, a split exposure image may be generated at 1/60 second intervals, and the two generated split exposure images may be combined and displayed. By doing so, it is possible to display the image at the time of live view more smoothly and at the same time prevent the blurring from becoming extremely large.

Further, when changing from the display of the divided exposure image to the display of the temporary composite image, the temporary composite image may be generated and the display may be changed while gradually increasing the number of composite images. By doing so, it is possible to control the switching between the display of the divided exposure image and the temporary composite image so as not to be conspicuous. That is, on the live view screen, the exposure time (total) of the images displayed while the user is adjusting the composition and the imaging parameters is controlled to be shorter than the exposure time at the time of the main imaging. Any method may be used.

Further, in the present embodiment, the case where the divided exposure images are combined by the addition averaging has been described, but the present invention does not limit the method for synthesizing the divided exposure images to this. For example, a tentative composite image may be generated by adding and synthesizing a predetermined number of divided exposure images and then multiplying the predetermined gain value to reduce the gain.

Further, whether or not to control the image display described in the present embodiment may be determined depending on the imaging situation. For example, if it can be determined from the set imaging mode and the amount of movement of the imaging device 100 that the imaging device 100 is fixed to the tripod, the tentative composite image is displayed even when SW1 is not pressed. You may control it.

According to the first embodiment, the control unit 101 automatically switches the image as the live view display and limits the image used for the tentative composite image so that the user can tentatively synthesize the image before the start of the main imaging. You can adjust the imaging settings while observing the image.

(Second embodiment)
The second embodiment will be described below with reference to the drawings. In the second embodiment, unlike the first embodiment, the image as the live view display is switched by using the posture information of the image pickup apparatus itself. Hereinafter, the second embodiment will be described with a focus on the differences from the first embodiment.

FIG. 4 is a block diagram for explaining the structure of the image pickup device as the image pickup device in the present embodiment.

The control unit 101 to the instruction input unit 110 in FIG. 4 is the same as the control unit 101 to the instruction input unit 110 of FIG. 1 in the first embodiment.

The angular velocity sensor 111 is, for example, a gyro sensor or the like, and periodically detects the angular velocity indicating the moving amount and the moving direction of the imaging device 100, converts it into an electric signal, and outputs it to the control unit 101 or the image processing unit 107.

FIG. 5 is a flowchart for explaining the operation of imaging in the present embodiment. Step S501 and steps S503 to S508 in FIG. 5 are the same as steps S201 and S203 to S208 of the first embodiment.

In the present embodiment, in step S502, the angular velocity sensor detects the amount of movement of the posture of the image pickup apparatus 100, and the control unit 101 compares the amount of movement with a predetermined threshold value. If the amount of movement is smaller than the threshold value, the control unit 101 determines that the image pickup apparatus 100 has less blurring and the divided exposure image may be displayed in live view. On the contrary, when the movement amount is equal to or more than the threshold value, the control unit 101 determines that the image pickup apparatus 100 has a large blur and may display the tentative composite image instead of the divided exposure image in the live view.

FIG. 6 is a diagram for explaining the display on the display unit 109 in the present embodiment.

The figure shows an example of capturing an optical image.

FIG. 6A is a diagram showing an operation of generating a divided exposure image. The contents shown in FIG. 6 (a) are the same as those in FIG. 3 (a) of the first embodiment.

R1, R2 ... Shown in FIG. 6B represent the divided exposure images captured and read out at time t1, t2 ... The captured divided exposure images shall be temporarily recorded in the RAM 103 for the number of images required to generate the composite image, and when a new divided exposure image is generated, the images shall be sequentially updated and replaced.

FIG. 6C shows a signal of a tentatively combined image in which a predetermined number of divided exposure images are combined (added and averaged). In FIG. 6, since the case where the exposure time T specified by the user is divided into four and the divided exposure is performed is described, the composite image is generated by synthesizing the four divided exposure images. The newest composite image of the generated composite image shall be temporarily recorded in the RAM 103, and when a new composite RAW image is generated, it shall be sequentially updated and replaced.

FIG. 6D shows a live view image displayed on the display unit 109.

FIG. 6E shows the relationship between the amount of movement of the image pickup apparatus and time. FIG. 6E shows that the movement amount of the imaging device is larger than the threshold value TH before the time t5, and the movement amount of the imaging device is smaller than the threshold value TH after the time t5.

From the time t1 when the first divided exposure image is captured to the time t5, the divided exposure images R1, R2 ... Are sequentially displayed as live view images. From time t5, the provisional composite images Rave5, Rave6 ... Are sequentially displayed as live view images.

In the present embodiment, the case where the magnitude of the movement of the imaging device is determined based on the detection result of the angular velocity sensor 111 has been described. However, in the present embodiment, the method of determining the magnitude of the movement of the imaging device is described. It is not limited to. For example, the divided exposure images that are repeatedly captured may be compared and the judgment may be made based on the difference value between the images and the size of the motion vector.

Further, instead of the movement of the imaging device, the live view display may be switched by determining a change in the angle of view imaged by the imaging unit 105. For example, the control unit 101 compares the divided exposure images that are repeatedly captured, and determines the change in the angle of view that is being captured based on the difference value between the images and the magnitude of the motion vector. Further, the control unit 101 takes an image based on information on whether or not the user controls the optical system by using an operation means such as the instruction input unit 110 to change the zoom position of the zoom lens. It may be determined whether or not the existing angle of view is changed.

Further, whether or not to control the image display described in the present embodiment may be controlled according to the imaging situation. For example, if it can be determined from the set imaging mode and the movement of the imaging device 100 that the user is taking a panning shot, the provisional composite image will be live even if the angle of view being captured is changed. Control to display as a view image. By doing so, the user can take an image while checking the degree of blurring in panning, which enhances convenience.

According to the second embodiment, the control unit 101 can automatically switch the image as the live view display based on the movement of the image pickup apparatus or the angle of view being imaged.

(Other embodiments)
Although the embodiments of the present invention have been described above, the above-described embodiments are merely examples and are not intended to limit the scope of the invention. The embodiments described above can be implemented in various forms, and various omissions, replacements, and changes can be made without departing from the gist of the invention.

In the present invention, a program that realizes one or more functions of the above-described embodiment is supplied to a system or device via a network or a recording medium, and one or more processors in the computer of the system or device program. It can also be realized by the process of reading and operating. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100 Imaging device 101 Control unit 102 ROM
103 RAM
104 Optical system 105 Imaging unit 106 A / D conversion unit 107 Image processing unit 108 Recording unit 109 Display unit 110 Instruction input unit 111 Angular velocity sensor

Claims (13)

An imaging means for performing temporary imaging and main imaging,
Synthetic means and
Has a display means,
The imaging means repeatedly captures an image in the provisional imaging, and the image is captured.
The synthesizing means synthesizes a first number or a number of images smaller than the first number among the images captured by the tentative imaging by the tentative imaging means to generate a tentatively composited image.
When the display means satisfies the predetermined conditions, the provisional composite image is displayed, and when the predetermined conditions are not satisfied, the imaging means displays the image captured by the provisional imaging.
The tentative composite image is an imaging device having a pseudo-exposure time longer than the exposure time of an image used for creating the tentative composite image. The imaging means captures a plurality of images in the main imaging,
The imaging device according to claim 1, wherein the compositing means creates a composite image using the plurality of images captured by the imaging means in the main imaging. The imaging apparatus according to claim 2, wherein the composite image has a pseudo-exposure time longer than the exposure time of the image used for creating the composite image. Any one of claims 1 to 3, wherein the synthesizing means creates the tentatively synthesized image by using the image captured by the tentative imaging means, which is later captured. The image pickup apparatus according to item 1. The imaging device according to any one of claims 1 to 4, wherein the predetermined condition is an instruction given by a user. The imaging device according to any one of claims 1 to 4, wherein the predetermined condition is a half-press of the shutter button. The imaging device according to any one of claims 1 to 4, wherein the predetermined condition is that the angle of view imaged by the imaging means is not changed. The claim is characterized in that the synthesis means detects whether or not there is a change in the angle of view captured by the imaging means by detecting a difference or a motion vector from the image captured by the imaging means. Item 7. The imaging apparatus according to item 7. It has a detecting means for detecting the amount of movement of the imaging device, and has
The imaging device according to any one of claims 1 to 4, wherein the predetermined condition is a movement amount detected by the detection means smaller than a predetermined threshold value. When the imaging means takes a panning shot,
The imaging device according to any one of claims 1 to 9, wherein the display means displays the tentative composite image regardless of the predetermined conditions. An imaging step for performing temporary imaging and main imaging,
Synthesis steps and
With display steps,
In the imaging step, the image is repeatedly imaged in the temporary imaging, and the image is captured.
In the synthesizing step, among the images captured by the tentative imaging in the imaging step, a first number or a number less than the first number is combined to generate a tentatively composited image.
In the display step, the tentative composite image is displayed when the predetermined conditions are satisfied, and when the predetermined conditions are not satisfied, the image captured by the tentative imaging in the imaging step is displayed.
The tentative composite image is an imaging method characterized in that the pseudo-exposure time is longer than the exposure time of the image used for creating the tentative composite image. A computer program that causes a computer to operate the imaging method according to claim 11. A computer-readable recording medium for recording the program according to claim 12.

Images