JP7545238B2 - Imaging device and control method thereof - Google Patents

Description

The present invention relates to an imaging device and a control method thereof.

There are known imaging devices that have a live view (hereinafter referred to as LV) function that captures images continuously at regular intervals and displays the captured images (live view images). A user can check the captured still image content using such LV images. In addition, a user can take continuous still images while checking a moving subject using the LV images.

When an imaging device has a single imaging sensor, the imaging sensor needs to output both image data for recording still images and image data for LV display, but it cannot drive for LV display while driving for still images. If still images are continuously shot (continuous shooting) while LV is displayed, the still image drive time may overlap with the display drive time depending on the timing of the release and the continuous shooting interval. Since image data output for LV display cannot be obtained while the imaging sensor is driving for still images, the LV display is either blacked out or the output of the previous frame is displayed multiple times as LV display (the same frame is displayed continuously).

When the same frame is displayed continuously, a phenomenon occurs in which the subject's movement appears unnatural in the LV display, as if it were accelerating. In addition, when a blackout is performed, the unnatural movement disappears, but there is a problem with visibility, such as eye fatigue of the photographer due to the large change in brightness. Patent Document 1 describes a configuration in which, when the LV display is blacked out during continuous shooting of still images, the brightness of the LV image display displayed between blackouts is brightened. In Patent Document 1, on the premise that a black image is displayed outside the LV display period during continuous shooting (blackout), the LV display brightness is brightened as the ratio of the black image display period to the LV display period increases. Patent Document 1 claims that such a configuration can prevent the display from appearing dark due to blackout during still image capture.

JP 2018-006827 A

However, Patent Document 1 is premised on black image display, and by displaying the luminance of the LV display brighter, the luminance difference with the black image display becomes large, so the deterioration in visibility due to the luminance difference is not improved. Furthermore, if the exposure time of each still image becomes longer, the black image display period or the period during which the LV image is not updated becomes longer, so visibility is further deteriorated in LV displays with blackouts and continuous display of the same frame.

The present invention provides a technology that improves the visibility of the LV display while capturing still images, etc.

An imaging device according to an aspect of the present invention has the following arrangement.
An imaging device that captures a live view image and displays the live view on a display unit,
a determination unit that determines whether or not exposure in capturing one still image is to be divided based on an exposure time for capturing the one still image and an imaging cycle of the live view image;
a division means for dividing an exposure time for capturing the single still image when the determination means determines that the exposure should be divided, and acquiring a plurality of still images to be combined into the single still image;
The display control means is configured to perform live view display during a period in which a live view image could not be captured due to capturing of the one still image by sequentially using one or more of the plurality of still images as a live view image.

The present invention improves the visibility of the LV display while capturing still images, etc.

1 is a block diagram showing an example of the configuration of an imaging apparatus according to a first embodiment. 5 is a timing chart relating to LV display during continuous still image shooting according to the first embodiment. 5 is a timing chart relating to LV display during continuous still image shooting according to the first embodiment. 5 is a timing chart relating to LV display during continuous still image shooting according to the first embodiment. 5 is a timing chart relating to LV display during continuous still image shooting according to the first embodiment. 5 is a timing chart relating to LV display during continuous still image shooting according to the first embodiment. 4 is a flowchart showing an example of processing according to the first embodiment. 4 is a flowchart showing an example of processing according to the first embodiment. FIG. 13 is a diagram for explaining the occurrence of overlapping read periods. 11A and 11B are diagrams for explaining a process for preventing overlapping of read periods. 10 is a flowchart showing an example of processing according to the second embodiment. 10 is a flowchart showing an example of processing according to the second embodiment. 10 is a flowchart showing an example of processing according to the second embodiment. 10 is a flowchart showing an example of processing according to the second embodiment.

The following embodiments are described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe multiple features, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicate explanations are omitted.

The image processing device will be described below as part of an imaging device. Also, a digital camera will be described below as an example of an imaging device, but the imaging device is not limited to this. For example, imaging devices such as digital video cameras, smartphones, mobile phones with cameras, and car-mounted cameras can also be used.

First Embodiment
1 is a block diagram showing an example of the configuration of a digital camera as an imaging device according to the first embodiment. The digital camera has an imaging unit 100, an image processing unit 101, a display unit 102, a display device 103, a control unit 104, a data transfer unit 105, a bus 106, a temporary recording unit 107, and a recording unit 108. The digital camera also has a compression/decompression unit that compresses image data or decompresses compressed data, and an external recording unit that records compressed data on an external medium (all not shown). The compressed data is image data compressed in a format such as JPEG or MPEG. The digital camera has a function of capturing a live view image and displaying the live view. Hereinafter, live view may also be abbreviated as LV.

The imaging unit 100 has an imaging sensor such as a CCD or CMOS sensor that converts the received subject image into an electrical signal to create image data, and an AD converter that converts the analog image signal obtained by the imaging sensor into a digital signal and outputs it as image data. The image data output from the imaging unit 100 is written to the temporary recording unit 107 via the data transfer unit 105.

The image processing unit 101 executes a number of processes, such as pixel correction, black level correction, shading correction, scratch correction, white balance adjustment, magnification chromatic aberration correction, gamma correction, luminance/color generation processing, geometric transformation, noise reduction, and enlargement/reduction. The image processing unit 101 performs appropriate image processing on image data by using one or more of these processes. In addition, in order to appropriately perform image processing, the image processing unit 101 acquires evaluation values such as average luminance, histogram, and amount of motion for each divided area of the image or for the entire image. In addition, the image processing unit 101 detects the subject area and then acquires evaluation values such as average luminance, histogram, and amount of motion for the subject area. These evaluation values are used, for example, for white balance adjustment, magnification chromatic aberration correction, gamma correction, luminance/color generation processing, and the like. In addition, the image processing unit 101 is connected to the data transfer unit 105, and writes the processing results of image processing performed on image data input from the data transfer unit 105 to the temporary recording unit 107 via the data transfer unit 105.

The display unit 102 controls the display of the display device 103. The display unit 102 receives image data processed by the image processing unit 101 from the temporary recording unit 107 via the data transfer unit 105, processes the image as necessary, generates a display image, and transfers it to the display device 103. The processes performed by the display unit 102 include, for example, brightness and color adjustment, adding pixels of a fixed color (e.g., blackened) around the image to fit the size of the display device 103, embedding the image capture time in the display data, superimposing an OSD (On-Screen Display) image, and format conversion to match the display device 103. The display device 103 is, for example, a liquid crystal display (LCD) and displays the image from the display unit 102. The display device 103 can be, for example, a display panel on the back of a digital camera or an electronic viewfinder (EVF). The display panel and electronic viewfinder display LV.

The control unit 104 includes one or more processors that control the operation of the digital camera, and issues various instructions to each functional block that constitutes the digital camera, and executes various controls and processes. For example, the control unit 104 controls the image processing unit 101, display unit 102, data transfer unit 105, temporary recording unit 107, and recording unit 108, which are connected via a bus 106. The control unit 104 also controls the shooting operation (exposure, readout) by the imaging unit 100. The one or more processors execute a program recorded in the recording unit 108 to realize each process of this embodiment. Note that at least some of the functions of the image processing unit 101, display unit 102, data transfer unit 105, etc. may be realized by the control unit 104 (processor).

The data transfer unit 105 includes multiple Direct Memory Access controllers (WRDMAC and RDDMAC) that transfer data. Image data is temporarily stored in the temporary storage unit 107 from the image processing unit 101 via the bus 106 by the WRDMAC. The image data stored in the temporary storage unit 107 is output to the image processing unit 101 and the display unit 102 connected to the data transfer unit 105 via the bus 106 by the RDDMAC. The bus 106 includes a system bus and a data bus, each of which has an independent bus configuration.

The temporary recording unit 107 includes a memory control unit and a memory (not shown). The memory control unit writes data to the memory and reads data from the memory in response to instructions from the control unit 104 or the data transfer unit 105. The memory has a sufficient storage capacity to store a predetermined number of still images, a predetermined duration of moving images, audio data, constants for the operation of the control unit 104, programs, etc., and is composed of, for example, a DRAM. The temporary recording unit 107 may include multiple memories.

The recording unit 108 is composed of a non-volatile memory control unit and a non-volatile memory (not shown). The non-volatile memory control unit writes data to the non-volatile memory and reads data from the non-volatile memory in response to instructions from the control unit 104. The non-volatile memory is an electrically erasable and recordable memory, such as an EEPROM. Constants, programs, etc. for the operation of the control unit 104 are stored in the non-volatile memory.

Figures 2A-2B and 3A-3C are timing charts relating to the LV display during continuous still image shooting with a constant release time and an adjusted continuous shooting interval. The vertical direction indicates timing signals or processing, and the horizontal direction indicates time. Figures 2A-2B and 3A-3C show two types of timing charts with different exposure times for capturing still images (hereinafter also referred to as still image capture). Note that the timing charts of Figures 2A-2B and 3A-3C are intended to show a schematic representation of the timing of each operation for the purpose of understanding the present invention, and do not strictly represent the time required for each process.

First, we will explain Figures 2A and 2B. The imaging display V sync 211 is the readout timing of the LV image from the imaging unit 100, and indicates the timing of the vertical synchronization signal for imaging the LV image (hereinafter also referred to as LV imaging). The imaging display V sync 211 is a timing signal in which, for example, 60 frames of images are output per second, and indicates the imaging cycle of the LV image.

Still image synchronization 212 indicates the timing of reading out a still image from the imaging unit 100, and is a synchronization signal that occurs at imaging intervals T1 during continuous shooting of still images after the first still image is captured. Timing 200 indicates the timing when the user presses the shutter button, and the still image synchronization signal is generated after release time T0. Release time T0 may be a fixed time or a variable time. However, if release time T0 is a fixed time, there is a high possibility that still image shooting will overlap with LV shooting. In contrast, if release time T0 is variable, it is possible to configure the start timing of still image shooting to be shifted to a time that does not overlap with LV shooting, for example.

The sensor output 213 indicates an exposure time TL for LV image capture, an exposure time T2 for still image capture, timings for starting and ending readout of an image signal from the imaging unit 100, and a sensor switching time. Here, the sensor switching time is the time required to switch between capturing an LV image and capturing a still image. In the sensor output 213, the imaging timing of the LV image is shown in display 1 to display 8, and the imaging timing of the still image is shown in still image 1 to still image 2. Display 1 to display 8 of the sensor output 213 include an exposure time TL and a readout time _tL in capturing an LV image. Still image 1 to still image 2 of the sensor output 213 include an exposure time T2 and a readout time _t2 for capturing a still image. Switching time 201 is the switching time of the imaging unit 100 from capturing an LV image to capturing a still image, and switching time 202 is the switching time of the imaging unit 100 from capturing a still image to capturing an LV image. In general, the image capturing time (exposure time+readout time) differs between Display 1 to Display 8 and Still Image 1 to Still Image 2. This is because the number of pixels and the exposure time differ between the LV image and the still image.

Image processing 214 indicates the timing of the start and end of image processing by the image processing unit 101. When the reading of the top line of the image sensor of the imaging unit 100 starts, the image processing unit 101 can start image processing of that frame even if the reading of all lines has not been completed (chasing processing). The timing indicated by display 1 to display 8 and still image 1 to still image 2 in image processing 214 includes the following processing. That is, it includes recording the image read from the imaging unit 100 to the temporary recording unit 107 via the data transfer unit 105, the image processing unit 101 reading and processing this image via the data transfer unit 105, and recording the processed image via the data transfer unit 105 to the temporary recording unit 107. In addition, the timing of image processing (processing 203) of display 1 to display 8 and still image 1 and still image 2 indicates that the image processing unit 101 starts processing before the entire screen of image data from the imaging unit 100 is recorded in the temporary recording unit 107 (chasing processing). The data transfer unit 105 controls the image processing unit 101 to read the image data written by the imaging unit 100. In other words, the image processing unit 101 is controlled so as not to read the image data before the imaging unit 100 writes it to the temporary storage unit 107. Since a still image has a larger number of pixels than an LV image, the time required for image processing is longer for a still image than for an LV image.

The display V sync 215 indicates the timing of the vertical synchronization signal for LV display on the display device 103, and is a timing signal for displaying, for example, 60 frames of images per second. The display V sync 215 occurs with a certain time delay from the imaging display V sync 211, which is the imaging cycle of the LV image. When the image processing of a certain frame is started (for example, from the top line of the imaging sensor), the display unit 102 can display using that frame even if the image processing of that frame is not completed (chasing process). The display V sync 215 is set to a timing shifted from the imaging display V sync 211 by the shortest time required from when the image is output from the imaging unit 100 to when it is displayed on the display device 103. The LV display 216 indicates the timing at which the display unit 102 displays the image processed by the image processing unit 101 on the display device 103. In the example of Figures 2A and 2B, the timing of capturing the still images (still image 1, still image 2) does not overlap with the timing of capturing the LV images, so LV images can be captured and displayed continuously from display 1 to display 8.

Next, we will explain Figures 3A to 3B. Figures 2A to 2B show a state in which the exposure time T2 of still images is shorter than the period of LV imaging (imaging display V synchronization) and does not affect LV imaging and LV display. Figures 3A to 3B show a time chart in which the exposure time T2 of still image imaging is longer than the period of imaging display V synchronization and still image imaging and LV imaging overlap.

The vertical items, imaging display V sync 211, still image sync 212, sensor output 213, image processing 214, display V sync 215, and LV display 216, are as described in Figs. 2A-2B. However, still image synthesis 351, which synthesizes multiple still images obtained by split exposure, described later, is added. Also, while the exposure time T2 of the still image is within one period of imaging display V sync (display V sync) in Figs. 2A-2B, the exposure time of the still image is long enough to span three periods of imaging display V sync in Figs. 3A-3B. Therefore, in Figs. 3A-3B, split exposure is performed to split the exposure time of one still image into multiple parts in order to obtain a still image with a long exposure time. In this embodiment, the exposure time of one still image is split in synchronization with imaging display V sync 211. One still image 1 is generated by combining (still image combination 351) multiple still images (still images 1-1 to 1-3) obtained by such divided exposure.

As shown in the timing of the sensor output 213 in FIG. 3B, the imaging unit 100 divides the exposure time T2 of the still image into exposure times T3, T4, and T5 and reads out from the imaging sensor (divided exposure). Note that in each exposure of the divided exposure, a readout time (TR: time from start to completion of still image readout) of the still image from the imaging sensor is required. Also, in FIG. 3B, the period required from completion of exposure for the previous still image capture (start of still image readout) to start of exposure for the next still image capture is set to _t3 = _t4 = _t5 . These periods are, for example, readout periods of image data of one line of the imaging sensor, and exposure for the next still image is started sequentially from the line where readout is completed. In the example of FIG. 3B, the imaging time required to obtain a still image of exposure time T2 (=T3+T4+T5) is T3+ _t3 +T4+ _t4 +T5+TR. If the image readout time in each divided exposure is equal, and this is taken as t (=t ₃ =t ₄ =t ₅ ), then the imaging time required to obtain a still image with exposure time T2 is T3+T4+T5+2t+TR.

In the example of FIG. 3B, the timing at which the exposure for LV imaging is completed is set as the split timing of exposure for still image imaging, based on the signal of the imaging display V synchronization 211. However, the method of determining the split timing of exposure for still image imaging is not limited to this. For example, the split timing of exposure for still image imaging may be determined based on the exposure start timing of LV imaging. By performing split exposure using the split timing determined based on the imaging display V synchronization 211 in this way, it is possible to match the delay from normal LV imaging to LV display with the display delay in the case of LV display using split exposure of a still image. In other words, it is possible to match the timing at which the LV image captured according to the imaging cycle of the imaging display V synchronization 211 is displayed with the timing at which each of the multiple still images obtained by splitting the exposure of one still image is displayed.

In image processing 214, the image processing unit 101 performs image processing on still images 1-1 (processing 303), 1-2 (processing 304), and 1-3 (processing 305) that have been subjected to divided readout. At this time, the image processing unit 101 adjusts the brightness of the multiple still images obtained by divided exposure based on the imaging conditions of the live view image. This is to use the still images for live view display. For example, the image processing unit 101 performs brightness adjustment so that the display brightness of still images 1-1 and 1-2 matches that of the live view image based on each exposure time T3, T4, and T5 and the imaging conditions during live view imaging. For example, assuming that the imaging conditions other than the exposure time are the same for LV imaging and still image imaging, and the LV exposure time is TL, the gain for brightness adjustment to generate an LV display image from each of still images 1-1 to 1-3 is expressed by the following formulas (1) to (3), respectively.

Gain1=TL/T3...(1)
Gain2=TL/T4...(2)
Gain3=TL/T5...(3)
In the above example of brightness adjustment, the brightness of the multiple still images obtained by divided exposure is adjusted according to the divided exposure times (T3 to T5) and the brightness of the live view image by using the formulas (1) to (3). The brightness is adjusted according to the ratio with the exposure time (TL) of the image capture. The brightness is adjusted by the following formula using the Gain of the corresponding exposure time among Gains 1 to 3 calculated by formulas (1) to (3): For example, in the case of adjusting the luminance of the still image 1-1, the luminance-adjusted still image is obtained by calculating the formula (4) using Gain1 as the Gain.

Still image after brightness adjustment = Still image captured by still image capture × Gain ... (4)
In the LV display 216, LV imaging cannot be performed for the frame 307 next to display 4 (frame 306) and the frame 308 next thereto because still image imaging is performed. Therefore, still images 1-1 and 1-2 are displayed as LV images for frames 307 and 308. In the illustrated example, since LV imaging of display 7 was performed, display 7 is used for LV display in the frame next to frame 308, and still image 1-3 is not used for LV display. If the exposure time of the still image is longer than that in the illustrated example and exposure time T5 (and/or its readout time) or switching time 202 overlaps with LV imaging of display 7, LV imaging of display 7 cannot be performed. In that case, still image 1-3 (305) is used as the LV image instead of display 7.

In still image synthesis 351, still images 1-1, 1-2, and 1-3, which are the divided exposed sensor outputs, are synthesized to generate a still image corresponding to exposure time T2. In the illustrated example, still images 1-1 and 1-2 are added together (synthesis process 309) in parallel with still image 1-2 being output from the sensor, to obtain a synthesized still image. Furthermore, in parallel with still image 1-3 being output from the sensor, the still image synthesized in synthesis process 309 and still image 1-3 are added together (synthesis process 310) to generate a synthesized still image. The still images thus generated by synthesis processes 309 and 310 are acquired as still images equivalent to exposure time T2 (=T3+T4+T5).

In FIG. 3B, the timing of splitting exposure in still image capture is determined in consideration of the timing of LV capture (completion of exposure time of LV capture), but this is not limited to this. For example, as shown in FIG. 3C, the timing of splitting exposure in still image capture may be determined according to the period Tv of the capture display V synchronization 211. In FIG. 3C, the exposure time T2 of the still image is divided by the period Tv of the capture display V synchronization, and T2=Tv+Tv+Tr. The final period Tr is the remaining exposure time shorter than Tv obtained as a result of division by Tv. Since split exposure is performed by the period Tv of the capture display V synchronization 211, the capture of still image 1-1 does not arrive in time for the display of frame 306' of the LV display 216. In this case, the frame 306 (display 4) immediately before it may be repeatedly displayed in frame 306', or frame 306' may be blacked out.

In addition, in Figures 2A-2B and 3A-3C, the exposure start timing for still image capture relative to timing 200 when the user presses the shutter button is set to after the exposure of the LV capture following timing 200 is completed (release time T0). However, the exposure start timing for still image capture is not limited to this, and for example, the still image exposure start timing may be the closest future capture display V synchronization timing relative to timing 200. In this case, since the LV capture at the still image exposure start timing is replaced by still image capture, at least one frame of the image obtained by still image capture will be displayed on the LV regardless of the length of the exposure time of the still image.

The processing procedure of the control unit 104 for realizing the operations described above with reference to Figures 2A to 3C will be described with reference to the flowcharts of Figures 4 and 5. First, the processing for realizing the operations described with reference to Figures 2A to 2B and Figures 3A and 3C will be described with reference to Figure 4. Figure 4 is a flowchart showing the divided exposure processing of a still image in LV display still image capture by the control unit 104. In Figure 4, the control unit 104 determines whether to perform divided exposure based on a certain fixed threshold value (Th), and performs divided exposure with capture display V synchronization (Tv).

In step S401, the control unit 104 sets the exposure time Ts of a still image as the initial value of the remaining exposure time Tr. The exposure time Ts of a still image is the time of the shutter speed determined by the user's specification or exposure control by the control unit 104. In FIG. 3C, Ts = Tv + Tv + Tr.

In step S402, the control unit 104 compares the remaining exposure time Tr with a fixed, predetermined threshold value Th. If the comparison shows that Tr is greater than Th (YES in S402), the process proceeds to step S403, and if Tr is equal to or less than Th (NO in S402), the process proceeds to step S406. Note that the threshold value Th is a fixed value that represents the maximum exposure time at which a still image can be captured without interfering with LV imaging. For example, the time required for the imaging unit 100 to switch between LV imaging and still image imaging (e.g., time 201 and time 202) is the time removed from the imaging display V synchronization cycle Tv, and is less than the imaging display V synchronization cycle Tv (Th<Tv).

In step S403, the control unit 104 performs split exposure by capturing a still image with the shorter of Tv and Tr as the exposure time. Here, Tv is the time of one period of the image display V synchronization 211. The shorter of Tv and Tr is used because there are cases where Th<Tr<Tv. The captured split exposure image is temporarily recorded in the temporary recording unit 107 via the data transfer unit 105. In step S404, the image processing unit 101 adjusts the brightness of the image captured by split exposure so as to match it with the normal LV captured image, and the display unit 102 displays the brightness-adjusted image on the display device 103. Note that the brightness adjustment can be performed using the method described in Figures 3A to 3B (Equations (1) to (4)), for example. In step S405, the control unit 104 updates the remaining exposure time Tr to a value obtained by subtracting Tv from the current Tr (Tr = Tr - Tv). After that, the process returns to step S402. In this way, steps S402 to S405 are repeated until the remaining exposure time Tr becomes equal to or less than the threshold value Th.

In step S406, the control unit 104 performs split exposure and captures a still image for the remaining exposure time Tr. However, if Tr<0 at this point (if Tv>Tr>Th, Tr<0 in step S405), the process ends. In step S407, the control unit 104 determines whether the split exposure performed in step S406 or the switching time of the image capture unit 100 overlaps with the start timing of LV imaging. If the split exposure overlaps with the start timing of LV imaging, the image processing unit 101 adjusts the brightness of the still image obtained by the split exposure and capture so as to match it with the normal LV captured image, and the display unit 102 displays the brightness-adjusted still image on the display device 103. Note that the brightness adjustment is as described above with reference to Figures 3A to 3B and formulas (1) to (4).

When split exposure of still images is performed, in step S408, the image processing unit 101 synthesizes multiple split exposure images obtained by split exposure to generate one still image corresponding to exposure time T2. In the example of FIG. 3C, two still images with exposure time Tv and a still image with exposure time Tr (= T2 - Tv - Tv) are synthesized. The generation method can be, for example, the method described in FIG. 3A to FIG. 3B. Note that in the flowchart, the synthesis of still images is shown collectively in step S408, but as shown in FIG. 3C, still images obtained by split exposure may be synthesized sequentially. Also, if the still image exposure time Ts is shorter than Th, the processing from S406 onwards is immediately executed, and the control unit 104 controls to perform operations according to the timing described in FIG. 2A to FIG. 2B, for example.

As a result, the LV display can continue to display updated images continuously without causing blackouts or interrupted updates due to the continuous display of the same image.

Figure 5 shows an example of a processing flow for determining whether to perform split exposure based on the period until the timing synchronized with the imaging display V synchronization. In Figure 5, it is determined whether to perform split exposure based on the period until the timing when LV imaging ends, and the exposure time for still image imaging is divided by the period until the timing when LV imaging ends. Through the processing shown in Figure 5, the control unit 104 controls to perform operations according to the timing shown in Figures 2A to 2B and Figures 3A to 3B, for example.

In step S501, the control unit 104 sets the remaining exposure time Tr to the exposure time Ts of the still image (Tr = Ts). The exposure time Ts is the shutter speed time determined by the user's specification or exposure control by the control unit 104, and in the example of FIG. 3B, Ts = T3 + T4 + T5.

In step S502, (period until the next LV imaging display V synchronization + LV exposure time TL) is acquired as Te. In this embodiment, Te is calculated to match the completion timing of the exposure time of LV imaging as shown in FIG. 3B, but this is not limited to this. For example, as described above, Te may be calculated based on specifications such as matching the exposure start timing of LV imaging. This makes it possible to match the delay from reading an image from the imaging unit 100 to displaying it on the display device 103 during LV imaging and during imaging by divided exposure, making it possible to more reliably use the LV image or still image (divided still image) as the LV display. In step S503, the control unit 104 compares the remaining exposure time Tr with Te acquired in step S502. As a result of the comparison, if Tr is greater than Te (YES in step S503), the next LV image cannot be captured due to the exposure time of the still image capture, so the process proceeds to step S504. In step S504 and after, a process is performed in which the divided still images obtained by capturing still images using divided exposures are used for LV display. On the other hand, if Tr is equal to or less than Te (NO in step S503), the process proceeds to step S507.

In step S504, the control unit 104 controls the imaging unit 100 to capture a still image by split exposure over an exposure time Te. The captured split exposure image is temporarily recorded in the temporary recording unit 107 via the data transfer unit 105. In step S505, the image processing unit 101 adjusts the brightness of the image captured by split exposure to match an image obtained by normal LV imaging. The display unit 102 displays the brightness-adjusted image on the display device 103. The brightness adjustment is as described in FIG. 3B and formulas (1) to (4).

In step S506, the control unit 104 updates the remaining exposure time Tr to a value obtained by subtracting Te from Tr, and returns the process to step S502. In step S502, the control unit 104 acquires a new Te, compares Tr and Te in step S503, and repeats the processes of steps S502 to S506 until Tr becomes equal to or less than Te. In this way, for example, divided exposure is performed as shown in FIG. 3B. In FIG. 3B, T3 is the Te calculated first, and T4 is the Te calculated next. T5 is Tr at the point when Tr becomes equal to or less than Te. The processes of steps S507 to S509 are the same as steps S406 to S408 in FIG. 4.

As described above, according to the above embodiment, in the LV display, updated images can be continuously displayed without causing phenomena such as blackouts or interrupted updates due to continuous display of the same image. Therefore, according to the above embodiment, it is possible to provide an imaging device that can perform smooth LV display with good visibility even when LV imaging and display cannot be performed due to a long exposure time for still images during still image capture or continuous shooting of still images.

Second Embodiment
For example, as shown in FIG. 3B and FIG. 3C of the first embodiment, consider a case where a still image 1-3 is obtained from a still image 1-1 by performing divided exposure. In this case, if the exposure time for obtaining the still image 1-3 is short, the period for reading out the still image 1-2 from the imaging sensor of the imaging unit 100 and the period for reading out the still image 1-3 may overlap. When such overlapping of the readout periods occurs, it becomes impossible to read out the still images. In the second embodiment, an imaging device that can reliably read out still images by avoiding such overlapping of the readout periods will be described. The configuration of the imaging device according to the second embodiment is the same as that of the first embodiment.

The timing chart for the case where LV images can be displayed continuously in continuous still image shooting is as shown in the first embodiment (FIGS. 2A to 2B). On the other hand, when the still image exposure time is long, still images are captured by divided exposure as shown in FIGS. 3A to 3C. In such divided exposure processing, if the exposure time in the last divided exposure is short and the readout period of the still image by the last divided exposure overlaps with the readout period of the still image by the previous divided exposure, still image shooting processing using a timing chart different from that in FIGS. 3B and 3C is required.

First, let us explain the overlap of readout periods. Figure 6 is a diagram showing still image shooting with divided exposure. Figures 6(a) and 6(b) are diagrams showing the timing of exposure and readout of still images 1-1 to 1-3 of Figures 3B and 3C. Figure 6(a) shows a case where the exposure time T5 of the last divided exposure (divided exposure of still image 1-3) is longer than the period To until the readout of the previous still image (still image 1-2) is completed. Period To is the period from the start of exposure of still image 1-3 to the readout of still image 1-2 is completed (To = TR - t4). When T5 > To, the readout period of still image 1-2 and the readout period of still image 1-3 do not overlap.

On the other hand, FIG. 6(b) shows a case where the exposure time T5 of the last divided exposure (divided exposure for still image 1-3) is shorter than the period To until the readout of the previous still image (still image 1-2) is completed. In this case, the readout period of still image 1-2 and the readout period of still image 1-3 overlap during period Tw, so still image 1-3 with exposure time T5 cannot be obtained. Note that T3 to T5 in the figure correspond to T3 to T5 in the timing chart of FIG. 3B, and Tv and Tr shown in parentheses correspond to the timing chart of FIG. 3C. Also, while FIG. 6 shows a case where the number of divisions for still image capture is three, this is not limited to this, and the number of divisions may be two, four or more.

In this embodiment, when there are still images captured by split exposure that overlap in their readout periods, the capture of the multiple still images by the image sensor is changed so as to eliminate the overlap. Figure 7 shows an example of how to deal with the case where the readout periods of still images captured by split exposure overlap (Figure 6(b)). Here, the four methods shown in Figures 7(a) to (d) are explained.

The first method is to delay the start timing of exposure of still images so as to avoid overlapping of readout periods. Figure 7 (a) shows a method of solving this problem by shifting the timing of split exposures at the timing when the still image readout periods overlap to a timing when they do not overlap. Considering the image quality of the composite image of the still images, it is desirable to expose still images 1-2 and 1-3 with as little time between them as possible. However, as shown in Figure 6 (b), since the exposure time of still image 1-3 is short, if exposure/readout is performed without any time between still images 1-2 and 1-3, overlap occurs in the readout periods of still images 1-2 and 1-3. Therefore, the control unit 104 delays the start timing of exposure of still image 1-3 in the imaging unit 100 so as to avoid overlapping of the readout periods. In other words, the overlap in the readout periods is eliminated by delaying the start timing of the exposure of still image 1-3 by Tx (Tx>To-T5) so that the start of readout of still image 1-3 occurs after the completion of readout of still image 1-2.

In the second method, the overlap of the readout periods is avoided by shifting a part of the exposure time allocated to at least one of the still images other than the still image of interest to the exposure time of the still image with which the readout periods overlap. In FIG. 7(b), the exposure time of the other still image (still image 1-1) is allocated to the exposure time of still image 1-3 so that the readout periods are divided so as not to overlap. For example, as in FIG. 3B, when the length of each divided exposure is determined in synchronization with the imaging display V synchronization 211, the exposure start timing of the still image (exposure start timing of still image 1-1) is shifted back by Tx as shown in FIG. 7(b). This reduces the exposure time of still image 1-1 by Tx, and the exposure time of still image 1-3 can be extended by Tx. By setting Tx so that the exposure time of still image 1-3 is T5+Tx>To, the overlap of the reading times of still images 1-2 and 1-3 can be avoided. Note that, as in FIG. 3C, when the divided exposure time is divided by a fixed length Tv, for example, the exposure time of still image 1-1 may be Tv-Tx, and the exposure time of still image 1-3 may be Tr+Tx>To. In this case, the still image whose exposure time is shortened is not limited to the first still image. Also, the exposure time for extending the last still image may be obtained from the exposure times of multiple still images. For example, the exposure time of still image 1-1 and still image 1-2 may be changed to Tv-Tx/2, and the exposure time of still image 1-3 may be extended by Tx.

The third method is to avoid overlapping readout periods by canceling capture of still images that would cause overlapping readout periods. Figure 7(c) shows how the overlapping readout periods described above is eliminated by not performing split exposure with overlapping still image readout periods. Because split exposure for the last still image 1-3 is not performed, the total exposure time for the still images is insufficient by the exposure time of the deleted still image 1-3. This lack of exposure time can be resolved, for example, by increasing the gain after image synthesis of still images 1-1 and 1-2.

In addition, when LV imaging becomes possible because the image for LV display of V of still image 1-3 that was canceled is no longer available, the control unit 104 performs LV imaging and displays the LV in order to obtain an LV display image.

The fourth method avoids overlapping readout periods by changing the shooting of two still images with overlapping readout periods to shooting a single still image with an exposure time equal to the sum of the exposure times of those two still images. For example, as shown in FIG. 7(d), when split exposures occur in which the still image readout periods overlap, this is resolved by extending the exposure time without splitting the exposure times. That is, when the readouts of the split exposures of still images 1-3 overlap in FIG. 7(d), the overlapping readout periods is eliminated by not splitting them and setting the exposure time of still image 1-2 to T4+T5, which eliminates the overlap.

Next, an example of operation for implementing the first to fourth methods for eliminating overlapping read periods described above with reference to Figures 7(a) to (d) will be described using the flowcharts in Figures 8 to 11.

Figure 8 is a flowchart showing the process for executing the first method described with reference to Figure 7 (a) (shifting the timing of split exposures where the still image readout periods overlap to timing where they do not overlap).

Figure 8 (a) shows the process added to the flowchart of Figure 4. If it is determined in step S402 that the remaining exposure time Tr is not greater than the threshold value Th (NO in step S402), the process proceeds to step S801. In step 801, the control unit 104 compares the remaining exposure time Tr with the still image readout period To. If the remaining exposure time Tr is longer than the still image readout period To (Tr>To), the process proceeds to step S406. The process from step S406 onwards is as shown in Figure 4. On the other hand, if the remaining exposure time Tr is less than or equal to the still image readout period To (Tr≦To), the process proceeds to step S802. In this way, the control unit 104 determines whether the still image readout periods overlap by comparing the remaining exposure time Tr with the still image readout period To. If the remaining exposure time Tr is longer than the still image readout period To, the still image readout periods do not overlap, and if the remaining exposure time Tr is less than or equal to the still image readout period To, the still image readout periods are determined to overlap.

In step S802, the exposure start timing of the remaining exposure time Tr is shifted back, for example as shown for still images 1-3 in FIG. 7(a), so that the readout periods of the still images do not overlap. Then, the process proceeds to step S406 in FIG. 4, where divided exposure is started at the timing set in step S802.

Figure 8 (b) shows the process added to the flowchart of Figure 5 to realize the operation shown in Figure 7 (a). If it is determined in step S503 that the remaining exposure time Tr is less than or equal to Te (NO in step S503), the process proceeds to step S811. In step 811, the control unit 104 compares the remaining exposure time Tr with the still image readout period To. If the remaining exposure time Tr is longer than the still image readout period To (Tr>To), the process proceeds to step S507. The process from step S507 onwards is as shown in Figure 5. On the other hand, if the remaining exposure time Tr is less than or equal to the still image readout period To (Tr≦To), the process proceeds to step S812. In step S812, the exposure start timing of the remaining exposure time Tr is shifted backward, for example, as shown for still images 1-3 in Figure 7 (a) so that the still image readout periods do not overlap. Processing then proceeds to step S507 in FIG. 5, where divided exposure begins at the timing set in step S812.

Figure 9 is a flowchart that explains the process for executing the second method described with reference to Figure 7 (b) (changing the period of the divided exposure of still images so that the still image readout periods are divided without overlapping).

9A shows the process executed prior to starting the process shown in the flowchart of FIG. 4. In step S901, the control unit 104 sets the remaining still image exposure time Tr to the still image exposure time Ts. The still image exposure time Ts is the shutter speed time determined by the user's specification or exposure control by the control unit 104. In step S902, the control unit 104 compares the remaining still image exposure time Tr with the threshold value Th. If Tr>Th, the process proceeds to step 903, and if Tr≦Th, the process proceeds to step 904. In step S903, the control unit 104 updates the remaining still image exposure time Tr to a value obtained by subtracting the period Tv of the image capture display V synchronization 211 from Tr. Steps S902 and S903 are repeated until Tr≦Th. The threshold value Th and period Tv are as described above with reference to FIG. 4.

In step S904, the control unit 104 compares the remaining still image exposure time Tr with the still image readout period To. As in the process of FIG. 8, by comparing the remaining still image exposure time Tr with the still image readout period To, it is determined whether the still image readout periods overlap. If the remaining still image exposure time Tr is longer than the still image readout period To (Tr>To), this process ends, and the control unit 104 starts the still image capture shown in FIG. 4. If the remaining still image exposure time Tr is equal to or shorter than the still image readout period To (Tr≦To), the process proceeds to step S905. In step S905, the control unit 104 sets the exposure time of the first divided exposure to a time obtained by subtracting Tx from Tv, and then starts the still image capture shown in FIG. 4. Note that in S403 to S405 of FIG. 4, Tv used for the first divided exposure is Tv-Tx. This allows the exposure time of the last divided exposure to be extended by Tx. Therefore, for example, if Tx is set to To-Tr, the readout period of the last divided exposure will not overlap with the readout period of the previous divided exposure.

9B shows the process executed prior to starting the process shown in the flowchart of FIG. 5. In step S911, the control unit 104 sets the remaining still image exposure time Tr to the still image exposure time Ts. Then, in step S912, the control unit 104 acquires (the period until the next LV imaging display V synchronization + the LV exposure time TL) as Te. In step S913, the control unit 104 compares the remaining still image exposure time Tr with Te. If Tr>Te, the process proceeds to step 914, and if Tr≦Te, the process proceeds to step 915. In step S914, the control unit 104 updates the remaining still image exposure time Tr to a value obtained by subtracting Te from the remaining still image exposure time Tr. Steps S913 and S914 are repeated until Tr≦Te.

In step S915, the control unit 104 compares the remaining still image exposure time Tr with the still image readout period To to determine whether the still image readout periods overlap. If the remaining still image exposure time Tr is longer than the still image readout period To (Tr>To), this process ends, and the control unit 104 starts still image capture as shown in FIG. 5. If the remaining still image exposure time Tr is equal to or shorter than the still image readout period To (Tr≦To), the process proceeds to step S916. In step 916, the exposure start timing of the still image is shifted as shown in FIG. 7(b). For example, if the exposure start timing is shifted back by To-Tr, the readout periods of the last divided exposure will not overlap. Then, the process is performed again from step S911. When the result of the final determination in step S915 is NO, divided exposure is performed at the exposure start timing of the still image set at that time.

Figure 10 is a flowchart showing the process for executing the third method described above (avoiding overlapping readout periods by canceling split exposures in which still image readout periods overlap) with reference to Figure 7 (c).

Figure 10 (a) shows the process added to the flowchart of Figure 4. In step S402, the control unit 104 compares the remaining still image exposure time Tr with the threshold value Th. If Tr is greater than Th (Tr>Th), the process proceeds to step S403. The processes in steps S403 to S405 are as described in Figure 4.

If it is determined in step S402 that Tr is equal to or less than Th (Tr≦Th), the process proceeds to step S1001. In step S1001, the control unit 104 compares the remaining still image exposure time Tr with the still image readout period To. If the remaining still image exposure time Tr is longer than the still image readout period To (Tr>To), the process proceeds to step S406. The process from step S406 onwards is as described in FIG. 4. On the other hand, if the remaining still image exposure time Tr is equal to or less than the still image readout period To (Tr≦To), the process proceeds to step S1002. As described above, by comparing the remaining still image exposure time Tr with the still image readout period To, it is determined whether the still image readout periods overlap.

In step S1002, the control unit 104 does not perform any further still image exposure and terminates still image capture. At this time, the control unit 104 stores the exposure time Tr of the exposure that was not performed due to the termination. In step S1003, if LV imaging is not completed in time, the image processing unit 101 adjusts the brightness of the still image obtained by the divided exposure and imaging to match the normal LV imaging image, and the display unit 102 displays the brightness-adjusted still image on the display device 103. This process is the same as the process of S407 described above with reference to FIG. 4.

In step S1004, the image processing unit 101 generates a single still image by combining the multiple divided exposure images acquired by divided exposure. The generation method may be the method described in step S408. Furthermore, since the total exposure time of the still images has been shortened by the remaining aborted exposure time Tr, the image processing unit 101 increases the gain by (still image exposure time Ts)/(Ts-Tr) in step S1005. This makes it possible to generate a still image equivalent to the still image with the still image exposure time Ts.

Figure 10 (b) shows the process added to the flowchart of Figure 5. In step S503, the control unit 104 compares the remaining still image exposure time Tr with Te. If Tr is greater than Te (Tr>Te), the process proceeds to step S504. The processes of steps S504 to S506 are as described in Figure 5. On the other hand, if it is determined in step S503 that Tr is equal to or less than Te (Tr≦Te), the process proceeds to step S1011. In step S1011, the control unit 104 compares the remaining still image exposure time Tr with the still image readout period To. If the remaining still image exposure time Tr is longer than the still image readout period To (Tr>To), the process proceeds to step S507. The processes from step S507 onwards are as described in Figure 5. On the other hand, if the remaining still image exposure time Tr is equal to or less than the still image readout period To (Tr≦To), the process proceeds to step S1012. As described above, whether the still image readout periods overlap is determined by comparing the remaining still image exposure time Tr and the still image readout period To. The processing of steps S1012 to S1015 is similar to the processing of steps S1002 to S1005 in FIG. 10(a).

Figure 11 is a flowchart showing the process for executing the fourth method described above with reference to Figure 7 (d) (extending the still image exposure time without dividing the exposure time when divided exposures occur in which the still image readout periods overlap).

Figure 11 (a) shows the process added to the flowchart of Figure 4. In step S402, the control unit 104 compares the remaining still image exposure time Tr with the threshold value Th. If Tr is greater than Th (Tr>Th), the process proceeds to step S1101. In step S1101, the control unit 104 determines whether Tr-Tv is greater than To. If Tr-Tv is greater than To, the process proceeds to step S403. The processes of steps S403 to S405 are as described above with reference to Figure 4. Also, if it is determined in step S402 that Tr is equal to or less than Th, the process proceeds to step S406. The processes from step S406 onwards are as described above with reference to Figure 4.

Also, if it is determined in step S1101 that Tr-Tv is equal to or less than To, processing proceeds to step S406. In this case, the exposure for the still image is not divided any further, and exposure is performed using a remaining still image exposure time Tr that is greater than Th. This is the case when exposure division is prohibited in anticipation of overlapping still image readout periods, and exposure and imaging are performed in step S406 with the remaining still image exposure time Tr being longer than one period Tv of the imaging display V synchronization. The subsequent processing in steps S407 and S408 is as described above.

As described above, according to the second embodiment, when split exposure is performed, it is possible to prevent overlapping of the image readout periods when reading out previous and subsequent split still images.

The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-mentioned embodiments to a system or device via a network or storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that realizes one or more of the functions.

The invention is not limited to the above-described embodiment, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to disclose the scope of the invention.

100: sensor, 101: image processing unit, 102: display unit, 103: display device, 104: control unit, 105: data transfer unit, 106: bus, 107: temporary recording unit, 108: recording unit

Claims (19)

An imaging device that captures a live view image and displays the live view on a display unit,
a first determination means for determining whether or not exposure in capturing one still image is to be divided based on an exposure time for capturing the one still image and an imaging cycle of the live view image;
a division means for dividing an exposure time for capturing the single still image when the first determination means determines that the exposure should be divided, and acquiring a plurality of still images to be combined into the single still image;
a second determination means for determining whether or not an overlap occurs between a readout period of a first still image from an image sensor among the plurality of still images divided by the division means and a readout period of a second still image subsequent to the first still image from the image sensor;
a change means for changing the capture of the plurality of still images by the image sensor so as to eliminate the overlap when the second determination means determines that the overlap occurs;
a display control means for sequentially using one or more of the plurality of still images as a live view image, thereby performing live view display during a period in which a live view image could not be captured due to capturing of the one still image;
An imaging device comprising: 2. The imaging device according to claim 1 , wherein the first determination means determines to divide the exposure for capturing the one still image when the exposure time for capturing the one still image is greater than a predetermined threshold value determined based on an imaging cycle of a live view image. The imaging device according to claim 2, characterized in that the predetermined threshold value is the imaging period of the live view image minus the time required to switch between capturing a live view image and capturing a still image. The imaging device according to claim 2 or 3, characterized in that the division means divides the exposure time for capturing the one still image by the length of the imaging cycle of the live view image. the first determination means determines a division timing based on an imaging timing of a live view image determined by an imaging cycle of the live view image, and determines to divide an exposure for imaging the one still image when an exposure period of the one still image exceeds the division timing;
2. The image pickup apparatus according to claim 1, wherein the dividing means divides an exposure time for capturing the one still image at the division timing. The imaging device according to claim 5, characterized in that the division timing is determined to be the timing at which exposure of a live view image that has started according to the imaging cycle ends. The imaging device according to claim 5, characterized in that the division timing is determined to be the timing at which capturing of a live view image starts according to the imaging cycle. The imaging device according to claim 5, characterized in that the division timing is determined so that the timing at which a live view image captured according to the imaging cycle is displayed coincides with the timing at which each of the multiple still images obtained by dividing the exposure of the single still image is displayed. The imaging device according to any one of claims 1 to 8, further comprising an adjustment means for adjusting the brightness of the still images obtained by the division means based on the imaging conditions of the live view image. The imaging device according to claim 9, characterized in that the brightness of the still images is adjusted according to the ratio of the divided exposure time of each image to the exposure time for capturing the live view image. 11. The imaging apparatus according to claim 1, wherein the change unit delays a start timing of exposure of the second still image so as to avoid the overlap of the readout periods. The imaging device according to any one of claims 1 to 10, characterized in that the change means shifts a portion of the exposure time assigned to at least one of the plurality of still images other than the second still image to the exposure time of the second still image so that the exposure time of the second still image is long enough to avoid the overlap . 11. The imaging apparatus according to claim 1, wherein the change unit stops capturing the second still image. A synthesizing unit is further provided for synthesizing the plurality of still images into the single still image,
The imaging device according to claim 13, characterized in that, when the capturing of the second still image is stopped, the combining means combines the multiple still images excluding the second still image, and generates a single still image by adjusting a luminance deficiency corresponding to an exposure time of the second still image. The imaging device according to any one of claims 1 to 10, characterized in that, instead of capturing the first still image and the second still image, the change means changes the exposure time so that a single still image is captured using an exposure time obtained by adding up the exposure time of the first still image and the exposure time of the second still image. 16. The imaging apparatus according to claim 1, wherein the second determination means determines that the overlap occurs when an exposure time given to the second still image is shorter than a predetermined time. 17. The image capture apparatus according to claim 16 , wherein the predetermined time is set based on a time required to read out a still image from the image sensor. A method for controlling an imaging device that captures a live view image and performs live view display, comprising the steps of:
a first determination step of determining whether or not exposure in capturing one still image is to be divided based on an exposure time for capturing the one still image and a timing for capturing the live view image;
a division step of dividing an exposure time for capturing the single still image to obtain a plurality of still images to be combined into the single still image when it is determined in the first determination step that the exposure is to be divided;
a second determination step of determining whether or not there is an overlap between a readout period of a first still image from an image sensor among the plurality of still images divided by the division step and a readout period of a second still image subsequent to the first still image from the image sensor;
a change step of changing the capturing of the plurality of still images by the image sensor so as to eliminate the overlap when it is determined that the overlap occurs in the second determination step;
a display control step of sequentially displaying one or more of the plurality of still images as a live view image on a display unit during a period in which a live view image cannot be captured due to capturing the one still image;
13. A method for controlling an imaging apparatus comprising: A program for causing a computer to function as each of the means of the imaging apparatus according to any one of claims 1 to 17 .

Images