JP2020184697A - Imaging apparatus and control method of the same - Google Patents

Abstract

To provide an imaging apparatus capable of achieving both still image shooting at a timing that suits a user's intention and stable moving image display during still image shooting and a control method of the same.SOLUTION: When an imaging apparatus shoots a still image while shooting a moving image and displaying the shot moving image, priority is given to shooting a still image. The imaging apparatus decides whether still image data or moving image data obtained after taking a still image is processed first to ensure that the number of times the same image is repeatedly displayed does not exceed a predetermined number of times in displaying the moving image.SELECTED DRAWING: Figure 2

Description

The present invention relates to an image pickup device and a control method thereof, and more particularly to an image pickup device capable of capturing a still image during moving image shooting and a control method thereof.

Some digital cameras and electronic devices having a camera function (hereinafter collectively referred to as an image pickup device) have a live view (LV) function. The LV function causes a display device for displaying a moving image to function as a finder by executing a moving image shooting and displaying a moving image obtained by the shooting in parallel. The user can adjust the shooting range of the still image or moving image to be recorded by the moving image displayed by the LV function.

Normally, since the resolution of the still image data and the moving image data are different, the operation of the image sensor also differs between the still image shooting and the moving image shooting. When the same image pickup sensor is used for moving image shooting and still image shooting, it is necessary to switch the operation of the image pickup sensor according to the type of the image to be shot. Depending on the timing of still image shooting, it is necessary to interrupt the moving image shooting, in which case the live view display is interrupted or the update frequency of the displayed image is reduced.

In Patent Document 1, a full-resolution frame and a lower-resolution frame are read out from an image sensor in a regular order to acquire a moving image, and when a still image shooting instruction is given, the latest full resolution is obtained. A method of outputting a resolution frame as a still image is disclosed.

Japanese Unexamined Patent Publication No. 2018-038088

As disclosed in Patent Document 1, by extracting one frame of a moving image being shot as a still image, it is possible to prevent the moving image shooting from being interrupted by the still image shooting or the frequency of updating the moving image is reduced. it can. However, the still image intended by the user may not be obtained due to the difference between the timing at which the full-resolution frame was shot most recently and the input timing of the still image shooting instruction. Further, when shooting a moving image, it is necessary to perform reading of two types having different resolutions while switching, for example, for each frame. Therefore, there is also a problem that the read control of the image sensor and the generation process of the display image data become complicated.

The present invention has been made in view of such problems of the prior art, and it is possible to achieve both still image shooting at a timing suitable for the user's intention and stable moving image display during still image shooting. An object of the present invention is to provide an image pickup apparatus and a control method thereof.

The above-mentioned purpose is to give priority to the image pickup element for taking a still image when the image pickup element, the image processing means, the image pickup element take a moving image, and the still image is taken while displaying the taken moving image. Control means for determining whether the still image data obtained by shooting a still image or the moving image data obtained by shooting a moving image after shooting a still image is processed first by the image processing means. The control means has a still image data obtained by shooting a still image and a still image so that the number of times the same image is repeatedly displayed does not exceed a predetermined number of times in displaying a moving image. This is achieved by an imaging apparatus characterized in that which of the moving image data obtained in the shooting of the moving image after the shooting of the moving image is first processed by the image processing means.

According to the present invention, it is possible to provide an image pickup apparatus capable of achieving both still image shooting at a timing suitable for the user's intention and stable moving image display during still image shooting, and a control method thereof.

Block diagram showing a functional configuration example of the image pickup apparatus according to the embodiment Timing chart related to operation during still image shooting during movie shooting in the first embodiment Flowchart for generating live view image data in the first embodiment The figure for demonstrating the details of the process of FIG. Timing chart related to operation during still image shooting during movie shooting in the second embodiment Flowchart for generating live view image data in the second embodiment The figure for demonstrating the details of the process of FIG. Timing chart related to operation during still image shooting during movie shooting in the third embodiment Flowchart for generating live view image data in the third embodiment A block diagram showing a functional configuration example of the data transfer unit 105 of FIG. Flowchart for generating live view image data in the fourth embodiment Timing chart related to operation during still image shooting during movie shooting in the fourth embodiment

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings based on an exemplary embodiment. The following embodiments do not limit the invention according to the claims. Further, although a plurality of features are described in the embodiment, not all of them are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the attached drawings, the same or similar configurations are designated by the same reference numbers, and duplicate description is omitted.

In the following embodiments, an example in which the present invention is applied to an imaging device such as a digital camera will be described. However, the present invention can be applied to any electronic device having an imaging function capable of capturing a still image during moving image imaging using one imaging sensor. Such electronic devices include video cameras, computer devices (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smartphones, game machines, robots, drones, and drive recorders. These are examples, and the present invention can be applied to other electronic devices.

● (First embodiment)
FIG. 1 is a block diagram showing a functional configuration example of a digital camera 1 as an example of an imaging device according to the present invention.
The image sensor 100 is, for example, a CCD image sensor or a CMOS image sensor, and a plurality of pixels having a photoelectric conversion unit are arranged. The image pickup device 100 converts an optical image formed on the image pickup surface by the image pickup lens 120 into an electric signal with a plurality of pixels, and generates image data composed of the plurality of pixel data. The image data generated by the image sensor 100 is written to the temporary storage unit (buffer memory) 107 via the data transfer unit 105.

The image processing unit 101 applies predetermined image processing to the moving image data and the still image data to generate image data for display and recording (live view image data, still image data for recording and display). .. The image processing unit 101 applies various image processing to the image data stored in the temporary storage unit 107. The image processing applied to the image data by taking a still image includes preprocessing, color interpolation processing, correction processing, data processing processing, and the like. Preprocessing includes noise reduction, signal amplification, reference level adjustment, defective pixel correction, and the like. The color interpolation process is a process of interpolating the values of color components not included in the image data read from the pixels, and is also called a demosaic process. The correction processing includes white balance adjustment, processing for correcting the brightness of an image, processing for correcting aberrations in the optical system of the image pickup lens 120, processing for correcting color, and the like. The data processing process includes scaling processing, coding and decoding processing, header information generation processing, and the like. It should be noted that these are examples of image processing capable of taking a still image, and do not limit the image processing performed by the image processing unit 101. The image processing unit 101 writes the generated image data to the temporary storage unit 107. When the live view image data is generated, the image processing unit 101 outputs it to the display unit 102 via the data transfer unit 105.

The image processing unit 101 may apply detection processing or evaluation value calculation processing to the generated live view image data. The detection process includes detection of a feature area (for example, a face area or a human body area) and its movement, and a person recognition process. Further, the evaluation value calculation process is used for the operation of the control unit 104, such as a pair of image signals for phase difference AF (automatic focus detection), an evaluation value for contrast AF, and an evaluation value used for automatic exposure control (AE). It is a process to calculate the evaluation value.

The display unit 102 receives the live view image data from the temporary storage unit 107 via the data transfer unit 105. The display unit 102 applies a processing process for displaying the live view image data on the display device 103 as necessary. The processing includes, for example, edge processing (letterbox processing, etc.) when the aspect ratios of the display screen of the display device and the live view image are different. In addition, processing includes processing for synthesizing auxiliary information such as shooting time, synthesizing GUI images such as OSD images, and converting signal formats.

The display unit 102 outputs the live view image data to which the necessary processing is applied to the display device 103. The display device 103 is, for example, an LCD (Liquid Crystal Display) or an OLED (Organic Light Emitting Diode) display. The display device 103 that displays the live view image data (moving image data) functions as an electronic viewfinder (EVF).

The shooting instruction unit 111 is an input member for the user to give an instruction for still image shooting to the digital camera 1. The shooting instruction unit 111 may be a release button provided on the digital camera 1, or may be another input member such as a remote controller. The user instruction through the shooting instruction unit 111 is detected by the control unit 104.

The control unit 104 has a CPU, and reads, for example, a program stored in the ROM 113 into the RAM 112 and executes the program. An FPGA (field-programmable gate array), which is a programmable processor, may be used instead of the CPU. The control unit 104 realizes the function of the digital camera 1 by controlling the operation of each functional block. The ROM 113 is, for example, a rewritable non-volatile memory, and stores programs, setting values, GUI data, and the like that can be executed by the CPU of the control unit 104. The RAM 112 is used to read a program executed by the CPU of the control unit 104 and to store a value required during the execution of the program. The control unit 104 also mainly controls the reading operation from the image sensor 100, which will be described later.

The data transfer unit 105 is composed of, for example, a plurality of DMA (Direct Memory Access) controllers (DMAC). Various image data are temporarily stored in the temporary storage unit 107 via the bus 106 by the DMAC. Further, the image data stored in the temporary storage unit 107 is read out to the bus 106 by the DMAC and supplied to each unit connected to the data transfer unit 105.

Further, the data transfer unit 105 has a synchronization signal control unit 114. The synchronization signal control unit 114 generates a vertical synchronization signal (imaging display V synchronization signal) 110 used by the image sensor 100 for shooting and reading moving image data (here, live view image data for display), and supplies the image sensor 100 to the image sensor 100. To do. For example, when the frame rate of the moving image is 60 frames / second, the synchronization signal control unit 114 generates an imaging display V synchronization signal having a period of 1/60 seconds.

When an instruction to shoot a still image is detected during shooting a moving image, the control unit 104 suspends the generation of the image pickup display V synchronization signal by the synchronization signal control unit 114, and then sets the reading mode of the image sensor 100 to the moving image reading mode. Switch to the still image read mode from. Then, the control unit 104 generates a synchronization signal (still image synchronization signal) according to the shooting timing of the still image and supplies it to the image sensor 100.

When the still image shooting for one frame is completed, the control unit 104 switches the read mode of the image sensor 100 from the still image read mode to the moving image read mode. Then, the control unit 104 causes the synchronization signal control unit 114 to restart the generation of the imaging display V synchronization signal 110.

The bus 106 has, for example, an independent system bus and a data bus. The functional block connected to the bus 106 is capable of bidirectional communication.
The temporary storage unit 107 is composed of a memory control unit and a memory, and writes data to the memory or reads data from the memory and outputs the data in response to an instruction from the control unit 104 or the data transfer unit 105. The temporary storage unit 107 is also used as a buffer for moving image data and still image data.

The recording unit 108 records the still image data and moving image data for recording generated by the image processing unit 101 on a recording medium such as a memory card under the control of the control unit 104. Further, the recording unit 108 reads out and outputs the image data recorded on the recording medium according to the control of the control unit 104. The control unit 104 stores the read image data in the temporary storage unit 107.

FIG. 2 is a diagram schematically showing timing control of shooting and display operation when still image shooting is performed while performing live view (LV) display (moving image display). The horizontal axis is the time axis, and the time advances from the left to the right in the drawing. Here, the case of continuous shooting in which instructions for still image shooting are continuously input is shown.

The image pickup display V synchronization (signal) indicates the output timing of the vertical synchronization signal indicating the timing at which the exposure and data reading of the LV image are started from the image pickup device 100. As described above, the vertical synchronization signal is output at a period T1 according to the frame rate of the LV image (moving image).

The still image synchronization (signal) is a signal indicating the timing of reading the still image data by the image sensor 100. The control unit 104 generates a still image synchronization signal at a predetermined timing while an instruction for still image shooting is input from the shooting instruction unit 111. In the present embodiment, when a still image shooting instruction is first detected, the control unit 104 outputs a still image synchronization signal after a certain period of time (release time lag). After that, when the instruction for still image shooting is continuously input, the control unit 104 outputs the still image synchronization signal at a predetermined continuous shooting interval T5.

The sensor output image is a period during which the image pickup element 100 performs imaging and reading of image data (LV image data, still image data) and a period during which LV image (moving image) reading and still image reading are switched (reading mode switching period). Is shown. Display 1 to display 9 are LV image imaging and readout periods (moving motion drive period or first drive period), and still images 1 to 3 are still image capture and readout periods (still image drive period or second drive period). Period) is shown. Further, 201 and 202 indicate the drive mode switching period of the image sensor 100. Reference numeral 201 denotes a switching period from the drive mode (moving image mode) for acquiring the LV image (moving image) to the driving mode (still image mode) for acquiring the still image, and 202 indicates the switching period from the still image mode to the moving image mode. The drive mode of the image sensor 100 is switched, for example, by the control unit 104 changing the setting of the image sensor 100.

The resolution and exposure time of the LV image, which is a moving image for display, and the still image for recording are different. In particular, since the resolution of the LV image is lower than the resolution of the still image data, the amount of image data to be read is small. Therefore, the moving image driving period is shorter than the still image driving period. Of the displays 1 to 9, the moving image driving periods shown by the displays 2, 4, 6, and 9 are shaded because the LV image data is not actually read out.

Of the displays 2, 4, 6 and 9, the displays 2 and 9 are not read out because they overlap with the still image driving period. The displays 4 and 6 that do not overlap with the still image drive period are not read out based on the determination result of the execution feasibility determination process of the LV image processing described later. However, the displays 4 and 6 may be read out for purposes other than the display. Applications other than display include, for example, obtaining an evaluation value for AF and detecting the position of a main subject or a subject to be tracked, but the present invention is not limited to these.

The image processing indicates the image processing period in the image processing unit 101. The displays 1, 3, 5, 7, 8 and the still images 1 and 2 show the sensor output images to be processed. The image processing period is a period until the image data read from the image sensor 100 is supplied to the image processing unit 101 via the temporary storage unit 107 and stored in the temporary storage unit 107 again after the image processing.

With respect to the LV image, the supply to the image processing unit 101 is started without waiting for all the image data for one screen to be stored in the temporary storage unit 107. On the other hand, with respect to the still image, the supply of the still image to the shooting is started after all the still image data for one screen read from the image sensor 100 is stored in the temporary storage unit 107. However, this is not essential, and as with the live view image, the storage operation from the image sensor 100 to the temporary storage unit 107 and the supply operation from the temporary storage unit 107 to the still image shooting may be performed in parallel. Good. The data transfer unit 105 supplies only the image data stored in the temporary storage unit 107 to the image processing unit 101. There is no image data that is not stored in the temporary storage unit 107 and is supplied for shooting a still image.

The display V synchronization is a vertical synchronization signal of the display device 103. By shortening the delay of the display V synchronization with respect to the imaging display V synchronization, the real-time property of the live view display is improved.

The display shows the display period on the display device 103 and the image to be displayed. The reason why the displays 1, 3 and 5 are output for two consecutive frames is that the live view image data corresponding to the displays 2, 4 and 6 is not generated.

In the drive control of the image sensor shown in FIG. 2, when a still image shooting instruction is given during moving image shooting, the still image shooting is preferentially executed so as to realize a predetermined release time lag and continuous shooting interval. On the other hand, in the display of the captured moving image, the moving image frame (live view image) to be displayed is selected so that the same image is not displayed three times (frames) or more in succession. As a result, it is possible to suppress a decrease in the update frequency of the moving image display during still image shooting.

Here, the generation control operation of the live view image data according to the shooting timing of the still image will be described with reference to the flowchart shown in FIG. Specifically, in the flowchart shown in FIG. 3, the image processing unit 101 determines whether or not to generate an LV image for a moving image frame obtained in the next moving image driving period starting after the still image driving period ends. The operation related to the process (LV image generation determination process) is shown. This operation is executed by the synchronization signal control unit 114. The moving image data of the frames (displays 4 and 6) determined not to be generated by the LV image generation determination process is not read from the image sensor 100. As described above, the moving image data of the frame determined not to be generated may be read out for use for a purpose other than LV image generation.

In S300, the synchronization signal control unit 114 determines whether or not the still image synchronization signal from the control unit 104 has been detected, proceeds to S301 if it is determined to be detected, and repeatedly executes S300 if it is not determined. ..
In S301, the synchronization signal control unit 114 determines whether or not the elapsed time from the latest imaging display V synchronization signal is the first threshold value or more and the second threshold value or less, and determines that the first threshold value or more and the second threshold value or less. If it is determined, the process proceeds to S303, and if it is not determined, the process proceeds to S302. Details of the determination process in S300 and S301 will be described later with reference to FIG. In the present embodiment, the imaging display V synchronization signal is used for determining the elapsed time, but the display V synchronization signal may be used.

In S302, the synchronization signal control unit 114 determines whether or not continuous shooting is completed in the next still image shooting, and if it is determined that continuous shooting is finished, the process proceeds to S303, and if not, the process proceeds to S304. .. The synchronization signal control unit 114 determines that the next still image is the last shot of continuous shooting, for example, when the continuously detected instruction input from the shooting instruction unit 111 is no longer detected. Can be done. Even in the case of single shooting, it can be determined that continuous shooting has ended. The synchronous signal control unit 114 may detect the instruction input, or the control unit 104 may perform the detection. When the control unit 104 detects the instruction input, the control unit 104 may notify the synchronization signal control unit 114 that the continuous shooting shooting will end in the next still image shooting.

In S303, the synchronization signal control unit 114 determines that the image processing unit 101 does not generate the LV image corresponding to the next moving image driving period after the end of the still image driving period, and ends the LV image generation determination process. .. Note that the image processing unit 101 does not generate only the LV image corresponding to the next moving image driving period after the end of the still image driving period, and the LV image corresponding to the next moving image driving period is the image processing unit. It is the target to be generated in 101. The synchronization signal control unit 114 notifies the control unit 104 of the determination result. The control unit 104 controls the image data transfer operation and the operation of the image processing unit 101 so that the image processing unit 101 immediately executes the image processing on the still image data obtained during the still image driving period. In FIG. 2, the image processing related to the still image 2 corresponds.

In S304, the synchronization signal control unit 114 determines that the image processing unit 101 generates an LV image corresponding to the next moving image driving period that starts after the end of the still image driving period, and ends the LV image generation determination process. To do. The synchronization signal control unit 114 notifies the control unit 104 of the determination result. The control unit 104 performs an image data transfer operation and an image so that the image processing for the still image data obtained in the still image driving period is executed after the image processing for the LV image data obtained in the next moving image driving period. It controls the operation of the processing unit 101. In FIG. 2, the image processing related to the display 3 and the still image 1 is applicable.

When it is determined in S302 that continuous shooting is completed in the next still image shooting by such a judgment process, the time until the confirmation image of the last shot still image is displayed on the display device 103 is shortened. It becomes possible to do. Further, for the frame in which the LV image data is not generated, the number of times the generated LV image data is redisplayed can be set to one. As a result, a decrease in the update frequency of the LV image is suppressed, and a smooth display is realized.

FIG. 4 is a diagram for explaining the details of the determination of S303 and S304 in the LV image generation determination process. In FIG. 4, the horizontal axis indicates time, and time progresses from left to right.

Reference numerals 400 and 401 indicate a moving image drive period in the image sensor 100. Further, reference numeral 402 denotes a still image driving period that is started after a certain time T0 (release time lag) from the input of the instruction for still image shooting and every time the continuous shooting interval T5 elapses thereafter. Further, the length of each line included in 403 to 405 represents the total time of the still image driving period and the mode switching time of the image sensor (switching time from the still image mode to the moving image mode).

Each of the dotted lines shown by 403 indicates the time range occupied by the still image driving period and the mode switching time when the still image driving period starts in the period T1 for different start timings. Similarly, each of the solid line indicated by 404 and the dotted line indicated by 405 indicate the time range occupied by the still image driving period and the mode switching time when the still image driving period starts in the periods T2 and T3, with respect to different start timings. Shown.

The still image synchronization signal detected in S300 of the LV image generation determination process defines the start time of the line indicated by 403 to 405. Further, the first threshold value of S301 indicates the start time point of the period T2 (or the end time point of the period T1), and the second threshold value indicates the end time point of the period T2 (or the start time point of the period T3). The period T2 corresponds to the range of the start time of the still image driving period in which the still image driving period does not overlap with the moving image driving period. That is, the still image shooting that starts within the period T2 does not affect the moving image shooting.

Consider that the image processing on the still image data is started immediately after the end of the still image driving period, as in the image processing of the still image 2 of FIG. When the still image drive period starts within the period T2, even if the image processing for the still image data is started immediately, the LV image is updated as usual from the second moving image drive period after the still image drive period ends. it can. Therefore, instead of the LV image data that cannot be generated by using the image processing unit 101 for the image processing of the still image data, the LV image data obtained in the moving image driving period 400 immediately before the still image driving period is displayed once. It's fine. That is, the number of times that the LV image data obtained in the moving image driving period 400 is continuously displayed may be two times.

On the other hand, when the still image driving period starts in the period T1 or T3, the still image driving period overlaps with the moving image driving period. When the still image driving period starts in the period T1, the moving image driving period immediately before the still image driving period starts, and when the still image driving period starts in the period T3, the moving image driving period immediately after the still image driving period. Each overlaps. When the still image driving period starts in the period T1, the LV image corresponding to the moving image driving period two before the still image driving period is displayed twice in succession. When the still image driving period starts in the period T3, the LV image corresponding to the moving image driving period immediately before the still image driving period is displayed twice in succession.

In S304, the next moving image driving period starting after the end of the still image driving period is the moving image driving period 401 when the still image driving period starts in the period T1. Further, when the still image driving period starts in the period T3, it becomes the next moving image driving period of the moving image driving period 401. If the image processing on the still image data corresponding to the still image driving period is performed before the image processing on the LV image data corresponding to these moving image driving periods, the moving image driving period before the still image driving period is supported. The LV image will be displayed three times in a row. Therefore, in order to prevent the LV image from being continuously displayed twice, the image processing for the still image data corresponding to the still image driving period is performed on the LV corresponding to the next moving image driving period that starts after the end of the still image driving period. It is performed after the image processing on the image data.

As described above, according to the present embodiment, when a still image shooting instruction is given during movie shooting, the still image shooting is prioritized so as to realize a predetermined release time lag and continuous shooting interval. And execute. On the other hand, in order to select the image to be image-processed according to the start timing of still image shooting so that the same image is repeatedly displayed twice or less in the moving image display during continuous still image shooting, the moving image is used. It is possible to suppress a decrease in the display update frequency.

● (Second embodiment)
Next, the second embodiment of the present invention will be described. The present embodiment is different from the first embodiment in that the shooting frame rate of the moving image is 120 fps. In the following, only the parts different from the first embodiment will be described, and detailed description of the common parts will be omitted.

FIG. 5 schematically shows the timing control of shooting and display operation when still image shooting is performed while performing live view (LV) display (moving image display), as in FIG. 2 described in the first embodiment. It is a figure. The description of the contents common to those in FIG. 2 will be omitted.

Also in this embodiment, still image shooting is prioritized over moving image shooting so that the release time lag T0 and the shooting interval T5 are constant. Since the shooting frame rate of the moving image is 120 fps, the frequency of occurrence of the imaging display V synchronization is twice that of the first embodiment (the generation cycle T1 is 1/2). The display frame rate of the moving image is 60 fps, which is the same as that of the first embodiment. Therefore, by default, the LV image data based on the moving image data taken in the odd-numbered moving image driving period (moving moving images 1, 3, 5, ...) Is used for the LV display.

Of the displays 1 to 14, the LV image data is not actually read out during the moving image driving period shown by the displays 4, 6, 9, 10 and 14, and is therefore shaded. The displays 4, 9 and 14 are not read out because they overlap with the still image driving period. The displays 6 and 10 that do not overlap with the still image drive period are not read out based on the determination result of the execution feasibility determination process of the LV image processing described later. However, the displays 6 and 10 may be read out for purposes other than the display. Further, the displays 2, 8 and 12 that generate an LV image but are not used for the display are used for purposes other than the display. The displays 1, 3, 5, 7, 8, 11, and 13 are used for displaying LV images and for purposes other than display.

Similar to the first embodiment, even in the drive control of the image sensor shown in FIG. 5, when a still image shooting instruction is given during moving image shooting, the still image shooting is performed so as to realize a predetermined release time lag and continuous shooting interval. Priority is given to execution. In addition, in the display of the captured moving image, the moving image frame (live view image) to be displayed is selected so that the same image is not displayed twice (frames) or more in succession. As a result, it is possible to suppress a decrease in the update frequency of the moving image display during still image shooting.

The generation control operation of the live view image data according to the shooting timing of the still image will be described with reference to the flowchart shown in FIG. The same operation as that of the first embodiment will be described by adding the same reference numbers as those in FIG.

When it is determined that the still image synchronization signal is detected in S300, the synchronization signal control unit 114 in S601 determines that the elapsed time from the imaging display V synchronization signal of the frame used for display (odd number frame) is equal to or longer than the first threshold value. And it is determined whether or not it is equal to or less than the second threshold value. Then, the synchronization signal control unit 114 proceeds to S302 if it is determined to be the first threshold value or more and the second threshold value or less, and proceeds to S303 if it is not determined. The details of the determination process in S601 will be described later with reference to FIG. 7. In the present embodiment, the imaging display V synchronization signal is used for determining the elapsed time, but the display V synchronization signal may be used.

In S302, the synchronization signal control unit 114 determines whether or not continuous shooting is completed in the next still image shooting, and if it is determined that continuous shooting is finished, the process proceeds to S303, and if not, the process proceeds to S304. ..

In S303, the synchronization signal control unit 114 determines that the image processing unit 101 does not generate the LV image corresponding to the next moving image driving period after the end of the still image driving period, and ends the LV image generation determination process. .. The synchronization signal control unit 114 notifies the control unit 104 of the determination result. The control unit 104 controls the image data transfer operation and the operation of the image processing unit 101 so that the image processing unit 101 immediately executes the image processing on the still image data obtained during the still image driving period. In FIG. 5, the image processing related to the still image 2 corresponds.

In S304, the synchronization signal control unit 114 determines that the image processing unit 101 generates an LV image corresponding to the next moving image driving period that starts after the end of the still image driving period, and ends the LV image generation determination process. To do. The synchronization signal control unit 114 notifies the control unit 104 of the determination result. The control unit 104 performs an image data transfer operation and an image so that the image processing for the still image data obtained in the still image driving period is executed after the image processing for the LV image data obtained in the next moving image driving period. It controls the operation of the processing unit 101. In FIG. 5, the image processing related to the display 5 and the still image 1 is applicable.

When it is determined in S302 that continuous shooting is completed in the next still image shooting by such a judgment process, the time until the confirmation image of the last shot still image is displayed on the display device 103 is shortened. It becomes possible to do. Further, for the frame in which the LV image data is not generated, the number of times the generated LV image data is redisplayed can be set to one. As a result, a decrease in the update frequency of the LV image is suppressed, and a smooth display is realized.

FIG. 7 is a diagram for explaining the details of the determination of S303 and S304 in the LV image generation determination process. In FIG. 4, the horizontal axis indicates time, and time progresses from left to right.

Reference numerals 700 to 702 indicate a moving image drive period in the image sensor 100. Here, the moving image driving periods 700 and 702 are odd-numbered moving image driving periods, and it is assumed that the obtained moving image data is used for display. Hereinafter, the odd-numbered moving image driving period is referred to as a display moving image driving period. Further, 703 indicates a still image driving period that is started after a certain time T0 (release time lag) from the input of the instruction for still image shooting and every time the continuous shooting interval T5 elapses thereafter. The length of each line included in 704 to 706 represents the total time of the still image driving period and the mode switching time of the image sensor (switching time from the still image mode to the moving image mode).

Each of the dotted lines shown by 704 indicates the time range occupied by the still image driving period and the mode switching time when the still image driving period starts in the period T1 for different start timings. Similarly, each of the solid line indicated by 705 and the dotted line indicated by 706 indicates the time range occupied by the still image driving period and the mode switching time when the still image driving period starts in the periods T2 and T3, with respect to different start timings. Shown.

The still image synchronization signal detected in S300 of the LV image generation determination process defines the start time of the line indicated by 704 to 706. Further, the first threshold value of S601 indicates the start time point of the period T2 (or the end time point of the period T1), and the second threshold value indicates the end time point of the period T2 (or the start time point of the period T3). The period T2 corresponds to the range of the start time of the still image driving period that does not overlap with the moving image driving period for display. That is, the still image shooting that starts within the period T2 does not affect the moving image shooting for the LV display. The moving image driving period 701 in FIG. 7 is an even-numbered moving image driving period, and it is assumed that the obtained image data is used for a purpose other than display.

When the still image driving period starts within the period T2, it is not necessary to immediately start the image processing on the still image data unless it is the last shooting of continuous shooting. Therefore, the control unit 104 controls the image processing unit 101 to perform image processing on the moving image data obtained in the moving image driving period for the next display after the still image driving period ends to generate and display the LV image data. To do. FIG. 5 corresponds to the control of image processing for the still image 1. On the other hand, in the case of the final shooting of continuous shooting, the control unit 104 starts image processing on the still image data immediately after the end of the still image driving period, as in the image processing of the still image 2 in FIG. Control.

Further, when the still image driving period starts in the period T1 or T3, the still image driving period overlaps with the moving image driving period for display. When the still image drive period starts in the period T1, it is for the video drive period for display immediately before the still image drive period, and when the still image drive period starts in the period T3, it is for the display immediately after the still image drive period. Overlap during the video drive period of. When the still image drive period starts in the period T1, the control unit 104 generates an LV image from image data obtained in the non-display moving image driving period immediately before the moving image driving period in which the still image driving periods overlap. Is controlled to be displayed. This corresponds to the control of displaying the LV image data generated from the image data obtained in the previous display 8 instead of the display 9 overlapping in the period of the still image 2 in FIG.

When the still image driving period starts in the period T3, the control unit 104 continuously displays the LV image data generated from the image data obtained in the moving image driving period for display immediately before the still image driving period twice. Control to do. In the example of FIG. 7, the control unit 104 corresponds to the moving image driving period 702 for display which overlaps the still image driving period with the LV image data generated from the moving image data obtained in the moving image driving period shown by 700. It is controlled so that it is also displayed as LV image data.

In S304, the next moving image driving period starting after the end of the still image driving period is the moving image driving period 702 when the still image driving period starts in the period T1. Further, when the still image driving period starts in the period T3, the moving image driving period for the next display of the moving image driving period 702 is set.

As described above, the same effect as that of the first embodiment can be realized by this embodiment as well. Further, when the shooting frame rate is higher than the display frame rate, the frequency of repeatedly displaying the same image data can be reduced by using the LV image data that is not originally used for display for display, and a smoother moving image display can be performed. Becomes possible.

● (Third embodiment)
Next, a third embodiment of the present invention will be described. Since the present embodiment corresponds to a modified example of the second embodiment, the parts different from the first and second embodiments will be mainly described below.

FIG. 8 is a diagram schematically showing the timing control of shooting and display operation when still image shooting is performed while performing live view (LV) display (moving image display), as in FIGS. 2 and 5. The description of the contents common to those in FIGS. 2 and 5 will be omitted.

In FIG. 8, 700 indicates the timing at which continuous shooting is detected to end in the next still image shooting. Similar to the second embodiment, even in the drive control of the image sensor shown in FIG. 8, when a still image shooting instruction is given during movie shooting, the still image shooting is performed so as to realize a predetermined release time lag and continuous shooting interval. Priority is given to execution. In addition, in the display of the captured moving image, the moving image frame (live view image) to be displayed is selected so that the same image is not displayed twice (frames) or more in succession. As a result, it is possible to suppress a decrease in the update frequency of the moving image display during still image shooting.

The present embodiment is different from the second embodiment in that the moving image driving period to be started next after the end of the still image driving period is always used as the driving period for display, except for the case where the continuous shooting is the last shooting. Specifically, instead of the display 11, the display 10 is set as the moving image driving period for display. Further, in FIG. 8, an operation of displaying the still image data obtained in the final shooting of continuous shooting is added. This can also be carried out in the first and second embodiments.

The generation control operation of the live view image data according to the shooting timing of the still image will be described with reference to the flowchart shown in FIG. The same operation as that of the first embodiment will be described by adding the same reference numbers as those in FIG.

When the synchronization signal control unit 114 determines that the still image synchronization signal has been detected in S300, the synchronization signal control unit 114 proceeds to S302. In S302, the synchronization signal control unit 114 determines whether or not continuous shooting is completed in the next still image shooting, and proceeds to S303 if it is determined that continuous shooting is completed, and to S304 if it is not determined. ..

In S303, the synchronization signal control unit 114 determines that the image processing unit 101 does not generate the LV image corresponding to the next moving image driving period after the end of the still image driving period, and ends the LV image generation determination process. .. The synchronization signal control unit 114 notifies the control unit 104 of the determination result. The control unit 104 controls the image data transfer operation and the operation of the image processing unit 101 so that the image processing unit 101 immediately executes the image processing on the still image data obtained during the still image driving period. In FIG. 8, the image processing related to the still image 3 corresponds.

In S304, the synchronization signal control unit 114 determines that the image processing unit 101 generates an LV image corresponding to the next moving image driving period that starts after the end of the still image driving period, and ends the LV image generation determination process. To do. The synchronization signal control unit 114 notifies the control unit 104 of the determination result. The control unit 104 performs an image data transfer operation and an image so that the image processing for the still image data obtained in the still image driving period is executed after the image processing for the LV image data obtained in the next moving image driving period. It controls the operation of the processing unit 101. In FIG. 8, the image processing related to the display 5, the still image 1, the display 10, and the still image 2 is applicable.

In this embodiment as well, the same effect as in the second embodiment can be realized. Further, when the still image shooting is not the last still image shooting of the continuous shooting, the LV image generated from the moving image data obtained in the next moving image driving period after the end of the still image driving period is used for the display. did. Therefore, a moving image frame whose timing is close to that at the time of still image shooting is displayed, and the update interval of the moving image display before and after the still image shooting can be shortened.

● (4th embodiment)
Next, a fourth embodiment of the present invention will be described. In the following, only the parts different from those of the first to third embodiments will be described, and the common parts will be designated by the same reference numerals and detailed description thereof will be omitted.

FIG. 10 is a block diagram showing in more detail the relationship between the functional configuration example and the image sensor 100 and the control unit 104 for the synchronization signal control unit 114 in the data transfer unit 105 of FIG.
The synchronization signal generation counter 1142 generates an imaging display V synchronization signal according to the shooting frame rate. The synchronization signal generation counter 1142 generates a shooting display V synchronization signal every time, for example, counts a number of clock pulses corresponding to a frame period corresponding to a shooting frame rate. The shooting display V synchronization signal is output to the selector 1141 and also to the control unit 104 as an interrupt signal.

The selector 1141 selects and outputs one of the image pickup display V synchronization signal generated by the synchronization signal generation counter 1142 and the still image synchronization signal 1143 supplied from the control unit 104. The selector 1141 outputs the still image synchronization signal 1143 in the still image mode, and outputs the imaging display V synchronization signal in the moving image mode. The mode of the selector 1141 is controlled by the control unit 104.

When the control unit 104 detects an instruction input for still image shooting from the shooting instruction unit 111, the control unit 104 generates a still image synchronization signal 1143 after the release time lag T0 and every time the continuous shooting interval T5 elapses thereafter, and causes the selector 1141 to generate a still image synchronization signal 1143. Output. The control unit 104 also generates a control signal 1041 for switching the drive mode of the image sensor 100 and supplies the control signal 1041 to the image sensor 100.

FIG. 11 is a flowchart relating to the generation control operation of the live view image data in the present embodiment.
In S1100, the control unit 104 determines whether or not LV shooting is being performed, proceeds to S1100 if it is in LV shooting mode, and ends processing if it is not in LV imaging mode.
In S1101, the control unit 104 sets the operation mode of the image sensor 100 to the moving image mode by using the control signal 1041. In the state where the instruction input for still image shooting is not detected, the image sensor 100 is driven in the moving image mode in order to perform moving image shooting for live view display.

In S1102, the control unit 104 determines whether or not there is a request (switching request) for switching the operation mode of the image sensor 100 to the still image mode, such as inputting an instruction for still image shooting from the shooting instruction unit 111. If the control unit 104 determines that there is a request to switch to the still image mode, the process proceeds to S1103, and the operation mode of the image sensor 100 is switched to the still image mode by the control signal 1041. If it is not determined that there is a request to switch to the still image mode, the control unit 104 repeatedly executes S1102 while controlling the moving image shooting, the generation and display of the LV image.

In S1104, the control unit 104 switches the selector 1141 to the still image mode. As a result, the still image synchronization signal generated by the control unit 104 is supplied to the image sensor 100 through the selector 1141. Further, the control unit 104 initializes the LV interrupt counter. The LV interrupt counter may be included in the control unit 104, or may be a variable whose value is stored in the RAM 112. Due to the initialization, the value of the LV interrupt counter becomes a predetermined initial value (for example, 0).

The control unit 104 generates a still image synchronization signal after the release time lag T0. The image sensor 100 is exposed in the still image mode, and the still image data is read out. The read still image data is transferred to the temporary storage unit 107 via the data transfer unit 105. When the data transfer unit 105 stores one frame of still image data in the temporary storage unit 107, the data transfer unit 105 notifies the control unit 104 of the completion of the still image shooting.

In S1105, the control unit 104 waits for the data transfer unit 105 to notify the completion of still image shooting. When the completion of still image shooting is notified, the control unit 104 executes S1105.

In S1106, the control unit 104 determines whether or not the completed still image shooting is the last shooting in single shooting or continuous shooting. If it is determined in S1110 that it is the last shooting in single shooting or continuous shooting, the control unit 104 immediately starts the image processing of the still image data stored in the temporary storage unit 107 in the image processing unit 101 and processes it. Is returned to S1100. If it is determined that continuous shooting is continuing, the control unit 104 advances the process to S1107.

On the other hand, the control unit 104 executes S1111 to S1113 as interrupt processing. In S1109, the control unit 104 monitors an interrupt due to the imaging display V synchronization signal from the synchronization signal generation counter 1142. While the image sensor 100 is being driven in the still image mode, the image sensor V synchronization signal is blocked by the selector 1141 and is not supplied to the image sensor 100, but is input to the control unit 104 as an interrupt signal 1144.

When the control unit 104 detects the interrupt signal 1144 in S1111, the control unit advances the process to S1112 and increments (counts up) the value of the LV interrupt counter by one. The control unit 104 continuously counts up the interrupt signal 1144 until it is determined that the still image shooting completed in S1113 is the last shooting in single shooting or continuous shooting. The value of the interrupt counter is the cumulative number of movie shooting timings (imaging display V synchronization signal) that occur during the period when the image sensor is driven in the still image mode after the still image shooting (including continuous shooting) is started. Equivalent to. Alternatively, the value of the interrupt counter can be said to be the cumulative number of moving image shots that were not taken to take a still image.

In S1107, the control unit 104 compares the value of the LV interrupt counter with a predetermined threshold value, and proceeds to S1110 if the value is less than the threshold value and to S1108 if the value is greater than or equal to the threshold value. This threshold value is a value for limiting the number of times that the same image is continuously displayed in the moving image display. When the value of the LV interrupt counter reaches the threshold value, the update of the moving image display is prioritized over the processing of the still image. Specifically, the control unit 104 performs the same processing as in S1101 in S1108, and proceeds to S1109. In S1109, the control unit 104 controls the moving image shooting, the generation and display of the LV image, waits until the processing is completed, and then proceeds to S1110. On the other hand, when the value of the LV interrupt counter is less than the threshold value, the control unit 104 causes the image processing unit 101 to execute image processing on the still image data in S1110, and returns to S1120.

FIG. 12 is a timing chart showing an example in which image processing of still image and moving image data is controlled according to the operation described with reference to FIG. Here, it is assumed that both the shooting frame rate and the display frame rate are 60 fps and the threshold value is 1. Further, FIG. 12 does not show the switching time of the operation mode of the image sensor 100 for convenience.

The instruction input 200 for still image shooting from the shooting instruction unit 111 is detected, the still image shooting is started after the release time lag T0, and then the still image shooting is repeatedly executed at the continuous shooting interval T5. Also in this embodiment, still image shooting is prioritized over moving image shooting so that the timing of still image shooting is constant.

When the first still image drive period is started, the control unit 104 initializes the value of the LV interrupt counter to 0. In the example of FIG. 12, the LV synchronization signal is not generated between the start of the still image driving period (still image 1) and the storage of one frame of still image data in the temporary storage unit 107. Therefore, the value of the LV interrupt counter remains 0. In this case, in FIG. 11, the processing transitions to S1105, S1106, S1107, and S1110, and the control unit 104 causes the image processing unit 101 to immediately start image processing on the still image data.

In the still image drive period (still image 2) corresponding to the second image of continuous shooting, since the LV synchronization signal is generated during the still image drive period, the control unit 104 counts up the value of the LV interrupt counter to 1. .. Since the value of the LV interrupt counter has reached the threshold value, the processing transitions to S1105, S1106, S1107, S1108, and S1109 in FIG. The control unit 104 causes the image processing unit 101 to first perform image processing on the moving image data obtained in the next moving image driving period, and then perform image processing on the still image data. In FIG. 12, the image processing for the still image 2 is performed after the image processing for the display 6 is completed.

As described above, in the present embodiment, image processing for still image data is performed according to the cumulative number of times of moving image shooting timing (imaging display V synchronization signal) generated during the period when the image sensor is driven in the still image mode. And, the priority of image processing for moving image data is controlled. Also in the present embodiment, as in the first to third embodiments, it is possible to take a still image at a fixed timing while suppressing a decrease in the update frequency of the moving image display.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The present invention is not limited to the contents of the above-described embodiments, and various modifications and modifications can be made without departing from the spirit and scope of the invention. Therefore, a claim is attached to make the scope of the invention public.

100 ... Image sensor, 101 ... Recording processing unit, 104 ... Control unit, 105 ... Data transfer unit, 111 ... Shooting instruction unit, 114 ... Synchronous signal control unit

Claims (10)

With the image sensor
Image processing means and
When shooting a still image while shooting a moving image with the image sensor and displaying the shot movie, the image sensor is made to preferentially shoot the still image, and the still image is obtained. It has a control means for determining which of the obtained still image data and the moving image data obtained in the shooting of the moving image after the shooting of the still image is processed by the image processing means first.
The control means captures the still image data obtained by shooting the still image and shooting the still image so that the number of times the same image is repeatedly displayed does not exceed a predetermined number of times in the display of the moving image. Which of the moving image data obtained in the subsequent shooting of the moving image is processed first by the image processing means is determined.
An imaging device characterized by this. When the control means does not overlap the first drive period, which is the drive period of the image sensor for shooting a moving image, with the second drive period, which is the drive period of the image sensor for shooting a still image. The image pickup apparatus according to claim 1, wherein it is determined that the still image data obtained in the second driving period is processed by the image processing means before the moving image data. When the control means overlaps the first drive period, which is the drive period of the image sensor for shooting a moving image, with the second drive period, which is the drive period of the image sensor for shooting a still image. The image pickup apparatus according to claim 1 or 2, wherein the moving image data is determined to be processed by the image processing means before the still image data obtained in the second driving period. .. The present invention according to any one of claims 1 to 3, wherein the most recently displayed moving image data is repeatedly displayed as the moving image data corresponding to the shooting of the moving image that was not performed for taking the still image. The imaging apparatus described. During the first driving period in which the shooting frame rate of the moving image is higher than the display frame rate of the moving image and is the driving period of the image pickup element for shooting the moving image, the image pickup element for shooting a still image When the second driving period, which is the driving period, overlaps, the control means uses the moving image data obtained in the most recent first driving period, which does not overlap the second driving period, for display. The image pickup apparatus according to any one of claims 1 to 4. When the cumulative number of times of shooting a moving image that was not performed to shoot the still image reaches a predetermined threshold value, the control means is more than the still image data obtained by shooting the latest still image. The image pickup apparatus according to claim 1, wherein the moving image data obtained by shooting a moving image after shooting the still image is determined to be processed first by the image processing means. 6. The control means according to claim 6, wherein the control means obtains the cumulative number of times by counting the number of times of synchronization signals of moving image shooting generated while the image pickup device is driven in the still image mode. Image sensor. When the still image is taken as a single shot or the last shot of continuous shooting, the control means determines that the still image data is processed by the image processing means before the moving image data. The imaging apparatus according to any one of claims 1 to 7, wherein the image pickup apparatus is characterized by the above. A control method for an image pickup device having an image pickup device and an image processing means.
When the control means shoots a still image while shooting a moving image by the image sensor and displaying the shot moving image, the image sensor is caused to preferentially shoot the still image and the still image is executed. It has a control step of determining which of the still image data obtained in the shooting of the image and the moving image data obtained in the shooting of the moving image after the shooting of the still image is processed by the image processing means first.
In the control step, the control means uses the still image data obtained by shooting the still image and the still image data so that the number of times the same image is repeatedly displayed does not exceed a predetermined number of times in the display of the moving image. It is determined which of the moving image data obtained by shooting the moving image after shooting the still image is processed by the image processing means first.
A control method for an imaging device, characterized in that. A program for causing a computer included in an imaging device to function as each means of the imaging device according to any one of claims 1 to 8.

Images