JP2020188335A - Imaging apparatus and method for controlling the same - Google Patents

Abstract

To provide an imaging apparatus capable of acquiring still image data at a higher speed during moving image shooting.SOLUTION: An imaging element 101 receives light from a subject and acquires image data by photoelectric conversion. A discrimination control unit 102 detects that an instruction of still image photographing is generated during acquisition of the moving image, and controls to extend a blanking period in reading of the moving image by changing a reading method of a frame image related to generation of a video file. A moving image determination unit 103 determines the encoding attribute (I, P, B frames) of the frame image of the video file, and stores data of the frame image to which the encoding attribute is added in a frame buffer 104. A moving image generation unit 105 generates a video file from the frame image stored in the frame buffer 104. A still image generation unit 106 generates a still image file using data read out during the extended blanking period.SELECTED DRAWING: Figure 1

Description

The present invention relates to an imaging device capable of capturing moving images and capturing still images, and a control method thereof. In particular, the present invention relates to a technique capable of appropriately generating a moving image and generating a still image at a higher speed when an instruction for shooting a still image is detected during movie shooting.

The moving image data obtained from the image sensor is efficiently compressed in units called GOP (Group of Picture). In-frame compression coding, called I (Intra picture) frame in GOP, allows coding and decoding independently of other frames. Although it cannot process with a high compression rate, it has the feature of being able to acquire high-quality image data. Further, in the inter-frame compression coding called P (Predictive Picture) frame, a high compression ratio can be achieved by using the difference information from the I frame. On the other hand, there is also a problem that encoding and decoding cannot be performed without referring to the I frame. In the compression coding between the front and rear frames called the B (Bidirectionally Predictive Picture) frame, the difference information from the previous and next frames such as the I frame and the P frame is used. This makes it possible to achieve a higher compression ratio than the P frame. However, there is also a problem that encoding and decoding cannot be performed without referring to the preceding and following frames. Such encoding and decoding formats are standardized and known as H.264 / MPEG-4 AVC in ISO / IEC 14496-10 "MPEG-4 Part 10 Advanced Video Coding", and thus a detailed description thereof will be given. Is omitted.

By the way, when still image shooting is performed when moving image shooting is performed by the moving image data generation method, there is a problem that moving image data is lost during the still image shooting period and a blank occurs. Therefore, Patent Document 1 discloses an imaging device that generates still image data without interrupting the generation of moving image data when still image shooting is performed during the generation of moving image data. This image pickup device holds the signal from the image pickup element in the moving image storage element and the still image storage element. Control is performed to read the still image data from the still image storage element by using the moving image signal reading blanking period when the moving image data is being read from the moving image storage element.

Japanese Unexamined Patent Publication No. 2010-4175

In the prior art disclosed in Patent Document 1, the time for not reading the moving image data, that is, the so-called moving image blanking time is shorter than the reading time for the moving image data based on the output of the image sensor. The moving image blanking time is allocated to the reading time of the still image data based on the output of the image sensor, and a large amount of moving image blanking time is required to read all the still image data. For example, it is assumed that the still image data is read while the moving image data is read at 60 fps (frames per second). When the moving image blanking time of 20 times is required, high-speed continuous shooting of still images cannot be performed, and the continuous shooting of still images is limited to 3 fps.

When a high-speed rate is required as the frame rate of moving image data (for example, 120 fps), one moving image blanking time is further shortened. In order to read all the still image data, a larger number of moving image blanking times are required, and as a result, continuous still image shooting becomes slower.
An object of the present invention is to provide an imaging device capable of acquiring still image data at a higher speed during moving image shooting.

The apparatus of the embodiment of the present invention is an imaging device capable of acquiring a still image while acquiring a moving image by the imaging element, and a method of reading a frame image related to the moving image from the imaging element. When the coding attribute of the frame image is determined by setting to the first reading method and an instruction for still image shooting is detected during acquisition of the frame image related to the moving image, the frame image related to the moving image is detected. A control means for changing the reading method of the above to a second reading method, a moving image generating means for generating moving image data from the data of the frame image to which the coding attribute is added, and the second reading by the control means. A still image generation means for generating still image data using the data acquired from the imaging element after the method is changed. The control means determines, as the second reading method, a method in which the number of pixels of the frame image related to the generation of the moving image data is reduced as compared with the first reading method. Control is performed to extend the blanking period in the reading, and the still image generating means generates the still image data from the data read in the blanking period.

According to the present invention, it is possible to provide an imaging device capable of acquiring still image data at a higher speed during moving image shooting.

It is a block diagram which shows the structure of the image pickup apparatus which concerns on embodiment of this invention. It is a schematic diagram which shows the structure of the image sensor of an embodiment. It is a schematic diagram which shows a pixel group and a column amplifier / ADC. It is a circuit diagram which shows the structure of one pixel. It is a flowchart explaining the generation method of a standard moving image file. It is a timing chart explaining the generation method of a standard moving image file. It is a timing chart explaining the detailed control corresponding to the pixel configuration of FIG. It is a flowchart explaining the process of 1st to 3rd Embodiment. It is a timing chart explaining the process of 1st Embodiment. It is a timing chart explaining the process of 2nd Embodiment. It is a timing chart explaining the processing of 3rd Embodiment.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. An example of an imaging device capable of appropriately generating moving image data and reading still image data at high speed when a still image shooting instruction is detected during movie shooting is shown.

FIG. 1 is a block diagram showing a main component of an image pickup apparatus 100 capable of capturing moving images and still images according to the present embodiment. The image sensor 101 receives light from a subject imaged by an image pickup optical system (not shown) and outputs an electric signal by photoelectric conversion. A CCD (charge coupling element) type image sensor, a CMOS (complementary metal oxide semiconductor) type image sensor, or the like is applied to the image pickup device 101, and an image signal corresponding to the light receiving sensitivity is output to the light image of the subject. .. As a driving method of the image pickup device 101, there is a global shutter method in which the image pickup device 101 is collectively exposed, the signal charge is temporarily stored in the storage element, and then the signal charges are sequentially read out. Further, there is a slit rolling method in which the image sensor 101 is exposed in one row or a plurality of rows, or one row or a plurality of columns, the signal charge is temporarily stored in the storage element, and then the signal charges are sequentially read out. In this embodiment, an example in which the global shutter method is applied is shown.

The discrimination control unit 102 controls conditions such as the frame pixel size, which is a moving image shooting condition, and the number of frame images per unit time. For example, in a general HDTV (High definition Television), the number of horizontal pixels is 1920 and the number of vertical pixels is 1080 as the frame image size, and the number of frame images per second is a number corresponding to 30 fps or 60 fps. The discrimination control unit 102 outputs a vertical synchronization signal (VSYNC: Vertical Synchronizing signal) 108 and a horizontal synchronization signal (HSYNC: Horizontal Synchronizing signal) to the image sensor 101. For example, when video recording is performed on an HDTV, VSYNC 108 is output to the image sensor 101 as a pulse of 30 or 60 per second, and HSYNC 109 is output as a pulse of 1080. As a result, the discrimination control unit 102 controls the start and stop of moving image shooting using the image sensor 101.

Further, the discrimination control unit 102 controls still image shooting conditions such as still image size, still image resolution, shutter speed, and the like. The discrimination control unit 102 controls the start and stop of still image shooting by the image sensor 101 by changing VSYNC 108 and HSYNC 109 for the image sensor 101 based on the still image shooting conditions. The discrimination control unit 102 determines whether or not a start instruction for still image shooting has been generated during movie shooting. Depending on the determination result, reset control of VSYNC 108 and HSYNC 109 is performed, and still image shooting becomes possible during moving image shooting.

The moving image discrimination unit 103 sequentially acquires the frame image signals based on the output of the image pickup device 101, and discriminates the compression method for the frame image signals. That is, it is determined whether the in-frame compression code or the inter-frame prediction compression code should be used, and a moving image file having an optimum amount of information can be constructed. The moving image discrimination unit 103 discriminates the compression method for the frame image signal of the moving image file even when the discriminating control unit 102 detects a start instruction for still image shooting during the moving image shooting. With respect to the frame image signal of the moving image file, it is determined whether to use the specified in-frame compression code or the inter-frame prediction compression code, and the moving image file having the optimum amount of information can be configured.

The frame buffer 104 is a storage device capable of storing the frame image signal obtained from the image pickup device 101 according to the instruction of the moving image discrimination unit 103, and stores and holds the image signals of a plurality of frames. The moving image generation unit 105 acquires the frame image signal stored in the frame buffer 104 based on the instruction of the moving image discrimination unit 103, and generates a moving image file. For example, a compression method based on ISO / IEC 14496-10 "MPEG-4 Part10 Advanced Video Coding" is applied to generate a GOP consisting of I frames, P frames, and B frames that compose a video file.

The still image generation unit 106 generates a still image file from a frame image signal based on the output of the image sensor 101. For example, a still image file is generated by applying a compression method by ISO / IEC 10918-1 "Joint Photographic Experts Group".

The recording medium 107 records the data of the moving image file and the still image file generated by the moving image generation unit 105 and the still image generating unit 106, respectively. Any recording device can be used for the recording medium 107 for storing data.

FIG. 2 is a schematic view showing a main configuration of the image sensor 101. The pixel group 201 constitutes a pixel array with a plurality of unit pixels. A unit pixel includes a single or a plurality of photodiodes (hereinafter, also referred to as PD), a storage element for storing the output of the photodiode, and the like. The PD constitutes a photoelectric conversion unit that receives light from a subject and outputs a pixel signal according to the light receiving sensitivity.

The timing control circuit 202 controls the operation timing of the pixel group 201 and the operation timing of the column amplifier / ADC (analog / digital converter) 203, the sensor output unit 204, and the like. The timing control circuit 202 controls the operation timing based on signals such as VSYNC (108) and HSYNC (109).

The column amplifier / ADC 203 performs analog-to-digital conversion after reading a signal from the pixel group 201 row by row. Further, the column amplifier / ADC 203 has a function of adding and reading signals of a plurality of columns or a plurality of rows of the pixel group 201 to perform AD conversion processing. The sensor output unit 204 outputs the signal digitized by the column amplifier ADC 203 to the moving image discrimination unit 103 and the still image generation unit 106 as a frame image signal.

FIG. 3 is a diagram schematically showing a part of the pixel group 201 in the range of 8 rows and 8 columns. As a general still image, the aspect ratio of aspect ratio is 2: 3. Inexpensive video and still image imagers have a total pixel count of about 24 million (length 4000 pixels x width 6000 pixels), and expensive video and still image imagers have a total pixel count of about 50 million (length 5800 pixels x). It is about 8600 pixels in width). In the present embodiment, the total number of pixels of the latter (horizontal pixel number 8600 × vertical pixel number 5800) is used.

The pixels 301 shown in FIG. 3 form a pixel group 201, and each of them is connected to the vertical read signal line VL_n (302) for each row. VL_n represents the vertical read signal line corresponding to the nth column. For example, the pixel 301 in the first column is connected to VL_1 (302) (Vertical Line 1), and the pixel 301 in the second column is connected to VL_2 (302). It is connected. Since the basic functions of VL_1 to VL_n are the same, the same reference numerals 302 are added.

Further, each of the pixels 301 is connected to the line selection signal line HL_SEL_m (303) line by line. HL_SEL_m represents a line selection signal line corresponding to the mth line. For example, pixel 301 in the first line is connected to HL_SEL_1 (303) (Horizontal Line Select 1), and pixel 301 in the second line is HL_SEL_2 (303). It is connected to the. Since the basic functions of HL_SEL_1 to HL_SEL_m are the same, the same reference numeral 303 is added.

As an example, when HL_SEL_1 (303) is selected, the signal of the first line of the pixel group 201 is read by the vertical read signal lines VL_1 (302) to VL_n (302). The column amplifier / ADC 203 converts the read signal into a digitized signal. Further, in the state where the signal of the pixel in the mth row of the pixel group 201 is read by the HL_SEL_m (303) to the vertical read signal lines VL_1 (302) to VL_n (302), the column amplifier ADC203 has a plurality of columns vertically. The signal from the read signal line is added and digitized. That is, a horizontal addition function that adds and converts a plurality of column signals is realized.

Further, by controlling a plurality of HL_SEL_m (303) at the same time, the signals of the pixels of the plurality of lines in the pixel group 201 are added to the vertical read signal lines VL_1 (302) to VL_n (302) and read. In this state, the column amplifier ADC203 realizes a vertical addition function that converts a signal from the vertical read signal line into a digitized signal. Each of the pixels 301 is also connected to various control signal lines for operation control, which will be described later.

FIG. 4 is a circuit diagram showing a configuration example of one pixel 301. When light from the subject is incident on the PD 401 included in the pixel 301, photoelectric conversion into an electric signal is performed according to the light receiving sensitivity. MOV_MEM402 is a storage element that stores the electrical signal of PD401 as a moving image signal, and is connected to PD401 via a transfer transistor Tr1. The STL_MEM403 is a storage element that stores the electrical signal of the PD401 as a still image signal, and is connected to the PD401 via a transfer transistor Tr2.

The transfer transistor Tr1 provided between PD401 and MOV_MEM402 transfers the electric signal of PD401 to MOV_MEM402 by controlling with the signal transfer signal TX_MOV_PM (404), and the moving image signal is stored.

The transfer transistor Tr3 is provided between MOV_MEM402 and a floating diffusion (hereinafter abbreviated as FD) unit 408 for output. The transfer transistor Tr3 transfers the moving image signal held in the MOV_MEM402 to the FD unit 408 by controlling it with the signal transfer signal TX_MOV_MF (405).

Similarly, the transfer transistor Tr2 provided between the PD401 and the STL_MEM403 transfers the electrical signal of the PD401 to the STL_MEM403 by controlling with the signal transfer signal TX_STL_PM (406), and the still image signal is stored. The transfer transistor Tr4 is provided between the STL_MEM403 and the FD unit 408. The transfer transistor Tr4 transfers the still image signal held in the STL_MEM403 to the FD unit 408 by controlling it with the signal transfer signal TX_STL_MF (407).

The signal transferred to the FD unit 408 by the signal transfer signal TX_MOV_MF (405) or the signal transfer signal TX_STL_MF (407) is output via the transistors Tr5 and Tr6. The transistor Tr5 is an amplification transistor, and its gate is connected to the FD unit 408. The transistor Tr6 is a selection transistor, and outputs a moving image signal or a still image signal to the vertical read signal VL_n (302) by the row selection signal HL_SEL_m (303).

The transistors Tr7 and Tr8 are reset transistors for setting the PD401 and FD unit 408 to the initialization potential. These are initialized by the reset signal RES (409) before acquiring the electric signal according to the subject and the moving image signal and the still image signal. The transistors Tr5, Tr7, and Tr8 are connected to a power supply line having a voltage of VCC, and a switching operation is performed by their respective control signals.

Next, a standard moving image file generation process will be described with reference to FIGS. 1 to 4 and FIGS. 5 to 7. FIG. 5 is a flowchart for explaining the processing method, and FIGS. 6 and 7 are timing charts showing processing examples. The pixel group 201 of the present embodiment has a total number of pixels of about 50 million (horizontal pixel number 8600 × vertical pixel number 5800). A processing example of generating a moving image file having a quantization bit rate of 12 bits and a frame number of 1 second corresponding to 60 fps from pixels having a predetermined number of pixels (horizontal pixel number 1920 x vertical pixel number 1080) among all pixels. Will be explained. However, the number of pixels, the number of frames, and the like are examples and are not limited to this example.

The method of reading the moving image signal is shown below.
-First moving image signal reading method For example, it is a method of adding and reading each signal of 4 pixels in the horizontal direction and 4 pixels in the vertical direction. In this addition reading, when light from the subject is applied to a pixel group having a predetermined number of pixels (horizontal pixel number 7920 x vertical pixel number 4320) among the pixel group 201, an electric signal corresponding to the light receiving sensitivity of each PD is generated. It is output. By adding signals in the horizontal direction and the vertical direction, respectively, a pixel signal with a reduced number of pixels (horizontal pixel number 1920 x vertical pixel number 1080) can be used as a moving image signal.

-Second moving image signal reading method For example, it is a method of reading a signal of only one pixel for every four pixels in the horizontal direction and every four pixels in the vertical direction. In this thinning-out reading, light from the subject is applied to pixels having a predetermined number of pixels (horizontal number of pixels 7920 x vertical number of pixels 4320) among the pixel group 201. An electric signal corresponding to the light receiving sensitivity is output from each PD of the pixel group discrete for each of the horizontal 4 pixels and the vertical 4 pixels, and the pixel signal with the reduced number of pixels (horizontal pixel number 1920 x vertical pixel number 1080) is a moving image signal. Can be used for.

-Third moving image signal reading method This method is uncountable reading. For example, in the pixel group 201, light from the subject is irradiated to pixels having a predetermined number of pixels (horizontal pixel number 1920 × vertical pixel number 1080) corresponding to the central portion of the screen. An electric signal corresponding to the light receiving sensitivity of each PD is output and can be used as a moving image signal.

In the first moving image signal reading method (additional reading), 16 pixels of 4 horizontal pixels and 4 vertical pixels are treated as a moving image signal corresponding to 1 pixel, so that it is suitable for high-sensitivity shooting, but the image as a whole is It has the aspect of low clarity. The image data acquired by the additive reading is enlarged. If a given resolution cannot be obtained, it is excluded from the application of I-frame (in-frame compression coding).

The second moving image signal reading method (thinning-out reading) treats only 1/16 pixel signal for every 4 horizontal pixels and 4 vertical pixels as a moving image signal, so it is not suitable for high-sensitivity shooting and is not read. The overall image resolution is reduced by the number of pixels. However, the signal reading is simpler than the first moving image signal reading method.

The third moving image signal reading method (non-additive reading) handles the moving image signal for each pixel, and is not suitable for high-sensitivity shooting, but has an aspect that the image becomes clear as a whole. Since only the pixel in the center of the screen is used in the pixel group 201, the angle of view of the pixel group 201 and the angle of view of the obtained moving image do not match. On the other hand, the signal reading is simpler than the first and second moving image signal reading methods.

For the purpose of applying the present invention, in the method of generating a moving image file, any of additive reading, thinning reading, and non-additive reading may be used. In the example of this embodiment, the first moving image signal reading method (additional reading) will be described as a premise.

A standard method for generating a moving image file will be described in detail with reference to FIGS. 5 and 6. FIG. 6 shows an example of a frame image of a moving image file generated by the moving image generation unit 105. The case where one I frame image, one P frame image, and two B frame images, a total of four frame images, constitutes one GOP is illustrated. VSYNC (108) is a synchronization signal indicating the start of each frame image of the moving image file, and corresponds to the number of frame images of the moving image file generated per second. The discrimination control unit 102 outputs VSYNC (108) to the image sensor 101. In this embodiment, the cycle of VSYNC (108) is treated as a cycle corresponding to the frame rate of 60 fps of HDTV. HSYNC (109) is a synchronization signal indicating the beginning of each horizontal line in the frame image, and corresponds to the vertical resolution of the frame image. The discrimination control unit 102 outputs HSYNC (109) to the image sensor 101. HSYNC (109) of this embodiment is a signal corresponding to the number of vertical pixels 1080 in the case of HDTV.

MOV_MEM (402) in FIG. 6 shows the operation of the global shutter system in which the image sensor 101 is collectively exposed, the signal is temporarily stored, and then the signals are sequentially read out. Each video frame is updated in sync with VSYNC (108). Below HSYNC (109) are the signals TX_MOV_PM (404) and TX_MOV_MF (405). The other symbols have already been explained in FIG. The details of FIG. 6 will be described later with reference to FIG. 7.

In the flowchart of FIG. 5, first, the determination control unit 102 sets the moving image shooting condition and the still image shooting condition (STEP 501). For example, the number of frame images per second is set to 60 as a moving image shooting condition. Next, the discrimination control unit 102 determines whether or not an instruction to start moving image shooting has occurred (STEP 502). If it is determined that the movie shooting start instruction has been generated, the process proceeds to STEP503, but the STEP502 determination process is repeated while the movie shooting start instruction is not generated.

In STEP503, a frame image is acquired from the image sensor 101 based on the synchronization signals (VSYNC and HSYNC). A total of four frame images, one I frame image, one P frame image, and two B frame images, constitutes one GOP. The moving image discrimination unit 103 gives a frame attribute to the frame image, adds the attribute data, and sequentially stores the data in the frame buffer 104 (STEP 504). The frame attribute in this case represents a coding attribute, that is, what kind of compression method is used to encode the frame image data. In this embodiment, I, P, and B are exemplified as frame attributes. The coded attribute data represents different attributes depending on the frame, such as coding in the intra-frame compression process and coding in the inter-frame compression process. However, this is an example of a coding attribute, and the present invention can be applied to an embodiment in which processing is performed by various methods such as inter-frame prediction and complementation according to frames. The frame image data stored in the frame buffer 104 is processed based on the frame attribute data by the moving image discrimination unit 103.

Next, the moving image generation unit 105 executes a moving image compression coding process (STEP 505). In the case of a frame image determined to be an I (Intra picture) frame, the moving image generation unit 105 executes an in-frame compression process. That is, the in-frame compression coding process for the I frame, which can be encoded and decoded independently of other frames, is performed. Further, in the case of a frame image determined to be a P (Predictive Picture) frame, the moving image generation unit 105 performs inter-frame compression coding processing related to the P frame that achieves a high compression rate by using the difference information from the I frame. .. In the case of a frame image determined to be a B (Bidirectionally Predictive Picture) frame, the moving image generation unit 105 performs inter-frame compression processing. That is, the processing of the compression coding between the previous and next frames related to the B frame that achieves a high compression rate by using the difference information from the I frame and the P frame is executed. The data of the moving image file thus generated by the moving image generation unit 105 is stored in the recording medium 107 (STEP 506).

FIG. 7 is a timing chart showing the timing shown in FIG. 6 in detail. Hereinafter, the description will be given according to the configuration of the pixel 301 shown in FIG. The time axis is the left-right axis, and the time T1, T8, and T9 are indicated by the RES (409) signal shown at the top. In addition, the times T0 to T7 are shown from HSYNC (109) to VL_n (302).

When VSYNC (108) and HSYNC (109) are input to the image sensor 101 at time T1, the PD401 and FD unit 408 of pixel 301 are initialized by the RES (409) signal (reset signal). Subsequently, at time T2, the TX_MOV_MF (405) signal is enabled to transfer the signal from MOV_MEM402, which stores the moving image signal, to the FD unit 408.

At time T3, by enabling the HL_SEL_1 (303) signal, a signal is output from the FD unit 408 of the pixel 301 in the first row of the pixel group 201 to the vertical read signal lines VL_1 (302) to VL_n (302). The period during which the HL_SEL_1 (303) signal is enabled is the period from time T3 to time T5, which is shorter than the period from time T1 to time T6 corresponding to the time interval of HSYNC (109). The time during which the HL_SEL_1 (303) signal is enabled is set to a time sufficient for the column amplifier / ADC 203 to digitize the moving image signal output to the vertical read signal line VL_n (302).

The maximum value of the HSYNC time interval in the present embodiment is determined by the number of vertical pixels 1080 and the frame rate of the moving image file of 60 fps. Further, the minimum value of the time interval of HSYNC (109) is determined from the operating frequency of the column amplifier ADC203 for performing the quantization of the moving image signal in 12 bits. Here, an example when the operating frequency of the column amplifier / ADC 203 is 1 GHz is shown in the following equation.
Equation 1: Maximum value calculation formula Frame rate / number of vertical pixels = 1/60/1080 ≒ 15.4 μs (microseconds)
Formula 2: Calculation formula for the minimum value
1 / Operating frequency of column amplifier / ADC203 x Number of quantization = 1/1 1GHz x 2 ^ 12 ≒ 4.1μs
The setting range of the time interval of HSYNC (109) is the range from the above minimum value to the maximum value.

The operating frequency of the sensor output unit 204 shown in FIG. 2 is an operating frequency capable of outputting 12 bits for a pixel having 1920 horizontal pixels within the time interval of HSYNC (109).
Equation 3: Calculation formula for operating frequency
1920 x 12 / HSYNC (109) time interval ≒ 1.5 to 5.6 GHz
For example, the time interval of HSYNC (109) in the standard moving image file generation method is 10 μs, and the operating frequency of the sensor output unit 204 is 2.3 GHz.

At time T6 in FIG. 7, HSYNC (109) for reading the pixel 301 in the second row is input to the image sensor 101. At time T7, by enabling the HL_SEL_2 (303) signal, a signal is output from the FD unit 408 of the second row pixel 301 of the pixel group 201 to the vertical read signal lines VL_1 (302) to VL_n (302).

Such an operation is repeated as many times as the number of vertical pixels is 1080. The period from time T5 to time T7 in FIG. 7 is a period in which each signal of the vertical read signal lines VL_1 (302) to VL_n (302) is indefinite. Since this period occurs in synchronization with the horizontal synchronization signal HSYNC, it is generally called a horizontal blanking period, or H blank for short.

When the time interval of HSYNC (109) is 10 μs, the time required to read one frame from the pixel group 201 is as shown in Equation 4 below.
Equation 4: Calculation formula for read time
10 μs x 1080 = 10.8 ms (milliseconds)
The read time is shorter than the VSYNC (108) time interval of about 16.7 ms. That is, during the period of about 5.8 ms, which is the time from the time T8 to the time T9 shown in FIG. 7, the operation of reading the moving image signal from the pixel group 201 is unnecessary. Therefore, it is possible to temporarily stop the operation of the image sensor 101 to save power. Since the period from time T8 to time T9 occurs in synchronization with the vertical synchronization signal VSYNC, it is generally called a vertical blanking period or V blank for short.
After finishing the description of the standard moving image file generation process, the still image file generation process during the generation of the moving image file will be described in each of the following embodiments.

[First Embodiment]
Next, a process of generating a still image file during generation of a moving image file in the imaging device according to the first embodiment of the present invention will be described with reference to FIGS. 1 to 4, 8 and 9. In the present embodiment, when a moving image file corresponding to a predetermined number of pixels (horizontal pixel number 1920 x vertical pixel number 1080), the number of quantization bits of each pixel is 12 bits, and the number of frames per second is 60 fps is generated. In addition, it is possible to generate a still image file. The still image file is generated by a pixel group having a total number of pixels of about 50 million pixels (horizontal pixel number 8600 x vertical pixel number 5800) with the number of quantization bits of each pixel being 14 bits. However, the number of pixels, the number of frames, and the like are examples and are not limited thereto. Further, the description will be made on the premise of the first moving image signal reading method (additional reading).

FIG. 8 is a flowchart illustrating the processing of the present embodiment. FIG. 9 is a timing chart showing an example of a frame image of a moving image file generated by the moving image generation unit 105. A GOP is composed of a total of four frame images, one I frame image, one P frame image, and two B frame images. An example in which the acquisition timing of the still image occurs during the acquisition of the first B frame image is shown. Hereinafter, differences from the matters described with reference to FIGS. 5 and 6 will be described, and the same description will be omitted. Such an abbreviation method is the same in the embodiments described later.

In the present embodiment, HSYNC (109) is a signal corresponding to the vertical pixels of the moving image file and the vertical pixels of the still image file. MOV_MEM (402) in FIG. 9 shows the operation of the global shutter method, and STL_MEM (403) shows the operation of acquiring a still image signal. Further, in FIG. 9, the signal TX_STL_PM (406) and the signal TX_STL_PM (407) are added. FIG. 9 shows an example in which the still image signal is stored in STL_MEM (403) while the signal of the moving image frame of B (1) is stored in MOV_MEM (402). Until the still image signal is stored in STL_MEM (403), it is the same as the standard moving image file generation method described above. After the still image signal is stored in STL_MEM (403), the method of reading the moving image file is changed in order to read the still image signal from STL_MEM (403).

Since the processes of STEP701, STEP702, and STEP704 in FIG. 8 are the same as the processes of STEP501, STEP502, and STEP503 in FIG. 5, their explanations are omitted. The vertical blanking period that occurs when the moving image generation unit 105 generates a moving image file is about 5.8 ms, which is the same as the standard moving image file generation method described above.

When the start instruction of movie shooting is detected in STEP702, the process proceeds to STEP703. The discrimination control unit 102 determines the occurrence of an instruction for still image shooting (STEP 703). The user can give an instruction to shoot a still image by using an operation unit (not shown). If it is determined that the still image shooting instruction has not been generated, the process proceeds to STEP704, and if it is determined that the still image shooting instruction has been generated, the process proceeds to STEP708.

When an instruction to shoot a still image is issued, a process of generating a still image file having a quantization bit rate of 14 bits and a total pixel number of about 50 million pixels (horizontal pixel number 8600 x vertical pixel number 5800) is executed. In the reading of the still image signal performed by using the column amplifier ADC203 having an operating frequency of 1 GHz, the time obtained from the following equation 5 is the minimum time required for reading.
Equation 5: Calculation formula for read time
1/1 GHz x 2 ^ 14 x 5800 ≒ 95ms

In the present embodiment, a process of reading a still image signal is performed using a vertical blanking period that occurs when a moving image file is generated. The number of frames is calculated by the following formula 6.
Equation 6: Calculation formula for the number of frames
95 / 5.8 ≒ 16.4 frames That is, the still image signal is read out using the vertical blanking period of 17 frames of the moving image file. As a result, the time required for actually reading the still image signal and generating the still image file is calculated by the following equation 7.
Equation 7: Calculation formula for required time
17 x 16.7 ≒ 284ms

In this calculation example, a very long time is required to process the still image file. Therefore, when a still image file is generated during the generation of the moving image file, a process of changing the reading method of the frame image related to the moving image file is performed. As a result, the vertical blanking period that occurs when the moving image file is generated can be lengthened.

If an instruction to shoot a still image is generated during movie shooting (STEP 703), the process of changing the reading method of the movie file is executed (STEP 708). In the first moving image signal reading method described above, a method of adding and reading each of the horizontal 4 pixel and vertical 4 pixel signals has been described. In the process of changing the reading method of the moving image file (STEP708), the addition reading of the vertical 4 pixels is changed to the addition reading of the vertical 8 pixels, so that the number of reading lines from the pixel group 201 can be reduced by half. As a result, the read time of one frame from the pixel group 201 is calculated by the following equation 8.
Equation 8: Calculation formula for read time
10 μs x 540 = 5.4 ms

The vertical blanking period in one frame is calculated by the following formula 9.
Equation 9: Calculation formula for vertical blanking period
16.7 − 5.4 = 11.3ms
Therefore, it is possible to extend the vertical blanking period compared to the standard 5.8 ms case. The number of frames of the still image signal read out using the expanded vertical blanking period is calculated by the following equation 10.
Equation 10: Calculation formula for the number of frames
95 / 11.3 ≒ 8.4 frames In other words, the still image signal can be read using the vertical blanking period of 9 frames of the moving image file. As a result, the time required to read the actual still image signal and generate the still image file is calculated from the following equation 11.
Equation 11: Calculation formula for required time
9 x 16.7 ≒ 150ms
Therefore, the processing time of the still image file can be shortened compared to the standard time required of 284 ms.

The frame image acquisition process is started in STEP 709 of FIG. 8, and the frame image is additionally read and read in STEP 710. A moving image frame with a predetermined number of pixels (1920 horizontal pixels x 540 vertical pixels) read by adding 4 horizontal pixels and 8 vertical pixels from the pixel group 201 is when the still image signal is not read. In comparison, the vertical resolution is halved.

As shown in FIG. 9, while the moving image frame of B (1) is stored in MOV_MEM (402), the still image signal is stored in STL_MEM (403). That is, the moving image frame of B (1) is encoded by referring to the moving image frames before and after the frame. Therefore, even if the vertical resolution of the video frame of B (1) is halved, the data of the video frame of B (1) is generated by performing appropriate complementation.

Further, as shown in FIG. 9, the reading of the still image signal is performed over a plurality of moving image frames. FIG. 9 illustrates a state in which a signal for a still image is read out in parallel with the signal reading of the moving image frame (B1) and the moving image frame (B2). The still image acquisition process is started in STEP711 of FIG. 8, and the still image data is read out in STEP712. As described above, the reading time of the still image is 95 ms (Equation 5), and the length of the vertical blanking period of the moving image file is 11.3 ms (Equation 9). In this case, a process of reading the still image signal is performed in parallel with reading at least 9 frames of the moving image file.

The read moving image signal is stored in the frame buffer 104 in STEP705, and then proceeds to the processing of STEP706 and STEP713. In STEP706, the moving image compression coding process is executed, and in STEP713, the still image compression coding process based on the still image signal from the image sensor 101 is executed. These are processed in parallel.

Each data of the moving image file generated by the moving image generation unit 105 and the still image file generated by the still image generating unit 106 is stored in the recording medium 107 (STEP 707).

The addition number of vertical pixels described in the present embodiment is an example, and is not limited thereto. In addition, an example of reading a still image signal and recording a still image file during recording of a B frame of a moving image file has been described, but a P frame that achieves a high compression ratio using the difference information from the I frame has been described. Is also applicable. In addition, when recording still image data while recording an I frame that can be encoded and decoded independently of other frames, the I frame is replaced with a P frame or a B frame for encoding processing. Is executed. This makes it possible to adapt the frame structure of an appropriate video file. Alternatively, when recording still image data during the recording of an I frame in a moving image file, the recording process is performed after the processing for the I frame is completed. The above items are the same in the embodiments described later.

In the present embodiment, a moving image file corresponding to a frame number of 60 fps per second is generated from each pixel signal having a quantization bit rate of 12 bits by a pixel group having a predetermined number of pixels (horizontal pixel number 1920 x vertical pixel number 1080). .. The method of generating a moving image file is changed when a still image shooting instruction is issued during movie shooting. A still image file can be generated from each pixel signal having a quantization bit rate of 14 bits by a pixel group 201 having a total pixel count of about 50 million (horizontal pixel count 8600 x vertical pixel count 5800). That is, if an instruction for still image shooting is generated during moving image shooting in which additional reading of horizontal 4 pixels and vertical 4 pixels is performed, the moving image signal reading method is switched to additional reading of horizontal 4 pixels and vertical 8 pixels, and the pixel group 201 starts from The number of lines read is halved. As a result, the reading time of the still image signal can be shortened.
According to the present embodiment, the moving image reading method can be adaptively changed when a still image is acquired during movie shooting, and a sufficient moving image blanking time associated with moving image reading can be secured.

[Second Embodiment]
Next, a second embodiment of the present invention will be described. In the present embodiment, the method of generating a moving image file is changed when a still image shooting instruction is issued during movie shooting in which additional reading of horizontal 4 pixels and vertical 4 pixels is performed, which is the first moving image signal reading method. .. By performing additional reading of 8 horizontal pixels and 4 vertical pixels, the number of reading columns from the pixel group 201 is halved. As a result, the reading time of the still image signal can be shortened.

The still image file generation process during the generation of the moving image file will be described with reference to FIG. The timing chart of FIG. 10 shows an example of a frame image of a moving image file generated by the moving image generation unit 105. When a total of four frame images of one I frame image, one P frame image, and two B frame images constitute one GOP, the still image data acquisition process is performed during the acquisition of the first B frame image. It shows the case to do. HSYNC (109) is a signal corresponding to the vertical pixels of the moving image file and the vertical pixels of the still image file.

The flow of the still image file generation process during the generation of the moving image file is the same as that of FIG. 8 described in the first embodiment. Therefore, only the differences from FIG. 8 will be described. Proceeding from STEP703 to STEP708, the process of changing the moving image signal reading method is performed. That is, the additional reading of the horizontal 4 pixels and the vertical 4 pixels is changed to the additional reading of the horizontal 8 pixels and the vertical 4 pixels, and the number of reading columns from the pixel group 201 is halved. By halving the number of columns of signals read from one row, parallel processing of the column amplifier / ADC203 becomes possible, and the processing time of the column amplifier / ADC203 is halved. As a result, the read time of one frame from the pixel group 201 is calculated by the following equation 12.
Equation 12: Calculation formula for read time
5 μs x 1080 = 5.4 ms

The vertical blanking period in one frame is calculated by the following equation 13.
Equation 13: Calculation formula for vertical blanking period
16.7 − 5.4 = 11.3ms
Therefore, it is possible to extend the vertical blanking period compared to the standard 5.8 ms case. The number of frames of the still image signal read out using the expanded vertical blanking period is calculated by the following equation 14.
Equation 14: Calculation formula for the number of frames
95 / 11.3 ≒ 8.4 frames That is, the still image signal is read out using the vertical blanking period of 9 frames of the moving image file. As a result, the time required to read the actual still image signal and generate the still image file is calculated from the above equation 11.
9 x 16.7 ≒ 150ms
Is calculated. It is possible to shorten the processing time of the still image signal compared to the standard time required of 284 ms.

Further, a moving image frame having a predetermined number of pixels (960 horizontal pixels x 1080 vertical pixels) read from the pixel group 201 by adding 8 horizontal pixels and 4 vertical pixels is when the still image signal is not read. In comparison, the horizontal resolution is halved. As shown in FIG. 10, while the signal of the moving image frame of B (1) is stored in MOV_MEM402, the still image signal is stored in STL_MEM403. That is, in the moving image frame B (1), the coding process is performed with reference to the moving image frames before and after the frame. Therefore, even if the horizontal resolution of the moving image frame B (1) is halved, the data of the moving image frame B (1) is generated by performing appropriate complementation.

As shown in FIG. 10, the signal reading for the still image is performed over a plurality of moving image frames. FIG. 10 illustrates how a still image signal is read in parallel with reading the moving image frame (B1) and the moving image frame (B2). When the still image read time is 95 ms and the vertical blanking time for the video file is 11.3 ms, the still image signal is read out in parallel with reading at least 9 frames of the video file (Fig. 8: STEP711). And 712).

In the present embodiment, an example of changing the addition of 4 horizontal pixels to the addition of 8 horizontal pixels by the moving image reading method has been described, but the number of additions of horizontal pixels is not limited to this.

[Third Embodiment]
Next, a third embodiment of the present invention will be described. In the present embodiment, the method of generating a moving image file is changed when a still image shooting instruction is issued during movie shooting in which additional reading of horizontal 4 pixels and vertical 4 pixels is performed, which is the first moving image signal reading method. .. By performing additional reading of 8 horizontal pixels and 8 vertical pixels, the number of rows read from the pixel group 201 is halved, and the number of columns read from the pixel group 201 is halved. As a result, the reading time of the still image signal can be further shortened.

A method of generating a still image file during generation of a moving image file in the present embodiment will be described with reference to FIGS. 1 to 4, 8 and 11. In FIG. 11, the difference from FIGS. 9 and 10 is shown in the timing of the signal output to the vertical read signal line VL_n (302).

The flow of the still image file generation process during the generation of the moving image file is the same as that of FIG. 8 described in the first embodiment. Therefore, only the differences from FIG. 8 will be described. Proceeding from STEP703 to STEP708, the process of changing the moving image signal reading method is performed. That is, the addition of 4 horizontal pixels is changed to the addition of 8 horizontal pixels, and the addition of 4 vertical pixels is changed to the addition of 8 vertical pixels, and the signal is read out. The number of read rows from the pixel group 201 is halved, and the number of read columns is halved. That is, in the present embodiment, the addition method shown in the first embodiment and the addition method shown in the second embodiment are used in combination and are applied at the same time. As a result, the read time of one frame from the pixel group 201 is calculated by the following equation 15.
Equation 15: Calculation formula for read time
5 μs x 540 = 2.7 ms

The vertical blanking period in one frame is calculated by the following equation 16.
Equation 16: Calculation formula for vertical blanking period
16.7 − 2.7 = 14ms
Therefore, it is possible to extend the vertical blanking period compared to the standard 5.8 ms case. The number of frames of the still image signal read out using the expanded vertical blanking period is calculated by the following equation 17.
Equation 17: Calculation formula for the number of frames
95/14 ≒ 6.8 frames In other words, the still image signal can be read using the vertical blanking period of 7 frames of the moving image file. As a result, the time required for actually reading the still image signal and generating the still image file is calculated by the following equation 18.
Formula 18: Calculation formula for required time
7 x 16.7 ≒ 117ms
The processing time of the still image signal can be shortened compared to the standard time required of 284 ms.

In the present embodiment, a moving image frame having a predetermined number of pixels (number of horizontal pixels 960 × number of vertical pixels 540) is acquired from the pixel group 201 by adding and reading 8 horizontal pixels and 8 vertical pixels. That is, the resolutions in the horizontal direction and the vertical direction are halved as compared with the case where the still image signal is not read. As shown in FIG. 11, while the signal of the moving image frame of B (1) is stored in MOV_MEM402, the still image signal is stored in STL_MEM403. That is, in the moving image frame B (1), the coding process is performed with reference to the moving image frames before and after the frame. Therefore, even if the resolutions in the horizontal direction and the vertical direction of the moving image frame B (1) are both halved, appropriate complementation is performed to generate the moving image frame B (1).

Further, as shown in FIG. 11, the reading of the still image file is performed over a plurality of moving image frames. FIG. 11 illustrates a state in which a still image signal is read in parallel with reading the moving image frame (B1) and the moving image frame (B2). When the still image read time is 95 ms and the vertical blanking time for the moving image file is 14 ms, the still image signal reading process is performed in parallel with reading at least 7 frames of the moving image file (Fig. 8: STEP711 and STEP711 and). See 712).

In the present embodiment, an example has been described in which the addition of 4 horizontal pixels is changed to the addition of 8 horizontal pixels and the addition of 4 vertical pixels is changed to the addition of 8 vertical pixels by the moving image signal reading method. The number of additions of horizontal pixels and the number of additions of vertical pixels are not limited to this.

In the above embodiment, it is possible to acquire still image data while acquiring moving image data formed by a plurality of frame images based on signals from an image sensor. It is determined whether or not an instruction to shoot a still image is detected during movie shooting, and the time required to acquire one frame image for generating a movie file and the method of generating a movie frame image are controlled by the judgment result. Will be done. By adaptively changing the moving image reading method, it is possible to secure a long moving image blanking period associated with moving image reading. That is, since the reading time of the still image data can be sufficiently secured, the moving image data can be encoded and generated at an appropriate timing without interrupting the generation of the moving image data during the still image shooting during the moving image data generation. According to the above embodiment, an imaging device capable of performing still image shooting (acquisition of still image data) at a higher speed without causing loss of moving image data during the still image shooting period and during moving image shooting. Can be provided.
Although the preferred embodiment of the present invention has been described above, the present invention is not limited to the above embodiment, and various modifications and modifications can be made within the scope of the gist thereof.

101 Image sensor 102 Discrimination control unit 103 Moving image discrimination unit 104 Frame buffer 105 Movie generation unit 106 Still image generation unit

Claims (10)

An image pickup device capable of acquiring a still image while a moving image is acquired by an image sensor.
The method for reading the frame image related to the moving image is set to the first reading method for the image sensor to determine the coding attribute of the frame image, and the frame image related to the moving image is stationary during acquisition. When an instruction to take an image is detected, a control means for changing the reading method of the frame image related to the moving image to the second reading method, and
A moving image generation means for generating moving image data from the frame image data to which the coding attribute is added, and
A still image generating means for generating still image data using the data acquired from the image sensor after the control means has changed to the second reading method.
The control means determines, as the second reading method, a method in which the number of pixels of the frame image related to the generation of the moving image data is reduced as compared with the first reading method. Controls to extend the blanking period in reading,
The still image generating means is an imaging device characterized in that the still image data is generated from the data read during the blanking period. The control means determines the coding attribute for each frame image, reads the data from the image pickup device by the first and second reading methods, and obtains the data of the frame image to which the coding attribute is added. Control the storage means to store
The imaging apparatus according to claim 1, wherein the moving image generating means generates the moving image data using the data acquired from the storage means. As the second reading method, the control means determines a method of adding and reading data of a plurality of rows or columns, or a plurality of rows and a plurality of columns to a frame image related to the moving image. The imaging apparatus according to claim 1 or 2. The coding attribute is an attribute that determines whether to perform in-frame compression processing or inter-frame compression processing on the frame image.
The imaging device according to claim 3, wherein the moving image generating means performs compression processing for each frame image according to the coding attribute. The control means said the frame when an instruction to shoot a still image is detected during acquisition of a frame image related to the moving image and the coding attribute of the frame related to the generation of the moving image represents inter-frame compression. The imaging apparatus according to claim 4, wherein the image reading method is changed to the second reading method. When the coding attribute related to the first frame represents the intra-frame compression of the image of the frame, and the coding attribute related to the second frame represents the inter-frame compression of the image of the frame.
The control means said when the instruction for still image shooting is detected during the acquisition of the frame image related to the moving image and the frame image related to the generation of the moving image is the image of the first frame. The image pickup apparatus according to claim 4, wherein the first frame is replaced with the second frame to control the compression process. The control means is concerned when an instruction for still image shooting is detected during acquisition of a frame image related to the moving image and the coding attribute of the frame related to the generation of the moving image represents intra-frame compression. The image pickup apparatus according to claim 4, wherein the still image generation means controls the generation of still image data after the processing of encoding the frame image is completed. When the instruction to shoot a still image is detected during the acquisition of the frame image related to the moving image and the still image data is read out during the generation of the moving image data, the moving image generating means has the image data of a plurality of frames. The imaging apparatus according to any one of claims 1 to 7, wherein the moving image data is generated by complementing with the above. The control means according to any one of claims 1 to 8, wherein as the first reading method, a method of reading moving image data from a part of a pixel group of the image pickup element is determined. The image sensor described. A control method executed by an image pickup device capable of acquiring a still image while a moving image is acquired by an image sensor.
The method for reading the frame image related to the moving image is set to the first reading method for the image sensor to determine the coding attribute of the frame image, and the frame image related to the moving image is stationary during acquisition. When an instruction to take an image is detected, a control step of changing the reading method of the frame image related to the moving image to the second reading method, and
A moving image generation step of generating moving image data from the frame image data to which the coding attribute is added, and
It has a still image generation step of generating still image data using the data acquired from the image sensor after the change to the second reading method.
In the control step, as the second reading method, a method of reducing the number of pixels of the frame image related to the generation of the moving image data as compared with the first reading method is determined, so that the moving image can be read. Control is performed to extend the blanking period in reading,
A control method for an imaging device, characterized in that, in the still image generation step, the still image data is generated from the data read during the blanking period.

Images