JP6239026B2 - Imaging device and imaging apparatus - Google Patents

Description

  The present invention relates to an imaging element and an imaging apparatus.

  In recent years, an imaging element such as a CMOS used in an imaging apparatus such as a digital camera is capable of capturing a high-resolution image by increasing the number of pixels by miniaturizing the pixels. An image pickup apparatus for consumer is generally provided with a pixel number of 10 million pixels or more (see Patent Document 1).

  FIG. 14 is a configuration block diagram of a general imaging apparatus. In FIG. 14, the image sensor 1500 includes a pixel unit 1501, an AD conversion unit 1502, and a P / S conversion unit 1503. The pixel unit 1501 converts the subject image into an electrical signal and outputs the electrical signal to the AD conversion unit 1502.

  The AD conversion unit 1502 converts the image signal read from the pixel unit 1501 into a digital signal. The P / S conversion unit 1503 performs parallel / serial conversion on the digital signal converted by the AD conversion unit 1502. The image signal processing circuit 1600 performs various signal processes on the image signal from the image sensor 1500.

JP 2013-26675 A

  In the imaging apparatus described above, since the transfer capacity of the transfer path for transferring the image signal from the imaging element 1500 to the image signal processing circuit 1501 is constant, the entire image of the subject is relatively increased by increasing the number of pixels of the imaging element. There is a problem that the signal transfer time becomes long.

  That is, the signal reading speed from the image sensor 1500 to the image signal processing circuit 1501 becomes a bottleneck of the image signal reading speed. Furthermore, when trying to realize high-speed transfer, there are problems such as power consumption, increase in heat generation, data transfer accuracy, and the like of the transfer circuit and processing circuit.

  The present invention provides an imaging device capable of making both a seamless moving image and a high-quality still image compatible when capturing a still image while capturing a moving image using an imaging device having a large number of pixels, and an imaging apparatus including the same. It was made for the purpose of providing.

In order to solve the above-described problems, an imaging device of the present invention includes an imaging unit that receives incident light and performs photoelectric conversion, and a plurality of AD conversions that convert an analog image signal output from the imaging unit into digital image data. Means, storage means for storing digital image data of at least one frame converted by the plurality of AD conversion means, generation means for generating moving image digital image data from still image digital image data, and for the still image Output means for outputting the digital image data and the moving image digital image data to the outside, and configured to transmit the image data in parallel between the plurality of AD conversion means and the storage means. together and, when shooting a moving image and a still image using the imaging means, the digital image data for the still picture the Symbol Stored in the unit and is characterized by being configured so as to be able to output the digital image data for the moving image to the outside earlier than the digital image data for the still image by said output means.

In addition, an imaging unit that receives incident light and performs photoelectric conversion , a plurality of AD conversion units that convert analog image signals output from the imaging unit into digital image data, and at least converted by the plurality of AD conversion units storage means for storing digital image data of one frame, comprising arithmetic means for performing arithmetic processing on the digital image data, and output means for outputting the digital image data to the outside, the, from the plurality of AD conversion means The image data is configured to be transmitted in parallel between the storage unit and the output unit. The output unit outputs the first size image data and the second size image data different from the first size. Yes to when outputting to the outside, and output the image data of the first size to the outside from the stored in the storage means It was constructed such and is characterized in.

Furthermore, an imaging apparatus according to the present invention includes an imaging unit that receives incident light and performs photoelectric conversion, a plurality of AD conversion units that convert analog image signals output from the imaging unit into digital image data, and the plurality of ADs. Storage means for storing at least one frame of digital image data converted by the conversion means, generation means for generating digital image data for moving image from digital image data for still image, digital image data for still image and moving image an imaging device having an output means for outputting the digital image data to the outside, and a signal processing section for performing predetermined signal processing on image data output from the imaging device, and a display unit for displaying images, the imaging element, between the signal processing unit, and a control section for controlling each of the display unit, from said plurality of AD conversion means to said memory means The image data is configured to be transmitted in parallel, wherein, when shooting a moving image and a still image using the imaging means, stores digital image data for the still image in the storage means, The output means outputs the moving image digital image data from the image sensor prior to the still image digital image data.

In addition, an imaging unit that receives incident light and performs photoelectric conversion , a plurality of AD conversion units that convert analog image signals output from the imaging unit into digital image data, and at least converted by the plurality of AD conversion units storage means for storing digital image data of one frame, and calculating means for performing a calculation process on the digital image data, an image pickup device and an output means for outputting the digital image data to the outside, an output from the imaging device a signal processing section for performing predetermined signal processing on the image data, a display unit for displaying the images, the imaging device, the signal processing unit, and a control section for controlling each of said display unit, The image data is configured to be transmitted in parallel between the plurality of AD conversion units and the storage unit, and the control unit is controlled by the output unit. When the image data of a second size different from the image data and the first size of a first size and output from the imaging device, the output image data of the first size from the stored in the storage means It is characterized by doing.

  According to the present invention, there is provided an imaging system capable of obtaining a seamless moving image and a high-quality still image even when still image shooting is performed during moving image shooting using an imaging element having a large number of pixels. can do.

1 is a diagram illustrating a schematic structure of an image pickup element in Embodiment 1. FIG. 3 is a diagram illustrating an example of a data bus configuration in Embodiment 1. FIG. 3 is a diagram illustrating a configuration of a pixel and column ADC block in Embodiment 1. FIG. FIG. 2 is a diagram illustrating a stacked configuration of an image sensor in Example 1. FIG. 3 is a cross-sectional view of an image sensor in Example 1. 1 is a system overview diagram of an imaging system in Embodiment 1. FIG. 3 is a flowchart of a shooting sequence in Embodiment 1. It is a figure which shows the resizing process of the image in an Example. FIG. 6 is a diagram illustrating an overview of image data size change processing in the first embodiment. FIG. 6 is a diagram illustrating an overview of image data size change processing in the first embodiment. 6 is a flowchart of an imaging sequence in Examples 2 and 3. FIG. 10 is a diagram for explaining an overview of image data size change processing in Embodiment 2. FIG. 10 is a diagram for explaining an overview of an image data size change process in Embodiment 3. 1 is a diagram illustrating a configuration of a general imaging device.

Example 1
Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the first embodiment, an imaging system having a shooting mode capable of shooting a still image during moving image shooting will be described. In this embodiment, the image pickup device always performs a shooting operation in the all-pixel readout mode, and shows an example of a method for creating both moving images and still images from the image signals output by driving in the all-pixel readout mode. .

  FIG. 1 is a block diagram showing an outline of an image sensor according to Embodiment 1 of the present invention.

  The imaging element 506 includes a first chip (first semiconductor substrate) 10 and a second chip (second semiconductor substrate) 11, and the second chip 11 and the first chip 10 are mutually connected. Are stacked. The first chip 10 has a pixel portion composed of a plurality of pixels 101 arranged in a matrix, and is arranged on the light incident side (light receiving side of the optical image) with respect to the second chip 11.

  In the pixel portion of the first chip 10, the plurality of pixels 101 arranged in a matrix are connected to the transfer signal line 103, the reset signal line 104, and the row selection signal line 105 for each row, and a plurality of pixels 101 for each column. It is connected to the column output line 102. Each of the plurality of column output lines 102 arranged in each column is connected to pixels arranged in different readout rows in the same column.

  The second chip 11 includes a plurality of AD converters (hereinafter referred to as ADC) 111, a row scanning circuit 112, a column scanning circuit 113, and a timing control circuit 114 provided for each column. Further, the second chip 11 includes a changeover switch 116, a frame memory 117, an in-element calculation unit 118, a parallel / serial conversion unit (hereinafter referred to as a P / S conversion unit) 119, and the like. The timing control circuit 114 is driven and controlled by the overall control calculation unit 509.

  In this manner, the pixel portion is formed in the first chip 10, and the driver circuit, the memory, the arithmetic unit, and the like of the pixel portion are formed in the second chip 11, so that the imaging layer and the circuit layer of the imaging element 506 are formed. The manufacturing process can be divided. In addition, it is possible to achieve high speed, downsizing, and high functionality by thinning and high density wiring in the circuit layer.

  The changeover switch 116 selectively inputs the digital image data of each channel output from the horizontal signal line 115-a and horizontal signal line 115-b provided for each channel to the in-element computing unit 118. The intra-element computing unit 118 rearranges the image data of each channel to generate one frame of image data, and sequentially outputs it to the frame memory 117. The frame memory 117 temporarily stores the output digital image data for at least one frame.

  The in-element calculation unit 118 performs calculation processing such as cutout and thinning on one frame of digital image data stored in the frame memory 117. Details will be described later. One frame of digital image data processed by the in-element calculation unit 118 is subjected to parallel / serial conversion by the P / S conversion unit 119 and is output to the image signal processing circuit 507 outside the image sensor 506.

  Here, the signal transfer path among the horizontal signal line 115-a, the horizontal signal line 115-b, the changeover switch 116, the in-element arithmetic unit 118, and the frame memory 117 is a digital signal line formed in the same chip. . Therefore, it is possible to secure the necessary data bus width and increase the speed so that the transfer of all the horizontal data is completed within the horizontal reading period.

  FIG. 2 is a diagram illustrating an example of a data bus configuration from the ADC 111 to the P / S conversion unit 119 in the second chip 11. As shown in FIG. 2, in the second chip 11, a column memory 111 a that temporarily holds the digital conversion output of the ADC 111 is provided between the ADC 111 and the in-element operation unit 118. In FIG. 2, the selector switch 116 is omitted.

  Image data held in the column memory 111a provided in each column in accordance with a control signal from the column scanning circuit 113 is converted into horizontal signal lines 115-a and 115-b provided in 16 channels in the horizontal transfer circuit 115. Are output in parallel. The image data output to the horizontal signal lines 115-a and 115-b is input to the frame memory 117 via the memory I / F circuit in the in-element arithmetic unit 118.

  For example, a case where image data of 32M pixels of 8K4K (horizontal 8000 pixels, vertical 4000 pixels) is output from the ADC 111 will be described. The required data bus bandwidth when image data of 32 M pixels is read out at a frame rate of 60 fps is 1920 M pixels / sec.

  Here, when the transfer capacity of each of the 16-channel horizontal signal lines 115-a and 115-b provided in the horizontal transfer circuit 115 is 12 bits, it is necessary to reduce the transfer capacity to a transferable frequency of 120 MHz. The column memory is sequentially selected by a control signal from the column scanning circuit 113, and image data of 120M pixels / sec per channel of the horizontal transfer circuit 115 is read in parallel by 16 channels.

  The image data input from the horizontal transfer circuit 115 to the frame memory 117 via the in-element operation unit 118 is partially read out from the frame memory and input to the in-element operation unit 118 again. Is done. For example, the image data output from the frame memory 117 is reduced to an image size of 1/16 times by the reduction scaling circuit in the in-element arithmetic unit 118. The data bus bandwidth required in that case is reduced to 120 Mpixel / sec. This is a data transfer capacity corresponding to the case of reading full HD size (2M pixels) image data at 60 fps.

  The image data output from the in-element operation unit 118 with the data bus band reduced is converted into a serial signal by the P / S conversion unit 119 in a two-channel configuration of 720 Mbps so as not to exceed the maximum serial transfer capacity of 1 Gbps. Is output.

  As described above, by providing the ADC 111, the in-element operation unit 118, and the frame memory 117 in the second chip 11, a wide data bus band necessary for processing image data is secured in the second chip 11, and the ADC 111. The high-speed moving image can be output with the serial transfer capacity that can be transferred to the outside of the image sensor while realizing the high transfer rate from the frame memory 117 to the frame memory 117.

  FIG. 3 is a diagram illustrating a detailed configuration of each pixel 101 and ADC 111 of the pixel portion of the image sensor 506 in the present embodiment. The outline of the operation of the image sensor according to the first embodiment will be described with reference to FIGS. 1 and 3.

  A photodiode (hereinafter referred to as PD) 201 photoelectrically converts received incident light into photocharge (here, electrons) having a charge amount corresponding to the amount of light. The cathode of the PD 201 is electrically connected to the gate of the amplification transistor 204 through the transfer transistor 202. A node electrically connected to the gate of the amplification transistor 204 forms a floating diffusion (hereinafter referred to as FD) unit 206.

  The transfer transistor 202 is provided between the cathode of the PD 201 and the FD unit 206, and is turned on when a transfer pulse φTRG is supplied to the gate via the transfer signal line 103 in FIG. Then, the photoelectric charge photoelectrically converted by the PD 201 is transferred to the FD unit 206.

  The reset transistor 203 is turned on when the drain is connected to the pixel power source Vdd, the source is connected to the FD unit 206, and the reset pulse φRST is supplied to the gate via the reset signal line 104 in FIG. Prior to the transfer of the signal charge from the PD 201 to the FD unit 206, the FD unit 206 is reset by discarding the charge of the FD unit 206 to the pixel power supply Vdd.

  The amplification transistor 204 has a gate connected to the FD unit 206 and a drain connected to the pixel power supply Vdd, and outputs the potential of the FD unit 206 after being reset by the reset transistor 203 as a reset level. Further, the amplification transistor 204 outputs the potential of the FD unit 206 after transferring the signal charge of the PD 201 by the transfer transistor 202 as a signal level.

  For example, the selection transistor 205 has a drain connected to the source of the amplification transistor 204 and a source connected to the column output line 102. When the selection pulse φSEL is applied to the gate via the row selection signal line 105 in FIG. 1, the pixel 101 is turned on, and a signal amplified by the amplification transistor 204 is output to the column output line 102.

  The selection transistor 205 may have a circuit configuration connected between the pixel power supply Vdd and the drain of the amplification transistor 204. As the transistors 202 to 205, for example, N-channel MOS transistors can be used. Further, the pixel 101 is not limited to the configuration including the four transistors described above, and may include a configuration including three transistors in which the amplification transistor 204 and the selection transistor 205 are combined into one transistor. .

  An analog image signal output from the pixel 101 via the column output line 102 is transmitted to the ADC 111. The ADC 111 includes a comparator 211, an up / down counter 212, a memory 213, and a DA converter (hereinafter referred to as DAC) 214.

  The comparator 211 includes a pair of input terminals, one of which is connected to the column output line 102 and the other is connected to the DAC 214. The output terminal of the comparator 211 is connected to the up / down counter 212. The timing control circuit 114 of FIG. 1 outputs a reference signal to the DAC 214 based on a command from the overall control calculation unit 509.

  The DAC 214 outputs a ramp signal whose level changes with time based on the reference signal input from the timing control circuit 114 of FIG. The comparator 211 compares the level of the ramp signal input from the DAC 214 with the level of the image signal input from the column output line 102.

  For example, the comparator 211 outputs a high-level comparison signal when the level of the image signal is lower than the level of the ramp signal, and outputs a low-level comparison signal when the level of the image signal is higher than the level of the ramp signal. Output. The up / down counter 212 counts a period during which the comparison signal is at a high level or a period during which the comparison signal is at a low level. By this counting process, the output signal of each pixel 101 is converted into a digital value.

  Note that an AND circuit may be provided between the comparator 211 and the up / down counter 212, a pulse signal may be input to the AND circuit, and the number of the pulse signals may be counted by the up / down counter 212.

  Further, the ADC 111 may count a count value corresponding to the reset level based on a reset signal when the pixel 101 is reset, and may further count the count value based on an optical signal after a predetermined imaging time has elapsed. Then, the difference value between the count value of the optical signal and the count value of the reset signal may be stored in the memory 213.

  The memory 213 is connected to the up / down counter 212 and stores the count value counted by the up / down counter 212. The count value stored in the memory 213 is transmitted as digital image data to the horizontal signal line 115-a and the horizontal signal line 115-b in FIG. 1 by driving control of the column scanning circuit 113 in FIG.

  FIG. 4 shows an outer configuration of the image sensor 506 according to the first embodiment described with reference to FIG. 4A is a perspective view of the image sensor 506 viewed from the light incident side, and FIG. 4B is a cross-sectional view of the image sensor 506.

  The image sensor 506 includes a first chip (imaging layer) 10 and a second chip (circuit layer) 11. The first chip 10 and the second chip 11 are integrated by electrically connecting a plurality of micropads 302 provided on each chip via a plurality of microbumps 301. That is, the first chip 10 and the second chip 11 are electrically directly connected via the plurality of macro bumps 101 and the plurality of micro pads 302. Note that the first chip 10 and the second chip 11 may be configured to be electrically directly connected by a method using other than the macro pad and the micro pad.

  FIG. 5 shows details of the cross-sectional structure of the image sensor 506 according to the first embodiment shown in FIGS. In FIG. 5, the imaging layer 401 corresponds to the first chip 10, and the circuit layer 402 corresponds to the second chip 11.

  In the imaging layer 401, a wiring layer 404 is formed on a silicon (hereinafter referred to as Si) substrate 403. In the Si substrate 403, an n-type diffusion region 407 to be the PD 201 is formed, and a p + diffusion region 408 is formed in a surface portion of the PD 201 (a boundary portion with the wiring layer 404).

  In the Si substrate 403, a plurality of n + diffusion regions 409 to be FD portions 206 and n + diffusion regions 410 for switching transistors are formed on the surface portion. In the wiring layer 404, a gate wiring 411 and a signal propagation wiring 412 of each transistor are formed in an insulating layer made of SiO 2 or the like, and a micropad 302a made of Cu or the like is formed on the surface thereof.

  The transfer transistor 202, the reset transistor 203, the amplification transistor 204, and the selection transistor 205 are configured by the n + diffusion region 409, the n + diffusion region 410, and the gate wiring 411 of the transistor. In the wiring layer 404, a via 414 for connecting the n + diffusion region 410 to the micropad 302a is formed.

  In the circuit layer 402, a wiring layer 406 is formed on the Si substrate 405. A plurality of transistor diffusion regions 416 are formed on the surface portion of the Si substrate 405. In the wiring layer 406, a plurality of gate wirings 417 and signal propagation wirings 418 of each transistor are formed in an insulating layer made of SiO 2 or the like, and a micropad 302b made of Cu or the like is formed on the surface portion thereof. .

  Various circuits are constituted by the transistor diffusion region 416 formed in the circuit layer 402, the gate wiring 417 of the transistor, the signal propagation wiring 418, and the like. Description of the circuit cross section is omitted. In the wiring layer 406, a via 420 for connecting the diffusion region 416 and the like to the micropad 302b is formed.

  The micropad 302 a formed on the wiring layer 404 of the imaging layer 401 and the micropad 302 b formed on the wiring layer 406 of the circuit layer 402 are electrically connected to each other by a microbump 301. Note that although FIG. 5 illustrates a configuration example in which the imaging layer 401 and the circuit layer 402 are connected using the micro bumps 301 as connection terminals, direct connection without using the micro bumps is also possible.

  FIG. 6 is a system schematic diagram of an image pickup apparatus using the image pickup device described with reference to FIGS. The subject image that has passed through the lens unit 501 is adjusted to an appropriate amount of light by the diaphragm 504 and formed on the imaging surface on the imaging device 506 having the configuration shown in FIGS.

  The subject image formed on the imaging surface on the image sensor 506 is photoelectrically converted by the PD 201 of the image sensor 506, and gain adjustment is performed by an intra-pixel amplifier or a column amplifier provided between the pixel 101 and the ADC 111. Then, A / D conversion processing is performed from an analog signal to a digital signal using the ADC 111, and is taken into the imaging signal processing circuit 507 as digital image signals of R, G, and B colors.

  The imaging signal processing circuit 507 performs various correction processes such as low-pass filter processing for reducing noise and shading correction, image signal processing such as white balance adjustment processing, and image data compression processing. Note that the imaging signal processing circuit 507 for performing these processes may be built in the imaging element 506 having a laminated structure.

  The lens unit 501 is driven by the lens driving unit 502 to control zoom, focus, and the like. The mechanical shutter (mechanical shutter) 503 and the diaphragm 504 are driven and controlled by a shutter / diaphragm driving unit 505.

  The overall control calculation unit 509 performs overall control of the imaging apparatus and various calculation processes. The first memory unit 508 temporarily stores image data. A removable recording medium 512 such as a semiconductor memory records image data. The recording medium control interface unit 510 records image data on the recording medium 512 or reads out image data recorded on the recording medium 512. Note that the overall control calculation unit 509 may be built in the imaging element 506 having a laminated structure.

  A display unit 511 displays image data and the like. The external interface unit 513 is an interface unit for communicating with an external computer or the like. The second memory unit 514 temporarily stores calculation results, parameters, and the like in the overall control calculation unit 509. Information regarding the driving conditions of the imaging apparatus set by the user via the operation unit 515 is sent to the overall control calculation unit 509, and the overall imaging apparatus is controlled based on these pieces of information.

  Next, the operation procedure of the imaging system in the present embodiment will be described with reference to FIGS. 1, 3, and 7.

  FIG. 7 is a flowchart illustrating a shooting sequence in a shooting mode in which a still image can be shot and recorded during moving image shooting in the imaging system of the first embodiment. FIG. 9 shows a moving image and a still image from image data obtained by driving the image sensor 506 in the all-pixel reading mode when shooting a still image of one frame during shooting of a plurality of frames of moving images. It is a figure explaining the method to produce | generate an image.

  In step S601, the overall control calculation unit 509 initializes various parameters based on settings input by the user from the operation unit 515. Then, the initialized parameters are recorded in the second memory unit 514.

  In step S602, the imaging system is driven in a monitor mode in which a captured subject image is displayed on the display unit 511 in real time. Specifically, the image signal is read out by driving the image sensor 506 in a mixed / thinning mode in which a plurality of pixels are mixed or a part of the pixels is thinned out and an image signal is read out. The read image signal is subjected to various signal processing in the imaging signal processing circuit 507 and then displayed on the display unit 511.

  When the monitor mode is driven, a photometric operation for automatic exposure adjustment (AE) control and a distance measurement operation for automatic focus adjustment (AF) control are appropriately performed based on the image signal read from the image sensor 506. Then, the imaging signal processing circuit 507 measures the brightness of the subject based on the photometric result, and calculates the aperture value Av and the shutter speed Tv so that the captured image has appropriate brightness. Further, the focal length of the subject image is calculated based on the distance measurement result.

  Here, AF control based on contrast information obtained from an image signal read from the image sensor 506 is performed. However, a focus detection pixel may be provided in the image sensor 506 in addition to the image sensor pixel. Then, AF control may be performed by detecting the imaging surface phase difference using the phase difference information (defocus amount) obtained from the focus detection pixels. Further, the contrast AF control and the imaging surface phase difference AF control may be used in combination as appropriate according to the imaging conditions, the subject, etc., or may be controlled so as to switch each AF method.

  Then, the overall control calculation unit 509 issues a command to the shutter / aperture driving unit 505 so that the calculated aperture value Av and shutter speed Tv are obtained. The overall control calculation unit 509 issues a command to the lens driving unit 502 so that the calculated lens position L is obtained. The shutter / aperture driving unit 505 drives the mechanical shutter 503 and the aperture 504 based on the received command. The lens driving unit 502 drives the lens 501 based on the received command. The calculated aperture value Av, shutter speed Tv, and lens position L are recorded in the second memory unit 514, respectively.

  When performing AE control, the accumulation time (exposure time) is varied for each row or pixel of the image sensor 506, or the gain for amplifying the image signal is varied for each row or pixel within one frame. Alternatively, it may be controlled to acquire images with different exposures. By controlling in this way, it is possible to generate an HDR image with a wide dynamic range.

  In step S603, the overall control calculation unit 509 determines ON / OFF of the moving image shooting trigger switch SW_1 in the operation unit 515. If the switch SW_1 is ON, the process returns to step S604, and if it is OFF, the process returns to step S602 again.

  In step S604, the shooting parameter i is reset to zero.

  In step S605, the aperture value Av, shutter speed Tv, and lens position L are read from the second memory unit 514. Then, based on the image data read from the image sensor 506, the aperture value Av read from the second memory unit 514, and the shutter speed Tv, the aperture value Av and the shutter speed Tv with appropriate brightness are calculated.

  In addition, the lens position L with an appropriate focal length is calculated from the image data read from the image sensor 506 and the lens position L read from the second memory unit 514. Then, the calculated aperture value Av, shutter speed Tv, and lens position L are recorded in the second memory unit 514.

  When the shooting parameter i = 0, the sensitivity difference between the drive mode in step S606 and the drive mode in step S602 is considered based on the aperture value Av and shutter speed Tv read from the second memory unit 514. The aperture value Av and the shutter speed Tv are recalculated. The lens position L uses a value read from the second memory unit 514.

When the shooting parameter i ≠ 0, the aperture value Av, shutter speed Tv, and lens position L read from the second memory unit 514 are used as they are. Then, the aperture value Av _i, the shutter speed Tv _i, so that the lens position _{L i,} the lens driving unit 502 from the total control calculation unit 509, issues a command to the shutter diaphragm driver 505, a lens 501, a mechanical shutter 503, the diaphragm 504 is driven.

  In step S606, unlike step S602, the image pickup device 506 is driven in an all-pixel read mode in which image signals are read from all the pixels on the image pickup device 506, and a shooting operation is performed. One frame of image data having the first data size obtained by the photographing operation is sent to the in-element computing unit 118.

  In step S607, the overall control calculation unit 509 determines ON / OFF of the still image shooting trigger switch SW_2 in the operation unit 515. If the switch SW_2 is ON, the process proceeds to step S608, and if it is OFF, the process proceeds to step S609.

  In step S608, the in-element computing unit 118 duplicates one frame of image data of the first data size photographed in step S606 and stores it in the frame memory 117 as still image data.

  In step S609, the in-element computing unit 118 resizes one frame of original image data having the first data size, and a one frame moving image image having a second data size smaller than the first data size. Process to convert to data.

  In resizing processing, image data is mixed or thinned out by mixing or thinning out several rows or columns in the horizontal direction (row direction) or vertical direction (column direction), or segmenting using only a part of the image data To reduce the image size. Alternatively, processing for changing the bit data amount of each pixel of the image data to a smaller size is performed. (For example, change from 16 bits to 10 bits.)

  FIG. 8 shows an example of the mixing / decimation process and the area cutout process. FIGS. 8A1 to 8A3 illustrate processing for performing horizontal 3-pixel mixing and vertical 1 / 3-pixel thinning as an example of the mixing / thinning processing, and FIGS. 8B1 to 8B2 illustrate region extraction processing. An example is shown.

In the mixing / decimation processing, the following equations (1) to (4) are calculated using only the pixel data shown in FIG. 8 (a2) for the image data having the first data size shown in FIG. 8 (a1). Then, by calculating the data R ′, Gr ′, Gb ′, and B ′ after the mixing / thinning processing, image data having the second data size is generated.
R ′ = (R + R + R) / 3 (Formula 1)
Gr ′ = (Gr + Gr + Gr) / 3 (Expression 2)
Gb ′ = (Gb + Gb + Gb) / 3 (Expression 3)
B ′ = (B + B + B) / 3 (Formula 4)

  The area cut-out process is performed on pixels in an area in which vertical, horizontal, vertical and horizontal directions surrounded by dotted lines in FIG. 8 (b2) are excluded from several to several hundred pixels with respect to the image data having the first data size in FIG. 8 (b1). By using only the data, image data of the second data size is generated.

  In step S610, the P / S conversion unit 119 performs parallel / serial conversion processing on the image data resized to the second data size in step S609, and transfers the image data to the imaging signal processing circuit 507. Here, the data size, the frame rate, and the like are set so that the image data output from the P / S conversion unit 119 is less than or equal to the output transfer capacity of the image sensor 506.

  Here, it is assumed that the number of pixels of the image data having the first data size is 24 million pixels, and the data amount of each pixel is 12 bits. In addition, transmission from the P / S conversion unit 119 of the imaging device 506 to the imaging signal processing circuit 507 is performed with 8 ports, and an output transfer capacity between the imaging device 506 and the imaging signal processing circuit 507 is 1 Gbps. .

  Further, the image data of the second data size is obtained by performing the horizontal three-pixel addition and the vertical 1/3 pixel thinning shown in FIG. 8 as resizing, and the number of pixels is 2 million pixels, and the data amount of each pixel Is 12 bits. Here, when the frame rate of the moving image is 120 fps and transmission is performed with 8 ports from the P / S conversion unit 119 of the image sensor 506 to the image signal processing circuit 507, 360 Mbps is required. Since the output transfer capacity between the image pickup device 506 and the image pickup signal processing circuit 507 is 1 Gbps, it is possible to transfer the image data for moving image with a sufficient margin.

  In step S611, the image signal processing circuit 507 performs various processing on the image data that has been resized in step S609 in response to a command from the overall control calculation unit 509 and transferred in step S610. Various processes performed by the imaging signal processing circuit 507 include various image signal processes such as a low-pass filter process for reducing noise, a defective pixel correction process, a shading correction process, a white balance process, a development process, and an image data compression process. Etc. Then, the image data subjected to various processes is recorded as a moving image on the recording medium 512.

  In step S612, 1 is added to the imaging parameter i. In step S613, the overall control calculation unit 509 determines ON / OFF of the switch SW_1. If the switch SW_1 is ON, it is determined that a still image shooting instruction has been issued, and the process proceeds to step S614. If the switch SW_1 is OFF, the process returns to step S605.

  In step S614, the P / S conversion unit 119 performs parallel / serial conversion processing on the still image data having the first data size stored in the frame memory 117 in step S608, and transfers the image data to the imaging signal processing circuit 507. To do.

  Note that the data size, frame rate, and the like of the still image image data output from the P / S conversion unit 119 are set so as to be less than or equal to the output transfer capacity of the image sensor 506.

  Here, the number of pixels of the image data of the first data size is 24 million pixels, and the data amount of each pixel is 12 bits. If the frame rate is set to 24 fps, 864 Мbps is required for transmission between the P / S conversion unit 119 and the imaging signal processing circuit 507 with 8 ports. Since the output transfer capacity from the image sensor 506 to the image signal processing circuit 507 is 1 Gbps, it is possible to transfer still image data with a margin.

  In step S615, the imaging signal processing circuit 507 performs various processes on the image data having the first data size transferred in step S614 in response to a command from the overall control calculation unit 509.

  Various processes performed by the imaging signal processing circuit 507 include various image signal processes such as a low-pass filter process for reducing noise, a defective pixel correction process, a shading correction process, a white balance process, a development process, and an image data compression process. Etc. Then, the image data subjected to various processes is recorded as a still image on the recording medium 512. Further, the moving image recorded in the storage medium 512 in step S611 is post-processed and saved in a predetermined moving image format.

  As described above, the moving image data is reduced in size from the first data size to the second data size in the image sensor and transferred to the subsequent stage of the image sensor. Then, the still image data with the first data size having a large data size is temporarily saved in a frame memory in the image sensor, and is transferred to the subsequent stage of the image sensor after the moving image data is transferred.

  In this way, in an imaging system equipped with an imaging device having a large number of pixels, it is possible to obtain a seamless moving image at a high frame rate even when still image shooting is performed during moving image shooting. In addition, since a still image taken at that time can be taken at a high shutter speed, a high-quality image in which the rolling distortion peculiar to the CMOS image sensor is not conspicuous can be obtained.

  In this embodiment, as shown in FIG. 7, a method of creating a moving image and a still image from an image signal obtained by always driving the image sensor 506 in the all-pixel reading mode has been described. However, as shown in FIG. 10, the image pickup device 506 is normally driven by driving the image pickup device 506 in the horizontal three-pixel mixing mode and the vertical one-third pixel thinning mode, and the image pickup device 506 is all turned on only when a still image shooting instruction is given. You may control to drive in pixel readout mode. In that case, the image data of the first data size obtained in the all-pixel read mode is resized to generate moving image data of the second data size, and the image data of the first data size is used for still images. It can be used as image data.

(Example 2)
Hereinafter, Embodiment 2 of the present invention will be described with reference to FIGS. 11 and 12. In the second embodiment, an imaging system having a shooting mode capable of shooting a high-resolution moving image during normal moving image shooting will be described as an example.

  The configuration of the image sensor is the same as that shown in FIGS. The outline of the imaging system is also the same as that shown in FIG. Further, the resizing process of the image data is the same as that shown in FIG.

  FIG. 11 is a flowchart illustrating an imaging sequence in the imaging system according to the second embodiment. FIG. 12 illustrates a method for generating a high-resolution moving image by driving the image sensor 506 in the all-pixel reading mode when a high-resolution moving image shooting instruction is given during normal moving image shooting in this embodiment. It is a figure explaining about.

  Steps S1001 to S1005 are the same as steps S601 to S605 shown in FIG.

  In step S1006, the switch SW_2 in the operation unit 515 is used as a high-resolution moving image shooting trigger switch. That is, the overall control calculation unit 509 determines ON / OFF of the switch SW_2. If the switch SW_2 is ON, the process proceeds to step S1007, and if it is OFF, the process proceeds to step S1008.

  In step S1007, in order to shoot a normal moving image in the first moving image shooting mode, the image pickup device 506 is driven in a horizontal three-pixel mixing and vertical 1/3 pixel thinning mode to perform a shooting operation. One frame of image data having the second data size obtained by the photographing operation is sent to the P / S conversion unit 109 via the in-element operation unit 118.

  In step S1008, in order to shoot a high-resolution moving image in the second moving image shooting mode, the image pickup device 506 is driven in the all-pixel reading mode to perform a shooting operation. One frame of image data having the first data size obtained by the photographing operation is sent to the in-element computing unit 118.

  In step S1009, one-frame image data of the first data size captured in step S1008 is duplicated in the element calculation unit 118 and stored in the frame memory 117 as high-resolution moving image data. Further, the in-element arithmetic unit 118 resizes one frame of image data having the original first data size, and one frame of normal moving image data having a second data size smaller than the first data size. Process to convert to.

  In step S1010, the P / S converter 119 performs parallel / parallel processing on the image data of the second data size captured in step S1007 or the image data of the second data size resized to the second data size in step S1009. Perform serial conversion processing. Then, the image is transferred to the imaging signal processing circuit 507.

  It is assumed that the number of pixels of image data having the second data size transferred from the P / S conversion unit 119 to the imaging signal processing circuit 507 in step S1010 is 2 million pixels, and the data amount of each pixel is 12 bits. In addition, it is assumed that the frame rate of the moving image is 120 fps, the transmission from the P / S conversion unit 119 of the imaging device 506 to the imaging signal processing circuit 507 is performed with 8 ports, and the imaging device 506 to the imaging signal processing circuit 507. The output transfer capacity is 1 Gbps. In this case, since the data size is the same as the data transferred in step S610 of the first embodiment, it is possible to transfer the normal moving image image data with a sufficient margin.

  Steps S1011 to S1013 are the same as steps S611 to S613 shown in FIG.

  In step S1014, the P / S conversion unit 119 performs parallel / serial conversion processing on the high-resolution moving image image data having the first data size stored in the frame memory 117 in step S1009, and the image signal processing circuit 507 outputs the parallel / serial conversion processing. Forward.

  Note that the data size and frame rate of the high-resolution moving image data output from the P / S conversion unit 119 are set so as to be less than or equal to the output transfer capacity of the image sensor 506.

  Here, the number of pixels of the image data of the first data size is 24 million pixels, and the data amount of each pixel is 12 bits. If the frame rate is set to 24 fps, 864 Мbps is required for transmission between the P / S conversion unit 119 and the imaging signal processing circuit 507 with 8 ports. Since the output transfer capacity between the image pickup device 506 and the image pickup signal processing circuit 507 is 1 Gbps, it is possible to transfer the high-resolution moving image data with a margin.

  In step S1015, the imaging signal processing circuit 507 performs various processes on the image data having the first data size transferred in step S1014 according to a command from the overall control calculation unit 509.

  Various processes performed by the imaging signal processing circuit 507 include various image signal processes such as a low-pass filter process for reducing noise, a defective pixel correction process, a shading correction process, a white balance process, a development process, and an image data compression process. And post-processing. Then, the image data subjected to various processes is recorded on the recording medium 512. The moving image recorded here is a high-resolution moving image corresponding to the entire effective imaging area (full screen) of the image sensor at a high frame rate of 120 fps.

  In addition, the moving image recorded in the storage medium 512 in step S1011 is also post-processed and stored as a 120 fps moving image corresponding to a predetermined moving image format whose image size is smaller than the image size of the entire screen.

  As described above, in this embodiment, as shown in FIG. 12, the image sensor 506 is driven in the horizontal three-pixel mixing and vertical 1/3 pixel thinning mode, and only when there is a high-resolution moving image shooting instruction. 506 is driven in the all-pixel readout mode. The image sensor 506 is obtained by driving the image sensor 506 in the all-pixel read mode and the image data of the second data size obtained by driving the image sensor 506 in the horizontal three-pixel mixing and vertical 1/3 pixel thinning mode. A normal moving image can be generated from the image data having the second data size obtained by resizing the image data having the first data size. Furthermore, a high-resolution moving image can be generated from the image data having the first data size obtained in the all-pixel reading mode.

  By configuring as described above, even when shooting a high-resolution video during normal video shooting in an imaging system equipped with an image sensor with a large number of pixels, seamless normal video and high-resolution video at a high frame rate Can be obtained.

(Example 3)
Hereinafter, Embodiment 3 of the present invention will be described with reference to FIGS. 11 and 13. In the second embodiment, a method of shooting a high-resolution moving image during normal moving image shooting has been described. In the third embodiment, a frame rate higher than normal (8 times here) during normal moving image shooting is used. A method for shooting a high-speed moving image will be described.

  The configuration of the image sensor, the outline of the imaging system, and the resizing process of the image data are the same as those in the first and second embodiments, and thus the description thereof is omitted. The flowchart of the imaging sequence of this embodiment will also be described with reference to FIG. 11 shown in the second embodiment. FIG. 13 is a diagram illustrating a method for generating a high-speed moving image with a high frame rate during normal moving image shooting in the present embodiment.

  Steps S1001 to S1005 and step S1007 are the same as those for shooting a high-resolution moving image during moving image shooting described in the second embodiment, and thus the description thereof is omitted.

  In step S1006, the switch SW_2 in the operation unit 515 is used as a high-speed moving image shooting trigger switch. That is, the overall control calculation unit 509 determines ON / OFF of the switch SW_2. If the switch SW_2 is ON, the process proceeds to step S1007, and if it is OFF, the process proceeds to step S1008.

  In step S1008, in order to shoot a high frame rate moving image in the second moving image mode, the image sensor 506 is driven with a horizontal three-pixel mixing, vertical ３ pixel thinning mode, and a frame rate that is eight times the normal shooting. Perform the action. When the normal frame rate is 120 fps, it is driven at 960 fps. Image data of one frame rate having the second data size obtained by the photographing operation is sent to the in-element computing unit 118.

  In step S1009, the image data of the second data size captured in step S1008 is duplicated in the element calculation unit 118, and is stored in the frame memory 117 as high frame rate moving image data. In addition, the in-element computing unit 118 thins out 7 frames per 8 frames from the original image data of the second data size and reduces the frame rate from 960 fps to 120 fps to 1/8. Process to convert to.

  In step S1010, the P / S conversion unit 119 performs parallel / serial conversion processing on the image data having the second data size (the frame rate is 120 fps) captured in step S1007. Alternatively, the P / S conversion unit 119 performs parallel / serial conversion processing on the image data having the second data size whose frame rate has been changed from 960 fps to 120 fps in step S1009. Then, the image is transferred to the imaging signal processing circuit 507.

  Steps S1011 to S1013 are the same as those in the second embodiment, and thus description thereof is omitted.

  In step S1014, the P / S conversion unit 119 performs parallel / serial conversion processing on the high frame rate moving image image data having the second data size stored in the frame memory 117 in step S1009, and the imaging signal processing circuit 507 is processed. Forward to.

  Here, the number of pixels of the image data of the second data size is 2 million pixels, and the data amount of each pixel is 12 bits. However, since the image data was captured at 960 fps, the image signal from the P / S converter 119 When transmitting to the processing circuit 507 with 8 ports, 2.88 Gbps is required. Therefore, when the output transfer capacity between the image sensor 506 and the image signal processing circuit 507 is 1 Gbps, the transfer capacity is insufficient.

  However, if the frame rate is reduced from 960 fps to 120 fps, which is 1/8 of the frame rate, and transferred from the image sensor 506 to the image signal processing circuit 507, a transfer capacity of 360 Mbps is sufficient. Here, since the output transfer capacity between the image sensor 506 and the image signal processing circuit 507 is 1 Gbps or less, it is possible to transfer the image data for high frame rate moving image with a sufficient margin.

  In step S1015, the imaging signal processing circuit 507 performs various processes on the image data having the second data size transferred in step S1014 in response to a command from the overall control calculation unit 509.

  Various processes performed by the imaging signal processing circuit 507 include various image signal processes such as a low-pass filter process for reducing noise, a defective pixel correction process, a shading correction process, a white balance process, a development process, and an image data compression process. And post-processing. Then, the image data subjected to various processes is recorded on the recording medium 512. The moving image recorded here is a high frame rate moving image having a reproduction frame rate of 960 fps corresponding to a predetermined moving image format.

  Further, the moving image recorded in the storage medium 512 in step S1011 is also post-processed and stored as a moving image having a playback frame rate of 120 fps corresponding to a predetermined moving image format.

  As described above, in this embodiment, as shown in FIG. 13, the image sensor 506 is driven in the horizontal three-pixel mixing mode, the vertical 1/3 pixel thinning mode, and 120 fps. Then, only when a high-speed moving image shooting instruction is given, the image sensor 506 is driven at a horizontal three-pixel mixing mode, a vertical 1/3 pixel thinning mode, and a frame rate of 960 fps.

  Then, a plurality of frames are thinned out from the image data of the second data size obtained by shooting at 120 fps and the image data obtained by shooting at 960 fps. A normal 120 fps moving image is created from the image data having the second data size set to 120 fps. In addition, a high-speed moving image of 960 fps can be generated from image data having the second data size obtained by performing the shooting operation at 960 fps.

  By configuring as described above, in an imaging system equipped with an imaging device having a large number of pixels, even when shooting a high-speed movie during normal movie shooting, a seamless normal movie and a high-speed movie can be obtained at a high frame rate. It becomes possible.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

101 pixels 10 first chip 11 second chip 117 frame memory 118 in-element arithmetic unit

Claims (24)

Imaging means for receiving incident light and performing photoelectric conversion;
A plurality of AD conversion means for converting analog image signals output from the imaging means into digital image data;
Storage means for storing digital image data of at least one frame converted by the plurality of AD conversion means;
Generating means for generating moving image digital image data from still image digital image data;
Output means for outputting the still image digital image data and the moving image digital image data to the outside ,
The image data is configured to be transmitted in parallel between the plurality of AD conversion units and the storage unit, and
When shooting a moving image and a still image using the imaging means, the still image digital image data stored in the storage means, the digital image data for the moving from the digital image data for the still image by the output means An image pickup device configured to be able to output to the outside first. It said generating means, the image pickup device according to claim 1, characterized in that said from still image digital image data is used to generate resize the digital image data for the moving image. The imaging device according to claim 1, wherein the digital image data for still image can be output to the outside after the digital image data for moving image is output to the outside by the output unit . 4. The image sensor according to claim 1, wherein the output unit converts the image data into a serial signal and outputs the serial signal to the outside. 5. Comprising a plurality of semiconductor substrates stacked on each other;
5. The imaging device according to claim 1, wherein at least one of the storage unit and the generation unit is provided on a semiconductor substrate different from the imaging unit. Before term memory means and the image pickup device according to claim 5, wherein the generating means and which are located on different said semiconductor substrate and each of said image pickup means. Imaging means for receiving incident light and performing photoelectric conversion;
A plurality of AD conversion means for converting analog image signals output from the imaging means into digital image data;
Storage means for storing digital image data of at least one frame converted by the plurality of AD conversion means ;
Arithmetic means for performing arithmetic processing on the digital image data;
Output means for outputting the digital image data to the outside ,
The image data is configured to be transmitted in parallel between the plurality of AD conversion units and the storage unit, and
When the image data of the first size and the image data of the second size different from the first size are output to the outside by the output means, the image data of the first size is stored in the storage means. An image sensor that is configured to be capable of being output to the outside . Comprising a plurality of semiconductor substrates stacked on each other;
The imaging device according to claim 7, wherein at least one of the storage unit and the calculation unit is provided on a semiconductor substrate different from the imaging unit. Before term memory means and the image pickup device according to claim 8, wherein the calculating means and which are located on different said semiconductor substrate and each of said image pickup means. The image sensor according to claim 7 , wherein the calculation unit changes at least one of the first size or the second size of image data to the other size. The first size or the second size, the imaging device according to any one of claims 7 to 10, characterized in that it is set to be equal to or less than transfer capacity of the output means. With respect to the first and second size, to any one of claims 7 to 10 frame rate of the image data to be equal to or less than transfer capacity of the output means, characterized in that it is set The imaging device described. Imaging means for receiving and photoelectrically converting incident light, a plurality of AD conversion means for converting analog image signals output from the imaging means into digital image data, and at least one frame converted by the plurality of AD conversion means Storage means for storing the digital image data, generation means for generating moving image digital image data from the still image digital image data, output for outputting the still image digital image data and the moving image digital image data to the outside An imaging device having means ;
A signal processing unit that performs predetermined signal processing on image data output from the image sensor;
A display unit for displaying the images,
A control unit for controlling each of the image sensor, the signal processing unit, and the display unit;
Have
The image data is configured to be transmitted in parallel between the plurality of AD conversion units and the storage unit,
Wherein, when shooting a moving image and a still image using the imaging means, stores digital image data for the still image in the storage means, the still image digital image data for the moving image by the output means An image pickup apparatus that outputs from the image pickup device prior to digital image data for use. It said generating means, the image pickup apparatus according to claim 13, wherein the the still image digital image data is used to generate resize the digital image data for the moving image. Claim 13 or 14, characterized by being configured to allow the output digital image data for the still image after printing the digital image data for the moving image to the outside of the image sensor to the outside of the imaging device by said output means The imaging device described in 1. 16. The imaging apparatus according to claim 13, wherein the output unit converts the image data into a serial signal and outputs the serial signal to the outside. The imaging device includes a plurality of semiconductor substrates stacked on each other,
17. The imaging apparatus according to claim 13, wherein at least one of the storage unit and the generation unit is provided on a semiconductor substrate different from the imaging unit. Before SL imaging apparatus according to claim 17, characterized in that provided in different said semiconductor substrate storage means and said generating means and each of said image pickup means. Imaging means for receiving and photoelectrically converting incident light, a plurality of AD conversion means for converting analog image signals output from the imaging means into digital image data, and at least one frame converted by the plurality of AD conversion means storage means for storing the digital image data, arithmetic means for performing arithmetic processing on the digital image data, an image pickup device and an output means for outputting the digital image data to the outside,
A signal processing unit that performs predetermined signal processing on image data output from the image sensor;
A display unit for displaying the images,
A control unit for controlling each of the image sensor, the signal processing unit, and the display unit;
Have
The image data is configured to be transmitted in parallel between the plurality of AD conversion units and the storage unit,
Wherein, when outputting the image data of a second size different from the first size image data and said first size by said output means from the imaging device, the image data of the first size An imaging apparatus that outputs the data after storing the data in the storage unit. The imaging device includes a plurality of semiconductor substrates stacked on each other,
The imaging apparatus according to claim 19, wherein at least one of the storage unit and the calculation unit is provided on a semiconductor substrate different from the imaging unit. Before SL imaging apparatus according to claim 20, characterized in that provided in different said semiconductor substrate storage means and the computing means and each of said image pickup means. The imaging device according to any one of claims 19 to 21 , wherein the calculation unit changes at least one of the first size or the second size of image data to the other size. The first size or the second size, the imaging apparatus according to any one of claims 19 to 22, characterized in that it is set to be equal to or less than transfer capacity of the output means. With respect to the first and second size, to any one of claims 19 to 23, characterized in that the frame rate of the image data to be equal to or less than transfer capacity of the output means is set The imaging device described.

Images