JP6733714B2 - Imaging device - Google Patents

Description

The present invention relates to an image pickup device.

There is a method of reducing deterioration in the quality of a moving image by acquiring and processing a plurality of images having different exposure times within one frame (see, for example, Patent Document 1).
Patent Document 1 JP 2013-081060 A

The amount of pixel signals read from the image pickup device is significantly increased, and the processing load at the image processing stage is also increased.

According to the first aspect of the present invention, a first pixel including a first photoelectric conversion unit that converts light into electric charge, and a second photoelectric conversion unit that is arranged next to the first pixel in the column direction and that converts light into electric charge. A second pixel including a conversion unit, a first control line connected to the first pixel and outputting a first control signal for controlling the first pixel, and a second control line connected to the second pixel to control the second pixel To the second pixel , a second control line for outputting a second control signal, and a first output line connected to the first pixel for outputting a first signal based on the charge of the first photoelectric conversion unit, and a second pixel is connected, is output and an image pickup device having a second output line second signal based on the electric charge of the second photoelectric conversion unit is output, and the first signal and the second output line which is output to the first output line and a calculating unit for calculating the correlation between the second signal, based on the correlation calculated by the calculation unit, the timing at which the first control signal is output to the first control line, the second control signal is the second control line imaging device is provided comprising a timing output, and wo system Gosuru control unit, to the.

Note that the above summary of the invention does not enumerate all the necessary features of the present invention. Further, a sub-combination of these feature groups can also be an invention.

FIG. 3 is a block diagram of an imaging device 500. 3 is a cross-sectional view of the image sensor 100. FIG. 6 is a schematic diagram illustrating a layout of a light receiving substrate 113. FIG. 3 is a block diagram of a unit block 131 and related circuits. FIG. It is a circuit diagram of the pixel 150 alone. 6 is a timing chart showing the operation of the pixel 150. 3 is a block diagram of a drive unit 502. FIG. It is a schematic diagram which shows operation|movement of rolling shutter mode. 3 is a schematic diagram of a subject 300. FIG. It is a schematic diagram which shows the shape of the captured images 311, 312, and 313. 9 is a flowchart showing the operation of the image capturing control unit 474. 6 is a flowchart showing the operation of the correlation calculation unit 472. 9 is a flowchart showing another operation of the image pickup control unit 474. FIG. 6 is a diagram illustrating an example of contents of a table referred to by an imaging control unit 474. It is a schematic diagram which shows operation|movement of global shutter mode.

Hereinafter, the present invention will be described through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. Further, not all of the combinations of features described in the embodiments are essential to the solving means of the invention.

FIG. 1 is a block diagram of the imaging device 500. The image pickup apparatus 500 includes an optical system including an image pickup lens 520, and an electronic circuit including an image pickup element 100, a system control unit 501, a driving unit 502, a photometry unit 503, a work memory 504, a recording unit 505, and a display unit 506.

The optical system including the image pickup lens 520 guides the incident light flux propagating along the optical axis OA to the image pickup element 100. In the figure, a virtual single imaging lens 520 arranged near the pupil is shown as a representative, but the optical system includes a plurality of optical lenses and forms an incident light flux on the imaging surface of the imaging device 100. .. The imaging lens 520 may be a replaceable type that can be attached to and detached from the imaging device 500.

The driving unit 502 executes storage control and readout control of pixel signals for the image sensor 100. As a result, the drive unit 502 executes storage control and readout control of pixel signals for the image sensor 100 under the control of the system control unit 501.

The photometric unit 503 has a light receiving element that receives a part of the light flux incident on the imaging lens 520 and detects the brightness. Accordingly, the photometric unit 503 detects the brightness distribution and the like of the imaged scene. The photometric unit 503 may use a part of the image sensor 100 as a light receiving element. In the illustrated imaging device 500, the drive unit 502 also refers to the detection result of the photometry unit 503.

The work memory 504 is used as a work area when the system control unit 501 executes processing. The work memory 504 also functions as a buffer for information recorded in the recording unit 505, information displayed by the display unit 506, and the like.

The system control unit 501 includes an image processing unit 511, a calculation unit 512, and the like. The image processing unit 511 executes image processing using the work memory 504 as a work area to generate image data. For example, when the image data to be generated is JPEG format image data, white balance processing, gamma processing, compression processing, etc. are executed.

The image data generated by the image processing unit 511 is recorded in the recording unit 505. Further, the generated image data may be displayed on the display unit 506. Further, it may be transferred to an external storage unit such as a cloud server through a communication line. The calculation unit 512 determines the shutter speed, the aperture value, the ISO sensitivity, and the like in the image pickup operation of the image pickup apparatus 500 based on the brightness information acquired by the system control unit 501 from the photometry unit 503, for example.

The image sensor 100 generates a pixel signal corresponding to the light flux incident on the imaging lens 520. The pixel signal generated by the image sensor 100 becomes image data in the image processing unit 511 of the system control unit 501.

FIG. 2 is a sectional view of the image sensor 100. The image sensor 100 includes a light receiving substrate 113, a signal processing substrate 111, and a memory substrate 112 that are stacked on each other. The light receiving substrate 113, the signal processing substrate 111, and the memory substrate 112 are electrically connected to each other by a conductive bump 109 such as Cu.

Note that, as indicated by the white arrow in the figure, the subject light flux enters the image sensor 100 in the Z-axis plus direction. In the present embodiment, the surface on which the subject light flux enters the light receiving substrate 113 will be referred to as the back surface in the following description. Further, as shown by the coordinate axes in the figure, the right direction in the figure orthogonal to the Z axis is described as the X axis plus direction, and the direction orthogonal to the Z axis and the X axis and intersecting the front side of the paper is referred to as the Y axis plus direction. To do.

The light receiving substrate 113 has a photoelectric conversion element layer 106 and a wiring layer 108 as electrical elements. The photoelectric conversion element layer 106 is arranged on the back surface side of the wiring layer 108. The photoelectric conversion element layer 106 includes a plurality of photodiodes 104 and a plurality of transistors 105 which are two-dimensionally arranged. Each of the plurality of transistors 105 is provided corresponding to one of the plurality of photodiodes 104.

The photoelectric conversion element layer 106 includes a photoelectric conversion element that outputs a pixel signal corresponding to incident light. An example of the light receiving substrate 113 is a backside illumination type CMOS image sensor, and the photoelectric conversion element layer 106 is provided with a photodiode 104 as a photoelectric conversion element.

In the light receiving substrate 113, the wiring layer 108 has the wiring 107 for transmitting the pixel signal from the photoelectric conversion element layer 106 to the signal processing substrate 111. The wiring 107 may be arranged in multiple layers, and may further include passive elements and active elements.

A plurality of bumps 109 are arranged on the surface of the wiring layer 108. The plurality of bumps 109 are aligned with the plurality of bumps 109 provided on the opposite surfaces of the signal processing substrate 111, and the light receiving substrate 113 and the signal processing substrate 111 are pressed and the like to be aligned. The bumps 109 are joined together and electrically connected.

Further, the light receiving substrate 113 has a color filter 102 and a microlens 101 as optical elements. In the light receiving substrate 113, the color filter 102 is arranged on the incident side of incident light in the photoelectric conversion element layer 106 via the passivation film 103. Therefore, incident light enters each of the plurality of photodiodes 104 through the color filter 102.

Further, the color filter 102 includes a plurality of types that transmit wavelength regions different from each other. The color filters 102 of different types have a specific array corresponding to each of the photodiodes 104. A set of the color filter 102, the photodiode 104, and the transistor 105 forms one pixel.

The microlens 101 is arranged on the incident side of the incident light with respect to the color filter 102 so as to correspond to each pixel. Each of the microlenses 101 collects incident light toward the corresponding photodiode 104.

The signal processing substrate 111 processes the pixel signal acquired from the light receiving substrate 113. The memory substrate 112 also stores pixel signals. A plurality of bumps 109 are arranged on the surfaces of the signal processing substrate 111 and the memory substrate 112 that face each other. The bumps 109 are aligned with each other, and the signal processing substrate 111 and the memory substrate 112 are pressed and the like, so that the aligned bumps 109 are bonded to each other and electrically connected.

It should be noted that the bonding between the bumps 109 is not limited to Cu bump bonding by solid phase diffusion, and micro bump bonding by solder melting may be adopted. Further, the bumps 109 may be provided, for example, about one for each output wiring described later. Therefore, the size of the bumps 109 may be larger than the pitch of the photodiodes 104. Further, bumps larger than the bumps 109 corresponding to the pixel area may be additionally provided in the peripheral area other than the pixel area in which the pixels are arranged.

The signal processing substrate 111 has TSVs (through silicon vias) 110 that connect circuits provided on the front and back surfaces to each other. The TSV 110 is preferably provided in the peripheral area. The TSV 110 may also be provided in the peripheral region of the light receiving substrate 113 and the memory substrate 112.

FIG. 3 is a schematic diagram showing the layout of the pixels 150 and the unit blocks 131 in the photoelectric conversion element layer 106 of the image sensor 100. The layout in the figure shows a state where the light receiving substrate 113 is viewed from the back surface side.

The photoelectric conversion element layer 106 has a plurality of unit blocks 131 arranged two-dimensionally in the row direction and the column direction. Further, each of the unit blocks 131 includes a plurality of pixels 150 arranged two-dimensionally in the same row direction and column direction as the unit block 131. Therefore, 20 million or more pixels 150 are two-dimensionally arranged in the image sensor 100.

In the illustrated example, each of the unit blocks 131 is formed by 16 pixels 150 of 4 pixels×4 pixels that are adjacent to each other. As shown in the partially enlarged view shown in the drawing, the 16 pixels 150 forming the unit block 131 are composed of four pixels of green pixels Gb, Gr, blue pixels B, and red pixels R according to the Bayer array, for example. Either can be assigned.

The green pixels Gb and Gr have a green filter as the color filter 102 and receive light in the green wavelength band of the incident light. Similarly, the blue pixel B has a blue filter as the color filter 102 and receives light in the blue wavelength band, and the red pixel R has a red filter as the color filter 102 and receives light in the red wavelength band. ..

Of course, the number of pixels 150 forming the unit block 131 is not limited to the above. Further, the shape of the unit block 131 formed by the plurality of pixels 150 is not limited to the square. Therefore, when the total number of pixels of the light receiving substrate 113 is about 20 million, the layout may be such that the array number of the unit blocks 131 is, for example, 64 rows and 32 columns, 48 rows and 114 columns.

FIG. 4 schematically shows the connection relationship with one of the unit blocks 131 formed in the image sensor 100. The illustrated unit block 131 includes pixels 150 in L rows and P columns, and is connected to the row control unit 200 and the peripheral circuit 133.

In the image sensor 100, the row control unit 200 is coupled to the unit block 131 via the wiring Rst_l (where l is an integer from 1 to L), the wiring Tx_l, and the wiring Sel_l. Each of the wiring Rst_l, the wiring Tx_l, and the wiring Sel_l is commonly connected to the P pixels 150 in the first row in the unit block 131. Similarly, the wiring Tx_l and the wiring Sel_l are also commonly connected to the P pixels 150 of the 1st row in the unit block 131.

The row control unit 200 may be called a row selection unit, a vertical scanning circuit, or the like. The row control unit 200 is provided for each unit block 131. In the image sensor 100, the row control unit 200 may be provided on the signal processing board 111 side.

In the image sensor 100, the peripheral circuit 133 is coupled to the unit block 131 through the column transmission line 170_p (where p is an integer from 1 to P). The column transmission path 170_p is provided for each pixel 150 in the same column.

Further, the column transmission path 170_p is commonly connected to the L pixels 150 of the p-th column in the unit block 131. As a result, the column transmission path 170 is shared by the pixels 150 in the same column in the unit block 131, and the pixel signal from the pixels 150 included in the column is transmitted.

In the image sensor 100, the column transmission path 170_p is connected from the light receiving substrate 113 side to the peripheral circuit 133 mounted on the signal processing substrate 111 through the bump 109. The peripheral circuit 133 in the signal processing board 111 is arranged for each unit block 131 at a position corresponding to the unit block 131 in the light receiving board 113.

Each of the peripheral circuits 133 includes an analog/digital conversion circuit 202 and a CDS circuit 204 which are connected in series for each column transmission path 170_p. A set of the analog/digital conversion circuit 202 and the CDS circuit 204 is provided in the same number of columns as the number of columns per unit block 131.

The peripheral circuit 133 further includes a shift register 206 arranged on the output side of the CDS circuit 204. In the illustrated example, the shift register 206 is arranged for each unit block 131. The output of the shift register 206 is connected to the pixel memory 414 through the column bus line 172. The shift register 206 may be called a horizontal scanning circuit, a multiplexer, or the like.

FIG. 5 is a circuit diagram showing a structure of the pixel 150 forming the unit block 131. However, the illustrated circuit includes two photodiodes 104-1 and 104-2, and corresponds to two pixels 150. These two pixels are included in the same column in the unit block 131.

The cathodes of the photodiodes 104-1 and 104-2 are individually coupled to one ends of the selection transistors 61-1 and 61-2 arranged corresponding to the photodiodes 104-1 and 104-2, respectively. The photodiodes 104-1 and 104-2 each store an electric charge according to the amount of received light.

The selection transistors 61-1 and 61-2 switch whether to output a voltage according to the charge amount accumulated in the photodiodes 104-1 and 104-2. The conduction state of the selection transistor 61 is controlled by, for example, a pixel selection signal provided through the wiring Tx_l provided from the row control unit 200.

Further, in the pixel 150, the reset transistor 59, the amplification transistor 60, the ground side transistor 65, the first transistor 78, the second transistor 63, the amplification transistor 66, the transfer transistor 64, which are provided in common for the photodiodes 104-1 and 104-2, are provided. , And a capacitor 10. These commonly provided transistors function as a switch unit that controls the state of the capacitor 10 (state that indicates what voltage is applied to both electrodes).

In the illustrated example, the pixel 150 sequentially holds the feedthrough voltage and the signal voltage at the outputs of the photodiodes 104-1 and 104-2 by the single capacitor 10. This makes it possible to reduce the area of the CDS circuit of each pixel 150 while providing each pixel 150 with the CDS circuit. Further, since the common capacitor 10 holds the feedthrough voltage and the signal voltage, the pixel value can be accurately detected.

The reset transistor 59 is connected to the photodiode 104 via each selection transistor 61. The reset transistor 59 switches whether or not the cathodes of the photodiodes 104-1 and 104-2 are connected to the reference voltage VCC. The reset transistor 59 is controlled by a reset signal applied through the wiring Rst_l of the row control unit 200.

The gate of the amplification transistor 60 is connected to the photodiodes 104-1 and 104-2 via the selection transistors 61-1 and 61-2, respectively. The amplifying transistor 60 amplifies the voltage output from one of the photodiodes 104-1 and 104-2 selected by turning on the selection transistor 61. The amplification transistor 60 is arranged between the reference voltage VCC and the ground potential.

A ground side transistor 65 is arranged between the amplification transistor 60 and the ground potential. A P control voltage is applied to the gate of the ground-side transistor 65.

The capacitor 10 has three electrodes, a first electrode and a second electrode are arranged to face each other, and a second electrode and a third electrode are arranged to face each other. As a result, the capacitor 10 forms a three-terminal capacitor, and is charged/discharged with a voltage according to the amount of charge accumulated in the photodiodes 104-1 and 104-2.

The first electrode of the capacitor 10 is connected to the output terminal of the amplification transistor 60 via the first transistor 78. The first transistor 78 functions as a first switch that switches whether to connect the outputs of the photodiodes 104-1 and 104-2 amplified by the amplification transistor 60 to the first electrode. The first transistor 78 is controlled by the first control signal SH1 applied to the gate.

The second electrode of the capacitor 10 is connected to the output terminal of the amplification transistor 60 via the second transistor 63. The second transistor 63 functions as a second switch that switches whether or not the output of the photodiode 104 amplified by the amplification transistor 60 is connected to the second electrode. The second transistor 63 is controlled by the second control signal SH2 applied to the gate.

The third electrode of the capacitor 10 is connected to the reference potential. The capacitance between the second electrode and the third electrode maintains the potential of the second electrode with respect to the reference potential. The capacitor 10 has a first electrode, a second electrode, and a third electrode that are formed so as to overlap each other in the same region. Therefore, the mounting area can be reduced as compared with the case where a plurality of two-terminal capacitors are used. However, the capacitor 10 may be formed by using two elements, a two-terminal capacitor provided between the node B and the node C and a two-terminal capacitor provided between the node B and the reference potential.

The amplification transistor 66 amplifies and outputs the voltage at the first electrode of the capacitor 10. The transfer transistor 64 functions as a transfer unit that switches whether to transfer the voltage output from the amplification transistor 66 to the read line 170-p. The transfer transistor 64 is controlled by a row selection signal supplied from the row control unit 200 through the wiring Sel_l.

The capacitor 10 sequentially transits between a first state in which a feedthrough voltage is applied to the second electrode and a second state in which a signal voltage is applied to the second electrode, in accordance with each control signal given from the row control unit 200. Let The peripheral circuit 133 turns on the transfer transistor 64 at least in the second state to transfer the voltage level of the first electrode of the capacitor 10. Thereby, the peripheral circuit 133 can calculate the pixel value of the pixel signal based on the transferred voltage level.

FIG. 6 is a timing chart showing the operation of the pixel 150, and shows the signal waveforms at the nodes A, B, and C shown in FIG. The node A corresponds to the output terminal of the amplification transistor 60, the node B corresponds to the second electrode of the capacitor 10, and the node C corresponds to the first electrode of the capacitor 10.

The case of reading the output of the photodiode 104-1 will be described. First, the reset signal RST becomes H level, and the reset transistor 59 is turned on. Further, the selection transistor 61-1 is controlled to be in the ON state. As a result, the cathode voltage of the photodiode 104-1 is reset and the voltage at the node A becomes the reference voltage VCC. The amplification factors of the amplification transistors 60 and 66 are 1.

After a lapse of time after the reset, the voltage at the node A becomes the feedthrough voltage Vft. The row control unit 200 sets the first control signal SH1 and the second control signal SH2 to the H level after the reset signal RST is input and a predetermined time has elapsed. As a result, both the first transistor 78 and the second transistor 63 are turned on. At this time, the reference voltage VCC is applied to the first electrode (node C) of the capacitor 10, and the feedthrough voltage Vft is applied to the second electrode (node B) (first state).

Next, the first control signal SH1 and the second control signal SH2 are set to L level. As a result, the first electrode of the capacitor 10 is brought into a floating state, and the inter-electrode voltage of the capacitor 10 is maintained.

Then, at the timing when the light reception of the frame in the photodiode 104-1 ends, the voltage at the node A becomes a voltage corresponding to the voltage sig1 corresponding to the accumulated charge amount in the photodiode 104-1. Specifically, it is a voltage that is lower than the feedthrough voltage Vft by the voltage sig1.

Further, the second control signal SH2 is set to H level while the first control signal SH1 is maintained at L level. As a result, the second transistor 63 is turned on (second state).

Since the first control signal SH1 is maintained at the L level, when the capacitor 10 transits to the second state, the first electrode becomes the floating state. As a result, the signal voltage output from the photodiode 104-1 is applied to the second electrode while maintaining the inter-electrode voltage of the capacitor 10 in the first state. Since the voltage of the second electrode drops by the voltage sig1 compared to the first state, the voltage of the first electrode also drops by the voltage sig1 compared to the first state.

The peripheral circuit 133 reads the voltage of the first electrode (node C) at least in the second state. Since the voltage of the first electrode in the first state is the reference voltage VCC, if the reference voltage is known, the peripheral circuit 133 does not have to read the voltage of the first electrode in the first state. In this case, the peripheral circuit 133 detects the voltage sig1 by subtracting the voltage of the first electrode in the second state from the known reference voltage VCC.

The peripheral circuit 133 may correct the value of the known reference voltage VCC according to the temperature of the image sensor 100 or the like. Further, the peripheral circuit 133 may read the voltage of the first electrode (node C) and calculate the difference in each of the first state and the second state. As a result, the voltage sig1 corresponding to the accumulated charge amount in the photodiode 104-1, that is, the pixel value is calculated.

Next, the output of the photodiode 104-2 is read. At this time, the reset signal RST becomes H level, and the reset transistor 59 is turned on. Further, the selection transistor 61-2 is controlled to be on. As a result, the cathode voltage of the photodiode 104-2 is reset and the voltage at the node A becomes the reference voltage VCC. The subsequent operation is similar to that of the photodiode 104-1. The peripheral circuit 133 calculates the voltage sig2 according to the amount of accumulated charge in the photodiode 104-2 from the voltage read in the second state.

With such a configuration and operation, correlated double sampling of the outputs of the photodiodes 104-1 and 104-2 can be performed using one capacitor 10. Therefore, the circuit area can be reduced, and the error due to the variation of the capacitors can be eliminated. Further, in the present embodiment, since one photodiode 10 and the peripheral switch section are shared by the two photodiodes 104, the circuit area can be further reduced.

Although the capacitor 10 and the switch unit are provided corresponding to the respective photodiodes 104, as shown in the figure, a set of the capacitor 10 and the switch unit is provided for the plurality of photodiodes 104-1 and 104-2. You may provide in common. In addition, one set of capacitor 10 and switch unit may be provided for one photodiode 104.

The switch unit is provided for each capacitor 10. When the capacitor 10 is shared by two or more photodiodes 104, the capacitor 10 holds the output values of the respective photodiodes 104-1 and 104-2 in order. Further, the plurality of capacitors 10 may simultaneously hold the output values of the corresponding photodiodes 104-1 and 104-2.

FIG. 7 is a block diagram of the drive unit 502 that drives the image sensor 100. The drive unit 502 includes a drive control unit 420, a sensor control unit 441, a block control unit 442, a synchronization control unit 443, a signal control unit 444, and a pixel memory 414.

In the drive unit 502, the sensor control unit 441 transmits, to the light receiving substrate 113, a control signal related to charge accumulation and charge reading of each pixel in the pixel 150 of the image sensor 100. More specifically, the sensor control unit 441 transmits a reset signal and a pixel selection signal to the row control unit 200 corresponding to each of the unit blocks 131, and starts and ends the charge accumulation of the pixels 150 to be controlled. And control. Further, the pixel signal is output to the column transmission path 170 by transmitting a column selection signal or the like to the read pixel.

In the drive unit 502, the block control unit 442 transmits a control signal that specifies the unit block 131 to the light receiving substrate 113. More specifically, the block control unit 442 transmits a specific signal that specifies the unit block 131 to be controlled.

In the unit block specifying signal, a transfer pulse or the like received by each of the pixels 150 via the wiring Tx_j or the like is a logical product of a selection signal sent by the sensor control unit 441 and a specifying signal sent by the block control unit 442. As a result, the pixels 150 arranged substantially evenly on the light receiving substrate 113 can be controlled as independent blocks for each region.

The block control unit 442 can simultaneously transmit measurement signals to the plurality of unit blocks 131. As a result, the image pickup device 100 can operate the plurality of unit blocks 131 in synchronization with each other, or can execute the operation across the plurality of unit blocks 131.

In the drive unit 502, the synchronization control unit 443 supplies a synchronization signal to the light receiving substrate 113. Thereby, the plurality of elements mounted on the light receiving substrate 113 can be activated synchronously. Therefore, for example, by adjusting the synchronization signal, it is possible to perform random control, thinning-out control, or the like in which only specific pixels of pixels belonging to the same unit block 131 are controlled.

In the driving unit 502, the signal control unit 444 is responsible for timing control for the analog/digital conversion circuit 202, the CDS circuit 204, and the shift register 206. Accordingly, the image processing unit 511 can generate one image data at high speed and high quality from the pixel signals read from the plurality of unit blocks 131.

In the driving unit 502, the pixel memory 414 has a memory space for storing the pixel signal read from the pixel 150 and digitized. The pixel memory 414 may be provided corresponding to each unit block 131. The I/F circuit 418 is responsible for communication between the drive control unit 420 and the system control unit 501.

The pixel memory 414 may further include a data transfer interface that transfers a pixel signal to the image processing unit 511 in accordance with a delivery request. The data transfer interface is connected to a data transfer line connected to the image processing unit 511. Such a data transfer line is formed by, for example, a part of the data bus, and a delivery request from the system control unit 501 to the drive control unit 420 is executed by addressing using the address bus.

In addition to the addressing method, the pixel signal transmission by the data transfer interface uses, for example, both the rising edge and the falling edge of the clock signal used for synchronizing each circuit when performing the data transfer, and the double data is transmitted. A rate system can also be used. Further, by omitting a part of the procedure such as address designation, a burst transfer method can be adopted in which data is transferred all at once to increase the speed.

Further, the pixel signal transmission by the data transfer interface may be a bus method using a line in which the control unit, the memory unit and the input/output unit are connected in parallel, a serial method in which data is serially transferred bit by bit. It can also be used in combination.

By using the data transfer interface as described above, the image processing unit 511 can receive only the requested pixel signal. Thereby, for example, when forming a low-resolution image, image processing can be completed at high speed. The drive unit 502, the row control unit 200 in FIG. 5, and the peripheral circuit 133 function as a read unit that sequentially reads pixel signals from the pixels 150 across the plurality of unit blocks 131.

In the drive unit 502, the drive control unit 420 refers to the timing memory 430, converts the instruction received by the drive unit 502 from the system control unit 501 into a control signal that can be executed by each control unit, and transfers the control signal. The timing memory 430 is formed by a flash RAM or the like. In this way, the drive control unit 420 centrally controls the sensor control unit 441, the block control unit 442, the synchronization control unit 443, and the signal control unit 444 described above.

Further, the illustrated driving unit 502 includes a correlation calculating unit 472 and an image capturing control unit 474. The correlation calculation unit 472 detects the magnitude of rolling shutter distortion included in the pixel signal based on the pixel signal read from the pixel memory 414. The imaging control unit 474 determines a value set as an imaging condition for each unit block 131, such as a frame rate or an exposure time, based on the correlation value calculated by the correlation calculation unit 472. Operations of the correlation calculation unit 472 and the imaging control unit 474 will be described again with reference to other drawings.

FIG. 8 is a timing chart explaining the rolling shutter mode in the image sensor 100. In the rolling shutter mode, in each of the unit blocks 131, the operation of selecting the pixel 150 and accumulating the pixel signal and the operation of reading the accumulated pixel signal are sequentially performed for each row.

In the illustrated example, the drive unit 502 first supplies a drive signal to the wirings Rst_1 and Tx_1 for the pixels 150 in the first row via the row control unit 200. As a result, charge accumulation in the pixels 150 in the first row of each unit block 131 is started.

Next, the driving unit 502 gives a driving signal to the wiring Sel_1 for the pixel 150 in the first row through the row control unit 200 at the end of the accumulation period. As a result, the reading of the pixel signals of the pixels 150 in the first row is started. The reading period is a time period until a pixel signal is read from each pixel 150 in the first row, processed by the analog/digital conversion circuit 202 and the CDS circuit 204, and sequentially written in the pixel memory 414 from the shift register 206. Including.

Similarly, a drive signal is applied to the wirings Rst_2, Tx_2, and Sel_2 for the pixels 150 in the second row to perform storage and reading. The length of the accumulation period for the pixels 150 in the second row is the same as that of the pixels 150 in the first row, but the pixels are delayed by a reading period in time. A drive signal is applied to the wirings Rst_l, Tx_l, and Sel_l from the third row to the L-th row, and accumulation and reading are sequentially performed.

The order of selected rows is the same for any of the plurality of unit blocks 131, and in the illustrated example, for each of them, one row is selected from the row on the −Y side to the row on the +Y side. To be done. Further, the storage and read timings of the same row are synchronized among the plurality of unit blocks 131. However, at least one of the accumulation and the reading of the same row may not be synchronized among the plurality of unit blocks 131.

FIG. 9 is a schematic diagram showing a subject 300 imaged in the rolling shutter mode by using the image sensor 100. The subject 300 is shown in a state where the image light immediately before entering the image sensor 100 is viewed from the +Z side.

The subject 300 has a mixture of a straight line image 301 that moves at high speed at V ₁ , a straight line image 302 that moves at low speed at V ₂ and a stationary straight line image 303. The straight line images 301, 302, and 303 are vertical straight lines when the image light immediately before entering the image sensor 100 is viewed from the +Z side. Further, the moving straight line images 301 and 303 both move horizontally in the drawing in the −X direction.

FIG. 10 is a schematic diagram showing captured images 311, 312, 313 obtained when the subject 300 including the moving linear images 301, 302, 303 is captured by the image sensor 100 operating in the rolling shutter mode. .. The picked-up images 311, 312, 313 are drawn in the direction in which they are output, displayed, etc.

It is assumed that the captured images 311, 312, 313 are captured by 12 unit blocks 131 each including four vertical and horizontal pixels 150. As described with reference to FIG. 8, in the rolling shutter mode, the timing at which pixel signals are accumulated in one unit block 131 is sequentially shifted row by row. Therefore, the linear images 301 and 303 moving on the subject 300 are imaged at different positions according to the timing of accumulating pixel signals.

In each of the unit blocks 131 of the A column and the B column that capture the moving linear images 301 and 302, the upper row in the drawing and the lower row in the drawing have different timings for accumulating pixel signals. Therefore, even though the straight line image 301 on the subject 300 is a vertical straight line, the captured images 311 and 312 are both inclined. That is, in the captured image 311, a row in which the pixel signal accumulation period is later in time is deviated in the +X direction, which is the moving direction, and an image is captured, and rolling shutter distortion occurs in each unit block 131.

It should be noted that the above-mentioned shift means a displacement in each of the four unit blocks 131 with the row in which the pixel signal is first accumulated as the initial position. In other words, in each unit block 131, it can be considered that the row in which the pixel signal is first stored (the bottom row in the illustrated example) has no deviation. Therefore, in the entire image sensor 100, as indicated by a dotted line B in the drawing, in the rows of the pixels 150 adjacent to each other at the boundary between the unit blocks 131 adjacent to each other, the captured image 311 is discontinuously displaced.

The shift due to the rolling shutter distortion as described above becomes more noticeable as the horizontal component of the moving speed V ₀ of the linear images 301 and 302 increases. In other words, by detecting the shift amount of the captured images 311 and 312 at the boundary of the unit block 131, it is possible to detect the moving speed of the linear images 301 and 302 moving on the subject 300.

FIG. 11 is a flowchart illustrating an operation procedure of the imaging control unit 474 in the drive unit 502. The illustrated procedure is started at the time when the pixel signals corresponding to the captured images 311, 312, and 313 are stored in the pixel memory 414 from the unit block 131 to be controlled.

The imaging control unit 474 first selects one of the unit blocks 131 included in the imaging device 100 as a control target (step S101). Next, the imaging control unit 474 detects the moving speed of the moving component included in the subject 300 of the selected unit block 131 (step S102). The contents of step S102 will be described later with reference to FIG.

When the moving speed of the moving component in the subject 300 is known, the imaging control unit 474 determines the value of the shooting condition in the unit block 131 to be controlled according to the found moving speed. The value of the shooting condition determined here is, for example, a frame rate when a moving image is captured using the image sensor 100.

That is, when the moving component of the subject 300 is moving at a high speed, the frame rate in the unit block 131 is increased to capture a clearer image. On the other hand, when the moving speed of the moving component of the subject 300 is low, or when the subject is stationary, the amount of information of the pixel signal output from the image sensor can be reduced by not increasing the frame rate or rather decreasing it. In this case, even if the frame rate is reduced, the quality of the captured image does not deteriorate.

Therefore, the imaging control unit 474 can calculate an appropriate frame rate in the unit block 131 to be controlled by acquiring the moving speed of the moving component of the subject 300 detected in step S102 (step S103). Furthermore, the imaging control unit 474 sets the calculated frame rate in the unit block 131 to be controlled (step S104), and causes the imaging operation to be performed under appropriate imaging conditions.

In this way, the imaging control unit 474, which has calculated and set the value of the imaging condition for one unit block 131, next checks whether or not there remains any unit block 131 for which the imaging condition has not been set (step S105), if such a unit block 131 remains, the unit block 131 is specified as a control target (step S101), and the procedure from step S102 is executed again.

If it is determined in step S105 that the imaging conditions have been set for all the unit blocks 131, the illustrated procedure is ended. However, when the image sensor 100 continues to image, the series of procedures from step S101 may be repeated again.

FIG. 12 is a flow chart showing an operation procedure in the case of detecting the moving speed of the moving component in the subject 300, which corresponds to the content of step S102. The illustrated procedure is started when the pixel signal from the image sensor 100 is stored in the pixel memory 414.

Each of the unit blocks 131 generates a pixel signal in the rolling shutter mode. The correlation calculation unit 472 reads, from the pixel memory 414, the pixel signals corresponding to the rows of the pixels 150 that are adjacent to each other at the boundary of the unit blocks 131 for the unit blocks 131 that are adjacent in the column direction (step S201).

Next, the correlation calculation unit 472 calculates the correlation value of the row of the pixels 150 adjacent to each other at the boundary of the unit block 131 (step S202). It can be said that the magnitude of the correlation value is the degree of coincidence of the pixel signals in one column, and the higher the correlation value, the more the images in the adjacent rows match.

An example of the correlation value is calculation of a residual sum of squares obtained by squaring the difference between pixel signals in rows of adjacent pixels and taking the sum thereof. Here, the smaller the residual sum of squares, the larger the correlation.

The correlation calculation unit 472 compares the value of the residual sum of squares with a predetermined threshold value to determine whether rolling shutter distortion has occurred (step S203). When the magnitude of the residual sum of squares is larger than the threshold value (step S203: NO), the correlation calculation unit 472 determines that the component of the subject 300 has not moved, and ends the process.

On the other hand, when the magnitude of the residual sum of squares is smaller than the threshold value (step S203: YES), it is determined that the moving component of the subject 300 is moving, and the correlation value corresponding to the moving speed is calculated. That is, the correlation calculation unit 472 shifts one of the rows of the adjacent pixels 150 at the boundary of the unit block 131 by one pixel in the row direction with respect to the other (step S204), and calculates the residual sum of squares again (step S204). Step S205). Hereinafter, the correlation calculation unit 472 repeats the calculation of the residual sum of squares until the number of columns of pixels 150 in the unit block 131 is shifted by p pixels (step S206: NO).

After shifting by p pixels (step S206: YES), the correlation calculation unit 472 specifies the shift that has occurred at the boundary of the unit block 131 due to the rolling shutter distortion as the number of pixels (step S207). In this way, the magnitude of the rolling shutter distortion generated by the moving component of the subject 300 in the unit block 131 is known, and the moving speed of the moving component can be detected.

FIG. 13 is a flowchart illustrating another operation procedure of the imaging control unit 474 in the drive unit 502. The illustrated operation procedure has the same contents as the operation procedure shown in FIG. 11 except for the steps described below. Therefore, common steps are given the same reference numerals to omit redundant description.

The operation procedure illustrated in FIG. 13 is similar to that of FIG. 11 in that the imaging control unit 474 adds step S106 that also refers to the brightness information obtained from the photometric unit 503 before determining the imaging condition in step S103. The procedure is different. That is, in the operation procedure illustrated in FIG. 13, the imaging control unit 474 also refers to the subject brightness detected by the photometry unit 503 in addition to the moving speed of the moving component of the subject 300 detected from the calculation result of the correlation calculation unit 472. , Determine imaging conditions.

In addition, in the operation procedure shown in FIG. 13, in place of step S103 in FIG. 11, when the imaging control unit 474 determines the value of the imaging condition, the table is searched to select the determined value in step S107. The operation procedure is different from that shown in FIG. That is, in the operation procedure illustrated in FIG. 13, the imaging control unit 474 searches for the imaging condition to be determined from the candidate values stored in the table in advance, using the acquired moving speed and subject brightness as keys.

FIG. 14 is a diagram exemplifying the contents of the table referred to by the imaging control unit 474 in step S107 as described above. As shown in the figure, the table is based on two reference conditions of the moving speed of the moving component of the subject 300 and the subject brightness, and two imaging conditions of the frame length and the exposure time length in the unit block 131. The values are stored in association with each other. As a result, even when the types of information to be referred to, the types of imaging conditions to be determined, the number of unit blocks 131 to be controlled, and the like increase, the processing load can be suppressed.

When the unit block 131 has a color filter such as a Bayer array, it is preferable to execute the above operation before performing the interpolation process. When the analog/digital converted pixel signal is output as a captured image converted into a predetermined format such as JPEG, the operation of FIG. 13 is performed after the analog/digital conversion and It is preferably performed before conversion to format. In addition, it is preferable that at least the correlation calculation unit 472 and the imaging control unit 474 be provided in the signal processing board 111.

As described above, in the image pickup device 100, the set value of the image pickup condition can be changed for each unit block 131 according to the image pickup target of the subject 300. Therefore, even when an object whose moving speed, brightness, etc. are different from each other is included in one object scene, it is possible to individually set appropriate imaging conditions for each unit block 131. As a result, the quality of the finally obtained image data can be improved.

Further, when the image pickup target corresponding to any of the unit blocks 131 is stationary or is sufficiently bright, the image pickup element 100 lowers the frame rate or shortens the exposure time to reduce the image pickup element. The pixel signal output from 100 can be reduced. This facilitates the transfer of pixel signals and reduces the load of image processing.

FIG. 15 is a timing chart illustrating a global shutter mode which is another operation mode of the unit block 131 in the image sensor 100. In the global shutter mode, all the pixels 150 included in the unit block 131 are simultaneously reset and then selected to accumulate pixel signals. In the global shutter mode, the accumulated pixel signals may be sequentially read row by row as in the rolling shutter mode.

In the global shutter mode as described above, since the pixel signals are accumulated at the same timing in the unit block 131, the rolling shutter distortion does not occur in the captured image 311. Therefore, when a particularly large rolling shutter distortion occurs in a unit block 131, the imaging control unit 474 may switch the accumulation control in the unit block 131 to the global shutter mode.

However, the speed of the moving component of the subject cannot be detected for the unit block 131 operating in the global shutter mode in which rolling shutter distortion does not occur. Therefore, the imaging control unit 474 does not execute the operation of setting the imaging condition according to the moving speed of the subject for the unit block 131 that stores the pixel signal in the global shutter mode, but operates in the rolling shutter mode. The processing shown in FIG. 11 and FIG. 13 is executed only for the unit block 131 that is present.

Further, in the global shutter mode, since all the pixels 150 in the unit block 131 are reset all at once, a large current flows, charge leakage easily occurs between adjacent rows, and so on. , Noise is likely to be added to the captured image. Therefore, when the speed of the moving component of the subject decreases, the unit block 131 switched to the global shutter mode may be returned to the rolling shutter mode.

Therefore, in the image sensor that operates in the global shutter mode, the operation mode of the unit block 131 may be temporarily or intermittently set to the rolling shutter mode for the purpose of detecting the speed of the moving component of the subject. Since the operation in the rolling shutter mode is executed for the purpose of detecting the moving component of the subject, it is not necessary to generate image data from the pixel signal obtained in this process.

Although the present invention has been described using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It is apparent to those skilled in the art that various changes or improvements can be added to the above-described embodiment. It is apparent from the description of the scope of claims that the embodiments added with such changes or improvements can be included in the technical scope of the present invention.

The execution order of each process such as the operation, procedure, step, and step in the device, system, program, and method shown in the claims, the specification, and the drawings is, in particular, “before” or “prior to”. It should be noted that the output of the previous process can be realized in any order unless it is used in the subsequent process. Even if the operation flow in the claims, the description, and the drawings is described by using “first,” “next,” and the like for convenience, it means that it is essential to carry out in this order. Not a thing.

10 capacitor, 60 amplification transistor, 61-1 and 61-2 selection transistor, 59 reset transistor, 65 ground side transistor, 78 first transistor, 63 second transistor, 66 amplification transistor, 64 transfer transistor, 100 image sensor, 101 micro Lens, 102 color filter, 103 passivation film, 104, 104-1, 104-2 photodiode, 105 transistor, 106 photoelectric conversion element layer, 107 wiring, 108 wiring layer, 109 bump, 110 TSV, 111 signal processing substrate, 112 Memory substrate, 113 light receiving substrate, 131 unit block, 133 peripheral circuit, 150 pixels, 170 column transmission line, 172 column bus line, 200 row control unit, 202 analog/digital conversion circuit, 204 CDS circuit, 206 shift register, 300 subject , 301, 302, 303 linear images, 311, 312, 313 captured images, 414 pixel memory, 418 I/F circuit, 420 drive control unit, 430 timing memory, 441 sensor control unit, 442 block control unit, 443 synchronization control unit 444 signal control unit, 472 correlation calculation unit, 474 imaging control unit, 500 imaging device, 501 system control unit, 502 driving unit, 503 photometric unit, 504 work memory, 505 recording unit, 506 display unit, 511 image processing unit, 512 arithmetic unit, 520 imaging lens

Claims (12)

A first pixel that includes a first photoelectric conversion unit that converts light into electric charge; and a second pixel that includes a second photoelectric conversion unit that is arranged next to the first pixel in the column direction and that converts light into electric charge. A first control line connected to the first pixel and outputting a first control signal for controlling the first pixel; and a first control line connected to the second pixel for controlling the second pixel. A second control line that outputs two control signals and a first output line that is connected to the first pixel and that outputs a first signal based on the charge of the first photoelectric conversion unit, and is connected to the second pixel And a second output line that outputs a second signal based on the charge of the second photoelectric conversion unit ,
A calculation unit calculating a correlation with the previous SL second signal output before Symbol first signal output to the first output line and to said second output line,
Based on the correlation calculated by the calculation unit, and the timing of the first control signal is output to the first control line, and the timing of the second control signal is output to the second control line, the and control Gosuru control unit,
An imaging device including. And a timing at which the first control signal is output to the first control line and a timing at which the second control signal is output to the second control line so that the correlation value becomes high. The image pickup apparatus according to claim 1, which controls the. The first pixel includes a first transfer unit that is connected to the first control line and transfers a charge of the first photoelectric conversion unit according to the first control signal.
The said 2nd pixel is connected to the said 2nd control line, The 2nd transfer part for transferring the electric charge of the said 2nd photoelectric conversion part according to the said 2nd control signal is included, The claim|item 1 or claim 2 characterized by the above-mentioned. Imaging device. The image pickup device is connected to the first pixel, is connected to a third control line that outputs a third control signal for controlling the first pixel, and is connected to the second pixel, and controls the second pixel. A fourth control line for outputting a fourth control signal for
And a timing at which the third control signal is output to the third control line and a timing at which the fourth control signal is output to the fourth control line, based on the correlation calculated by the calculation unit. and timing that, the imaging apparatus according to claim 3 Gosuru wo system. And a timing at which the third control signal is output to the third control line and a timing at which the fourth control signal is output to the fourth control line so that the correlation value becomes high. The image pickup apparatus according to claim 4, which controls the. The first pixel is connected to a first floating diffusion to which the charge of the first photoelectric conversion unit is transferred and the third control line, and resets the potential of the first floating diffusion according to the third control signal. Including 1 reset section,
The second pixel is connected to the second floating diffusion to which the charge of the second photoelectric conversion unit is transferred and the fourth control line, and resets the potential of the second floating diffusion according to the fourth control signal. The imaging device according to claim 4 or 5 , including a 2 reset unit. The first pixel is connected to the first floating diffusion to which the charge of the first photoelectric conversion unit is transferred and the first control line, and resets the potential of the first floating diffusion according to the first control signal. Including 1 reset section,
The second pixel is connected to a second floating diffusion to which a charge of the second photoelectric conversion unit is transferred and the second control line, and resets a potential of the second floating diffusion according to the second control signal. The image pickup apparatus according to claim 1, further comprising a 2 reset unit. The image pickup device includes a first conversion circuit used to convert the first signal output to the first output line into a digital signal, and the second signal output to the second output line as a digital signal. The 2nd conversion circuit used in order to convert into an image pickup device according to any one of claims 1 to 7 . The imaging device includes a first semiconductor substrate on which the first photoelectric conversion unit and the second photoelectric conversion unit are arranged, and a second semiconductor substrate on which the first conversion circuit and the second conversion circuit are arranged. The image pickup apparatus according to claim 8 , further comprising: The imaging device according to claim 9, wherein the control unit is disposed on the second semiconductor substrate. The control unit includes a first circuit connected to the first control line and outputting the first control signal, and a second circuit connected to the second control line and outputting the second control signal. The imaging device according to claim 10, which has. The image pickup device according to claim 9 , wherein the first semiconductor substrate is laminated by the second semiconductor substrate.

Images