JP7321741B2 - Imaging device and its control method - Google Patents

Description

The present invention relates to an image pickup apparatus having a counting section that counts signals based on the output of a photoelectric conversion section, and a control method thereof.

During live view operation, in which the captured image of the subject is displayed in real time by the imaging device, the amount of data read from the imaging device is reduced, improving the frame rate and reducing power consumption.

On the other hand, as described in Japanese Unexamined Patent Application Publication No. 2002-200010, an image pickup device having a 1-bit AD converter and a counter for each pixel has been proposed. In such an image sensor, since AD conversion is performed for each pixel on the signal of the light receiving element, it is possible to eliminate the trade-off between the number of scanning lines and the readout speed of the image sensor that performs AD conversion for each column.

JP 2015-173432 A

With the image pickup device of Patent Document 1, it is not possible to reduce readout data even during the above-described live view operation, and as a result, it is not possible to improve the frame rate and reduce power consumption.
An imaging device according to the present invention includes a counting section that counts a signal based on an output of a photoelectric conversion section included in a pixel section. SUMMARY OF THE INVENTION It is an object of the present invention to provide an imaging apparatus capable of reducing readout data in a mode in which signals are read out from some of all pixels while suppressing an increase in circuit size.

A device according to an embodiment of the present invention is an imaging device including an imaging device having a plurality of pixel units, in which a signal generated from an output of a photoelectric conversion unit using the avalanche effect of the pixel unit is counted and held. counting means for controlling the counting means, control means for controlling the counting means, and signal processing means for performing signal processing using the output of the counting means. A mode is set in which the signal of the first photoelectric conversion unit among the plurality of photoelectric conversion units is read and the signal of the second photoelectric conversion unit is not read, thereby performing a thinning readout operation for the signal based on the output of the photoelectric conversion unit. The control means holds a first count value in the current frame of the captured image by the first counting means corresponding to the first photoelectric conversion section, and the second counting means corresponds to the second photoelectric conversion section. The second counting means performs control to hold a second count value corresponding to the first count value in the previous frame of the captured image, and the signal processing means controls the first and second count values. Signal processing is performed using the count value.

According to the present invention, it is possible to provide an imaging device capable of reducing readout data in a mode in which signals are read out from some pixel groups of all pixels while suppressing an increase in circuit size.

It is a figure which shows the structure of the unit pixel which concerns on 1st Embodiment of this invention. It is a figure which shows the structure of the counter which the unit pixel which concerns on 1st Embodiment has. It is a figure which shows the structure of the imaging device which concerns on 1st Embodiment. 4 is a timing chart showing driving of a unit pixel according to the first embodiment; It is a figure which shows the structure of the compression part which concerns on 1st Embodiment. 4 is a flowchart for explaining compression processing according to the embodiment of the present invention; It is a figure which shows the structure of the imaging device which concerns on embodiment of this invention. It is a figure which shows the structure of the unit pixel which concerns on 2nd Embodiment of this invention. It is a figure which shows the structure of the counter which the unit pixel which concerns on 2nd Embodiment has. It is a figure showing composition of an image pick-up element concerning a 2nd embodiment. 9 is a timing chart showing driving of a unit pixel according to the second embodiment;

Preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings.
[First embodiment]
A unit pixel 100, which is a constituent element of a pixel array that constitutes the solid-state imaging device according to the present embodiment, will be described with reference to FIG. FIG. 1 is a diagram showing the configuration of a unit pixel 100. As shown in FIG. A unit pixel 100 includes an avalanche photodiode (hereinafter referred to as APD) 101 , a quench resistor 102 , a waveform shaping circuit 103 and a counter 104 .

The APD 101 as a photoelectric conversion element is a semiconductor element utilizing the avalanche effect. The cathode of APD 101 is connected to a voltage source of reverse bias voltage _VAPD through quench resistor 102, and the anode of APD 101 is grounded. The APD 101 generates charges by avalanche multiplication when photons are incident. The generated charge is discharged through quench resistor 102 .

The waveform shaping circuit 103, whose input terminal is connected to the cathode of the APD 101, amplifies and edge-detects changes in potential associated with the generation and discharge of charges in response to incident photons, thereby generating a voltage pulse. Generate. Thus, the APD 101, the quench resistor 102, and the waveform shaping circuit 103 function as a 1-bit AD converter by converting the presence or absence of incident photons into voltage pulses.

The counter 104 counts the number of voltage pulses output by the waveform shaping circuit 103 and outputs a count result (count value). This makes it possible to output pixel values during the exposure period in multiple bits. Based on the enable signal CNT_EN, the control signal LOAD_EN, and the data signal LOAD_DATA input to the unit pixel 100, the counter 104 sets data in a flip-flop (400 in FIG. 2), which will be described later.

A specific configuration and operation of counter 104 will be described with reference to FIG. The counter 104 includes a flip-flop (hereinafter also referred to as FF) 400 , an addition section 401 , a counter selection section 402 and an AND element 403 . The enable signal CNT_EN and the voltage pulse PLS output from the waveform shaping circuit 103 are input to the AND element 403 to perform a logical product operation. The enable signal CNT_EN is a control signal that determines whether to count voltage pulses PLS. The counter 104 counts voltage pulses based on the enable signal CNT_EN.

Adder 401 adds the output of AND element 403 and the output of FF 400 and outputs the result to counter selector 402 . The output of the addition section 401 and the data signal LOAD_DATA are input to the counter selection section 402 . The counter selection unit 402 determines whether to set the data signal LOAD_DATA or the output of the addition unit 401 based on the control signal LOAD_EN.

The FF 400 holds the output signal determined by the counter selection unit 402 and outputs a counter value (count value) corresponding to the held signal. The FF 400 is initialized to an initial value of zero asynchronously to the clock by an asynchronous reset signal. In this embodiment, the clock and asynchronous reset signal are signals common to the entire imaging section of the imaging element.

Next, with reference to FIG. 3, the configuration of the imaging device 300 according to this embodiment will be described. The imaging device 300 has a configuration in which a plurality of unit pixels 100 are arranged in a two-dimensional array. For convenience, FIG. 3 shows only 3 columns×4 rows of pixels in the pixel array. A data signal LOAD_DATA for the unit pixels 100 on the first and third rows from the top of FIG. 3 is set to zero. The data signal LOAD_DATA for the unit pixels 100 on the second and fourth rows from the top in FIG. 3 is set with the counter values of the unit pixels 100 on the first and third rows in the same column. That is, when natural number variables i and j are used, a data signal corresponding to the counter value of the unit pixel 100 in the 2×i−1 row and j-th column is input to the unit pixel in the 2×i-th row and j-th column. be done.

The output of each unit pixel 100 is defined as follows with the direction from left to right in FIG. 3 as a reference direction.
・CNT00, CNT01, CNT02 in the first line.
- CNT10, CNT11, and CNT12 in the second line.
- CNT20, CNT21, and CNT22 in the third line.
· CNT30, CNT31, and CNT32 in the fourth line.
In the present embodiment, the thinning rate in the vertical direction is set to 1/2 when thinning readout is performed, and pixel values in the second and fourth rows are thinned out and read out.

The imaging device 300 includes a plurality of switch elements 301 and 303, a horizontal selection circuit 302, a timing generator (also referred to as TG) 304, a control section 305, a compression section 306, a vertical selection circuit 307, control signal lines 308, 310 and 311, line A memory 309 is provided. A control signal LOAD_EN is output from the control unit 305 to the control signal line 308, and the unit pixels in the same column in the first and second rows and the third and fourth rows operate like a shift register in units of two. . Control related to the output of the control signal LOAD_EN is performed using a switch element (not shown) provided inside the control unit 305 .

An enable signal CNT_EN for controlling whether or not the counter 104 of each unit pixel counts is output from the control unit 305 to the control signal lines 310 and 311 . The control unit 305 supplies the enable signals CNT_EN of the first and third rows to the unit pixels through the control signal line 310, and supplies the enable signals CNT_EN of the second and fourth rows to the unit pixels through the control signal line 311. supply the pixels. In the thinning readout mode of the imaging device 300, only the enable signal CNT_EN output to the control signal line 310 during the accumulation operation is controlled to be high (H) level. In addition, in the mode in which the all-pixel readout operation is performed, the enable signals CNT_EN output to each of the control signal lines 310 and 311 during the accumulation operation are controlled to become H level at the same timing.

Based on a counter (not shown), the TG 304 generates signals for notifying timings such as an imaging period and a transfer period, and outputs the signals to the vertical selection circuit 307 and the horizontal selection circuit 302 . In addition, in order to determine whether the coordinates of the pixels for which signal readout is being performed are odd rows or even rows, the TG 304 stores the vertical line numbers for which readout is being performed in the line memory 309 and compresses them. 306 is notified.

The vertical selection circuit 307 controls ON/OFF of the plurality of switch elements 301 for the vertical transmission line based on the timing notified by the TG 304 . When the switch element 301 is turned on, the counting result of the unit pixel 100 is transmitted to the vertical transmission line corresponding to the switch element 301 .

The horizontal selection circuit 302 controls a plurality of switching elements 303 for horizontal transmission lines based on the timing notified by the TG 304 . When the switch element 303 is turned on, the outputs of the vertical transmission lines are sequentially transmitted to the horizontal transmission lines.

The control unit 305 controls the control signal LOAD_EN and the enable signal CNT_EN supplied to the unit pixel 100 via the control signal lines 308 , 310 and 311 based on the imaging period and frame count notified by the TG 304 .

A line memory 309 stores the pixel data output to the horizontal transmission line based on the timing notified by the TG 304 . The compression unit 306 acquires the pixel data output to the horizontal transmission line and the pixel data from the line memory 309 based on the timing notified by the TG 304, and performs compression processing.

A method of driving the imaging device according to the present embodiment will be described with reference to FIG. FIG. 4 is a timing chart for explaining the thinning reading operation of the image sensor 300. FIG. FIG. 4 shows imaging driving in the unit pixel 100. By performing imaging driving in parallel in a plurality of unit pixels 100, an optical image is converted into a digital signal. In FIG. 4, the waveform generated by the APD 101 and the quench resistor 102 provided in the unit pixel 100 corresponding to CNT00 is denoted as APD00, and the voltage pulse output from the waveform shaping circuit 103 is denoted as PLS00. Also, the control signal output to the control signal line 308 is denoted as LOAD_EN308, and the enable signals output to the control signal lines 310 and 311 are denoted as CNT_EN310 and CNT_EN311. Furthermore, the output data of the horizontal transmission line is written as READ_DATA. CNT00 to CNT32 have already been described in FIG. Under the control of switch elements 301 and 303, count results of unit pixels 100 are sequentially output via vertical and horizontal transmission lines.

Time t200 to time t210 shown in FIG. 4 represent various timings related to the driving of the unit pixel 100 . A period from time t200 to time t203 is an initialization period, and a period from time t203 to time t204 is a first imaging period (first frame). The period from time t204 to time t207 is the first read and initialization period. A period from time t207 to time t208 is a second imaging period (second frame). A predetermined period after time t208 is a second reading and initialization period.

Time t200 represents the timing of starting initialization of the value of the FF 400 of the unit pixel 100. At this time, the control signal LOAD_EN308 becomes H level. At this time, the values of CNT00 to CNT32 are undefined (denoted as X).

At the next time t201, zeros are loaded into the unit pixels corresponding to CNT00 to CNT02 on the first row and the unit pixels corresponding to CNT20 to CNT22 on the third row in synchronization with a clock (not shown). Also, the CNT outputs of the unit pixels in the same columns in the first and third rows are loaded into the unit pixels corresponding to CNT10 to CNT12 in the second row and the unit pixels corresponding to CNT30 to 32 in the fourth row, respectively. . At time t201, since the count output of each unit pixel of the first and third rows at time t200 is X, X is loaded into each unit pixel of even-numbered rows.

At time t202, the unit pixels corresponding to CNT10 to CNT12 in the second row and the unit pixels corresponding to CNT30 to CNT32 in the fourth row are respectively output values of the unit pixels in the same columns in the first and third rows. Initialization is done by loading some zeros. At the same timing (time t202), the signal level of the control signal LOAD_EN308 becomes L.

At time t203, the signal level of the enable signal CNT_EN310 becomes H, and the exposure period starts. When light is incident on the APD 101 and the voltage pulse PLS00 rises, the value of CNT00 changes to a value obtained by adding 1 to the initial value at substantially the same timing. CNT01 and CNT02 on the first row and CNT20 to CNT22 on the third row are also processed in the same manner by changing the APD 101 of the corresponding unit pixel.

At time t204, the signal level of the enable signal CNT_EN310 becomes L, and the exposure period ends (end of the first imaging period). Reading of the counting results of CNT00 to CNT02 is started as C00 to C02, respectively, and reading of the counting results of CNT20 to CNT22 is started as C20 to C22, respectively. When the switch element 301 in the first row is enabled (ON), the values of CNT00 to CNT02 are output to the vertical transmission lines of each column. Thereafter, the switch elements 303 in the first column to the switch elements 303 in the third column are sequentially enabled, thereby outputting the pixel data of the first row to the horizontal transmission line. The same operation is performed up to the fourth row, and each pixel data is output to the line memory 309 and the compression section 306 .

At time t205, the signal level of the control signal LOAD_EN308 becomes H, and initialization is started in the first and third rows. At time t206, zero is loaded into the unit pixels corresponding to CNT00 to CNT02 on the first row and the unit pixels corresponding to CNT20 to CNT22 on the third row. C00 to C02 are loaded to the unit pixels corresponding to CNT10 to CNT12 on the second row, and C20 to C22 are loaded to the unit pixels corresponding to CNT30 to CNT32 on the fourth row. At the same timing (time t206), the signal level of the control signal LOAD_EN308 becomes L.

At time t207, the signal level of the enable signal CNT_EN310 becomes H, and the exposure period starts as at time t203. At time t208, similarly to time t204, the exposure period is completed and the readout operation is started. That is, the counting results obtained by the current exposure are output from the unit pixels 100 in the first and third rows. The count results obtained by the previous exposure are output from the unit pixels 100 in the second and fourth rows. At time t209, initialization is started as at time t205, and initialization is executed at time t210, as at time t206.

By performing processing in this way, it is not necessary to provide a special frame memory for thinning readout. That is, it is possible to output the output of the unit pixel 100 of the previous frame and the output of the unit pixel 100 of the current frame to the line memory 309 and the compression unit 306 in the data of one frame.

The line memory 309 holds the pixel data of the 1st and 3rd rows read out via the horizontal transmission path, and synchronizes with the period when the pixel data of the 2nd and 4th rows is read out. It outputs the held pixel data to the compression unit 306 .

Next, the compressor 306 will be described in detail with reference to FIGS. 5 and 6. FIG. FIG. 5 is a block diagram showing the configuration of the compression section 306. As shown in FIG. Compression section 306 includes quantization section 500 , entropy coding section 501 , code amount measurement section 503 , code amount control section 502 , subtractor 505 and selection section 506 . A subtractor 505 subtracts the pixel data of the second and fourth rows from the pixel data of the first and third rows read from the line memory 309 to calculate a difference. Selection section 506 determines whether or not to output the inter-frame difference signal from subtractor 505 based on the inter-frame difference selection signal. The inter-frame difference selection signal is a signal corresponding to the period during which the inter-frame difference signal is output.

The readout data of the unit pixels 100 in the second and fourth rows are data output from the unit pixels 100 arranged in the same column in the first and third rows of the previous frame. Therefore, the difference between the read data of the first and third rows and the read data of the second and fourth rows is inter-frame difference data.

The selection unit 506 outputs the inter-frame difference signal during the period in which the unit pixels 100 in the second and fourth rows output. On the other hand, the selection unit 506 outputs data 0, which is invalid data, while data is being input from the unit pixels 100 on the first and third rows. In this case, the compression section 306 does not output encoded data. In the first frame, the outputs of the unit pixels of the second and fourth rows, which are the outputs of the previous frame, are zero at time t202 (FIG. 4). Therefore, the difference output between frames is the difference between the output of the current frame and zero, so it is synonymous with the output of only the current frame.

In this embodiment, the configuration in which the unit pixels in the second and fourth rows are initialized at time t202 has been described, but this is not the only option. For example, by selecting the output of the current frame only for the first frame, intra-frame compression can be performed for the first frame. Also, by compressing only the current frame in units of a predetermined number of frames, it is possible to construct a GOP (Group Of Picture).

A quantization unit 500 shown in FIG. 5 calculates a difference value between adjacent pixels in an encoding block and generates quantized data of the difference value. An entropy coding unit (hereinafter simply referred to as a coding unit) 501 assigns codes to the quantized difference values to generate coded data. Golomb coding and Huffman coding are known as entropy coding methods, but any coding method can be used. A code amount measurement unit 503 measures the amount of encoded data for each encoded block. The code amount control unit 502 controls the code amount based on the encoded data amount measured by the code amount measurement unit 503 .

FIG. 6 is a flowchart for explaining a series of processes performed by the quantization unit 500 and subsequent components in the compression unit 306 . Components after the quantization unit 500 are an encoding unit 501 , a code amount control unit 502 , and a code amount measurement unit 503 . When compression processing is started in FIG. 6, pixel data in units of encoding blocks is input to the quantization unit 500 .

First, in S1101, the quantization unit 500 calculates the difference value between adjacent pixels in the encoding block. Next, in S1102, the quantization unit 500 performs quantization processing using a predetermined value on the difference value calculated in S1101. The predetermined value is a parameter that determines the quantization step, and is dynamically determined by code amount control section 502 . Hereinafter, the predetermined value will be referred to as QP (Quantization Parameter). The calculated difference value is divided by QP and quantized by rounding off the fractional part of the division result.
Next, in S1103, the encoding unit 501 assigns codes to the difference values quantized in S1102. Next, in S1104, the code amount measurement unit 503 measures the amount of encoded data (encoded data amount). The code amount measuring unit 503 determines whether or not the encoded data amount is within a target data amount (hereinafter referred to as a target encoded data amount). In this embodiment, an encoded data amount that fits within the output band of the image sensor 300 is set as the target encoded data amount.

If it is determined in S1104 that the encoded data amount is not within the target encoded data amount, the process proceeds to S1105, and if it is determined that the encoded data amount is within the target encoded data amount , the process proceeds to S1106.

In the Q value control shown in S1105, the code amount control unit 502 selects QP. In general, QP includes a parameter with a high compression ratio for a natural image but with significant deterioration in image quality, and a parameter with a low compression ratio but little deterioration in image quality. In fixed-length compression, a plurality of such quantization parameters are used, and when the target compression cannot be achieved, the QP is changed again and compression processing is performed. After the process of S1105, the process returns to S1102 to continue the process, and code allocation is performed again according to the difference between the encoded data amount and the target encoded data amount.

Also, when proceeding from S1104 to S1106, output processing of encoded data is performed in S1106. At the time of output, the QP used for quantization of each encoded block, code allocation information (encoding parameter) at the time of encoding, and a flag signal indicating whether intra-frame compression or inter-frame compression are output. It is output in association with data (encoded data).
By performing processing in this way, it is possible to perform compression with reference to information between frames in the output of the image sensor 300 without having a frame memory.

In this embodiment, the configuration having only one horizontal transmission line has been described, but this is not the only option. For example, as a configuration of the imaging device 300, a configuration is assumed in which two horizontal transmission lines are provided and two lines can be read out simultaneously. In this case, by deleting the line memory 309 and synchronizing the odd line data and the even line data and inputting them to the compression unit 306, the memory can be further reduced. Also, in the present embodiment, processing performed at a thinning rate of 1/2 in the vertical direction has been described, but the thinning rate in the vertical direction is not limited. For example, an imaging device that performs processing at a thinning rate of 1/4 in the vertical direction can simultaneously output data for four frames. It may be applied not only to the vertical thinning process but also to the horizontal thinning process.

Further, in an image pickup device having functional pixels, counters assigned to functional pixels may be utilized. A functional pixel is, for example, a part of sub-pixels in a split-pupil imaging device having a plurality of sub-pixels (photoelectric conversion units) sharing one microlens. For example, one pixel section has a microlens, and first and second photoelectric conversion sections that receive light from an object via the microlens and perform photoelectric conversion. Acquiring a phase difference signal from the output of the first photoelectric conversion unit and the output of the second photoelectric conversion unit by correlation calculation, and acquiring the imaging signal by adding each output of the first and second photoelectric conversion units. can be done.

Next, with reference to FIG. 7, an overview of the configuration of an imaging device 600 using the imaging device 300 will be described. The optical system unit 601 includes a focus lens for focus adjustment, a movable lens (shift lens, etc.) that constitutes an imaging optical system, a fixed lens, a shutter, an aperture, a lens control unit, and the like. An optical system unit 601 forms an optical image on the imaging device 300 . The imaging device 300 photoelectrically converts the subject image formed by the optical system unit 601 into an electrical signal, and further compresses the pixel values by the compression unit 306 and outputs the signal to the decompression unit 602 .

The decompression unit 602 performs decompression processing based on the encoding parameter associated with the compressed data in the compression unit 306 and the flag signal indicating intra-frame compression or inter-frame compression. The decompression unit 602 synthesizes the decompression results of the even-numbered lines and the odd-numbered lines into one sheet of image data, and outputs the image data to the capture unit 603 . At this time, if there is a flag signal indicating intra-frame compression, the decompression unit 602 does not read the image data of the previous frame from the storage unit (not shown), and stores the decompression result of the intra-frame compressed data in the capture unit 603 and the storage unit (not shown). output to the When there is a flag signal indicating inter-frame compression, the decompression unit 602 reads the decompression result of the previous frame stored in the storage unit (not shown), performs inter-frame reference, and decompresses the current frame.

The capture unit 603 determines the valid period and type of the pixel signal, and outputs data to the digital signal processing unit 604 . A digital signal processing unit 604 performs predetermined processing on the data acquired from the capture unit 603 . The predetermined processing is digital signal processing typified by white balance correction, synchronization processing, and moving image encoding processing. Image data (stream data) processed by the digital signal processing unit 604 is output to the external display device 605 . The external display device 605 has a display device such as a liquid crystal display, and displays an image on the screen according to the image data output from the digital signal processing section 604 .

In this embodiment, the flip-flops of the counters not used in the thinning readout mode are loaded with the output of the counter in the same column of another row at a predetermined timing, thereby storing the output of the counter one frame before. According to the present embodiment, in an imaging device having a 1-bit AD conversion section and a counter and having a thinning readout mode, compression can be performed by referring to signals between frames without using a frame memory. By adopting such a configuration, it is possible to reduce the frame rate drop due to the throughput of the transmission path of the captured image and the power consumption due to the readout of the image sensor.

[Second embodiment]
Next, a second embodiment of the invention will be described. In this embodiment, as an example other than the configuration in which the output of the counter is loaded into another flip-flop, the effect similar to that of the first embodiment can be obtained by switching the counting destination counter for the output pulse from the waveform shaping circuit 103. An image sensor that can be used will be described. In the following, differences from the first embodiment will be explained, and detailed explanations thereof will be omitted by using the same reference numerals and symbols as those of the first embodiment.

The configuration of the imaging element in this embodiment will be described with reference to FIGS. 8 to 10. FIG. A unit pixel 700 of this embodiment has the same configuration as the unit pixel 100 of FIG. Differs from one embodiment. That is, the unit pixel 700 has components for two pixels in the first embodiment.

The first pixel configuration section shown on the upper side of FIG. 8 has a first APD 101, a first quench resistor 102, a first waveform shaping circuit 103, an output switching section 701, and a first counter 702, and is always read out. corresponds to pixels. On the other hand, the second pixel configuration portion shown on the lower side of FIG. and correspond to pixels to be thinned out.

The enable signal CNT_EN, the reset signal SRESET0, and the output signal of the output switching unit 701 are input to the first counter 702 . The first counter 702 is loaded with zero by the reset signal SRESET0. The output switching unit 701 switches the output destination of the waveform shaping circuit 103 and outputs a signal to the first counter 702 or the OR element 704 .

Also, the enable signal CNT_EN, the reset signal SRESET1, and the output signal of the OR element 704 are input to the second counter 702 . The second counter 702 is loaded with zero by the reset signal SRESET1. An AND element 703 is a logic element for performing control according to whether or not the thinning readout mode is set. An OR element 704 is a logic element that performs a logical sum operation between the output of the output switching section 701 and the output of the AND element 703 .

The operation of each part will be described. The output switching unit 701 of the first pixel configuration unit switches which of the two counters 702 to output the output of the first waveform shaping circuit 103 based on the selection signal CNT_SEL output from the control unit 305. . When the signal level of the selection signal CNT_SEL is L, the output switching section 701 outputs a pulse to the first counter 702 . Also, the output switching unit 701 outputs a pulse to the second counter 702 via the OR element 704 when the signal level of the selection signal CNT_SEL is H.

A control signal MBK_EN indicating the thinning mode and the output of the second waveform shaping circuit 103 are input to the AND element 703 . The AND element 703 masks the output of the second waveform shaping circuit 103 corresponding to the pixels to be thinned out when the signal level of the control signal MBK_EN is H. When the signal level of the control signal MBK_EN is L, the AND element 703 passes the output of the second waveform shaping circuit 103 as it is and outputs it to the OR element 704 .

The output from the output switching section 701 and the output from the AND element 703 are input to the OR element 704 . The OR element 704 outputs to the second counter 702 a signal representing the result of the OR operation between the output of the output switching section 701 operating based on the selection signal CNT_SEL and the pulse based on the second waveform shaping circuit 103 . .

The configuration of the counter 702 will be described with reference to FIG. Since the first and second counters 702 have the same configuration, they will be described together. A difference from the configuration shown in FIG. 2 is that the counter selection unit 402 described in the first embodiment is changed to a counter selection unit 1002 .

The output of the addition unit 401 and data 0 are input to the counter selection unit 1002 , and a signal selected according to the reset signal SRESET (SRESET0 or SRESET1) is output to the FF 400 . When the reset signal SRESET is input, the counter selection unit 1002 selects data 0 in synchronization with a clock (not shown). Thereby, control is performed to set the value held in the FF 400 to zero. Further, the counter selection unit 1002 selects the output of the addition unit 401 and outputs it to the FF 400 when the reset signal SRESET is not input.

Next, referring to FIG. 10, the configuration of the imaging device 800 will be described. The difference from the imaging element 300 described in the first embodiment is that the pixels in each of the odd-numbered rows and the even-numbered rows are each composed of two pixels per column, that is, the unit pixels 700 having first and second pixel configuration portions. It is the point that For example, among the six unit pixels 700 shown in FIG. 10, the unit pixel 700 shown at the upper left end is the unit pixel 700 shown at the left end of the first row and the second row among the twelve unit pixels 100 shown in FIG. It corresponds to one unit pixel. The control unit 305 outputs the control signal MBK_EN, the selection signal CNT_SEL, and the reset signals SRESET0 and SRESET1 to the unit pixel 700, respectively.

The thinning readout operation of the image sensor 800 will be described with reference to the timing chart of FIG. FIG. 11 shows imaging driving in one unit pixel 700. By performing imaging driving in parallel in a plurality of unit pixels 700, an optical image is converted into a digital signal. The symbols shown in FIG. 11 follow the notation used in FIG.

Time t900 to time t909 shown in FIG. 11 represent various timings related to the driving of the unit pixel 700. FIG. A period from time t900 to time t902 is an initialization period, and a period from time t902 to time t903 is a first imaging period (first frame). The period from time t903 to time t906 is the first read and initialization period. A period from time t906 to time t907 is a second imaging period (second frame). A predetermined period after time t907 is a second reading and initialization period.

In the unit pixel 700 corresponding to CNT00, APD00 represents a waveform generated by the first APD 101 and the quench resistor 102 of the first pixel configuration portion that performs constant readout. Similarly, the voltage pulse output from the first waveform shaping circuit 103 is represented by PLS00.

Time t900 is the timing for initializing the value of FF 400 in unit pixel 700, and the signal level of reset signal SRESET0 for synchronous resetting of odd rows becomes H. At this time, the values of CNT00 to CNT32 are X (undefined). Further, the signal level of the control signal MBK_EN is set to H from time t900 as the setting of the thinning readout mode.

At time t901, zero is loaded into FF400 of the counter that outputs CNT00 to CNT02 on the first row and CNT20 to CNT22 on the third row in synchronization with a clock (not shown), and the signal level of the reset signal SRESET0 becomes L. becomes.

At time t902, the initialization period is completed, the signal level of the enable signal CNT_EN becomes H, and the exposure period starts. When light is incident on the APD 101 and the voltage pulse PLS00 rises, the value of CNT00 changes to a value obtained by adding 1 to the initial value at substantially the same timing. Similar processing is performed for CNT01 and CNT02 on the first row and CNT20 to CNT22 on the third row by changing the APD 101 of the corresponding pixel configuration portion.

At time t903, the signal level of the enable signal CNT_EN becomes L, and the exposure period is completed. Reading out of the count results of CNT00 to CNT02 is started as C00 to C02, and the count result of CNT20 to CNT22 is started as C20 to C22. When the switch element 301 in the first row is enabled (ON), the values of CNT00 to CNT02 are output to the vertical transmission lines of each column. Thereafter, the switch elements 303 in the first column to the switch elements 303 in the third column are sequentially enabled (ON), thereby outputting the pixel data of the first row to the horizontal transmission line. After that, the same operation is performed up to the fourth line. The pixel data are output to the line memory 309 and compression section 306 .

In this embodiment, unlike the first embodiment, the counting result of the even-numbered rows does not become zero in the first frame. Therefore, it is possible to perform intra-frame compression by inputting the odd-numbered row output to the quantization unit 500 without calculating the difference between the odd-numbered row output and the output from the line memory 309 . Alternatively, by setting the signal level of the reset signal SRESET1 to H at time t900 and setting the signal level to L at time t901, even-numbered rows can be initialized in the same manner as odd-numbered rows.

At time t904, the signal level of the reset signal SRESET1 becomes H, and initialization of the even-numbered rows is started. Also, by setting the signal level of CNT_SEL to H, the count destination is changed to FF400 of the second counter corresponding to the pixels to be thinned out (second pixel component). At time t905, zero is loaded into the FF400 of the counter that outputs CNT10 to CNT12 on the second row and CNT30 to CNT32 on the fourth row in synchronization with a clock (not shown), and the signal level of the reset signal SRESET1 becomes L. becomes.

At time t906, the signal level of the enable signal CNT_EN becomes H as at time t902, and the exposure period starts. When light is incident on the APD 101 and the voltage pulse PLS00 rises, the value of the CNT 10 changes to a value obtained by adding 1 to the initial value at substantially the same timing. CNT11 and CNT12 on the second row and CNT30 to CNT32 on the fourth row are processed in the same way by changing the APD 101 of the corresponding second pixel configuration portion.

At time t907, the exposure period is completed, the signal level of the enable signal CNT_EN becomes L, and reading is started. At time t903, the first to fourth row counter FF 400 outputs were sequentially transferred. done. The reason for this is that, in the first frame, the odd rows are the counting result of the current frame, whereas in the second frame, the FFs 400 that are counting destinations are replaced with the FFs 400 corresponding to the pixels in the even rows. By controlling in this way, it is possible to align the data output to READ_DATA for each frame.

At time t908, the signal level of the selection signal CNT_SEL becomes L, the signal level of the reset signal SRESET0 becomes H, and initialization starts. At time t909, the FF400 of each counter in the first and third rows is set to zero, and the signal level of the reset signal SRESET0 becomes L. Thereafter, the operation control from time t902 to time t909 described above is repeated.

In the present embodiment, control is performed to switch the counter, which is the counting destination, on a frame-by-frame basis. Therefore, as in the first embodiment, the counting results of the previous frame and the current frame can be read out in one frame of data without using a frame memory for thinning out the signals of the image sensor.

[Other embodiments]
In the above-described embodiment, signals are read out simultaneously for odd-numbered rows and even-numbered rows, data between lines are referred to in the compression unit before output from the image sensor, and the data of the previous frame is thinned out and output from the image sensor by not outputting the data. . However, the present invention is not applied only to data compression for the purpose of reducing power consumption. For example, in an embodiment in which long-time exposure is performed in even-numbered frames and short-time exposure is performed in odd-numbered frames, image data in dark and bright areas are synthesized by an image synthesizing unit like HDR (high dynamic range) processing. The amount of information may be reduced.

HDR processing is a well-known technique for expanding the dynamic range, so only a brief description will be given. First, the short-second exposure time in the imaging unit is denoted as t1, and the long-second exposure time is denoted as t2. In this case, a process of multiplying the signal acquired by short-second exposure by a gain value corresponding to t2/t1 is performed. Next, a process of comparing the pixel value with the upper limit value is performed. For example, the upper limit of pixel values is set to a value corresponding to 12 bits. When the pixel value of the image obtained by the long-second exposure exceeds the upper limit value, the process of selecting the pixel value of the image obtained by multiplying the signal obtained by the short-second exposure by the gain value is performed. Further, when the pixel values of the image obtained by the long-second exposure are equal to or less than the upper limit value, a process of selecting the pixel values of the image obtained by the long-second exposure is performed.

In the above embodiment, the configuration using the register group composed of the FFs 400 included in the counter of the pixel unit has been described. In other words, the register group corresponds to a plurality of pixel portions periodically arranged in the row direction or the column direction in the pixel array, and is composed of counter flip-flops included in each pixel portion. The technical scope of the present invention is not limited to such examples. An embodiment is possible in which the register group is configured using various storage elements (SRAM, etc.) capable of holding the count value of the signal.

Further, in the above-described embodiment, the configuration example in which the image sensor 300 includes a signal processing unit (compression unit 306) inside has been shown. The present invention is not limited to this example, and can be applied to a configuration in which a signal processing section is provided outside the imaging device 300. FIG. Alternatively, in a stacked-type imaging device, the substrate portion of the imaging layer includes a pixel array composed of a large number of unit pixels 100 or 700, and the substrate portion of the circuit layer includes a signal processing portion.

In an example of application to an imaging device having a split-pupil imaging device, each pixel unit has a microlens and a plurality of photoelectric conversion units that receive light through the microlens and perform photoelectric conversion. For example, in a two-part configuration having first and second photoelectric conversion units, the first register group is composed of registers provided with a plurality of counters respectively corresponding to the plurality of first photoelectric conversion units. The second register group is configured by registers included in a plurality of counters respectively corresponding to the plurality of second photoelectric conversion units. Performing focus adjustment based on the defocus amount calculated by imaging plane phase difference detection, acquiring a plurality of image signals from different viewpoints based on each output of the first and second photoelectric conversion units, and performing stereoscopic display. etc.

According to the above-described embodiment, in an imaging device in which each unit pixel, which is a constituent element of a pixel array, has a 1-bit type AD converter and a counter, a circuit scale is set in a mode in which signals are read out from some pixel groups in all pixels. It is possible to suppress the increase in the production cost and the increase in the production cost. By referring to the signals of the current frame and the previous frame related to the captured image, it is possible to provide an imaging device capable of signal processing represented by data compression, image synthesis, and the like.

100,700 unit pixel 101 APD (avalanche photodiode)
104,702 counter 300,800 image sensor 305 control unit 306 compression unit

Claims (11)

An imaging device comprising an imaging element having a plurality of pixel units,
counting means for counting and holding a signal generated from an output of a photoelectric conversion unit using an avalanche effect of the pixel unit;
a control means for controlling the counting means;
A signal processing means for performing signal processing using the output of the counting means,
A mode is set in which the signal of the first photoelectric conversion unit among the plurality of photoelectric conversion units is read and the signal of the second photoelectric conversion unit is not read, thereby performing a thinning readout operation for the signal based on the output of the photoelectric conversion unit. The control means holds a first count value in the current frame of the captured image by the first counting means corresponding to the first photoelectric conversion section, and the second counting means corresponds to the second photoelectric conversion section. performing control to hold a second count value corresponding to the first count value in the previous frame of the captured image by the second counting means;
The imaging apparatus, wherein the signal processing means performs signal processing using the first and second count values. The imaging element has a pixel array in which the plurality of pixel units are arranged,
the counting means has a register holding the count value;
a first register group comprising the registers of the counting means corresponding to a plurality of pixel portions belonging to a first row or column in the pixel array;
a second register group comprising the registers of the counting means corresponding to a plurality of pixel portions belonging to a second row or column in the pixel array;
2. The method of claim 1, wherein in said mode said first register group holds said first count value respectively and said second register group holds said second count value respectively. imaging device. 3. The control means controls reset timing for the pixel section, and transfers the plurality of the first count values held by the first register group to the second register group. 3. The imaging device according to 2. 2. The control means initializes or sets the count value by outputting a signal for controlling counting operation or a signal for loading data to the counting means. 4. The imaging device according to any one of 3. each of the pixel units includes a plurality of photoelectric conversion units, a plurality of counting means, and a switching means for switching a signal output from the photoelectric conversion unit for each frame and outputting the signal to the plurality of counting means;
The control means outputs a signal from the first photoelectric conversion section among the plurality of photoelectric conversion sections to the first counting section among the plurality of counting sections by controlling the switching section. 2. The imaging apparatus according to claim 1, wherein the signal from the first photoelectric conversion unit is output to second counting means among the plurality of counting means. 6. The method according to claim 5, wherein in said mode, said control means outputs a control signal to said pixel section, and performs control such that a signal from said second photoelectric conversion section is not inputted to said second counting means. The imaging device described. 7. The signal processing means according to any one of claims 1 to 6, wherein the signal processing means obtains the first and second count values and compresses the image data or performs processing for expanding a dynamic range. 10. The image pickup device according to claim 1. The pixel unit has the photoelectric conversion unit and counting means,
3. The imaging according to claim 2, wherein the first and second register groups correspond to the plurality of pixel units periodically arranged in the row direction or the column direction in the pixel array. Device. The pixel section has a microlens and the first and second photoelectric conversion sections that perform photoelectric conversion by receiving light through the microlens,
The first register group includes registers included in the plurality of counting means corresponding to the plurality of first photoelectric conversion units, and the second register group includes the plurality of second photoelectric conversion units. 3. The image pickup apparatus according to claim 2, wherein the register comprises a plurality of registers of the counting means respectively corresponding to . The image pickup apparatus according to any one of claims 1 to 9 , wherein the image pickup element includes the signal processing means. A control method executed by an imaging device including an imaging device having a plurality of pixel units,
The imaging device is
counting means for counting and holding a signal generated from an output of a photoelectric conversion unit using an avalanche effect of the pixel unit;
a control means for controlling the counting means;
A signal processing means for performing signal processing using the output of the counting means,
The control method is
A mode is set in which the signal of the first photoelectric conversion unit among the plurality of photoelectric conversion units is read and the signal of the second photoelectric conversion unit is not read, thereby performing a thinning readout operation for the signal based on the output of the photoelectric conversion unit. and the control means holds a first count value in the current frame of the captured image by the first counting means corresponding to the first photoelectric conversion section, and corresponding to the second photoelectric conversion section. a step of controlling to hold a second count value corresponding to the first count value in the previous frame of the captured image by the second counting means;
and a step of performing signal processing by the signal processing means using the first and second count values.

Images