JP6957202B2 - Solid-state image sensor, image sensor and imaging method - Google Patents

Description

The present invention relates to a solid-state image sensor, an image pickup device, and an image pickup method.

In recent years, an image pickup device provided with a solid-state image sensor such as a CMOS image sensor has become widespread. As a new type of image sensor, an image sensor as shown in Patent Document 1 has been proposed. The image sensor disclosed in Patent Document 1 is provided with the following signal processing circuits in each pixel. In Patent Document 1, a comparator that stores a charge generated by a photoelectric conversion element, a voltage of the storage capacitance is compared with a reference voltage, and a pulse is output when both are matched, and an output of the comparator is used. Each pixel is provided with a reset means for returning the voltage of the storage capacity to the reset voltage. On the other hand, Patent Document 2 describes an optical distance measuring device that uses an avalanche diode as a light receiving element to perform distance measurement based on the Time Of Flight method.

JP-A-2015-173432 Japanese Unexamined Patent Publication No. 2014-77658

However, the conventional technique has not always obtained an image having a sufficiently wide dynamic range.
An object of the present invention is to provide a solid-state image sensor, an image pickup device, and an image pickup method capable of improving the dynamic range.

According to one aspect of the embodiment, a plurality of pixels each provided with a sensor unit that emits a pulse at a frequency corresponding to the light receiving frequency of the photon, and a first counter that counts the number of pulses emitted from the sensor unit. , when the count value of said first counter has not reached the predetermined value during a predetermined period, selecting a first signal corresponding to the count value of the first counter, the count value of said first counter when it reaches the predetermined value during the predetermined period, a selection unit the count value of said first counter to select a second signal corresponding to the time reaches the predetermined value, the second signal Based on this, a solid-state imaging device is provided that includes an image processing unit that estimates the number of the pulses during the predetermined period.

According to the present invention, it is possible to provide a solid-state image sensor, an image pickup device, and an image pickup method capable of improving the dynamic range.

It is a figure which shows the solid-state image sensor by 1st Embodiment. It is a timing chart which shows the example of the operation of the solid-state image sensor according to 1st Embodiment. It is a timing chart which shows other example of the operation of the solid-state image sensor according to 1st Embodiment. It is a figure which shows the structure of the solid-state image sensor by 1st Embodiment. It is a block diagram which shows the image pickup apparatus according to 1st Embodiment. It is a figure which shows the example of the structure of a pixel signal value. It is a figure which shows the solid-state image sensor by 2nd Embodiment. It is a timing chart which shows the example of the operation of the solid-state image sensor by 2nd Embodiment. It is a figure which shows the solid-state image sensor according to 3rd Embodiment. It is a timing chart which shows the example of the operation of the solid-state image sensor according to 3rd Embodiment. It is a figure which shows the example of the structure of a pixel signal value. It is a figure which shows the solid-state image sensor according to 4th Embodiment. It is a timing chart which shows the example of the operation of the solid-state image sensor according to 4th Embodiment. It is a figure which shows the example of the structure of a pixel signal value.

Embodiments of the present invention will be described below with reference to the drawings. The present invention is not limited to the following embodiments, and can be appropriately modified. In addition, the following embodiments may be combined as appropriate. In the following description, the High level of the digital signal may be referred to as "High". Similarly, the Low level of the digital signal may be expressed as "Low". Further, the High level of the digital signal may be treated as a digital value of 1 and may be expressed as "0x1" in hexadecimal. Similarly, the Low level of the digital signal may be treated as a digital value of 0 and may be expressed as "0x0" in hexadecimal.

[First Embodiment]
The solid-state image sensor, the image pickup device, and the image pickup method according to the first embodiment will be described with reference to FIGS. 1 to 6. FIG. 1 is a diagram showing a solid-state image sensor according to the present embodiment.

As shown in FIG. 1, the solid-state imaging device 100 according to the present embodiment includes a vertical scanning unit 101, a timing generator (TG: Timing Generator) 102, a column memory 103, a horizontal scanning unit 104, and a time counter 106. I have. Further, the solid-state image sensor includes a plurality of pixels 110 arranged in a matrix, that is, in a matrix. Although four pixels 110a, 110b, 110c, and 110d are shown here for simplification of the description, a large number of pixels 110 are actually provided in the solid-state image sensor 100. Further, when describing the pixels in general, reference numerals 110 will be used, and when describing individual pixels, reference numerals 110a to 110d will be used. [3: 0] indicates that the signal has a bit width of 4.

The timing generator 102 generates a signal for controlling each unit of the solid-state image sensor 100 based on a control signal or the like supplied from the control unit 504 (see FIG. 5). A synchronization signal VD or the like is supplied to the timing generator 102 from the control unit 504. The timing generator 102 supplies various signals and the like to each of the vertical scanning unit 101, the column memory 103, the horizontal scanning unit 104, and the time counter 106. Further, the timing generator 102 supplies the control signal WRT to each pixel 110. The timing generator 102 can function as a control unit (control means) that controls each unit of the solid-state image sensor 100.

Each pixel 110 is provided with a photodiode 111, a quench element 112, an inverter 113, a photon counter 114, a selection switch 115, and a pixel memory 116, respectively. Further, each pixel 110 is further provided with an OR circuit 117, a delay circuit 118, and a read switch 119.

The photodiode 111 is an avalanche photodiode. The anode of the photodiode 111 is connected to the ground potential, and the cathode of the photodiode 111 is connected to one end of the quench element (quenching resistor) 112. A bias voltage Vbias is applied to the other end of the quench element 112. A bias voltage Vbias larger than the yield voltage of the photodiode 111 is applied to the photodiode 111 via the quench element 112. Therefore, the photodiode 111 operates in the Geiger mode. That is, when a photon is incident on the photodiode 111, it causes an avalanche multiplication phenomenon. As a result, an avalanche current is generated, and a voltage drop occurs in the quench element 112. The quench element 112 is a resistance element for stopping the avalanche multiplication phenomenon of the photodiode 111. Here, the quench element 112 is configured by utilizing the resistance component of the MOS transistor. When an avalanche current is generated by the avalanche multiplication phenomenon, a voltage drop occurs in the quench element 112, and the bias voltage applied to the photodiode 111 drops. When the bias voltage drops to the breakdown voltage, the avalanche multiplication phenomenon stops. As a result, the avalanche current stops flowing, and the bias voltage Vbias is applied to the photodiode 111 again. The cathode of the photodiode 111, one end of the quenching element 112, and the input terminal of the inverter 113 are connected to each other at the node PLSa. The output terminal of the inverter 113 and the input terminal of the photon counter 114 are connected to each other at the node PLSd. When a photon is incident on the photodiode 111, the above phenomenon occurs, so that a voltage change occurs at the node PLSa. The inverter 113 generates a pulse signal PLS according to a voltage change in the node PLSa, and outputs the generated pulse signal PLS to the node PLSd. In this way, the waveform-shaped pulse signal PLS is output from the inverter 113. As described above, in the sensor unit 403 (see FIG. 4), when a photon is incident on the photodiode 111, the pulse signal PLS is output from the inverter 113 at a frequency corresponding to the frequency of receiving the photon. The bias voltage Vbias can be, for example, about + 20V, but is not limited to this. For example, the anode of the photodiode 111 may be connected to a negative potential.

The photon counter 114 counts the number of pulses of the pulse signal PLS output from the inverter 113. The bit width of the photon counter 114 is, for example, 16. The upper limit value that can be counted by the photon counter 114 having a bit width of 16, that is, the upper limit value of counting is 0xFFFF (65535 in decimal). However, here, in order to simplify the description, the photon counter 114 having a bit width of 4 will be used for the description. The upper limit of the count of the photon counter 114 having a bit width of 4 is 0xF (decimal number 15). In the following description, the count value by the photon counter 114 will be referred to as a photon count value. A reset signal (reset pulse) RES output from the timing generator 102 is input to the photon counter 114. The count value of the photon counter 114 is reset to 0x0 by the reset signal RES. When the reset is released, the photon counter 114 starts counting the pulse signal PLS. The photon counter 114 outputs a signal FLAG according to the photon count value. More specifically, the photon counter 114 sets the value of the signal FLAG to 0x0 when the photon count value does not reach the predetermined value Cmax. Further, the photon counter 114 sets the value of the signal FLAG to 0x1 when the photon count value reaches the predetermined value Cmax. The bit width of the signal FLAG is, for example, 1. When the bit width of the photon counter 114 is 16, the predetermined value Cmax is set to, for example, 0xFFFF (65535 in decimal). However, here, as described above, in order to explain the case where the bit width of the photon counter 114 is 4, the predetermined value Cmax is set to, for example, 0xF (decimal number 15).

A clock signal CLK output from the timing generator 102 at a predetermined cycle is input to the time counter 106. The time counter 106 counts the number of pulses of the clock signal CLK, that is, the number of clock pulses. The bit width of the time counter 106 is 16, which is the same as the bit width of the photon counter 114, and the upper limit of the count of the time counter 106 is 0xFFFF (65535 in decimal). However, here, in order to simplify the description, the time counter 106 having a bit width of 4 will be used for the description. In the following description, the count value by the time counter 106 will be referred to as a time count value or TIME. The reset signal RES output from the timing generator 102 is input to the time counter 106. The count value of the time counter 106 is reset to 0x0 by the reset signal RES.

The selection switch 115 is controlled by the signal FLAG output from the photon counter 114. Specifically, when the signal FLAG is 0x0, the selection switch 115 is set so that the photon count value output from the photon counter 114 is input to the pixel memory 116. On the other hand, when the signal FLAG is 0x1, the selection switch 115 is set so that the time count value output from the time counter 106 is input to the pixel memory 116. In other words, if the photon count value does not reach the predetermined value Cmax, the photon count value is input to the pixel memory 116, and if the photon count value reaches the predetermined value Cmax, the time count value is the pixel memory. It is designed to be input to 116. In addition to the photon count value or the time count value input via the selection switch 115, the value of the signal FLAG is also input to the pixel memory 116. That is, a pixel signal value including either a photon count value or a time count value and a signal FLAG value (information) is input to the pixel memory 116. The pixel memory 116 holds the pixel signal value input via the selection switch 115 at the timing when the level of the output signal of the OR circuit 117 changes from 0x0 to 0x1. When the bit width of the photon count value or the time count value is, for example, 16 and the bit width of the signal FLAG is 1, the bit width of the pixel signal value including these is 17. Therefore, the bit width of the pixel memory 116 is 17. However, here, in order to simplify the explanation, since the description is given using the photon counter 114 and the time counter 106 having a bit width of 4, the pixel signal value having a bit width of 5 is held in the pixel memory 116. The explanation is as follows.

A signal output from the delay circuit (delay) 118 and a control signal WRT output from the timing generator 102 are input to the OR circuit (OR gate) 117. The OR circuit 117 takes the logical sum of the signal output from the delay circuit 118 and the control signal WRT output from the timing generator 102, and outputs the calculation result. The OR circuit 117 outputs a high level signal when one of the input signals is high level. The signal FLAG output from the photon counter 114 is input to the delay circuit 118. The delay circuit 118 delays the signal FLAG by a predetermined time, and outputs the delayed signal FLAG to the OR circuit 117.

As described above, in the present embodiment, when the photon count value reaches the predetermined value Cmax, the time count value is input to the pixel memory 116. Then, when a predetermined delay time elapses after the photon count value reaches the predetermined value Cmax, the level of the signal output from the OR circuit 117 changes from 0x0 to 0x1, and the pixel memory 116 displays the time count value and the signal. The FLAG value of 0x1 is retained. On the other hand, when the photon count value does not reach the predetermined value Cmax, the photon count value is input to the pixel memory 116. Then, when the control signal WRT changes from 0x0 to 0x1, the pixel memory 116 holds the photon count value and the signal FLAG value 0x0. The count value of the pixel memory 116 is reset to 0x0 by the reset signal RES output from the timing generator 102.

A plurality of control lines extending in the horizontal direction are connected to the vertical scanning unit 101. The vertical scanning unit 101 sequentially supplies the read signal READ to these control lines. When the read signal in general is described, the reference numeral READ is used, and when the individual read signals are described, the reference numerals READn and READn + 1 are used. The read signal READn is a read signal applied to the control line located in the nth line. The read signal READ n + 1 is a read signal applied to the control line located in the n + 1st line. A read signal READ is supplied to the plurality of pixels 110 located in the same row via the same control line.

A plurality of signal lines (vertical signal line, output signal line) 105 extending in the vertical direction are connected to the column memory 103. Reference numerals 105 will be used when describing the signal lines in general, and reference numerals 105a and 105b will be used when describing the individual signal lines. The signal output from the pixel memory 116 is output to the column memory 103 via the read switch 119, the output unit 120, and the signal line 105. The read switch 119 is turned off when the read signal READ supplied from the vertical scanning unit 101 is 0x0, and is turned on when the read signal READ is 0x1. The signal values output from the plurality of pixels 110 located in the row selected by the read signal READ supplied from the vertical scanning unit 101, that is, the pixel signal values are written to the column memory 103 via the signal lines 105, respectively. Is done. The column memory 103 holds pixel signal values read from each pixel 110. The horizontal scanning unit 104 sequentially outputs each pixel signal value held in the column memory 103 to the image processing unit 502 (see FIG. 5) via the output line Output.

FIG. 2 is a timing chart showing an example of the operation of the solid-state image sensor according to the present embodiment. Here, the operation of the pixel 110a among the plurality of pixels 110 will be described. FIG. 2 shows an example in which the count value of the photon counter 114 provided in the pixel 110a does not reach the predetermined value Cmax during the counting period corresponding to the imaging period of one image, that is, during the predetermined period. .. [0] indicates that the signal has a bit width of 1, [3: 0] indicates that the signal has a bit width of 4, and [4: 0] indicates that the signal has a bit width of 5. It shows that.

The period from timing t201 to timing t205 is a 1V period corresponding to the period of the synchronization signal. At the timing t201, when the synchronization signal (synchronous pulse) VD reaches the high level, the timing generator 102 sets the reset signal (reset pulse) RES to the high level. As a result, the time counter 106, the photon counter 114 of each pixel 110, and the pixel memory 116 of each pixel 110 are reset. The synchronization signal VD returns to the Low level at the timing t202.

The period from the timing t203 to the timing t204 is a counting period, which is a period during which the pulse signal PLS is counted. During the counting period, as described above with reference to FIG. 1, when a photon is incident on the photodiode 111, a signal is generated by the photodiode 111, and a waveform-shaped pulse signal PLS is output from the inverter 113. The photon counter 114 counts the number of pulse signals PLS output from the inverter 113. At the timing t203, the timing generator 102 sets the reset signal RES to the Low level. When the reset signal RES reaches the Low level, the reset of the photon counter 114 is released, and the photon counter 114 starts counting the pulse signal PLS output from the inverter 113. Further, when the reset signal RES reaches the Low level, the reset of the time counter 106 is also released, and the time counter 106 starts counting the number of pulses of the clock signal CLK output from the timing generator 102 in a predetermined cycle. Note that FIG. 2A shows a photon count value and a time count value (TIME) in hexadecimal.

FIG. 2B is a graph showing an example of a photon count value and a time count value. FIG. 2B shows how the photon count value increases and how the time count value (TIME) increases with the passage of time. The photon count value and the time count value shown in FIG. 2B correspond to the photon count value and the time count value shown in FIG. 2A, respectively. The horizontal axis of FIG. 2B is the time, and the vertical axis of FIG. 2B is the count value. In the example shown in FIG. 2, at the timing t204 when the counting period ends, the photon count value of the pixel 110a is 0xC, and the time count value (TIME) is 0xF.

When the control signal WRT changes from 0x0 to 0x1 at the timing t204, the pixel signal value including the photon count value and the signal FLAG value is held in the pixel memory 116 provided in the pixel 110a. The photon count value at the timing t204 is 0xC, and the value of the signal FLAG at the timing t204 is 0x0. Therefore, for example, 0x0C is held as a pixel signal value in the pixel memory 116 provided in the pixel 110a.

In the period from the timing t204 to the timing t205, the pixel signal value is read out. First, the vertical scanning unit 101 sets the read signal READ0 to the High level. As a result, the pixel signal values of the plurality of pixels 110 located in the 0th row are written to the column memory 103 via the signal line 105. After that, the horizontal scanning unit 104 sequentially outputs each pixel signal value held in the column memory 103 to the image processing unit 502 (see FIG. 5) via the output line Output. In this way, the pixel signal values of the plurality of pixels 110 located in the 0th row are output. After that, in the same manner, the pixel signal values of the plurality of pixels 110 located in the first row are output. After that, the pixel signal values are output in the same manner, and the processing of the output of all the pixel signal values is completed by the timing t205. In this way, one frame of image data is output to the image processing unit 502.

FIG. 3 is a timing chart showing another example of the operation of the solid-state image sensor according to the present embodiment. FIG. 3 shows an example in which the count value of the photon counter 114 provided in the pixel 110a reaches a predetermined value Cmax during the counting period.

The period from timing t301 to timing t305 corresponds to a 1V period. Since the operation during the period from timing t301 to timing t303 is the same as the operation during the period from timing t201 to timing t203 described above with reference to FIG. 2, the description thereof will be omitted.

The period from timing t303 to timing t304 is a counting period. At the timing t303, the timing generator 102 sets the reset signal RES to the Low level. When the reset signal RES reaches the Low level, the reset of the photon counter 114 is released, and the photon counter 114 starts counting the pulse signal PLS output from the inverter 113. Further, when the reset signal RES reaches the Low level, the reset of the time counter 106 is also released, and the time counter 106 starts counting the number of pulses of the clock signal CLK output from the timing generator 102 in a predetermined cycle.

FIG. 3B is a graph showing another example of the photon count value and the time count value. FIG. 3B shows how the photon count value increases and how the time count value increases with the passage of time. The photon count value and the time count value shown in FIG. 3B correspond to the photon count value and the time count value shown in FIG. 3A, respectively. In the example shown in FIG. 3, the photon count value of the pixel 110a reaches a predetermined value Cmax, that is, 0xF at the timing t306 during the counting period. Therefore, at the timing t306, the value of the signal FLAG changes from 0x0 to 0x1. When the value of the signal FLAG is switched from 0x0 to 0x1, the selection switch 115 is switched. As a result, the selection switch 115 is set so that the time count value output from the time counter 106 is input to the pixel memory 116. After a predetermined delay time elapses from the timing t306, the output signal of the OR circuit 117 changes from 0x0 to 0x1. As a result, the time count value and the signal FLAG value are held in the pixel memory 116 as the pixel signal value of the pixel 110a. The time count value at the timing when the output signal of the OR circuit 117 changes from 0x0 to 0x1 is 0xA, and the value of the signal FLAG at the timing is 0x1. Therefore, 0x1A is held in the pixel memory 116 as the pixel signal value of the pixel 110a.

Since the operation during the period from timing t304 to timing t305 is the same as the operation during the period from timing t204 to timing t205 described above with reference to FIG. 2, the description thereof will be omitted. At the timing t304, the control signal WRT changes from 0x0 to 0x1, but since the output signal of the OR circuit 117 is already 0x1 in the pixel 110a, the change in the control signal WRT is ignored in the pixel 110a. ..

FIG. 4 is a diagram showing a solid-state image sensor according to the present embodiment. FIG. 4A is a perspective view showing a solid-state image sensor according to the present embodiment. As shown in FIG. 4A, the solid-state image sensor 100 is configured by laminating two substrates (semiconductor chips) 401 and 402. FIG. 4B shows the pixels provided in the solid-state image sensor 100 according to the present embodiment. In FIG. 4B, one pixel 110 of the plurality of pixels 110 provided in the solid-state image sensor 100 is extracted and shown.

As shown in FIG. 4A, the solid-state image sensor 100 includes a substrate (upper substrate) 401 that receives an optical image formed by an optical system (imaging optical system) (not shown), and mainly a digital circuit. It is composed of a substrate (lower substrate) 402. As shown in FIG. 4B, the pixel 110 is composed of a sensor unit (light receiving unit, pixel unit) 403 and a counting unit 404. The sensor unit 403 of the pixels 110 is formed on the substrate 401. The counting unit 404 of the pixels 110 is formed on the substrate 402. A plurality of sensor units 403 are arranged in a matrix on the substrate 401. A plurality of counting units 404 are arranged in a matrix on the substrate 402. Each of the plurality of sensor units 403 and each of the plurality of counting units 404 corresponding to these sensor units 403 are electrically connected to each other. In this way, the plurality of pixels 110 are arranged in a matrix. The sensor unit 403 includes a photodiode 111, a quench element (quenching resistor) 112, and an inverter 113. Since the sensor unit 403 is provided with the inverter 113, the waveform-shaped pulse signal PLS is transmitted from the sensor unit 403 to the counting unit 404. Therefore, the transmission from the sensor unit 403 to the counting unit 404 is relatively robust. The counting unit 404 includes a photon counter 114, a selection switch 115, a pixel memory 116, an OR circuit 117, a delay circuit 118, and a read switch 119. The vertical scanning unit 101, the timing generator 102, the column memory 103, the horizontal scanning unit 104, and the time counter 106 are located in either the peripheral circuit unit 405 of the board 401 or the peripheral circuit unit 406 of the board 402. It is equipped. Here, a case where the vertical scanning unit 101, the timing generator 102, the column memory 103, the horizontal scanning unit 104, and the time counter 106 are arranged in the peripheral circuit unit 406 of the substrate 402 will be described as an example.

As described above, in the present embodiment, the sensor unit 403 is formed on the substrate 401, and the counting unit 404 is formed on the substrate 402. Since the counting unit 404 having a large circuit scale is provided on the substrate 402 separate from the substrate 401 on which the sensor unit 403 is provided, a sufficient area of the sensor unit 403 can be secured. Therefore, a sufficient opening area of the sensor unit 403 can be secured.

The structure of the solid-state image sensor 100 is not limited to the above. The structure of the solid-state image sensor 100 can be appropriately changed according to the purpose and application. For example, the solid-state image sensor 100 may be configured by laminating three or more substrates, or the solid-state image sensor 100 may be configured by one substrate. Each of the plurality of substrates (semiconductor chips) may be manufactured according to different process rules. Further, another circuit for performing signal processing, a frame memory, or the like may be provided on the substrate 402. For example, a signal processing circuit for performing noise reduction processing, a detection circuit for detecting an imaged subject, and the like may be provided on the substrate 402.

FIG. 5 is a block diagram showing an imaging device according to the present embodiment. The optical system (imaging optical system) 501 includes a focus lens, a zoom lens, an aperture, and the like. The optical system 501 forms an optical image of the subject, and the formed optical image is incident on the image pickup surface of the solid-state image pickup device 100. The solid-state image sensor 100 captures an optical image formed by the optical system 501 as described above. The solid-state image sensor 100 sequentially reads pixel signal values from each of the plurality of pixels 110, and sequentially outputs the read pixel signal values to the image processing unit 502. As described above, the pixel signal value includes any one of the photon count value and the time count value and the value of the signal FLAG.

The image processing unit 502 sequentially processes the pixel signal values output from the solid-state image sensor 100, and estimates the number of photons incident on each pixel 110 during the counting period, that is, the number of incident photons. The estimated number of incident photons is taken as the brightness value of the pixel 110. In this way, an image, that is, image data is generated. Such an image may be a still image or a frame constituting a moving image. In the process of generating an image, the image processing unit 502 can further perform signal rearrangement, defect pixel correction, noise reduction, color conversion, white balance correction, gamma correction, resolution conversion, data compression, and the like. The image processing unit 502 estimates the number of photons incident on the pixel 110 during the counting period as follows based on the pixel signal value.

As described above, the pixel signal value includes any one of the photon count value and the time count value and the value of the signal FLAG. Therefore, the pixel signal value may include a photon count value or a time count value.

FIG. 6 is a diagram showing an example of the configuration of pixel signal values. FIG. 6A shows an example in which the photon count value does not reach the predetermined value Cmax during the counting period, and specifically shows an example of the pixel signal value corresponding to FIG. 2. FIG. 6B shows an example when the photon count value reaches a predetermined value Cmax during the counting period, and specifically shows an example of the pixel signal value corresponding to FIG. The upper 1 bit of the 5-bit pixel signal value is assigned to the signal FLAG. Therefore, it is possible to determine whether the value of the lower 4 bits is the photon count value or the time count value based on the value of the most significant bit of the pixel signal values.

The image processing unit 502 converts the pixel signal value into the number of incident photons during the counting period according to the value of the most significant bit of the pixel signal value. For example, as shown in FIG. 6A, when the pixel signal value is, for example, 0x0C, the value of the most significant bit is 0x0. The fact that the number of the most significant bits is 0x0 means that the lower 4 bits of the pixel signal value indicate the photon count value at the timing t204. In this case, the image processing unit 502 determines that the number of photons incident on the pixel 110a during the counting period is 0xC, that is, 12 decimal numbers. On the other hand, as shown in FIG. 6B, when the pixel signal value is, for example, 0x1A, the value of the most significant bit is 0x1. The fact that the value of the most significant bit is 0x1 means that the lower 4 bits of the pixel signal value indicate the time count value at the timing t306. In this case, the image processing unit 502 estimates the number of photons incident on the pixel 110 during the counting period from the time count value of 0xA as follows.

As described above, the time count value shown in the lower 4 bits of the pixel signal value is 0xA, that is, the decimal number 10. The time count value indicates the timing when the photon count value reaches the predetermined value Cmax, that is, 0xF. That is, the time count value indicates the timing at which 15 photons are detected. On the other hand, as described above with reference to FIG. 3, the time count value changes from 0x1 to 0xF in the counting period from the timing t303 to the timing t304. Therefore, the number of photons incident on the pixel 110 during the counting period can be estimated using the following equation (1).
Estimated number of incident photons = Cmax × (time count value corresponding to the counting period / time count value when the photon count value reaches Cmax) ... (1)

The predetermined value Cmax is, for example, 0xF, that is, 15 in decimal. The time count value corresponding to the counting period is, for example, 0xF, that is, 15 in decimal. In the case of the pixel signal value shown in FIG. 6B, the time count value when the photon count value reaches Cmax is 0xA, which is 10 in decimal. Therefore, in the case of the pixel signal value shown in FIG. 6B, the estimated number of incident photons = 15 × (15/10) = 22.5. That is, it is estimated that 22.5 photons were incident on the pixel 110 during the counting period. The image processing unit 502 uses the number of photons calculated in this way as the brightness value of the pixel 110, and generates an image based on the brightness value thus obtained. It should be noted that such processing may be performed in the solid-state image sensor 100 instead of being performed by the image processing unit 502. In this case, the pixel signal value output from the solid-state image sensor 100 includes only the photon count value.

The memory 503 is used when the image processing unit 502 performs arithmetic processing and the like. As the memory 503, for example, a DRAM (Dynamic Random Access Memory), a flash memory, or the like can be used. The memory 503 can also be used as a buffer memory during continuous shooting. The control unit (overall control / calculation unit) 504 controls the entire image pickup apparatus 500 according to the present embodiment. The control unit 504 is provided with a CPU (Central Processing Unit) and the like. Further, the control unit 504 outputs the image signal processed by the image processing unit 502 to the recording unit 507 and the display unit 506. The operation unit 505 is composed of operation members such as buttons, switches, and electronic dials. When a user or the like operates the operation unit 505, a signal corresponding to the operation content is supplied from the operation unit 505 to the control unit 504. The display unit 506 displays an image supplied from the control unit 504. A recording medium (not shown) is attached to the recording unit (recording control unit) 507. As such a recording medium, for example, a memory card or the like is used. A hard disk or the like may be used as the recording medium. The optical system drive unit 508 controls the focus lens, zoom lens, aperture, and the like provided in the optical system 501. The image pickup apparatus 500 may further include a wired or wireless communication interface for communicating with the external device. In this case, the image pickup apparatus 500 can transmit the generated image or the like to the external device or the like or receive the control signal or the like from the external device via the communication interface. Further, the image pickup device 500 may further include a light source device that projects light onto the subject. In this case, the light source device can emit light in a pulse shape in synchronization with, for example, a synchronization signal VD or the like. In addition, the light source device can always emit light. Since the light source device can irradiate the subject with light, the subject can be recognized more reliably.

As described above, according to the present embodiment, when the photon counter value does not reach the predetermined value Cmax during the counting period, the first signal corresponding to the photon count value is output from the pixel 110. On the other hand, when the photon count value reaches the predetermined value Cmax during the counting period, a second signal corresponding to the time when the photon count value reaches the predetermined value Cmax is output from the pixel 110. When the second signal is output from the pixel 110, the number of photons incident on the pixel 110 during the counting period is estimated based on the second signal. Therefore, according to the present embodiment, even when a number of photons exceeding the count limit value of the photon counter 114 is incident on the pixel 110, the number of photons incident on the pixel 110 can be reliably estimated. .. Moreover, an accurate pixel signal value can be obtained even when the number of photons incident on the pixel 110 is small. Therefore, according to the present embodiment, it is possible to provide a solid-state image sensor having a wide dynamic range.
In the present embodiment, the case where the predetermined value Cmax and the maximum value of the photon counter 114 match has been described as an example, but the present invention is not limited to this. For example, a value smaller than the maximum value of the photon counter 114 may be set as a predetermined value Cmax. Further, a plurality of predetermined values Cmax may be set up to the maximum value of the photon counter 114. By setting a plurality of predetermined values Cmax, it is possible to more accurately grasp the amount of change in the photon counter during the counting period. Then, the estimated number of incident photons may be calculated by using an equation different from the equation (1) described above. For example, the estimated number of incident photons may be calculated by using a second-order or higher function, a logarithmic function, or the like. By using a photon counter value corresponding to a plurality of set predetermined values Cmax, it is possible to calculate the estimated number of incident photons with high accuracy.
Further, in the present embodiment, there is an example in which a selection switch 115 for selecting which of the output from the photon counter 114 and the output from the time counter 106 is input to the pixel memory 116 is provided in the pixel 110. However, it is not limited to this. A memory (not shown) for holding the output from the time counter 106 is provided outside the pixel 110 separately from the pixel memory 116, and when the signal is transferred to the column memory 103, the output from the memory is selected. You may do so.

[Second Embodiment]
The solid-state image sensor, the image pickup device, and the image pickup method according to the second embodiment will be described with reference to FIGS. 7 and 8. The same components as those of the solid-state image sensor and the like according to the first embodiment shown in FIGS. 1 to 6 are designated by the same reference numerals, and the description thereof will be omitted or simplified.
The solid-state image sensor according to the present embodiment does not need to hold the value of the signal FLAG in the pixel memory 716.

FIG. 7 is a diagram showing a solid-state image sensor according to the present embodiment. As shown in FIG. 7, the solid-state imaging device 700 according to the present embodiment includes a vertical scanning unit 101, a timing generator 102, a column memory 103, a horizontal scanning unit 104, and a time counter 106. Further, the solid-state image sensor 700 includes a plurality of pixels 710 arranged in a matrix. Although four pixels 710a, 710b, 710c, and 710d are shown here for simplification of the description, a large number of pixels 710 are actually provided in the solid-state image sensor 700. Further, reference numerals 710 will be used when describing the pixels in general, and reference numerals 710a to 710d will be used when describing specific individual pixels.

The pixel 710 includes a photodiode 111, a quench element 112, an inverter 113, a photon counter 714, a selection switch 715, a pixel memory 716, and a read switch 717. The photodiode 111, the quench element 112, and the inverter 113 are provided in the sensor unit 403 arranged on the substrate 401. The photon counter 714, the selection switch 715, the pixel memory 716, and the read switch 717 are provided in the counting unit 404 arranged on the substrate 402.

The photon counter 714 counts the number of pulses of the pulse signal PLS output from the inverter 113, similarly to the photon counter 114 described above with reference to FIG. The bit width of the photon counter 714 is, for example, 16. However, here, in order to simplify the description, a photon counter 714 having a bit width of 4 will be described as an example. The reset signal RES output from the timing generator 102 is input to the photon counter 714. The count value of the photon counter 714 is reset to 0x0 by the reset signal RES. When the reset is released, the photon counter 714 starts counting. The photon counter 714 outputs a signal FLAG according to the photon count value. More specifically, the photon counter 714 sets the value of the signal FLAG to 0x0 when the photon count value does not reach the predetermined value Cmax. Further, the photon counter 714 sets the value of the signal FLAG to 0x1 when the photon count value reaches the predetermined value Cmax. The bit width of the signal FLAG is, for example, 1. When the bit width of the photon counter 714 is 16, the predetermined value Cmax can be 0xFFFF (65535 in decimal). However, here, as described above, in order to explain the photon counter 714 having a bit width of 4 as an example, the predetermined value Cmax is set to 0xF (decimal number 15). The photon counter 714 includes an enable control terminal EN for switching between a pulse counting state and a pulse non-counting state. An enable signal ENABLE output from the timing generator 102 is input to the enable control terminal EN. When the enable signal ENABLE is at the High level, the photon counter 714 is in the enable state in which the pulse is counted. On the other hand, when the enable signal ENABLE is at the Low level, the photon counter 714 is in the disabled state in which the pulse is not counted. The enable signal ENABLE signal controls the count start timing and the count end timing of the photon counter 714.

The selection switch 715 switches the connection according to the value of the signal FLAG. Specifically, when the value of the signal FLAG is 0x0, the selection switch 715 connects the photon counter 714 and the read switch 717. On the other hand, when the value of the signal FLAG is 0x1, the selection switch 715 connects the pixel memory 716 and the read switch 717.

The pixel memory 716 holds the time count value output from the time counter 106 at the timing when the value of the signal FLAG changes from 0x0 to 0x1. When the photon count value reaches Cmax, the pixel memory 716 holds the count value of the time counter 106 at the timing when the photon count value reaches Cmax. The count value of the pixel memory 716 is reset to 0x0 by the reset signal RES output from the timing generator 102. When the photon count value is less than Cmax, the pixel memory 716 holds the reset value in the pixel memory 716.

The read switch 717 changes from the OFF state to the ON state when the read signal READ supplied from the vertical scanning unit 101 changes from 0x0 to 0x1. When the value of the signal FLAG is 0x0, the pixel signal value including the photon count value output from the photon counter 714 and the value of the signal FLAG is sent to the column memory 103 via the switches 715 and 717 and the signal line 105. It is output. On the other hand, when the value of the signal FLAG is 0x1, the pixel signal value including the time count value output from the pixel memory 716 and the FLAG value is the column memory 103 via the switches 715 and 717 and the signal line 105. Is output to.

FIG. 8 is a timing chart showing an example of the operation of the solid-state image sensor according to the present embodiment. Here, the operation of the pixel 710a among the plurality of pixels 710 will be described. FIG. 8 shows an example in which the count value of the photon counter 714 provided in the pixel 710a reaches a predetermined value Cmax during the counting period.

The period from timing t801 to timing t805 corresponds to a 1V period. In the period from the timing t801 to the timing t803, the enable signal ENABLE is at the Low level. Therefore, during the period from timing t801 to t803, the photon counter 714 does not count the pulse signal PLS.

The period from timing t803 to timing t804 is a counting period. At the timing t803, the timing generator 102 changes the reset signal RES from the High level to the Low level and changes the enable signal ENABLE from the Low level to the High level. The photon counter 714 is enabled and starts counting the pulse signal PLS output from the inverter 113. Further, when the reset signal RES reaches the Low level, the reset of the time counter 106 is also released, and the time counter 106 starts counting the number of pulses of the clock signal CLK output from the timing generator 102 in a predetermined cycle.

FIG. 8B is a graph showing an example of a photon count value and a time count value. FIG. 8B shows how the photon count value increases and how the time count value increases with the passage of time. The graph shown in FIG. 8B corresponds to the photon count value and the time count value shown in FIG. 8A. In the example shown in FIG. 8, the photon count value of the pixel 710a reaches a predetermined value Cmax, that is, 0xF at the timing t806 during the counting period. Therefore, at the timing t806, the value of the signal FLAG changes from 0x0 to 0x1. When the value of the signal FLAG is switched from 0x0 to 0x1, the selection switch 715 is switched. As a result, the selection switch 715 is set to a state in which the time count value output from the pixel memory 716 is transmitted to the read switch 717. Further, when the value of the signal FLAG changes from 0x0 to 0x1, the pixel memory 716 holds the time count value output from the time counter 106 at that timing. The time count value at the timing when the value of the signal FLAG changes from 0x0 to 0x1 is, for example, 0xA. Therefore, for example, 0xA, which is a time count value, is held in the pixel memory 716.

At the timing t804, the timing generator 102 changes the enable signal ENABLE from the High level to the Low level. As a result, the photon counter 714 ends counting the pulse signal PLS output from the inverter 113. The photon counter 714 continues to hold the photon count value at timing t804. Here, since the photon count value has reached 0xF, which is a predetermined value Cmax, at a stage prior to the timing t804, 0xF, which is a predetermined value Cmax, is continuously held in the pixel memory 716.

The operation during the period from timing t804 to timing t805 is the same as the operation during the period from timing t204 to timing t205 described above with reference to FIG. That is, the pixel signal value including any one of the photon count value and the time count value and the value of the signal FLAG is sequentially output.

Even when configured as in the present embodiment, the same pixel signal value as in the first embodiment can be obtained. According to the present embodiment, since it is not necessary to hold the value of the signal FLAG in the pixel memory 716, the bit width of the pixel memory 716 can be made smaller than the bit width of the pixel memory 116 in the first embodiment. Therefore, according to the present embodiment, the circuit scale can be reduced, which in turn can contribute to an increase in the number of pixels.

[Third Embodiment]
The solid-state image sensor, the image pickup device, and the image pickup method according to the third embodiment will be described with reference to FIGS. 9 to 11. The same components as those of the solid-state image sensor and the like according to the first or second embodiment shown in FIGS. 1 to 8 are designated by the same reference numerals, and the description thereof will be omitted or simplified.
The solid-state image sensor according to the present embodiment is provided with a multi-counter 914 that counts the clock signal CLK after the count value of the pulse signal PLS reaches a predetermined value Cmax.

FIG. 9 is a diagram showing a solid-state image sensor according to the present embodiment. As shown in FIG. 9, the solid-state imaging device 900 according to the present embodiment includes a vertical scanning unit 101, a timing generator 102, a column memory 103, and a horizontal scanning unit 104. In this embodiment, the time counter 106 (see FIG. 1) is not provided. Further, the solid-state image sensor 900 includes a plurality of pixels 910 arranged in a matrix. Although four pixels 910a, 910b, 910c, and 910d are shown here for simplification of the description, a large number of pixels 910 are actually provided in the solid-state image sensor 900. Further, reference numerals 910 will be used when describing the pixels in general, and reference numerals 910a to 910d will be used when describing specific individual pixels.

The pixel 910 includes a photodiode 111, a quench element 112, an inverter 113, a multi-counter 914, a selection switch 915, a flip-flop 916, and a read-out switch 917. The photodiode 111, the quench element 112, and the inverter 113 are provided in the sensor unit 403 arranged on the substrate 401. The multi-counter 914, the selection switch 915, the flip-flop 916, and the read switch 917 are provided in the counting unit 404 arranged on the substrate 402.

The multi-counter 914 counts the number of pulses of the pulse signal PLS output from the inverter 113, similarly to the photon counter 114 described above with reference to FIG. The bit width of the multi-counter 914 is, for example, 16. However, here, in order to simplify the description, a multi-counter 914 having a bit width of 4 will be described as an example. In the following description, the count value by the multi-counter 914 will be referred to as a multi-count value. The multi-counter 914 can not only count the number of pulses of the pulse signal PLS output from the inverter 113, but can also count the number of pulses of the clock signal CLK output from the timing generator 102 in a predetermined cycle. The signal input to the multi-counter 914 is switched by the selection switch 915. The reset signal RES output from the timing generator 102 is input to the multi-counter 914. The count value of the multi-counter 914 is reset to 0x0 by the reset signal RES. The multi-counter 914 includes an enable control terminal EN for switching between a pulse counting state and a pulse non-counting state. An enable signal ENABLE output from the timing generator 102 is input to the enable control terminal EN. When the enable signal ENABLE is at the high level, the multi-counter 914 is in the enable state in which the pulse is counted. On the other hand, when the enable signal ENABLE is at the Low level, the multi-counter 914 is in a disabled state in which the pulse is not counted. The enable signal ENABLE signal controls the count start timing and the count end timing of the multi-counter 914. When the bit width of the multi-counter 914 is 16, the predetermined value Cmax is, for example, 0xFFFF (65535 in decimal). However, as described above, in order to explain the multi-counter 914 having a bit width of 4 as an example, the predetermined value Cmax is set to, for example, 0xF (decimal number 15). The multi-counter 914 outputs the signal FLAG1 according to the multi-count value. More specifically, the multi-counter 914 sets the value of the signal FLAG1 to 0x0 when the multi-count value does not reach the predetermined value Cmax. Further, the multi-counter 914 sets the value of the signal FLAG1 to 0x1 when the multi-count value reaches a predetermined value Cmax. The bit width of the signal FLAG1 is, for example, 1.

The selection switch 915 switches the connection according to the value of the signal FLAG2. Specifically, when the value of the signal FLAG2 is 0x0, the selection switch 915 is set so that the pulse signal PLS output from the inverter 113 is input to the multi-counter 914. On the other hand, when the value of the signal FLAG2 is 0x1, the selection switch 915 is set so that the clock signal CLK output from the timing generator 102 is input to the multi-counter 914.

The flip-flop 916 outputs the value of the input terminal D to the output terminal Q at the timing when the value of the input terminal CK changes from the Low level to the High level. The flip-flop 916 is provided with a reset terminal R, and a reset signal RES output from the timing generator 102 is input to the reset terminal R. When the reset terminal R provided on the flip-flop 916 reaches the High level, the signal FLAG2 output from the output terminal Q of the flip-flop 916 is reset to 0x0. FLAG1 output from the multi-counter 914 is input to the input terminal CK of the flip-flop 916. The input terminal D of the flip-flop 916 is fixed to 0x1. When the multi-count value is less than Cmax, the value of the signal FLAG2 output from the flip-flop 916 is 0x0. When the multi-count value reaches Cmax and the value of the signal FLAG1 output from the multi-counter 914 changes from 0x0 to 0x1, the value of the signal FLAG2 output from the flip-flop 916 becomes 0x1.

The read switch 917 changes from the OFF state to the ON state when the read signal READ supplied from the vertical scanning unit 101 changes from 0x0 to 0x1. When the read switch 917 is turned on, the pixel signal value including the signal value held in the multi-counter 914 and the value of the signal FLAG2 is output to the column memory 103 via the read switch 917.

FIG. 10 is a timing chart showing the operation of the solid-state image sensor according to the present embodiment. Here, the operation of the pixel 910a among the plurality of pixels 910 will be described. FIG. 10 shows an example in which the count value of the multi-counter 914 provided in the pixel 910a reaches a predetermined value Cmax during the counting period.

The period from timing t1001 to timing t1005 corresponds to a 1V period. In the period from the timing t1001 to the timing t1003, the enable signal ENABLE is at the Low level. Therefore, during the period from timing t1001 to t1003, the multi-counter 914 does not count the pulse signal PLS.

The period from timing t1003 to timing t1004 is a counting period. At the timing t1003, the timing generator 102 changes the reset signal RES from the high level to the low level and changes the enable signal ENABLE from the low level to the high level. The multi-counter 914 is enabled and starts counting the pulse signal PLS output from the inverter 113.

FIG. 10B is a graph showing an example of a photon count value and a time count value. FIG. 10B shows how the multi-count value increases and how the multi-count value increases with the passage of time after the multi-count value reaches a predetermined value Cmax. The graph shown in FIG. 10B corresponds to the photon count value and the time count value shown in FIG. 10A. In the example shown in FIG. 10, the photon count value of the pixel 910a reaches a predetermined value Cmax, that is, 0xF at the timing t1006 during the counting period. Therefore, at the timing t1006, the value of the signal FLAG1 changes from 0x0 to 0x1. When the value of the signal FLAG1 changes from 0x0 to 0x1, the value of the signal FLAG2 output from the flip-flop 916 changes from 0x0 to 0x1. As a result, the selection switch 915 is set so that the clock signal CLK is input to the multi-counter 914.

When the multi-count value reaches Cmax at the timing t1006, the multi-counter 914 resets the multi-count value to 0x0 at the timing t1007 immediately after the timing t1006. At this time, the value of the signal FLAG1 output from the multi-counter 914 changes from 0x1 to 0x0, but the value of the signal FLAG2 is maintained at 0x1. The multi-counter 914 starts counting the number of pulses of the clock signal CLK output from the timing generator 102.

At the timing t1004, the timing generator 102 changes the enable signal ENABLE from the High level to the Low level. As a result, the multi-counter 914 ends counting the clock signal CLK. The multi-counter 914 continues to hold the multi-count value at timing t1004. Here, a case where the multi-count value is, for example, 0x5 is shown as an example. This means that the period from the timing t1006 when the multi-count value reaches Cmax to the timing t1004 at the end of the counting period is a period corresponding to 0x5 clock signals CLK.

The operation during the period from timing t1004 to timing t1005 is the same as the operation during the period from timing t204 to timing t205 described above with reference to FIG. Here, the pixel signal value including the multi-count value and the value of the signal FLAG2 is sequentially output.

FIG. 11 is a diagram showing an example of pixel signal values. FIG. 11A shows an example in which the multi-count value does not reach the predetermined value Cmax during the accumulation period. FIG. 11B shows an example in which the multi-count value reaches a predetermined value Cmax during the accumulation period, and specifically, the pixel signal value corresponding to FIG. 10 is shown. The upper 1 bit of the 5 bits constituting the pixel signal value is assigned to the signal FLAG2. Therefore, it is possible to determine what the value of the lower 4 bits indicates based on the value of the most significant bit of the pixel signal values.

For example, as shown in FIG. 11A, when the pixel signal value is 0x0C, the value of the most significant bit is 0x0. Since the value of the most significant bit is 0x0, the lower 4 bits of the pixel signal value indicate the photon count value at the timing t1004 (see FIG. 10). In this case, the image processing unit 502 determines that the number of photons incident on the pixel 910a during the counting period is 0xC, that is, 12 decimal numbers. On the other hand, as shown in FIG. 11B, when the pixel signal value is 0x15, the value of the most significant bit is 0x1. Since the value of the most significant bit is 0x1, the lower 4 bits of the pixel signal value indicate that the period from the timing t1006 to the timing t1004 is a period corresponding to 0x5 clock signals CLK. The image processing unit 502 estimates the number of photons incident on the pixel 110 during the counting period from the time corresponding to, for example, 0x5 as follows.

The period corresponding to 0x5, which is the value of the lower 4 bits of the pixel signal value, that is, the decimal number 5, corresponds to the period from when the multi-count value reaches the predetermined value Cmax to the timing t1004. Therefore, the number of photons incident on the pixel 910 during the counting period can be estimated using the following equation (2).
Estimated number of incident photons = Cmax × {time count value corresponding to the counting period / (time count value corresponding to the counting period-multi-count value)} ... (2)

The predetermined value Cmax is, for example, 0xF, that is, 15 in decimal. The time count value corresponding to the counting period is, for example, 0xF, that is, 15 in decimal. The multi-count value is 0x5 (decimal number 5) in the case of the pixel signal value shown in FIG. 10 (b). Therefore, in the case of the pixel signal value shown in FIG. 10B, the estimated number of incident photons = 15 × (15 / (15-5)) = 22.5. That is, it can be estimated that 22.5 photons were incident on the pixel 110 during the counting period. The image processing unit 502 uses the number of photons calculated in this way as the brightness value of the pixel 910, and generates an image based on the brightness value thus obtained.

As described above, according to the present embodiment, the multi-counter 914 that counts the clock signal CLK after the count value of the pulse signal PLS reaches the predetermined value Cmax is provided. According to the present embodiment, since it is not necessary to separately provide the photon counters 114 and 714 and the pixel memories 116 and 716, the circuit scale can be reduced and the number of pixels can be increased.

[Fourth Embodiment]
The solid-state image sensor, the image pickup device, and the image pickup method according to the fourth embodiment will be described with reference to FIGS. 12 to 14. The same components as those of the solid-state image sensor and the like according to the first to third embodiments shown in FIGS. 1 to 11 are designated by the same reference numerals, and the description thereof will be omitted or simplified.
In the solid-state image sensor according to the present embodiment, the predetermined value Cmax is set smaller than that of the first to third embodiments, and the bit width of the pixel memory 1216 or the like is set smaller than that of the first to third embodiments. It is set small.

FIG. 12 is a diagram showing a solid-state image sensor according to the present embodiment. As shown in FIG. 12, the solid-state image sensor 1200 according to the present embodiment includes a vertical scanning unit 101, a timing generator 102, a column memory 103, a horizontal scanning unit 104, a time counter 1206, and a plurality of pixels 1210. I have. Although four pixels 1210a, 1210b, 1210c, and 1210d are shown here for simplification of the description, a large number of pixels 1210 are actually provided in the solid-state image sensor 1200. Further, reference numerals 1210 will be used when describing the pixels in general, and reference numerals 1210a to 1210d will be used when describing specific individual pixels.

The pixel 1210 includes a photodiode 111, a quench element 112, an inverter 113, a photon counter 1214, a selection switch 1215, a pixel memory 1216, a flip-flop 1218, and a read switch 1217. The photodiode 111, the quench element 112, and the inverter 113 are provided in the sensor unit 403 arranged on the substrate 401. The photon counter 1214, the selection switch 1215, the flip-flop 1218, and the read switch 1217 are provided in the counting unit 404 arranged on the substrate 402.

The photon counter 1214 counts the number of pulses of the pulse signal PLS output from the inverter 113 in the same manner as the photon counter 714 described above with reference to FIG. 7. The bit width of the photon counter 1214 is, for example, 16. However, here, in order to simplify the description, a photon counter 1214 having a bit width of 4 will be described as an example. The reset signal RES1 output from the timing generator 102 is input to the photon counter 1214. The count value of the photon counter 1214 is reset by the reset signal RES1. When the reset is released, the photon counter 1214 starts counting. The photon counter 1214 outputs the signal FLAG1 according to the photon count value. More specifically, the photon counter 1214 sets the value of the signal FLAG1 to 0x0 when the photon count value is less than the predetermined value Cmax. Further, the photon counter 1214 sets the value of the signal FLAG1 to 0x1 when the photon count value reaches the predetermined value Cmax. The signal FLAG1 is, for example, a 1-bit signal. In the first embodiment, the predetermined value Cmax when the bit width of the photon counter 114 is 16 is set to 0xFFFF (65535 in decimal). On the other hand, in the present embodiment, the predetermined value Cmax when the bit width of the photon counter 1214 is 16 is set to 0x7FFF (32767 in decimal). However, here, as described above, the photon counter 1214 having a bit width of 4 will be described as an example. In the second embodiment, the predetermined value Cmax when the bit width of the photon counter 114 is 4 is set to 0xF, but in the present embodiment, the predetermined value Cmax when the bit width of the photon counter 1214 is 4 is set to 0x7. Similar to the photon counter 714 described above with reference to FIG. 7, the photon counter 1214 includes an enable control terminal EN for switching between a pulse counting state and a pulse non-counting state. An enable signal ENABLE output from the timing generator 102 is input to the enable control terminal EN. When the enable signal ENABLE is at the High level, the photon counter 1214 is in the enable state in which the pulse is counted. On the other hand, when the enable signal ENABLE is at the Low level, the photon counter 1214 is in the disabled state in which the pulse is not counted. The enable signal ENABLE signal controls the count start timing and the count end timing of the photon counter 1214. Further, the photon counter 1214 outputs the signal FLAG1 according to the photon count value. More specifically, the photon counter 1214 sets the value of the signal FLAG1 to 0x0 when the photon count value is less than the predetermined value Cmax. Further, the photon counter 1214 sets the value of the signal FLAG1 to 0x1 when the photon count value reaches the predetermined value Cmax. Even after the value of FLAG1 is changed from 0x0 to 0x1, the photon counter 1214 continues to count the pulse signal PLS output from the inverter 113.

A clock signal CLK output from the timing generator 102 at a predetermined cycle is input to the time counter 1206. The time counter 1206 counts the number of pulses of the clock signal CLK. In the first embodiment, the bit width of the time counter 106 is 16 bits, but in the present embodiment, the bit width of the time counter 1206 is 15. The upper limit of the count of the time counter 1206 having a bit width of 15 is 0x7FFF (32767 in decimal). However, as described above, here, in order to simplify the description, a photon counter 114 having a bit width of 4 will be described as an example. Therefore, here, the time counter 1206 having a bit width of 3 will be described as an example. The upper limit of the count of the time counter 1206 having a bit width of 3 is 0x7 (decimal number 7). In the following description, the count value by the time counter 1206 will be referred to as a time count value or TIME. The reset signal RES2 output from the timing generator 102 is input to the time counter 1206. The count value of the time counter 1206 is reset by the reset signal RES2.

The flip-flop 1218 outputs a value obtained by logically inverting the value of the input terminal D to the output terminal Q at the timing when the value of the input terminal CK changes from the Low level to the High level. The flip-flop 1218 includes a reset terminal R, and a reset signal RES1 output from the timing generator 102 is input to the reset terminal R. When the reset terminal R provided on the flip-flop 1218 reaches the High level, the signal FLAG output from the output terminal Q of the flip-flop 1218 is reset to 0x0. The signal FLAG1 output from the photon counter 1214 is input to the input terminal CK of the flip-flop 1218. The reset signal RES2 output from the timing generator 102 is input to the input terminal D of the flip-flop 1218. When the photon count value is less than Cmax, the value of the signal FLAG2 output from the flip-flop 1218 is 0x0. When the photon count value reaches Cmax, that is, 0x7, and the value of the signal FLAG1 output from the photon counter 1214 changes from 0x0 to 0x1, the signal FLAG2 output from the flip-flop 1218 becomes the logical inversion value of the reset signal RES2. .. That is, when the photon count value reaches Cmax, that is, when the reset signal RES2 reaches the Low level, the signal FLAG2 output from the flip-flop 1218 becomes 0x1. On the other hand, that is, when the photon count value reaches Cmax, that is, 0x7, when the reset signal RES2 is at the High level, the signal FLAG2 output from the flip-flop 1218 becomes the Low level. In other words, when the photon count value reaches Cmax, that is, when the time counter 1206 is reset, the signal FLAG2 output from the flip-flop 1218 changes from 0x0 to 0x1. On the other hand, when the photon count value reaches Cmax, that is, when the time counter 1206 reaches 0x7, the signal FLAG2 output from the flip-flop 1218 remains 0x0 and does not change.

The selection switch 1215 switches the connection according to the value of the signal FLAG2. Specifically, when the value of the signal FLAG2 is 0x0, the selection switch 1215 connects the photon counter 1214 and the read switch 1217. On the other hand, when the value of the signal FLAG2 is 0x1, the selection switch 1215 connects the pixel memory 1216 and the read switch 1217.

The pixel memory 1216 holds the time count value output from the time counter 1206 at the timing when the value of the signal FLAG2 changes from 0x0 to 0x1. In the first embodiment, the bit width of the pixel memory 116 is 16 bits, but in the present embodiment, the bit width of the pixel memory 1216 is 15 bits. The pixel memory 1216 having a bit width of 15 can hold data for 15 bits, that is, data of 0x7FFF (decimal number 32767) or less. However, as described above, here, in order to simplify the description, a photon counter 1214 having a bit width of 4 will be described as an example. Therefore, here, a case where the bit width of the pixel memory 1216 is 3 will be described as an example. The pixel memory 1216 having a bit width of 3 can hold data for 3 bits, that is, data of 0x7 (decimal number 7) or less. When the photon count value reaches Cmax, that is, 0x7, when the reset of the time counter 1206 is released, the pixel memory 1216 holds the count value of the time counter 1206. On the other hand, when the photon count value reaches Cmax, that is, when the time counter 1206 is in the reset state at the timing when it reaches 0x7, the pixel memory 1216 continues to hold the value at the time when the pixel memory 1216 is reset. .. Further, when the photon count value does not reach Cmax, that is, 0x7, the pixel memory 1216 continues to hold the value when the pixel memory 1216 is reset. Further, the pixel memory 1216 is reset to 0x0 by the reset signal RES1 output from the timing generator 102.

The read switch 1217 changes from the OFF state to the ON state when the read signal READ supplied from the vertical scanning unit 101 changes from 0x0 to 0x1. When the read switch 1217 is turned on, the result is as follows. That is, when the photon counter 1214 and the read switch 1217 are connected, the pixel signal values including the photon counter value held in the photon counter 1214 and the value of the signal FLAG2 are arranged in a row via the read switch 1217. It is output to the memory 103. On the other hand, when the pixel memory 1216 and the read switch 1217 are connected, the pixel signal value including the signal value held in the pixel memory 1216 and the value of the signal FLAG2 is stored in the column memory via the read switch 1217. It is output to 103. As described above, the bit width of the pixel memory 1216 is one bit smaller than the bit width of the photon counter 1214. Therefore, when the pixel signal value including the signal value held in the pixel memory 1216 and the value of the signal FLAG2 is output from the pixel 1210, the data width is expanded. The extended 1-bit value may be fixed at 0x0 or 0x1.

FIG. 13 is a timing chart showing an example of the operation of the solid-state image sensor according to the present embodiment. Here, the operation of the pixel 1210a among the plurality of pixels 1210 will be described. FIG. 13 shows an example in which the photon count value of the photon counter 1214 provided in the pixel 1210a reaches a predetermined value Cmax, that is, 0x7 during the counting period.

The period from timing t1301 to timing t1305 corresponds to a 1V period. In the period from the timing t1301 to the timing t1303, the enable signal ENABLE is at the Low level. Therefore, during the period from timing t1301 to t1303, the photon counter 1214 does not count the pulse signal PLS.

The period from timing t1303 to timing t1304 is a counting period. Here, a case where the counting period is a period corresponding to 15 counts from 0x1 to 0xF will be described as an example. However, since the time counter 1206 having a bit width of 3 is used here, the time counter 1206 can count up to 7.

At the timing t1303, the timing generator 102 changes the reset signal RES1 and the reset signal RES2 from the high level to the low level. Further, at the timing t1303, the timing generator 102 changes the enable signal ENABLE from the Low level to the High level. Then, the photon counter 1214 starts counting the pulse signal PLS output from the inverter 113. Further, the time counter 1206 starts counting the number of pulses of the clock signal CLK output from the timing generator 102 at a predetermined cycle.

FIG. 13B is a graph showing a photon count value and a time count value. FIG. 13B shows a state in which the photon count value increases and a state in which the time count value increases with the passage of time. The graph shown in FIG. 13B corresponds to the photon count value and the time count value shown in FIG. 13A. In the example shown in FIG. 13, the photon count value of the pixel 1210a reaches a predetermined value Cmax, that is, 0x7 at the timing t1306 during the counting period. When the photon count value reaches the predetermined value Cmax, the value of the signal FLAG1 output from the photon counter 1214 changes from 0x0 to 0x1. At the timing t1306, since the reset signal RES2 is at the Low level, the signal FLAG2 output from the flip-flop 1218 changes from 0x0 to 0x1. When the signal FLAG2 becomes 0x1, the selection switch 1215 is in a state of connecting the pixel memory 1216 and the read switch 1217. The pixel memory 1216 holds the time count value of the time counter 1206 when the value of the signal FLAG2 changes from 0x0 to 0x1. Here, for example, 0x5 is held in the pixel memory 1216. As described above, when the photon count value reaches the predetermined value Cmax in the period from the timing t1303 to the timing t1307, that is, in the period when the reset of the time counter 1206 is released, the result is as follows. That is, the time count value corresponding to the timing when the photon count value reaches the predetermined value Cmax is held in the pixel memory 1216. The period from timing t1303 to timing t1307 is half the period from timing t1303 to timing t1304.

At the timing t1307, when the reset signal RES2 changes from the Low level to the High level, the time count value of the time counter 1206 is reset to 0x0. That is, the timing generator 102 resets the time counter 1206 by changing the reset signal RES2 from the Low level to the High level at the timing when the time count value reaches the count upper limit value of the time counter 1206.

At timing t1304, the timing generator 102 changes the enable signal ENABLE from a high level to a low level. As a result, the photon counter 1214 stops counting the pulse signal PLS and continues to hold the photon count value at the timing t1304.

The operation during the period from timing t1304 to timing t1305 is the same as the operation during the period from timing t204 to timing t205 described above with reference to FIG. Here, the pixel signal value including any one of the photon count value and the time count value and the value of the signal FLAG2 is sequentially read out.

FIG. 14 is a diagram showing an example of pixel signal values. FIG. 14A shows an example in which the photon count value does not reach the predetermined value Cmax during the accumulation period. FIG. 14B shows an example in which the photon count value reaches a predetermined value Cmax during the accumulation period, and specifically, the pixel signal value corresponding to FIG. 13 is shown. The upper 1 bit of the 5 bits constituting the pixel signal value is assigned to the signal FLAG2. Therefore, it is possible to determine what the value of the lower 4 bits indicates based on the value of the most significant bit of the pixel signal values.

For example, as shown in FIG. 14A, when the pixel signal value is 0x0C, the value of the most significant bit is 0x0. The fact that the value of the most significant bit is 0x0 means that the photon count value does not reach the predetermined value Cmax, that is, 0x7 during the period from the timing t1303 to the timing t1307, which is the state in which the reset of the time counter 1206 is released. It means that. Therefore, in this case, the image processing unit 502 determines that the number of photons incident on the pixel 1210a during the counting period is 0xC, that is, 12 decimal numbers. On the other hand, as shown in FIG. 14B, when the pixel signal value is 0x15, the value of the most significant bit is 0x1. The value of the most significant bit being 0x1 means that the photon count value reached the predetermined value Cmax during the period from the timing t1303 to the timing t1307, which is the state in which the reset of the time counter 1206 is released. do. In this case, the lower 3 bits of the pixel signal value means that the period until the photon count value reaches the predetermined value Cmax, that is, the period from the timing t1303 to the timing t1306 is the time corresponding to 0x5 clock signals CLK. Is shown. The image processing unit 502 estimates the number of photons incident on the pixel 1210a during the counting period from the time corresponding to 0x5 by using the following equation (2).

The period corresponding to 0x5, which is the value of the lower 3 bits of the pixel signal value, that is, the decimal number 5, is the timing when the photon count value reaches the predetermined value Cmax, that is, the timing when the photon count value becomes 0x7. ..
Estimated number of incident photons = Cmax × (time count value / time count value corresponding to the counting period) ... (2)

The predetermined value Cmax is, for example, 0x7, that is, 7 in decimal. The time count value corresponding to the counting period is, for example, 0xF, that is, 15 in decimal. In the case of the pixel signal value shown in FIG. 13B, the time count value is 0x5, which is 5 in decimal. Therefore, in the case of the pixel signal value shown in FIG. 13B, the estimated number of incident photons = 7 × (15/5) = 21. That is, it can be estimated that 21 photons were incident on the pixel 1210a during the counting period. The image processing unit 502 uses the number of photons calculated in this way as the brightness value of the pixel 1210a, and generates an image based on the brightness value thus obtained.

As described above, according to the present embodiment, the value of the predetermined value Cmax is set to be smaller than that in the case of the first to third embodiments. Then, image data is generated based on the time count value when the photon count value reaches the predetermined value Cmax. Therefore, according to the present embodiment, the bit width of the pixel memory 1216 or the like can be reduced. Since the bit width of the pixel memory 1216 or the like can be reduced, the circuit scale can be reduced, which in turn can contribute to an increase in the number of pixels. It should be noted that the high-luminance signal does not necessarily require high resolution in generating an image. Therefore, no particular problem occurs even in the case of the present embodiment.
In the present embodiment, the case where the image data is generated based on the time count value when the predetermined value Cmax is reached has been described as an example, but the present invention is not limited to this. For example, a first threshold value may be provided for the time count value, and the photon count value when the time count value reaches the first threshold value may be held in the pixel memory 1216. In this case, for example, the estimated number of incident photons can be calculated using the following equation (3).
Estimated number of incident photons = (photon count value when the time count value reaches the first threshold value) × (time count value corresponding to the counting period / first threshold value)
Also in this case, a plurality of values may be set as threshold values.

[Modification Embodiment]
Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and modifications can be made within the scope of the gist thereof.
For example, in the above embodiment, the period of the clock signal CLK may be adjusted according to the counting period. For example, when the counting period is twice the counting period shown in FIG. 2, the period of the clock signal CLK may be doubled. Further, when the counting period is half of the counting period shown in FIG. 2, the period of the clock signal CLK may be halved.

Further, in the above embodiment, the case where the image processing unit 502 is provided separately from the solid-state image sensor 100, 700, 900, 1200 has been described as an example, but the image processing is performed inside the solid-state image sensor 100, 700, 900, 1200. The unit 502 may be provided. In this case, for example, the image processing unit 502 may be provided on the substrate 402. Further, the image processing unit 502 is arranged on a substrate (not shown) separate from the substrates 401 and 402 (see FIG. 4), and the substrate on which the image processing unit 502 is arranged is laminated with the substrates 401 and 402. May be good.
Further, in the first, second and fourth embodiments, the case where the time counters 106 and 1206 are provided separately from the timing generator 102 has been described as an example, but the present invention is not limited to this. For example, the timing generator 102 and the time counters 106 and 1206 may be integrated.
Further, in the above embodiment, the case where the output unit 120 is provided in each of the plurality of pixels has been described as an example, but the present invention is not limited to this. For example, the output unit 120 may be shared by a plurality of pixels adjacent to each other.
Further, in the above embodiment, the case where the photon counter 114, the pixel memory 116, and the like are provided in each of the plurality of pixels has been described as an example, but the present invention is not limited to this. For example, these may be shared by a plurality of pixels adjacent to each other.
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100, 700, 900, 1200 ... Solid-state image sensor 101 ... Vertical scanning unit 102 ... Timing generator 103 ... Column memory 104 ... Horizontal scanning unit 105 ... Vertical signal line 106, 1206 ... Time counter 110, 710, 910, 1210 ... Pixel 111 ... Photodiode 112 ... Quenching element 113 ... Inverter 114 ... Photon counter 115 ... Selection switch 116 ... Pixel memory 117 ... OR circuit 118 ... Delay circuit 119 ... Read switch

Claims (11)

A plurality of pixels each equipped with a sensor unit that emits a pulse at a frequency corresponding to the frequency of receiving photons, and
A first counter that counts the number of pulses emitted from the sensor unit, and
When the count value of said first counter has not reached the predetermined value during a predetermined period, selecting a first signal corresponding to the count value of the first counter, the count value of the first counter is the when it reaches the predetermined value during a predetermined period, a selection unit the count value of said first counter to select a second signal corresponding to the time reaches the predetermined value,
An image processing unit that estimates the number of pulses during the predetermined period based on the second signal.
A solid-state imaging device, characterized in that it comprises a. The solid-state imaging device according to claim 1, wherein each of the plurality of pixels is provided with the first counter. The solid-state imaging device according to claim 1 or 2, wherein each of the plurality of pixels is provided with the selection unit. The solid-state imaging device according to any one of claims 1 to 3, wherein the sensor unit includes an avalanche photodiode. The first signal includes information indicating that the count value of the first counter has not reached the predetermined value during the predetermined period.
The second signal according to any one of claims 1 to 4, wherein the second signal includes information indicating that the count value of the first counter has reached the predetermined value during the predetermined period. Solid-state image sensor. Further equipped with a second counter that counts the number of clock pulses,
When the count value of the first counter reaches the predetermined value during the predetermined period , the selection unit uses the second counter when the count value of the first counter reaches the predetermined value. The solid-state imaging device according to any one of claims 1 to 5, wherein the second signal is selected according to the count value of the above. The solid-state imaging device according to claim 6, further comprising a memory for storing the previous SL count value of the second counter. When the count value of the first counter reaches the predetermined value during the predetermined period, the first counter is a clock pulse after the count value of the first counter reaches the predetermined value. The second signal according to the number of clock pulses counted by the end of the predetermined period is counted, and the second signal is output according to any one of claims 1 to 5. Solid-state image sensor. The solid-state imaging device according to any one of claims 1 to 8, wherein the predetermined period is shorter than a period in which the number of pulses is counted by the first counter. A plurality of pixels each provided with a sensor unit that emits a pulse at a frequency corresponding to the light receiving frequency of a photon, a first counter that counts the number of pulses emitted from the sensor unit, and a count value of the first counter. If but does not reach the predetermined value during a predetermined period, selecting a first signal corresponding to the count value of said first counter, to the predetermined value a count value of said first counter during said predetermined time period when it reaches a solid state imaging device and a selector for selecting second signal corresponding to the time count value of said first counter has reached a predetermined value,
And the solid based on the previous SL second signal that will be output from the imaging device, an image processing unit for estimating the number of said pulses during said predetermined period,
An imaging device characterized by having. When the count value of the counter for counting the number of pulses emitted from the sensor unit at a frequency corresponding to the received light frequency of photons does not reach the predetermined value during a predetermined period, the first signal corresponding to the count value of the counter selected, if the count value of said counter reaches a predetermined value during the predetermined period, selecting a second signal when the count value of the counter corresponding to the time reaches the predetermined value,
A step of based on said second signal, estimating the number of said pulses during said predetermined period,
An imaging method characterized by having.

Images