JP2020107973A - Image processing apparatus and method, imaging apparatus, and imaging element control method - Google Patents

Abstract

To accurately compensate for the linearity of a signal obtained from an image sensor using an SPAD.SOLUTION: In an image processing apparatus that processes an image signal obtained from an imaging element having a plurality of pixels that receive light in a plurality of different wavelength bands, each of the plurality of pixels includes sensor means that outputs a signal pulse in response to the incidence of photons, and counting means that counts the number of the signal pulses, and the image processing apparatus includes storage means for storing a parameter for compensating the linearity of the image signal output from the imaging element for each wavelength band, and compensating means for compensating each of the image signals obtained from the imaging element by using the parameter according to the wavelength band of the light received by the pixel outputting the image signal.SELECTED DRAWING: Figure 1

Description

The present invention relates to an image processing apparatus and method, an image pickup apparatus, and an image pickup element control method.

Conventionally, there is a light receiving element that enables detection of a single photon by using an avalanche photodiode (APD). When a reverse bias voltage larger than the breakdown voltage is applied to the avalanche photodiode to drive in the Geiger mode, electrons and holes generated by the incidence of a single photon cause avalanche multiplication and generate a large current. By converting this current into a voltage, the incidence of a single photon can be detected as a voltage pulse. Then, the number of incident photons can be acquired by counting the number of detected pulses. A light receiving element using the APD as described above is called a Single Photon Avalanche Diode (SPAD).

On the other hand, it is known that the light receiving element (SPAD) using the APD has a problem in input/output linearity. In particular, when the illuminance is high, that is, when the amount of light received per unit time is large, the counted value becomes lower than the number of photons actually incident. The factors that deteriorate the linearity include the fact that a plurality of voltage pulses are connected and counted as a single pulse in the state where the illuminance is high, and photons are incident during the dead time.

In order to solve such a problem, in the light receiving element described in Patent Document 1, it is possible to obtain an output that monotonically increases with an increase in the amount of incident light by integrating the widths of the voltage pulses emitted from a plurality of SPADs. It is disclosed. Therefore, the light receiving element of Patent Document 1 includes a voltage-current conversion unit that converts a voltage signal into a current signal for each pixel, and further includes an integration unit that integrates current signals of a plurality of pixels. With this configuration, the problem of linearity due to the above factors is improved.

JP, 2014-81253, A

However, considering that the SPAD having the configuration described in Patent Document 1 is applied to an image pickup device such as a camera, it is necessary to form a plurality of SPADs in one pixel.

Further, considering that SPAD is applied to an image pickup apparatus such as a camera, it is covered with color filters of a plurality of colors in order to obtain spectral information of a subject. Since the light passing through the color filter has a wavelength deviation for each pixel, the amplification factor of the electron-hole pair generated due to the difference in wavelength changes, and the overlapping manner of the voltage pulses also changes. Therefore, when viewed as an image signal, there is a problem in that the linearity of input and output differs for each color of the color filter, and color bending occurs in the high-luminance portion.

The present invention has been made in view of the above problems, and an object of the present invention is to accurately compensate for linearity of a signal obtained from an image sensor using SPAD. Another object is to suppress color bending in the high-luminance portion.

In order to achieve the above object, the image processing device of the present invention is an image processing device that processes an image signal obtained from an image sensor having a plurality of pixels that receive light in a plurality of different wavelength bands, Each of the plurality of pixels has a sensor unit that outputs a signal pulse in response to the incidence of a photon, and a counting unit that counts the number of the signal pulses, and the image processing device captures an image for each wavelength band. A storage unit that stores a parameter for compensating the linearity of the image signal output from the element, and, for each of the image signals obtained from the image pickup element, of the light received by the pixel that outputs the image signal Compensation means for compensating using the parameter according to the wavelength band.

According to the present invention, it is possible to accurately compensate the linearity of a signal obtained from an image sensor using SPAD. In addition, it is possible to suppress color bending in the high brightness portion.

FIG. 3 is a block diagram showing a schematic configuration of an image pickup apparatus according to the first embodiment of the present invention, and a schematic diagram showing a configuration of an image pickup element. 3 is a flowchart showing processing according to the first embodiment. FIG. 4 is a diagram showing parameters recorded in a storage unit according to the first embodiment. It is a schematic diagram shown about 1 type among the parameters currently recorded on the memory|storage part in 1st Embodiment. FIG. 3 is a schematic diagram for explaining an interpolation calculation in the first embodiment. The block diagram which shows the schematic structure of the image sensor in 2nd Embodiment. The flowchart which shows the process in 2nd Embodiment. The figure which shows the parameter currently recorded on the memory|storage part in 2nd Embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. The following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiment, not all of the plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar components are designated by the same reference numerals, and duplicated description will be omitted.

<First Embodiment>
A first embodiment of the present invention will be described. FIG. 1A is a block diagram showing the configuration of the image pickup apparatus according to the first embodiment. In the image pickup apparatus shown in FIG. 1A, the image pickup element 101 converts a subject image into an electric signal by photoelectric conversion and outputs the electric signal as an image signal. The linearity compensating unit 102 performs a process of compensating the linearity on the image signal output from the image sensor 101.

The parameter determination unit 103 determines a parameter used in the linearity compensation unit 102 for compensating for linearity. The storage unit 104 records parameters for compensating the linearity. The temperature sensor 105 is arranged near the image sensor 101 and measures the temperature around the image sensor 101. The control unit 106 transfers control of the entire image pickup apparatus and drive information such as the number of pixels read from the image pickup element 101.

FIG. 1B is a schematic diagram showing the configuration of the image sensor 101. The configuration of the image sensor 101 and the method of reading an image signal from the image sensor 101 will be described below with reference to FIG. As shown in FIG. 1B, the image sensor 101 of this embodiment has a configuration including a plurality of pixels 301, a vertical scanning unit 308, a column memory 309, and an output unit 310. Note that in FIG. 1B, for the sake of explanation, two pixels 301 having the same structure are arranged in the row direction and two in the column direction, but in reality, more pixels 301 are arranged. There is. Further, the two pixels 301 arranged in the column direction are connected to the vertical signal line 307.

The image pickup device 101 is a light receiving device using a single photon avalanche diode (SPAD), and each pixel 301 includes an avalanche photodiode (APD) 302, a quench resistor 303, an inverter 304, a counter 305, and a read switch 306. Have. The pixel 301 is covered with color filters of a plurality of colors to obtain a color image, and light of different wavelength bands transmitted through the color filter enters each pixel. Various types of color filters are conceivable. For example, the type is not limited as long as it can be a primary color system or a complementary color system and can acquire a color image. As an example, a case where R, G, and B color filters of Bayer array are arranged will be described.

The APD 302 is an avalanche photodiode (APD) that operates in Geiger mode to detect single photons. The Geiger mode is an APD operation mode in which a reverse bias voltage larger than the breakdown voltage is applied to operate. In the APD 302, when the reverse bias voltage Vbias larger than the breakdown voltage Va is applied to enter the Geiger mode, the carriers generated by the incidence of a single photon cause avalanche multiplication and generate a large current. Here, Vbias is -20V, for example.

The quench resistor 303 is a resistor element for stopping the avalanche multiplication of the APD 302, and a resistor is used in the first embodiment. When a photon enters the APD 302 and a current is generated due to avalanche multiplication, a voltage drop (voltage fluctuation) occurs in the quench resistor 303. When the voltage of the cathode of the APD 302 swings from Vbias to the breakdown voltage Va or less due to this voltage drop, the avalanche multiplication stops. After that, when the cathode of the APD 302 is charged through the quench resistor 303, the voltage returns to Vbias (voltage before fluctuation).

In this way, one voltage signal pulse is generated at the cathode of the APD 302 in response to the incidence of a single photon. At this time, the width of the voltage signal pulse changes according to the time constant determined by the magnitude of the resistance value of the quench resistor 303. A sensor means is constituted by the APD 302 and the quench resistor 303.

The inverter 304 as the buffer means receives the voltage signal pulse generated in the APD 302 as an input as described above, and outputs the waveform-shaped signal pulse to its output terminal.

The counter 305 receives the signal pulse output from the inverter 304 as an input and counts the rising edge thereof. Note that the inverter 304 is used as the buffer means because the counter 305 counts the rising edges. Therefore, when a buffer whose polarity is not inverted is used, the counter 305 detects the falling edges of the signal pulses output by the buffer. Just count.

In the present embodiment, the counter 305 is assumed to be configured to count the number of pulses of 16 bits as an example. In the following description, the “count value” refers to the value counted by the counter 305. Further, the counter 305 receives the signal pulse res output from the vertical scanning unit 308, and controls the reset operation of the count value and the start timing of the count operation.

The read switch 306 is turned on in response to the signal pulse sel that the vertical scanning unit 308 sequentially outputs for each row, and the count value held in the counter 305 is written to the column memory 309 via the vertical signal line 307. ..

As described above, the vertical scanning unit 308 sequentially selects rows of pixels arranged in a matrix and outputs a signal pulse sel to each row. The vertical scanning unit 308 also outputs a signal pulse res for controlling the count operation of the counter 305 and the start timing of the count operation.

In the column memory 309, the count value of each pixel 301 of the row selected by the vertical scanning unit 308 by the signal pulse sel is written via the vertical signal line 307, and the count value of each column is held. Further, the vertical scanning unit 308 sequentially outputs the count value of each pixel 301 to the output unit 310 by sequentially selecting the count value of each pixel 301 held in the column memory 309 for each column.

The output unit 310 performs digital processing such as gain processing and signal rearrangement on the count value of each pixel 301 output from the column memory 309, and outputs the processed image signal to the outside of the image sensor 101. ..

Next, with reference to FIG. 2, a process in the first embodiment in the image pickup apparatus having the above configuration will be described.

First, in S101, the control unit 106 outputs drive information for outputting an image signal from the image sensor 101, and drives the image sensor 101 as described above.

In S102, the parameter determination unit 103 acquires the temperature information of the image sensor 101 from the temperature sensor 105 provided near the image sensor 101. Further, in step S103, the parameter determination unit 103 acquires, from the control unit 106, information regarding the color filter of the pixel currently processed in the image signal read from the image sensor 101.

In step S104, the parameter determination unit 103 refers to the parameter set required for linearity compensation and reads it from the storage unit 104. FIG. 3 schematically shows parameter sets stored in the table 400 recorded in the storage unit 104. It is assumed that the table 400 holds a total of nine types of parameter sets corresponding to three types of temperatures T0, T1, and T2 for each of the three types of R, G, and B color filters. For example, R_T0 indicates a parameter set when the temperature of the R color filter is T0. Each parameter set stores a data length of 32 bits and a word count of 16 words, and two 16-bit coefficients are stored in the 32-bit data length. In S104, the parameter determination unit 103 reads the corresponding parameter set from the table 400 based on the temperature information obtained from the temperature sensor 105 and the color filter information obtained from the control unit 106, and the linearity compensation unit Send to 102.

In step S105, the linearity compensating unit 102 processes the parameter set received from the parameter determining unit 103. FIG. 4 schematically shows a plurality of parameters included in the parameter set. The horizontal axis represents the signal level of the image signal output from the image sensor 101, which is input to the linearity compensation unit 102, and the vertical axis represents the level of the signal output from the linearity compensation unit 102. A dotted line 401 indicates an ideal state in which it is not necessary to change the input signal level and the output signal level.

However, as described above, in the light receiving element using the SPAD, the counted value becomes smaller than the number of photons actually incident, especially in the state where the illuminance is high, that is, the state where the amount of received light per unit time is large. I will end up. Therefore, the linearity compensating unit 102 uses a parameter set having characteristics as shown by the solid line 402 in order to perform compensation so as to recover the signal level counted less than the number of photons actually incident, Convert the input signal.

A plurality of (16 points in the example shown in FIG. 4) black circles on the solid line 402 in FIG. 4 correspond to the parameters included in one parameter set held in the table 400 in FIG. The parameter corresponding to one black circle consists of a total of 32 bits of 16-bit x-coordinate value and 16-bit y-coordinate value. The input signal is a signal output from the image sensor 101, and has a 16-bit value range in the present embodiment as described above. Therefore, 16 points are at equal intervals on the range of the input signal level. As described above, the parameters are designed in advance.

The linearity compensating unit 102 selects and processes two points close to the input signal level from the parameters designed discretely in advance. In FIG. 4, when the signal level of the pixel currently being processed is a, the two points (403 and 404) in the vicinity thereof are extracted and the output signal level is calculated. When the input signal level a is the same as the x coordinate value of any parameter, only that one point is extracted.

Next, a method for calculating the linearity compensation value (output signal level) from the extracted two points will be described with reference to FIG. FIG. 5 shows the relationship between the two points (403, 404) extracted by the linearity compensating unit 102 and the input signal level a of the pixel currently being processed. A point 501 indicates a linearity compensation value corresponding to the input signal level a. Here, the x coordinate value of the parameter 403 is x0, the y coordinate value is y0, the x coordinate value of the parameter 404 is x1, and the y coordinate value is y1. Since the x-coordinate value of the linearity compensation value to be calculated is already known as the input signal level a, in the present embodiment, the y-coordinate value of the linearity compensation value to be calculated is calculated by interpolation calculation. When the y coordinate value of the linearity compensation value (point 501) is set to b, it can be calculated by the equation (1).

In S106, the linearity compensating unit 102 corrects the image signal by replacing the input signal level with the linearity compensation value calculated in S105 and outputting the value.

In S107, it is determined whether or not the image signals of all the pixels output from the image sensor 101 have been processed. If the image signals of all the pixels have been processed, the process ends, and the image signals of the unprocessed pixels remain. In that case, it returns to S103 and continues the above-mentioned processing.

As described above, according to the first embodiment, attention is paid to the characteristics relating to the wavelength of the photon to be counted, and the deviation of the linearity caused by the difference in the wavelength of the light incident on each pixel by the color filter is accurately measured. It is possible to compensate, and it is possible to suppress color distortion in the high-luminance portion.

In the first embodiment, the parameter set stored in the storage unit 104 is set for each color and temperature of the color filter, but the present invention is not limited to this. At least, it suffices to set different parameter sets only depending on the colors of the color filters. On the other hand, by further setting the parameter for each exposure time, it becomes possible to compensate the non-linearity of the input/output determined by the illuminance on the image sensor 101 (the amount of light received per unit time) according to the exposure time. It becomes possible to perform compensation with higher accuracy.

Further, in the above-described example, the parameter set is held in the table 400, but it is also possible to hold the equations and coefficients showing the characteristics shown in FIG. 4 and perform compensation by calculation.

<Second Embodiment>
Next, a second embodiment of the present invention will be described. FIG. 6 is a block diagram showing a schematic configuration of the image pickup apparatus according to the second embodiment. The same components as those in FIG. 1 are designated by the same reference numerals and the description thereof will be omitted.

In FIG. 6, the voltage control unit 701 controls the gate voltage of the quench element 702 included in the pixel 301 described later according to the parameter received from the parameter determination unit 103. The quench element 702 is an element for stopping the avalanche multiplication of the APD 302, and uses the resistance component of a P-type MOS transistor (PMOS) in the second embodiment. The width of the voltage signal pulse changes according to the time constant determined by the magnitude of the resistance value of the resistance component of the quench element 702. Further, to the gate terminal of the PMOS of the quench element 702, any one of the first voltage supply line 703, the second voltage supply line 704, and the third voltage supply line 705 output from the voltage control unit 701 is output. It is connected. The APD 302 and the quench element 702 form a sensor unit.

Also in the second embodiment, the pixel 301 is covered with the R, G, and B color filters in the Bayer array. Hereinafter, for the sake of distinction, a pixel covered with an R color filter is referred to as a pixel 301R, a pixel covered with a G color filter is referred to as a pixel 301G, and a pixel covered with a B color filter is referred to as a pixel 301B. The first voltage supply line 703 is connected to the pixel 301R, the second voltage supply line 704 is connected to the pixel 301G, and the third voltage supply line 705 is connected to the pixel 301B. In FIG. 6, for convenience of explanation, only two pixels are shown in the row direction and two pixels are shown in the column direction, but in reality, more pixels are formed.

Next, with reference to FIG. 7, a process in the second embodiment in the image pickup apparatus having the above configuration will be described.

First, in S201, the parameter determination unit 103 acquires the temperature information of the image sensor 101 from the temperature sensor 105 provided near the image sensor 101. Further, in step S202, the parameter determination unit 103 refers to the parameter for controlling the gate voltage of the quench element 702 from the storage unit 104 and reads it.

FIG. 8 schematically shows the parameters held in the table 800 recorded in the storage unit 104. It is assumed that the table 800 holds a total of nine types of parameters corresponding to the three types of temperatures T0, T1, and T2 for each of the three types of R, G, and B color filters. For example, Vr_T0 represents a parameter when the temperature of the R color filter is T0. Each parameter is stored with a data length of 16 bits, and data regarding the value of the voltage applied to the quench element 702 is stored.

By changing the resistance value by adjusting the gate voltage of the quench element 702, it is possible to control the width of the signal pulse generated with the incidence of photons.

In step S202, the parameter determination unit 103 reads the corresponding parameter from the table 800 based on the temperature information obtained from the temperature sensor 105 in accordance with the control timing sent from the control unit 106. Then, in step S<b>203, the parameter determination unit 103 transmits the parameters read in step S<b>202 to the voltage control unit 701 included in the image sensor 101.

In S204, the quench element 702 is controlled. The voltage control unit 701 receives the voltage value supplied from the first voltage supply line 703 and the second voltage value based on the data related to the value of the voltage applied to the quench element 702, which is the parameter sent from the parameter determination unit 103. The voltage value supplied from the voltage supply line 704 and the voltage value supplied from the third voltage supply line 705 are respectively determined and supplied to a plurality of pixels forming the image sensor 101.

When a release member (not shown) is pressed in S205, the process proceeds to S206 for shooting. If the button has not been pressed, the process returns to S201 and the above process is repeated.

In S206, the control unit 106 outputs drive information for outputting an image signal from the image sensor 101. The drive information includes the settings of the vertical scanning unit 308, the column memory 309, and the output unit 310 of the image sensor 101, and the image sensor 101 outputs an image signal according to the drive information.

In step S207, an image signal is output from the image sensor 101, is subjected to signal processing (not shown), and is then sent to the recording device or the display device.

As described above, according to the second embodiment, attention is paid to the characteristics of the wavelength of the photon to be counted, and the width of the signal pulse generated with the incidence of the photon is controlled. Thereby, the color filter can suppress the color bending of the high-luminance portion due to the linearity shift caused by the difference in the wavelength of the light incident on each pixel.

In the second embodiment, the case where the parameters held in the storage unit 104 are set for all colors and temperatures of the color filter has been described as an example, but the present invention is not limited to this. .. For example, it is possible to reduce the amount of parameters to be held by holding only the parameters related to the color filter whose linearity is particularly deteriorated.

Further, in the above-described second embodiment, the color filter has different parameters depending on the color and the temperature. It is possible to suppress color bending in the.

<Other Embodiments>
The present invention may be applied to a system including a plurality of devices (for example, a camera, an interface device, an image processing device) or an apparatus including one device.

Further, the present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus execute the program. It can also be realized by a process of reading and executing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Therefore, the following claims are attached to open the scope of the invention.

Reference numeral 101: image sensor, 102: linearity compensation unit, 103: parameter determination unit, 104: storage unit, 105: temperature sensor, 106: control unit, 301, 706, 707, 708: pixel, 302: avalanche photodiode, 303 : Quench resistor, 701: voltage controller, 702: quench element

Claims (17)

An image processing device for processing an image signal obtained from an image pickup device having a plurality of pixels for receiving light in a plurality of different wavelength bands, wherein each of the plurality of pixels outputs a signal pulse in response to incidence of a photon. The image processing apparatus, and a sensor unit for counting the number of the signal pulses,
For each wavelength band, storage means for storing parameters for compensating the linearity of the image signal output from the image sensor,
Compensation means for compensating for each of the image signals obtained from the image sensor using the parameter according to the wavelength band of the light received by the pixel that outputs the image signal. Processing equipment. The parameter is a value indicating a signal level after compensation for a plurality of predetermined signal levels,
The image processing apparatus according to claim 1, wherein the compensation unit compensates the signal level of the image signal using the parameter. Further comprising a measuring means for measuring the temperature around the image sensor,
The parameter is further set for each of a plurality of predetermined temperatures,
The image processing apparatus according to claim 1, wherein the compensating unit further compensates using the parameter according to the measured temperature. The parameter is further set for each of a plurality of predetermined exposure times,
The image processing apparatus according to claim 1, wherein the compensating unit further compensates using the parameter according to the exposure time of the image signal. The image processing according to any one of claims 1 to 4, wherein the plurality of pixels are covered with color filters of a plurality of colors that mainly transmit the lights of the plurality of different wavelength bands. apparatus. The image processing apparatus according to claim 1, wherein each pixel includes an avalanche photodiode driven in a Geiger mode. An image processing apparatus according to any one of claims 1 to 6,
An image pickup device comprising: the image pickup device. An image pickup device having a plurality of pixels for receiving light in a plurality of different wavelength bands,
At least one of the plurality of wavelength bands, storage means for storing parameters for controlling the image sensor,
A control unit that controls the image sensor using the parameter,
The voltage of each pixel of the image sensor varies depending on the incidence of photons, and the sensor means has a resistor for returning the varied voltage to the voltage before the variation, and the sensor means according to the variation of the voltage. Counting means for counting the number of signal pulses output from
The parameter is a value for controlling the resistance value of the resistor, and the control unit controls the image sensor by using the parameter according to the wavelength band of the light received by each of the plurality of pixels. An imaging device characterized by the above. The image pickup apparatus according to claim 8, wherein the resistance is a MOS transistor, and the parameter is a value for controlling a gate voltage of the MOS transistor. Further comprising a measuring means for measuring the temperature around the image sensor,
The parameter is further set for each of a plurality of predetermined temperatures,
The image pickup apparatus according to claim 8 or 9, wherein the control unit further controls using the parameter according to the measured temperature. The imaging device according to any one of claims 8 to 10, wherein the plurality of pixels are covered with color filters of a plurality of colors that mainly transmit the light of the plurality of different wavelength bands. .. The image pickup apparatus according to claim 8, wherein each of the pixels includes an avalanche photodiode driven in a Geiger mode. Each of a plurality of pixels that receives light of a plurality of different wavelength bands has a sensor unit that outputs a signal pulse in response to the incidence of a photon, and a counting unit that counts the number of the signal pulses. An image processing method for processing an image signal obtained from an image sensor for outputting an image signal obtained by counting by a pixel counting means,
An acquisition step in which the acquisition means acquires a parameter for compensating the linearity of the image signal output from the image sensor, which is set for each wavelength band;
A compensating means for compensating each of the image signals obtained from the image sensor by using the parameter according to the wavelength band of the light received by the pixel outputting the image signal. Characteristic image processing method. A sensor having a plurality of pixels that receive light in a plurality of different wavelength bands, each pixel having a voltage that varies depending on the incidence of a photon, and a resistor that returns the varied voltage to the voltage before the variation. A method for controlling an image pickup device, comprising: a counting unit that counts the number of signal pulses output from the sensor unit according to a change in the voltage.
An acquisition step in which the acquisition means acquires a parameter according to each of the wavelength bands,
And a control step of controlling the resistance value of the resistance of each pixel using the parameter according to the wavelength band of the light received by each pixel. A program for causing a computer to function as each unit of the image processing apparatus according to claim 1. A program for causing a computer to execute each step of the control method according to claim 14. A computer-readable storage medium that stores the program according to claim 15 or 16.

Images