WO2023151675A1

WO2023151675A1 - Improved cmos image sensor

Info

Publication number: WO2023151675A1
Application number: PCT/CN2023/075600
Authority: WO
Inventors: 倪扬
Original assignee: 上海虹感微电子科技有限公司
Priority date: 2022-02-14
Filing date: 2023-02-13
Publication date: 2023-08-17
Also published as: CN116647769A

Abstract

The title of the present application is an "improved CMOS image sensor". A CMOS image sensor comprises a pixel matrix and a pixel control circuit (VS), each pixel has a pixel structure comprising a photodetector (PD) and a memory unit (MEM), and the pixel control circuit is capable of sequentially writing a photoelectric detection value representing the exposure to the memory unit (MEM) of the pixel. According to the present invention, the image sensor further comprises comparison circuits (DMU, RB, CE), and the comparison circuits are used for: storing sequential photoelectric detection values of at least some of the pixels; comparing the sequential photoelectric detection values; generating comparison signals (NUL, POS, NEG) according to the sequential photoelectric detection values of the pixels; and generating a comparison matrix reflecting an image change from the comparison signals.

Description

Improved CMOS image sensor

technical field

The present invention generally relates to CMOS image sensors.

Background technique

CMOS image sensors have seen very significant advances in spatial resolution and performance in recent decades, driven primarily by the camera-equipped smartphone market.

Today, these high-resolution sensors produce pictures of extremely high quality and contain increasingly large amounts of data.

Today, high-quality images are around 2Mb to 10Mb in size before image compression.

Such images require increased processing when used for computer vision, which can be problematic. More specifically, a significant portion of computing resources is devoted to processing parts of images that make little or no sense. In addition, high-performance computing resources potentially require expensive hardware and high energy consumption.

These problems can be solved if one can focus on parts of the image that are likely to contain important and useful information.

In many applications, the presence of motion in a scene constitutes a visual event to be processed. This is the case, for example, in video surveillance systems, people counting systems, gesture control systems, etc.

To solve these problems, so-called Dynamic Vision Sensors (DVS) have been developed, where each pixel of the sensor includes a brightness change detector. When a brightness change is detected, a corresponding event is generated. The event includes a timestamp and the X,Y coordinates of the associated pixel in the image matrix.

Such known sensors further comprise a conflict manager, usually located at the edge of the image array, in order to resolve conflicts between events when multiple events are generated.

Document US10567679B2 discloses an example of such a dynamic vision sensor.

The basic principle of this known sensor is that it allows to recreate the spatiotemporal trajectories of moving objects in a scene.

However, such known sensors have complex architectures that are incompatible with conventional image computing environments.

Contents of the invention

The present invention is based on the discovery that for many computer vision applications, events generated by simple image change detection between two consecutive image frames can be effective.

Therefore, the present invention provides a new sensor architecture that allows detecting changes between successive images and encoding the changes in a simple data format.

According to the present invention, for each captured image, a bit matrix having the same size as the image is generated, wherein each bit indicates whether the corresponding pixel has changed, preferably whether the brightness has changed. Optionally, the bit matrix also encodes the nature of the change, ie increase or decrease in brightness.

More specifically, the present invention provides a CMOS image sensor, which includes a pixel matrix and a pixel control circuit, each pixel has a pixel structure including a photodetector and a storage unit, and the pixel control circuit can sequentially write the photoelectric The detection value is stored in the pixel, the image sensor further includes a comparison circuit, the comparison circuit is used for storing at least part of the sequential photodetection values of the pixels, for comparing the sequential photodetection values, and for comparing the sequential photodetection values according to the The sequential photodetection values of the pixels generate a comparison signal and are used to generate a comparison matrix reflecting image changes from the comparison signal.

The invention allows a significant simplification of the per-pixel structure and does not require the provision of a collision manager.

Furthermore, the present invention can provide output compatible with existing image processing environments. Since less power is required, it can be used in battery-powered vision systems, for example in the Internet of Things (IoT) field.

The image sensor may further include the following optional features alone or in any technology compatible combination:

- said comparison circuit comprises a dual memory cell comprising a set of memory cell pairs selected at different times corresponding to two successive exposures of a pixel associated with said read bus are selectively connectable to corresponding read busses, which are selectively connectable to memory cells of associated pixel groups.

- said comparison circuit comprises a dual bus unit for selectively connecting a given pair of two memory cells to a pair of inputs of a comparator capable of generating said comparison signal.

- said read bus and memory cell pairs are respectively associated with columns or rows of a pixel matrix.

- The image sensor comprises a first scanning unit for sequentially connecting said pair of pixel memory cells of said dual memory cell to said comparator.

- The image sensor comprises a second scanning unit for sequentially connecting the pixel storage units of a selected pixel group to said read bus.

- The relationship between said sequential photodetection values is a comparison function with a dead zone.

- Said comparison function with dead zone is performed by a CMOS differential pair with offset adjustment.

- The comparison circuit is capable of generating comparison signals for only said fraction of pixels, each comparison signal being associated with a respective group of pixels.

- The comparison circuit is capable of generating a comparison signal for only said fraction of pixels, and the image sensor further comprises an interpolation circuit for interpolating the comparison signal and for associating the interpolated comparison signal with the corresponding pixel.

- The image sensor further comprises a combining circuit capable of combining pixel values generated by the pixel matrix and forming a main pixel image and a corresponding comparison signal contained in said comparison matrix and associated with a corresponding pixel or with a corresponding group of pixels.

- The image sensor further comprises a detection circuit for determining an image change level from said comparison matrix, and a storage circuit for storing at least one main image only if said image change level is above a threshold value.

Description of drawings

Other features, objects and advantages of the present invention will become more apparent from the following description of preferred embodiments of the present invention given with reference to the accompanying drawings.

In the attached picture:

1 is an overall diagram of a CMOS image sensor according to an embodiment of the present invention,

Figures 2(A) to 2(D) illustrate four possible embodiments of photodetectors at the pixel level,

Figures 3(A) to 3(C) illustrate three possible embodiments of photodetector value readout at the pixel level,

Figure 4 is a timing diagram of the signals involved in the circuit of Figure 1,

Figure 5 is an exemplary embodiment of a comparator with a dead zone and three output values, and

Figure 6 shows the deadband of the comparator response.

Detailed ways

A) General Architecture

Fig. 1 illustrates the overall architecture of an image sensor according to the present invention.

The sensor consists of five main components:

- matrix M of pixels P,

-Double storage unit DMU,

- read bus RB,

- Comparator-encoder unit CE,

- Horizontal scanning unit HS and vertical scanning unit VS.

Each pixel P of the matrix M comprises a photodetector PD and a memory unit MEM connected to the photodetector via a read switch RS.

The memory units MEM of pixels P of a selected column are connected via respective memory read switches MRS to respective column read buses CRBj, each said bus CRBj being connectable via respective switches RD1, RD2 to respective ones of the dual memory units DMU. The storage unit pair CMEM1, CMEM2. It can be understood that the double storage unit includes a plurality of storage unit pairs CMEM1 and CMEM2 equal to the number of columns.

The read bus comprises a pair of buses BS1, BS2 which are respectively connectable to each memory cell CMEM1, CMEM2 via a corresponding switch RD3, RD4.

The two buses BS1, BS2 of the read bus RB are connected to respective inputs of a comparator-encoder unit CE arranged to output POS and NEG output signals respectively, as will be explained below.

Referring now to FIGS. 1 to 4 , the vertical scanning unit VS is configured to be sequentially applied to successive rows L1 of corresponding matrix row exposure pulses TX1, . . . , TXn and corresponding row selection pulses SEL1 , . , ..., the pixels of Ln.

For each column of the matrix, the horizontal scanning unit HS is configured to be applied sequentially to the switches RD1, RD2 of the dual memory unit DMU:

- Input switching control pulses (rows RD1, RD2 in Fig. 4) applied to switches RD1, RD2 to selectively connect a given column bus CRBj to either memory cell CMEM1 or memory cell CMEM2, such a command is valid for all The columns are all generic,

- output memory control pulses CS1, CS2, ..., CSm (column specific) which are applied to the respective switches RD3, RD4 for selectively connecting the memory cells CMEM1, CMEM2 of a given column to the bus BS1 respectively , BS2.

The general operation of an image sensor is as follows:

1) The vertical scanning unit VS selects a row of pixels, and applies the current pixel value stored in each local pixel memory unit MEM for the selected row to the column bus CRBij by controlling the first SEL pulse of the MRS switch;

2) loading the pixel value applied to the corresponding bus CRBj into the corresponding memory cell CMEM1 by applying to the switch RD1 a control pulse substantially superimposed on the first SEL pulse;

3) The exposure pulse TX is applied to the switch RS of the row of pixels, so as to connect with the photodetector during the exposure The new pixel value corresponding to the dose of photons received during the period is loaded into the pixel storage unit MEM;

4) The control signals SEL and RD2 are activated to connect the pixel memory unit MEM to the corresponding column bus CRBj, and load the new pixel value to the corresponding memory unit CMEM2 of the dual memory unit DMU;

5) The contents of the memory CMEM1, CMEM2 are sequentially read through the signals CS1, CS2, ..., CSm, and the signals CS1, CS2, ..., CSm respectively connect the pixel values of each column stored in the cells CMEM1, CMEM2 to the read Take bus BS1, BS2;

6) The bus lines BS1, BS2 compare the contents of the existing memories CMEM1, CMEM2 successively in the comparator-encoder unit CE and the corresponding outputs (NEG, POS) are binary coded, as will be explained below.

Repeat the above process for all rows of the pixel matrix.

It can be seen from the above that at a given time, a given pair of memory cells CMEM1, CMEM2 in a dual memory cell contains the currently selected i-th row and the j-th column defined by the position of said given pair in the unit DMU contains The two sequential pixel values of pixel P.

The comparator-encoder unit CE is configured to produce a pixel-by-pixel output according to the following rules:

- if there is no change between the pixel values contained in CMEM1 and CMEM2 for a given pixel, cell CE outputs a NULL value, where both POS and NEG outputs are invalid (e.g. low state of binary zero),

- If there is a change, the cell CE activates its POS (positive) output in case of an increase in pixel value (increased brightness), or its NEG (negative) output in case of a decrease in pixel value (decrease in brightness), active state is for example a high state or a binary 1.

Thus, the outputs generated by unit CE for all pixels P of matrix M constitute a comparison matrix reflecting the results of a comparison between two sequential readings of the image sensor at the pixel level.

B) Actual Example

Each pixel P of the image sensor may include a photodetector PD having a monotonic response to exposure. It will be appreciated that linearity or uniformity of response is not required, but a high dynamic range is preferred to allow the sensor to detect brightness changes in dark and bright environments, thereby avoiding saturation in bright light conditions as much as possible (with neutralization of pixel value changes Effect).

A photodetector with a logarithmic response is the optimal choice for the sensor, although a high dynamic range linear photodetector may also be suitable.

Figures 2(A) to 2(D) illustrate various possible configurations of logarithmic photodetectors, each Both structures are based on photodiode D1. These structures are known per se to those skilled in the art.

The pixel memory unit MEM may consist of a capacitor selectively connectable to the photodetector, the voltage of which will represent the number of photons received by the photodetector during the exposure time.

The capacitor of the memory cell MEM can be, for example, a MOS capacitor or a MIM (metal-insulator-metal) capacitor. MOS capacitors would be preferred when good compactness is required.

3(A) to 3(C) illustrate different pixel embodiments that may be used in the present invention.

The above-mentioned embodiments respectively use the photodetection structures shown in FIG. 2(A), FIG. 2(B) and FIG. 2(D).

Those skilled in the art will appreciate that the embodiments of Figures 3(A) and 3(B) are more compact, while the embodiment of Figure 3(C) requires more components and thus more substrate space, although it is less compact However, since the feedback amplifier reduces the adverse effect of the internal capacitance of photodiode D1, it can operate at a higher rate.

In each of the embodiments of FIG. 3(A) to FIG. 3(C), the switching unit T2/T3 is driven by the selection signal SEL. This approach is important for the accuracy of change detection. In fact, if the switching unit T2/T3 is controlled by a constant voltage such as in a conventional CMOS pixel structure, the source of the signal transistor T2 will be floating when the pixel is in the non-selected state. This causes an error when writing the photodiode voltage VLOG to the memory MEM.

With the select signal SEL driving the follower T2/T3, when the pixel is selected, the transistor T2 is in a resistive state, so that the gate capacitance is stabilized.

Vertical scanning and horizontal scanning are performed in a manner known per se by shift registers or address decoders.

A suitably programmed sequencer generates control signals to apply to the circuit. As noted above, FIG. 4 provides an exemplary timing diagram of control signals for the circuit of FIG. 1 .

CMEM1 and CMEM2 memories are constructed of CMOS capacitors or MIMs, just like conventional CMOS processes.

As already explained, the output NUL/POS/NEG of each pixel of the comparison matrix is produced by comparing the values of CMEM1, CMEM2 transmitted by the buses BS1, BS2 to the comparator-encoder unit CE.

It will be understood that a basic comparator will allow detection of a change in positive or negative direction, thereby generating a POS signal and a NEG signal. However, such a basic comparator usually cannot generate a NUL signal corresponding to the situation where the pixel value is stable, in fact the comparator will always behave in a more or less random way in this case. The formula switches from POS values to NEG values, thereby generating significant noise in the comparison matrix.

To solve this potential problem, the preferred comparator-encoder unit CE according to the invention is designed with a fixed or programmable dead working zone (DWZ).

Figure 5 illustrates an exemplary embodiment of such a comparator. The differential signal present on the bus is applied to a differential pair consisting of transistors TC1, TC2, TC3, TC4.

The generated differential current is buffered by transistors TC9, TC10 for comparison with the current generated by the differential pair TC7, TC8 from the bias current Ib.

The value of the current generated by the differential pair TC7, TC8 is determined by the value of a resistor R inserted between Vcc and the drain of each of the transistors TC7, TC8. The reduction in current thus obtained generates a dead zone in the comparison between the signals present at the inputs (signals on the buses BS1 , BS2 ). In other words, as long as the difference between the input signals on BS1, BS2 is lower than the threshold, the outputs POS and NEG are kept at low level.

When the difference exceeds a positive or negative threshold defining the deadband, then the corresponding output POS or NEG switches to a high state.

Therefore, with proper selection of the R value, only brightness changes above a given threshold d generate POS or NEG output, thereby avoiding unstable POS/NEG output in the case of unchanged pixels.

The resistor R can use an off-chip resistor or an on-chip programmable resistor such as an I2C standard interface. In addition, the resistance R may also be a fixed value.

Fig. 6 illustrates the response of the comparator-encoder circuit CE with the dead zone described above.

The solid diagonal line indicates the possible values of the inputs (difference between BS2 and BS1). The dashed line shows the response of the comparator, where the output switches from NUL to POS when the difference between the voltage values on BS2 and BS1 exceeds the positive threshold +TV, and only when the (negative) difference between the voltage values on BS2 and BS1 exceeds the negative threshold -When TV, the output switches from NUL to NEG. The width of the dead zone DWZ is determined by the values of +TV and -TV, which are themselves determined by the resistor R.

The technology involved in this embodiment as described above is entirely analog, thus resulting in a sensor with very low power consumption, typically below 1 mW for standard video frame rates (25/30 Hz).

In alternative embodiments, the read and compare stages may be implemented by digital circuitry implementing appropriate analog-to-digital converters for each column and digital calculation circuitry for generating the comparison matrix.

Depending on the nature of subsequent processing, the output of the circuit for each pixel can be a dual output POS-NEG as shown, or a two-bit word derived from these outputs, such as the following encoding:

00: No brightness change (in dead zone)

01: Increased brightness

10: Reduced brightness

11: N/A

It will be appreciated that since the sensor according to the invention generates a lightweight comparison matrix, whatever the application (typically monitoring image changes), the subsequent processing of such a matrix can be performed by simple, low cost and low power consumption circuitry.

Those skilled in the art will appreciate that various modifications and changes can be made to the above description.

For example, comparison matrices can be constructed by scanning schemes other than conventional row and column scanning.

In addition, the comparison matrix can be constructed from a part of the pixels of the matrix sensor, for example, there are (N>1) 1 pixel in every N pixels, where N may be different in the horizontal and vertical directions, so that the power consumption and downstream calculation requirements are further reduced .

Alternatively, the pixel values of the sub-matrices within the pixel matrix can be accumulated in sub-matrix level memory cells, thereby generating an average reading for each sub-matrix, and from there a NUL, POS, or NEG output at the sub-matrix level.

According to another variant, the image sensor according to the invention can operate in a hybrid mode, in which:

- a low-power image change detection process using a comparison matrix, and

- Generate and record high-resolution and high-quality images as soon as some degree of change in the captured scene is detected from the contents of the comparison matrix.

This can be achieved by generating a hybrid image that combines the nominal (highest) resolution main image with pixels from a comparison matrix of the same or lower resolution.

In the case of lower resolutions, the pixel values of the comparison matrix associated with the corresponding sub-matrix of pixels of the main image may be interpolated before being assigned to the pixels of the corresponding sub-matrix.

Claims

A CMOS image sensor comprising a pixel matrix and a pixel control circuit (VS), each pixel having a pixel structure comprising a photodetector (PD) and a memory unit (MEM), the pixel control circuit capable of sequentially writing representative exposure After the photodetection value is sent to the storage unit (MEM) of the pixel, the image sensor further includes a comparison circuit (DMU, RB, CE), and the comparison circuit is used to store the sequential photodetection value of at least part of the pixels, for comparing the sequential photodetection values, for generating comparison signals (NUL, POS, NEG) from the sequential photodetection values of the pixels, and for generating a comparison matrix reflecting image changes from the comparison signals.
The image sensor according to claim 1, wherein the comparison circuit comprises a dual memory unit (DMU) and a read bus (CRBj) corresponding to a pair of memory units, the double memory unit (DMU) comprising a group of memory cells For (CMEM1, CMEM2), the read bus can selectively write the photodetection values of two sequential exposures of the associated pixel group (Li) into each memory cell pair.
Image sensor according to claim 2, wherein said comparison circuit comprises a dual bus unit (RB) for selectively connecting a given pair of two memory units (CMEM1, CMEM2) to A pair of inputs of a comparator (CE) capable of generating said comparison signal.
Image sensor according to claim 2 or 3, wherein said read bus (CRBj) and memory cell pairs (CMEM1, CMEM2) are respectively associated with columns or rows of said pixel matrix (M).
The image sensor according to claim 4, comprising a first scanning unit (HS) for sequentially connecting said pair of pixel memory cells of said double memory cell to said comparator.
Image sensor according to claim 5, comprising a second scanning unit (VS) for sequentially connecting the pixel storage units (MEM) of a given pixel group (Li) to the read bus (CRBj).
The image sensor according to any one of claims 1 to 6, wherein the relationship between the sequential photodetection values is a comparison function with a dead zone.
The image sensor of claim 7, wherein the comparison function with dead zone is performed by a CMOS differential pair with bias adjustment.
An image sensor according to any one of claims 1 to 8, wherein the comparison circuit is capable of generating comparison signals (NUL, POS, NEG) for only a fraction of the pixels in the pixel matrix, each comparison signal corresponding to group of pixels are associated.
An image sensor according to any one of claims 1 to 8, wherein said comparison circuit is capable of generating comparison signals (NUL, POS, NEG) for only a portion of the pixels in said pixel matrix, and further comprising comparing the comparison signals An interpolation circuit performs interpolation and correlates the interpolated comparison signal to each pixel.
An image sensor according to any one of claims 1 to 10, further comprising combining circuitry capable of combining pixel values generated by said pixels of said matrix and forming a primary pixel image and a corresponding pixel or pixels The comparison signal for the corresponding pixel in the group's comparison matrix.
The image sensor of claim 11 , further comprising detection circuitry for determining a level of image change from said comparison matrix, and storage circuitry for storing at least one primary image only when said level of image change is above a threshold.