WO2024188243A1

WO2024188243A1 - Image sensor, image processing method, and electronic device

Info

Publication number: WO2024188243A1
Application number: PCT/CN2024/081230
Authority: WO
Inventors: 李杨; 潘撼; 邓志鹏; 胡慧; 赵晓锦; 郝嘉诚; 左海彪
Original assignee: 华为技术有限公司
Priority date: 2023-03-14
Filing date: 2024-03-12
Publication date: 2024-09-19
Also published as: CN118678243A

Abstract

The embodiments of the present application relate to the technical field of image processing. Provided are an image sensor, an image processing method, and an electronic device, which are used for providing an image sensor with low power consumption and a low cost, such that frame-by-frame imaging can be realized, and a dynamic visual signal of a moving object in an image frame can also be output. The image sensor comprises: a photosensitive circuit and a column line circuit. The photosensitive circuit is configured to receive incident light in the same image frame, so as to generate an exposure signal. The column line circuit comprises a dual sampler, a switch array circuit and a digital-to-analog converter, wherein the dual sampler is coupled to an output end of the photosensitive circuit, and is configured to receive and process the exposure signal, output an image signal of the same image frame in a first operating mode, and output a dynamic visual signal of the same image frame in a second operating mode; the switch array circuit is coupled to an output end of the dual sampler, and is used for gating signals of different branches and transmitting the signals to an analog-to-digital converter; and the analog-to-digital converter is configured to convert a received signal of the switch array circuit into a digital signal and then output the digital signal.

Description

Image sensor, image processing method, and electronic device

Technical Field

The present application relates to the field of image processing technology, and in particular to an image sensor, an image processing method, and an electronic device.

Background Art

The scene captured by the image sensor contains both static and moving objects. In some applications, imaging both static and moving objects helps to analyze and understand the scene. In some applications, only moving objects need to be captured.

For example, most of the environment in the field of view is static, with only one or a few moving objects. If a conventional image sensor is used, each photosensitive circuit will image frame by frame, which will generate a lot of repeated and redundant information, increasing meaningless storage, calculation and power consumption.

Moreover, in these applications, moving objects are the information that deserves more attention, and static background information is relatively less important. However, traditional sensors used to capture images of moving objects have problems such as complex manufacturing processes and large pixel sizes.

Therefore, how to design a sensor that can not only achieve frame-by-frame imaging (including stationary objects and moving objects), but also output images of moving objects within the image frame, with a simple manufacturing process and low cost and area, has become a research direction for technical personnel in this field.

Summary of the invention

The embodiments of the present application provide an image sensor, an image processing method, and an electronic device, which are used to provide a low-cost, low-power image sensor that can not only achieve frame-by-frame imaging, but also output dynamic visual signals of moving objects in the image frame.

In order to achieve the above purpose, this application adopts the following technical solutions:

According to one aspect of an embodiment of the present application, an image sensor is provided, including: a photosensitive circuit and a column line circuit.

The photosensitive circuit is configured to receive incident light within the same image frame, and in a first working mode, generate one exposure signal; in a second working mode, generate multiple exposure signals. The column line circuit includes a dual sampler, a switch array circuit, and a digital-to-analog converter. The dual sampler is coupled to the output end of the photosensitive circuit and is configured to receive and process the exposure signal, output an image signal of the same image frame in the first working mode, and output a dynamic visual signal of the same image frame in the second working mode. The switch array circuit is coupled to the output end of the dual sampler and is used to select the signal transmission of different branches to the analog-to-digital converter. The analog-to-digital converter is configured to convert the received signal of the switch array circuit into a digital signal and output it.

The image processor provided by the embodiment of the present application selects the signal transmission of different branches to the analog-to-digital converter through the switch array circuit, thereby realizing different working modes of the circuit. The dual sampler can output image signals and dynamic visual signals, which can realize frame-by-frame imaging and output moving objects in the image frame. If the photosensitive unit divides the complete exposure period in an image frame into multiple sub-exposure periods, and outputs the photogenerated electrons generated in multiple sub-exposure periods to the column line circuit for processing in sequence, and obtains the brightness information corresponding to different sub-exposure periods, the presentation of the image brightness information in the complete exposure period in an image frame can be realized. Moreover, the output dynamic visual signal is the dynamic change between multiple sub-exposure periods in an image frame, which is equivalent to interpolation within the frame, rather than the dynamic change of the previous/next two frames. By adding an analog-to-digital converter in the column line circuit, it can be used to realize the conversion of analog signals into digital signals. The image sensor in the present application is an improvement made on the basis of the traditional CIS, and can reuse the mature CIS process and image algorithm, with low technical difficulty and short product development cycle.

On the premise of satisfying the imaging function, the dynamic vision function is added to become a "dual-mode vision sensor" that can switch working modes. And no digital image signal processor is required, which reduces the cost and area of image sensor manufacturing.

In a possible implementation, the switch array circuit includes a first switch, a second switch, a third switch, a fourth switch, and a fifth switch. The second end of the first switch is coupled to the first end of the third switch. The second end of the second switch is coupled to the second end of the third switch. The first end of the fourth switch is coupled to the first end of the third switch. The first end of the fifth switch is coupled to the second end of the third switch. By controlling the on and off of the first switch, the second switch, the third switch, the fourth switch, and the fifth switch, signals of different branches are transmitted to the analog-to-digital converter, thereby realizing flexible switching of the circuit working mode.

In a possible implementation, the analog-to-digital converter includes a sixth switch, a digital-to-analog converter, and a comparator. The first end of the sixth switch is coupled to the second input end of the comparator, and the second end of the sixth switch is coupled to the digital-to-analog converter. The sixth switch is used to connect the digital-to-analog converter to the circuit in the first working mode, and digitally quantize the analog signal imaged frame by frame by judging whether the comparator is flipped.

In a possible implementation, the column line circuit also includes a dual sampler, and the dual sampler includes: a first capacitor, a second capacitor, a first operational amplifier, and a seventh switch. The first end of the first capacitor is coupled to the output end of the photosensitive circuit, and the second end of the first capacitor is coupled to the first end of the second capacitor and the first input end of the first operational amplifier. The second end of the second capacitor is coupled to the output end of the first operational amplifier. The seventh switch is connected in parallel with the second capacitor. The second input end of the first operational amplifier is coupled to a reference voltage. The output end of the first operational amplifier is coupled to the output end of the dual sampler. A buffer can also be set in the dual sampler to improve the driving capability of the dual sampler and make the image sensor faster and more stable.

In a possible implementation, the photosensitive circuit divides the exposure period in the same image frame into two different periods, namely a first exposure period and a second exposure period, in the second working mode, and compares the difference in the cumulative intensity of incident light in different exposure periods to sense the change in illumination, thereby realizing a dynamic visual function.

In a possible implementation, the first exposure period and the second exposure period are two adjacent periods in the same image frame. By dividing the complete exposure period of the photosensitive unit in an image frame into multiple sub-exposure periods, the photogenerated electrons generated in the multiple sub-exposure periods are sequentially output to the column line circuit for processing, and the brightness information corresponding to the different sub-exposure periods is obtained, so as to realize the judgment of the brightness change in the complete exposure period in an image frame.

In a possible implementation, the dynamic visual signal represents the difference between the intensity of the incident light in the second exposure period and the intensity of the incident light in the first exposure period. By comparing the intensity of the incident light in the second exposure period with the intensity of the incident light in the first exposure period, the change in light intensity between the two periods in the image frame can be obtained.

In a possible implementation, the sum of the first exposure period and the second exposure period is the third exposure period; the image signal represents the cumulative intensity of the incident light in the third exposure period. The third exposure period lasts for a long time, and the number of photogenerated electrons generated is large. Therefore, the signal-to-noise ratio of the image signal is large, the flicker noise is small, and the image quality is good.

In a possible implementation, the photosensitive circuit is further configured to generate a reset voltage, and the double sampler is further configured to receive the reset voltage.

According to a second aspect of an embodiment of the present application, an image sensing method is provided, the image sensing method comprising: selecting a corresponding working mode and generating an exposure signal. According to the exposure signal, an image signal of an image frame is generated in a first working mode. In a second working mode, a dynamic visual signal of the image frame is generated. The image signal and the dynamic visual signal in the corresponding working mode are converted into digital signals and output. The beneficial effects of the image sensing method provided in the embodiment of the present application are the same as those of the image sensor, which will not be repeated here.

In a possible implementation, generating an exposure signal includes: in a first working mode, receiving incident light and generating a third exposure signal. In a second working mode, receiving incident light in a first exposure period and generating a first exposure signal, and then receiving incident light in a second exposure period and generating a second exposure signal. The first exposure period and the second exposure period are two adjacent periods in the same image frame. By dividing the complete exposure period of a photosensitive unit in an image frame into a plurality of sub-exposure periods, the photogenerated electrons generated in the plurality of sub-exposure periods are sequentially output to the column line circuit for processing, and the brightness information corresponding to the different sub-exposure periods is obtained, so as to realize the judgment of the brightness change in the complete exposure period in an image frame. Moreover, the output dynamic visual signal is the dynamic change between the plurality of sub-exposure periods in an image frame, which is equivalent to intra-frame interpolation, rather than the dynamic change of the previous/next two frames.

In a possible implementation manner, the image processing method further includes: generating a reset voltage.

In a possible implementation, the first exposure period and the second exposure period are two adjacent periods in the same image frame. The dynamic visual signal represents the difference between the intensity of the incident light in the second exposure period and the intensity of the incident light in the first exposure period. By comparing the intensity of the incident light in the second exposure period with the intensity of the incident light in the first exposure period, the light intensity change between the two periods in the image frame can be obtained.

In a possible implementation, the sum of the first exposure period and the second exposure period is the third exposure period. The image signal represents the cumulative intensity of the incident light in the third exposure period. The third exposure period lasts for a long time, and the number of photogenerated electrons generated is large. Therefore, when it is used as an image signal, the signal-to-noise ratio of the image signal is large, the flicker noise is small, and the image quality is good.

According to a third aspect of the present application, an electronic device is provided, comprising an image sensor and a printed circuit board, wherein the image sensor is arranged on the printed circuit board, and the image sensor comprises any one of the image sensors of the first aspect. A pin of the image sensor is used to output signals of different functions.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of a framework of an electronic device provided in an embodiment of the present application;

FIG2 is a schematic diagram of a framework of an image sensor provided in an embodiment of the present application;

FIG3 is a schematic diagram of a framework of another image sensor provided in an embodiment of the present application;

FIG4 is a schematic diagram of an output of a frame difference method provided in an embodiment of the present application;

FIG5 is a schematic diagram of a framework of another image sensor provided in an embodiment of the present application;

FIG6 is a schematic diagram of the relationship between exposure time periods provided in an embodiment of the present application;

FIG7 is a schematic diagram of the structure of a column line circuit provided in an embodiment of the present application;

FIG8 is a schematic diagram of the structure of a photosensitive circuit provided in an embodiment of the present application;

FIG9 is a timing diagram of a photosensitive circuit provided in an embodiment of the present application;

FIG. 10 is a timing diagram of another photosensitive circuit provided in an embodiment of the present application.

DETAILED DESCRIPTION

The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments.

In the following, the terms "second", "first", etc. are used only for convenience of description and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "second", "first", etc. may explicitly or implicitly include one or more of the feature. In the description of this application, unless otherwise specified, "plurality" means two or more.

In addition, in the embodiments of the present application, directional terms such as "up", "down", "left" and "right" may be defined including but not limited to the orientation relative to the schematic placement of the components in the drawings. It should be understood that these directional terms may be relative concepts, which are used for relative description and clarification, and may change accordingly according to changes in the orientation of the components in the drawings.

In the embodiments of the present application, unless otherwise clearly specified and limited, the term "connection" should be understood in a broad sense. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be directly connected or indirectly connected through an intermediate medium. In addition, the term "coupled" can be a direct electrical connection or an indirect electrical connection through an intermediate medium. The term "contact" can be a direct contact or an indirect contact through an intermediate medium.

In the embodiments of the present application, "and/or" describes the association relationship of the associated objects, indicating that there may be three relationships. For example, A and/or B may represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B may be singular or plural. The character "/" generally indicates that the associated objects are in an "or" relationship.

The embodiment of the present application provides an electronic device, which may be, for example, a camera, an internet protocol camera (IPC), a mobile phone with a front and/or rear camera, a tablet with a front and/or rear camera, a digital camera, a digital video camera, a vehicle-mounted camera, or an industrial camera, etc., which has an image acquisition function. In addition, the electronic device may be applied to the fields of security, photography, automotive electronics, or industrial machine vision, etc.

As shown in Fig. 1, the electronic device may include an image sensor 10, a lens 20 and an image processor 30. The lens 20 is used to focus the light emitted or reflected by the photographed object onto the image sensor 10, and the image sensor 10 is used to convert the received optical image into a digital signal; the image processor 30 is used to process the digital signal and output the image of the photographed object.

The image sensor 10 is an important component of the electronic device and affects the performance of the electronic device.

At present, commonly used image sensors10 include dynamic vision sensor (DVS) and complementary metal oxide semiconductor image sensor (CMOS image sensor, CIS).

In some technologies, a DVS is provided as shown in Fig. 2. The DVS includes an optoelectronic conversion module, a signal conversion module, a signal processing module, a differential module, and a comparison module which are sequentially connected in series.

The photoelectric conversion module in the pixel is used to linearly convert the incident light intensity into photocurrent, and then the photocurrent is converted into a logarithmic exposure signal through the signal conversion module. The logarithmic exposure signal is translated or amplified by the signal processing module and sent to the differential module. The module calculates the difference between the exposure signal at the current moment and the exposure signal at the next moment. The difference is then sent to the comparator, compared with the set value, and an "event" is output (the event is defined as a brightness change, including "dark to bright", "bright to dark", and "brightness unchanged", which is a digital quantity). If the incident light intensity becomes brighter, the exposure signal value will rise. When it is higher than the set value, the event will be triggered, that is, "dark to bright". If the incident light intensity becomes darker, the exposure signal value will decrease. When it is lower than the set value, the event will be triggered, that is, "bright to dark". If the incident light intensity remains unchanged or changes very little, the exposure signal remains unchanged, and the event will not be triggered, that is, "brightness remains unchanged".

Although DVS can realize dynamic vision functions and output events, it cannot image frame by frame. In addition, DVS has the disadvantages of large pixel size and low spatial resolution.

Based on this, in some technologies, as shown in FIG3 , a CIS is also provided, which includes a photosensitive unit, a CDS, a programmable gain amplifier (PGA), an analog-to-digital converter (ADC) and a memory.

The photosensitive unit is used to convert the light signal into an electrical signal and then output it. The CDS is used to collect the reset voltage VRST and exposure signal VSIG output by the photosensitive unit, and then make a difference (VRST-VSIG) between the reset voltage VRST and the exposure signal VSIG to eliminate most of the noise and make the difference (=VRST-VSIG) proportional to the product of the incident light intensity and the exposure time.

The PGA linearly amplifies the aforementioned difference, that is, provides analog gain to meet the input swing requirements of the ADC.

ADC quantifies the voltage value and converts the analog signal into a digital signal, that is, the brightness value.

The memory stores the quantization result of the ADC. At this point, the work of the image sensor is complete.

CIS has the advantages of correlated double sampling, improved image quality, low read noise, low dark current, high photoelectric conversion efficiency, color display, and small footprint. However, it can only image frame by frame and cannot output dynamic visual events.

Based on this, in some technologies, motion detection is implemented in the back-end processing of CIS, namely the frame difference method. The brightness value of the same pixel in two adjacent output frames of CIS is subtracted, and the dynamic event is judged by the difference. As shown in Figure 4, compared with the first frame, the second frame has an additional "triangle", so the result of dynamic vision is that only the pixel points corresponding to the "triangle" are output. Compared with the second frame, the third frame has an additional "rectangle", and the result of dynamic vision shown is only the "rectangle". This solution is easy to implement, and only a differential operation can be added to the image signal processing unit.

Although the frame difference method can output images and dynamic vision in one frame at the same time. However, it requires that all rows of the current frame complete exposure and analog-to-digital conversion before the exposure of the next frame can begin. Specifically, after the analog-to-digital conversion of the last row of the current frame is completed, there is an idle time between the two frames, and the first exposure end time of the first row of the next frame must be later than the end of the idle time. This will result in the brightness change being judged only based on the results of the previous and next frames. The time interval is long, and it is impossible to judge the brightness change in a shorter time. In addition, this requires storing the brightness values of all rows of the previous frame. After the brightness values are output row by row in the next frame, the corresponding pixels are differentiated, which greatly increases the storage requirements.

Based on this, the embodiment of the present application provides an image sensor with low cost, low power consumption and low storage requirement, which can realize both static frame-by-frame imaging function and dynamic visual function in the same frame image.

As shown in FIG5 , an embodiment of the present application provides an exemplary image sensor 10 , which includes a photosensitive circuit 11 and a column line circuit 12 . The output end of the photosensitive circuit 11 is coupled to the input end of the column line circuit 12 , and the column line circuit 12 is used to receive and process the photoelectric conversion signal output by the photosensitive circuit 11 .

The photosensitive circuit 11 is configured to receive incident light in the same image frame, and to generate one exposure signal in a first working mode; and to generate multiple exposure signals in a second working mode.

The first working mode mentioned above refers to the working mode of static frame-by-frame imaging of the image sensor; the second working mode refers to the working mode of dynamic vision of the image sensor.

For example, as shown in FIG6 , the first exposure period and the second exposure period are two adjacent periods in the same image frame. The first exposure period may be in front and the second exposure period may be in the back. The first exposure period may be in the back and the second exposure period may be in the front. Of course, the first exposure period and the second exposure period may be set adjacent to each other, and other periods may be spaced between the first exposure period and the second exposure period.

The first exposure signal corresponds to the incident light intensity in the first exposure period and the first exposure duration corresponding to the first exposure period. The second exposure signal corresponds to the product of the incident light intensity in the second exposure period and the second exposure duration corresponding to the second exposure period.

The embodiment of the present application does not limit the relationship between the first duration of the first exposure period and the second duration of the second exposure period. The first duration and the second duration can be in any relationship.

For example, as shown in FIG. 7 , the column line circuit 12 includes a dual sampler 120 , a switch array circuit 121 , and a digital-to-analog converter 122 .

The dual sampler 120 is coupled to the output terminal of the photosensitive circuit 11 and is configured to receive and process the exposure signal, output the image signal of the same image frame in the first working mode, and output the dynamic visual signal of the same image frame in the second working mode;

The image signal and the dynamic visual signal are both output from the same output terminal of the column line circuit 12 .

The switch array circuit 121 is used to select signals of different branches to be transmitted to the analog-to-digital converter 122. In some embodiments, the internal structure of the switch array circuit 121 includes a first switch S6, a second switch S7, a third switch S8, a fourth switch S9, and a fifth switch S10. Among them, the second end of the first switch S6 is coupled to the first end of the third switch S8. The second end of the second switch S7 is coupled to the second end of the third switch S8. The first end of the fourth switch S9 is coupled to the first end of the third switch S8. The first end of the fifth switch S10 is coupled to the second end of the third switch S8. By controlling the on and off of the first switch S6, the second switch S7, the third switch S8, the fourth switch S9, and the fifth switch S10, the signals of different branches are transmitted to the analog-to-digital converter 122.

In some embodiments, the analog-to-digital converter 122 includes a sixth switch S11 , a digital-to-analog converter DAC, and a comparator. A first terminal of the sixth switch S11 is coupled to a second input terminal of the comparator, and a second terminal of the sixth switch S11 is coupled to the analog-to-digital converter 122 .

In some embodiments, the dual sampler 120 includes: a first capacitor C3, a second capacitor C4, a first operational amplifier A1, and a seventh switch S3. The first end of the first capacitor C3 is coupled to the output end of the photosensitive circuit 11, and the second end of the first capacitor C3 is coupled to the first end of the second capacitor C4 and the first input end of the first operational amplifier A1. The second end of the second capacitor C4 is coupled to the output end of the first operational amplifier A1. The seventh switch S3 is connected in parallel with the second capacitor C4. The second input end of the first operational amplifier A1 is coupled to a reference voltage. The output end of the first operational amplifier A1 is coupled to the first end of the link 1 switch S4 and the first end of the link 2 switch S5.

In addition, in some other embodiments, the column line circuit 12 further includes a link 1 switch S4, a link 2 switch S5, a first link capacitor C5, and a second link capacitor C6. The second end of the link 1 switch S4 is coupled to the first end of the first switch S6. The first end of the first link capacitor C5 is coupled to the first end of the first switch S6, and the second end of the first link capacitor C5 is coupled to the reference ground voltage terminal GND. The second end of the link 2 switch S5 is coupled to the first end of the second switch S7. The first end of the second link capacitor C6 is coupled to the first end of the second switch S7, and the second end of the second link capacitor C6 is coupled to the reference ground voltage terminal GND.

In some embodiments, the image sensor 10 further includes a memory for storing the digital signal output by the analog-to-digital converter 122 .

The working principle and circuit signal flow state of the image sensor 10 are introduced below. As shown in FIG8 , an exemplary structure of a photosensitive circuit 11 provided in an embodiment of the present application is shown. The photosensitive circuit 11 includes a first transistor M1, a second transistor M2, a third transistor M3, a fourth transistor M4 and a photodiode (PD).

One end of the photodiode PD is coupled to the reference ground voltage terminal GND, and the other end of the photodiode PD is coupled to the first input-output stage of the first transistor M1.

A control electrode of the first transistor M1 is coupled to the first control signal terminal TG, and a second input-output electrode of the first transistor M1 is coupled to the node FD.

The control electrode of the second transistor M2 is coupled to the reset control signal terminal RST, the first input-output electrode of the second transistor M2 is coupled to the power supply voltage terminal VDD, and the second input-output electrode of the second transistor M2 is coupled to the node FD.

The control electrode of the third transistor M3 is coupled to the node FD, the first input-output electrode of the third transistor M3 is coupled to the power supply voltage terminal VDD, and the second input-output electrode of the third transistor M3 is coupled to the first input-output electrode of the fourth transistor M4.

A control electrode of the fourth transistor M4 is coupled to the third control signal terminal PC, and a second input-output electrode of the fourth transistor M4 is coupled to the reference ground voltage terminal GND.

The control electrode of the fifth transistor M5 is coupled to the fourth control signal terminal S1, and the second input and output electrodes of the fifth transistor M5 are connected to the fourth control signal terminal S1. The fifth transistor M5 is coupled to the first end of the first capacitor C1 , and the first input-output electrode of the fifth transistor M5 is coupled to the second input-output electrode of the third transistor M3 .

The control electrode of the sixth transistor M6 is coupled to the fifth control signal terminal S2 , the second input-output electrode of the sixth transistor M6 is coupled to the first end of the second capacitor C2 , and the first input-output electrode of the sixth transistor M6 is coupled to the second input-output electrode of the third transistor M3 .

The second ends of the first capacitor C1 and the second capacitor C2 are both coupled to the reference ground voltage terminal GND.

The control electrode of the seventh transistor M7 is coupled to the second input-output electrode of the third transistor M3 , the first input-output electrode of the seventh transistor M7 is coupled to the power supply voltage terminal VDD , and the second input-output electrode of the seventh transistor M7 is coupled to the first input-output electrode of the eighth transistor M8 .

A control electrode of the eighth transistor M8 is coupled to the sixth control signal terminal RS, and a second input-output electrode of the eighth transistor M8 is coupled to the column line circuit 12 .

The photodiode PD operates under a reverse bias voltage and is used to convert the incident light irradiating itself into electrons (called "photogenerated electrons"). The first transistor M1 can be understood as a charge transfer gate, the second transistor M2 can be understood as a reset transistor, the third transistor M3 and the seventh transistor M7 can be understood as source followers, the fourth transistor M4 can be understood as a row selector, and the fifth transistor M5 and the sixth transistor M6 control the first capacitor C1 and the second capacitor C2 respectively. These two transistors can be understood as level temporary state controllers. The node FD can be understood as a floating diffusion potential (FD) node.

Under the control of the reset control signal rst received by the reset control signal terminal RST, the second transistor M2 is turned on, and the reset voltage V3 of the power supply voltage terminal VDD is transmitted to the node FD, the node FD is reset, the voltage of the node FD is changed to the reset voltage V3, and the reset voltage V3 generated by the pixel is stored in the first capacitor C1 through the third transistor M3 and the fifth transistor M5. When the pixel needs to be read, the eighth transistor M8 is turned on through the control signal rs of the sixth control signal terminal RS, and the reset voltage V3 stored in the first capacitor C1 is output to the column line circuit 12 through the seventh transistor M7 and the eighth transistor M8, and is processed by the column line circuit 12.

Under the control of the first control signal tg received by the first control signal terminal TG, the first transistor M1 is turned on, and the photogenerated electrons generated in the photodiode PD enter the node FD for storage, changing the voltage VFD of the node FD. The change in the voltage VFD of the node FD is proportional to the product of the incident light intensity and the exposure time, thereby converting the light signal into a voltage signal. The voltage VFD of the node FD is stored in the second capacitor C2 through the third transistor M3 and the sixth transistor M6. When the pixel needs to be read, the eighth transistor M8 is turned on through the control signal rs of the sixth control signal terminal RS, and the photoelectric conversion voltage of the node FD is output to the column line circuit 12 through the seventh transistor M7 and the eighth transistor M8, and is processed by the column line circuit 12. At this point, the pixel has completed the photoelectric conversion and the output of the electrical signal.

See the timing diagram of the photosensitive circuit 11 shown in Figures 9 and 10. When the image sensor is in the second working mode, that is, the working mode of dynamic vision, the exposure period of a single image frame in the photosensitive circuit 11 is divided into multiple sub-periods. In the embodiment of the present application, the total exposure period of a single image frame is the third exposure period t3, and the third exposure period t3 is divided into the first exposure period t1 and the second exposure period t2. It is equivalent to dividing the original full exposure time into two.

The photosensitive circuit 11 receives the incident light in the first exposure period t1 and generates a first exposure signal corresponding to the first exposure period t1, and receives the incident light in the second exposure period t2 and generates a second exposure signal corresponding to the second exposure period t2. The relative magnitude of the incident light intensity in the second exposure period t2 and the incident light intensity in the first exposure period t1 is determined, thereby realizing the dynamic visual function in the exposure period of an image frame.

In some embodiments, as shown in FIG. 9 , the first control signal tg and the reset control signal rst are both turn-on signals for the first time, and the node FD and the photodiode PD are reset.

Then, the first control signal tg and the reset control signal rst are both cut-off signals, the first transistor M1 and the second transistor M2 are both cut off, and the photodiode PD starts to expose. After the first exposure period t1 ends, the first control signal tg is the on signal for the second time, the first transistor M1 is turned on, and the photodiode PD imports the photogenerated electrons generated in the first exposure period t1 into the node FD through the first transistor M1 and stores them, causing the voltage VFD of the node FD to drop. At this time, the voltage VFD of the node FD = the first voltage VSIG1, and is stored in the first capacitor C1 through the fifth transistor.

Then, the reset control signal rst is turned on to clear the photogenerated electrons at the node FD. At this time, the voltage at the node FD is VFD = reset voltage V3. After clearing, the photodiode PD begins to accumulate the photogenerated electrons generated during the second exposure period t2.

When the second exposure period t2 ends, the first control signal tg is a conduction signal again, the first transistor M1 is turned on, and the photodiode PD conducts the photogenerated electrons generated in the second exposure period t2 to the node FD through the first transistor M1, and at the same time causes the voltage VFD of the node FD to further decrease. At this time, the voltage VFD of the node FD = the second voltage VSIG2, and is stored in the second capacitor C2 through the sixth transistor.

Finally, the reset control signal rst is a turn-on signal again, and the power supply voltage terminal VDD is transmitted to the node FD via the second transistor M2. At this time, the voltage VFD at the node FD is restored to the reset voltage V3 again.

That is, during the driving process of the photosensitive circuit 11 , the photosensitive circuit 11 sequentially outputs the first voltage VSIG1 (first exposure signal), the second voltage VSIG2 (second exposure signal) and the reset voltage V3 during the exposure period in an image frame.

The column line circuit 12 sequentially receives the first voltage VSIG1 (first exposure signal), the second voltage VSIG2 (second exposure signal) and the reset voltage V3 output by the photosensitive circuit 11 and performs corresponding processing.

The double sampler 120 in the column line circuit 12 processes the first voltage VSIG1 and the second voltage VSIG2, and outputs a first node voltage Vout1 = V _REF + C ₃ /C ₄ *(V _SIG1 - _{V SIG2} ), where V _REF is a reference voltage, usually half of the power supply voltage Vdd, and turns on the link 1 switch S4. The first node voltage Vout1 is temporarily stored in the first link capacitor C5.

Then, the first switch S6 and the fourth switch S9 in the switch array circuit 121 are turned on to transmit the first node voltage Vout1 temporarily stored in the first link capacitor C5 to the first input terminal (ie, V _int3 ) of the comparator.

At the same time, the second switch S7, the third switch S8, and the fifth switch S10 are turned off, and the sixth switch S11 is turned on. The digital-to-analog converter DAC outputs a reference voltage V _REF which is sent to the second input terminal (ie, V _int4 ) of the comparator.

The signals at the two input terminals of the comparator are compared to obtain the brightness changes of exposure 1 and exposure 2. If the first node voltage Vout1>reference voltage V _REF, then (V _SIG1 -V _SIG2 )>0, that is, the first voltage V _SIG1 is greater than the second voltage V _SIG2 . This indicates that the incident light intensity of the first exposure period t1 is less than the incident light intensity of the second exposure period t2. Because the higher the voltage value, the smaller the difference with the reset voltage Vrst, that is, the smaller the voltage change, the smaller the intensity of the incident light. Conversely, the same applies.

In some other embodiments, as shown in FIG. 10 , the first control signal tg and the reset control signal rst are both turn-on signals for the first time to reset the node FD and the photodiode PD.

Different from the embodiment shown in FIG9 , when the first exposure period t1 ends, the embodiment shown in FIG9 turns on the reset control signal rst to clear the photogenerated electrons at the node FD, so that the voltage VFD of the node FD returns to the reset voltage V3, and the photodiode PD starts to accumulate the photogenerated electrons generated in the second exposure period t2. In the embodiment shown in FIG10 , when the first exposure period t1 ends, the voltage VFD at the node FD drops from the reset voltage V3 to the first voltage V _SIG1 . At this time, the reset control signal rst is not turned on, and the photodiode PD directly starts to accumulate the photogenerated electrons in the second exposure period t2. When the second exposure period t2 ends, the voltage VFD at the node FD further drops from the first voltage V _SIG1 to the second voltage V _SIG2 .

The first voltage V _SIG1 and the second voltage V _SIG2 are stored in the first capacitor C1 and the second capacitor C2 respectively, and then the double sampler 120 in the column line circuit 12 performs signal processing on the first voltage V _SIG1 and the second voltage V _SIG2 respectively.

In the embodiment of the present application, as shown in FIG. 7 , the second node voltage Vint1 = (C3/C4)*(Vrst-V _SIG1 )+Vref; the third node voltage Vint2 = (C3/C4)*(V _SIG1 -V _SIG2 )+Vref. Wherein, Vrst is a reset voltage, and Vref is a reference voltage. (Vrst-V _SIG1 ) is used to characterize the incident light intensity in the first exposure period t1, and (V _SIG1 -V _SIG2 ) is used to characterize the incident light intensity in the second exposure period t2.

Under the control of the switch array circuit 121, the third switch S8 and the sixth switch S11 are turned off, the first switch S6 and the fourth switch S9 are turned on, and the second node voltage Vint1 is sent to the first input terminal Vint3 of the comparator; the second switch S2 and the fifth switch S10 are turned on, and the third node voltage Vint2 is sent to the second input terminal Vint4 of the comparator, and the brightness change between the first exposure period and the second exposure period is obtained through the comparator.

When the second node voltage Vint1 is greater than the third node voltage Vint2, the incident light intensity in the first exposure period is greater, changing from bright to dark. Otherwise, it changes from dark to bright.

In summary, the image sensor 10 provided in the embodiment of the present application divides the complete exposure period of the photosensitive unit 11 in an image frame into multiple sub-exposure periods (not limited to two exposure periods) by controlling the on-off of the transistor, and then The photogenerated electrons generated in multiple exposure periods are converted into electrical signals and output to the column line circuit 12 for processing in sequence, so as to obtain the brightness information corresponding to the different exposure periods and realize the judgment of the brightness change.

The comparator can be used to directly output the dynamic changes (events) of the multiple exposure periods. The judgment of dynamic events does not require quantization of the output image, thereby reducing storage space and power consumption. Moreover, the output dynamic visual signal is the dynamic change between multiple sub-periods within the exposure period of the same image frame, which is equivalent to interpolation within the frame, rather than the dynamic change of the previous/next two frames. The judgment of dynamic vision is not limited by the frame rate.

Furthermore, the image sensor 10 in the present application is an improvement made based on the hardware architecture of the traditional CIS. It meets the function of frame-by-frame imaging, supports dynamic vision functions, realizes a "dual-mode vision sensor", and can reuse mature CIS process and image algorithms. It has low technical difficulty and a short product development cycle. At the same time, there is no need for an ISP processing circuit, thereby greatly reducing the number of transistors in the circuit and reducing the size of the chip. A variety of image processing functions can be realized within a limited chip area. Under the control of the switch array circuit, low power consumption can be achieved.

Compared with the rolling shutter exposure method, this solution can eliminate the "jelly effect" because all pixels are exposed at the same time. Therefore, it is more suitable for neural network training and reasoning, reducing learning complexity.

The above is merely an illustration of an example in which the image sensor 10 provided in an embodiment of the present application includes one photosensitive unit 11. In some embodiments, the image sensor 10 includes a plurality of photosensitive units 11 arranged in an array, and the photosensitive units 11 in the same column are coupled to the same column line circuit 12.

The embodiment of the present application also provides an image sensing method, the image sensing method comprising:

According to the selected working mode, a corresponding exposure signal is generated.

For example, the process can be completed by the photosensitive unit 11 in the image sensor 10. By controlling the driving timing of the photosensitive unit 11, the photosensitive unit 11 can output signals of different exposure periods in an image frame. Please refer to the above description of the photosensitive unit 11.

In the first working mode, an image signal of an image frame is generated according to the exposure signal. In the second working mode, a dynamic visual signal of the image frame is generated according to the exposure signal.

For example, the process may be completed by the dual sampler 120 and the switch array circuit 121 in the image sensor 10 , and reference may be made to the above descriptions of the two.

Converts image signals into digital signals and outputs them.

For example, the process may be completed by the analog-to-digital converter 122 in the image sensor 10 , and reference may be made to the above description of the analog-to-digital converter 122 .

In some embodiments, the third exposure period is equal to the first exposure period plus the second exposure period, and the first exposure period and the second exposure period are two adjacent periods in the image frame. The first exposure period may be in front and the second exposure period may be in the back. Alternatively, the first exposure period may be in the back and the second exposure period may be in the front.

In some embodiments, the image sensing method further includes: generating a reset voltage.

The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

An image sensor, characterized in that the image sensor comprises:

The photosensitive circuit is configured to receive incident light in the same image frame, generate one exposure signal in a first working mode, and generate a plurality of said exposure signals in a second working mode;

Column line circuits, including dual samplers, switch array circuits, and digital-to-analog converters;

The dual sampler is coupled to the output terminal of the photosensitive circuit and is configured to receive and process the exposure signal, output the image signal of the same image frame in the first working mode, and output the dynamic visual signal of the same image frame in the second working mode;

The switch array circuit is coupled to the output end of the double sampler and is used to select signals of different branches to be transmitted to the analog-to-digital converter;

The analog-to-digital converter is configured to convert the received signal from the switch array circuit into a digital signal and then output it.
The image sensor according to claim 1, wherein the switch array circuit comprises a first switch, a second switch, a third switch, a fourth switch, and a fifth switch;

The second end of the first switch is coupled to the first end of the third switch;

The second end of the second switch is coupled to the second end of the third switch;

The first end of the fourth switch is coupled to the first end of the third switch;

The first terminal of the fifth switch is coupled to the second terminal of the third switch.
The image sensor according to claim 1 or 2, characterized in that the analog-to-digital converter comprises a sixth switch, a digital-to-analog converter and a comparator;

A first terminal of the sixth switch is coupled to the second input terminal of the comparator, and a second terminal of the sixth switch is coupled to the digital-to-analog converter.
The image sensor according to any one of claims 1 to 3, characterized in that the dual sampler comprises: a first capacitor, a second capacitor, a first operational amplifier, and a seventh switch;

A first end of the first capacitor is coupled to the output end of the photosensitive circuit, and a second end of the first capacitor is coupled to a first end of the second capacitor and a first input end of the first operational amplifier;

The second terminal of the second capacitor is coupled to the output terminal of the first operational amplifier;

The seventh switch is connected in parallel with the second capacitor;

The second input terminal of the first operational amplifier is coupled to a reference voltage;

The output terminal of the first operational amplifier is coupled to the output terminal of the double sampler.
The image sensor according to any one of claims 1 to 4 is characterized in that, in the second operating mode, the photosensitive circuit divides the exposure period in the same image frame into two different periods, namely a first exposure period and a second exposure period.
The image sensor according to claim 5, characterized in that the first exposure period and the second exposure period are two adjacent periods in the same image frame.
The image sensor according to any one of claims 1 to 5, characterized in that the dynamic visual signal represents a difference between the intensity of the incident light in the second exposure period and the intensity of the incident light in the first exposure period.
The image sensor according to claim 5 or 6, characterized in that the sum of the first exposure period and the second exposure period is a third exposure period;

The image signal represents the cumulative intensity of the incident light during the third exposure period.
The image sensor according to any one of claims 1 to 8, characterized in that

The light-sensitive circuit is further configured to generate a reset voltage;

The dual sampler is further configured to receive the reset voltage.
An image processing method, characterized in that the image processing method comprises:

Generate a corresponding exposure signal according to the selected working mode;

In the first working mode, generating an image signal of an image frame according to the exposure signal;

In the second working mode, generating a dynamic visual signal of the image frame according to the exposure signal;

The signal generated in the corresponding working mode is converted into a digital signal and outputted.
The image processing method according to claim 10, characterized in that generating the exposure signal comprises:

In the first working mode, receiving incident light and generating a third exposure signal;

In the second working mode, incident light in a first exposure period is received and a first exposure signal is generated, and incident light in a second exposure period is received and a second exposure signal is generated;

The first exposure period and the second exposure period are two adjacent periods in the same image frame.
The image processing method according to claim 10 or 11 is characterized in that the image processing method further comprises: generating a reset voltage.
The image processing method according to claim 11, characterized in that the first exposure period and the second exposure period are two adjacent periods in the same image frame;

The dynamic visual signal represents a difference between an intensity of incident light during the second exposure period and an intensity of incident light during the first exposure period.
The image processing method according to claim 13, characterized in that the sum of the first exposure period and the second exposure period is a third exposure period;

The image signal represents the cumulative intensity of the incident light during the third exposure period.
An electronic device, characterized in that it comprises an image sensor and a printed circuit board, wherein the image sensor is arranged on the printed circuit board, the image sensor comprises the image sensor according to any one of claims 1 to 9, and a pin of the image sensor is used to output signals of different functions.