CN115118895A

CN115118895A - Vision system

Info

Publication number: CN115118895A
Application number: CN202210907267.3A
Authority: CN
Inventors: 蒋寓文; 傅正明
Original assignee: Shanghai Qigan Electronic Information Technology Co ltd
Current assignee: Shanghai Qigan Electronic Information Technology Co ltd
Priority date: 2022-07-29
Filing date: 2022-07-29
Publication date: 2022-09-27

Abstract

The invention discloses a vision system, which comprises an image sensor, wherein the image sensor is used for acquiring an original image, when initial voltage signals of the same pixel point in an nth frame original image and an n-1 th frame original image have changes, analog-to-digital conversion is carried out on analog signals of the pixel point to obtain a digital signal of the pixel point, and a trigger signal is sent to an event counting module. The event counting module is used for determining the number of target pixel points with changed initial voltage signals in the original image by counting the number of trigger signals from the image sensor within the exposure time of the original image. The image processing module is used for outputting the digital signals of all the pixel points of the original image in a synchronous transmission mode when the ratio of the number of the target pixel points to the total number of the pixel points of the original image is greater than a set threshold value; otherwise, outputting the digital signal of the target pixel point in the original image. The vision system provided by the invention is used for reducing the data transmission bandwidth on the premise of ensuring the image resolution and the image quality.

Description

Vision system

Technical Field

The invention relates to the technical field of semiconductor image sensing, in particular to a vision system.

Background

In recent years, the field of computer vision has been developed at a high speed, and in the field, a great deal of research is carried out on intelligently detecting, identifying and tracking targets by using cameras and computers instead of human eyes, and the field of computer vision has been widely applied to scenes such as security monitoring, automatic driving, medical care and the like. A typical industrial machine vision system includes an optical system (light source, lens, etc.), an image sensor, an image processing unit, a communication unit, etc. Wherein the image sensor is used for acquiring visual data. A Complementary Metal Oxide Semiconductor Image Sensor (CIS) is a key component of a digital camera as a typical solid-state imaging Sensor, and is widely applied to the fields of industrial cameras, consumer electronics, digital products, security monitoring and the like. A conventional CIS outputs each frame of image at a fixed frame rate (about 25 frames/sec), and each pixel on the image reflects a luminance value of the position, and there is a large amount of background information in such a picture. Considering that the picture regions belonging to the background information are always consistent in consecutive frames, and this part of data belongs to redundant data for the subsequent image processing algorithm, transmitting such redundant data to the data encoding module wastes both computing power and transmission bandwidth and power.

Although the prior art provides an Event-based visual Sensor (Event-based Camera Sensor), also called an Event Camera, typically a Dynamic Event Sensor (DVS), each pixel independently generates an asynchronous Event under the stimulation of illumination change, and each time the illumination change on a pixel is greater than a set threshold, a corresponding Event pulse is sent out, although the Event stream generated in this way eliminates the static redundant data of the conventional image Sensor, extracts the motion information in the scene, and reduces the power consumption and the bandwidth occupation, the following disadvantages still exist: many tasks in computer vision rely on comprehensive information in images, including absolute brightness, color, contrast, saturation, texture, etc., and the output of an event camera loses the information and only retains coordinates and time of pixels with changed brightness; (2) because the number of events in the event stream is not fixed and the total length of data is not fixed, the events cannot be directly input into a Convolutional Neural Network (CNN) for processing, a luminance image reconstructed through the event stream can be input into the CNN, but the reconstructed image increases extra computational power and cannot ensure accuracy; (3) although static redundant information on a two-dimensional picture is removed from the event stream, the event stream has great redundancy in the time dimension, most monitoring scenes do not need to pay attention to microsecond-level high-speed motion, and the event stream does not have good grasp on the motion characteristics of a low-texture area.

Based on the above background, how to control and optimize the system power consumption without reducing the image resolution and the image quality becomes a technical problem currently facing.

Disclosure of Invention

The embodiment of the invention provides a vision system, which is used for reducing data transmission bandwidth and saving system power consumption on the premise of ensuring image resolution and image quality.

In a first aspect, the present invention provides a vision system comprising an image sensor, an event counting module, and an image processing module: the image sensor is used for acquiring an original image, when the initial voltage signals of the same pixel point in the nth frame original image and the nth-1 frame original image are changed, the analog-to-digital conversion unit is triggered to perform analog-to-digital conversion on the analog signals of the pixel point to obtain digital signals of the pixel point, and the trigger signals are sent to the event counting module. The event counting module is used for determining the number of target pixel points with changed initial voltage signals in the nth frame original image by counting the number of trigger signals from the image sensor within the exposure time of the nth frame original image. The image processing module is used for outputting digital signals of all pixel points of the nth frame original image in a synchronous transmission mode when the ratio of the number of target pixel points to the total number of the pixel points of the nth frame original image is greater than a set threshold; and when the ratio of the number of the target pixel points to the total number of the pixel points of the nth frame original image is less than or equal to a set threshold, outputting the digital signals of the target pixel points in the nth frame original image in an asynchronous transmission mode.

The visual system provided by the invention has the beneficial effects that: the vision system supports two modes of outputting data of the CMOS image sensor, and the two modes are synchronous and asynchronous. When the initial voltage signal has more changed target pixel points, the digital signal of the whole frame image is output, which is beneficial to furthest retaining the comprehensive information in the image, including absolute brightness, color, contrast, saturation, texture and the like. When the target pixel point with the changed initial voltage signal is smaller, the pixel value of the changed pixel point is transmitted, and the data of the pixel point has strong space sparsity, so that the data redundancy can be effectively reduced.

Optionally, the image processing module outputs the digital signal of each pixel point of the nth frame original image in a synchronous transmission manner, including: aiming at a target pixel point with a changed initial voltage signal, the image processing module outputs a digital signal of the target pixel point of the nth frame original image and an identifier that the initial voltage signal of the target pixel point has a change in a synchronous transmission mode; and aiming at the non-target pixel points of which the initial voltage signals are not changed, the image processing module outputs the digital signals of the non-target pixel points of the n-1 th frame of original image in a synchronous transmission mode. Therefore, the embodiment can realize that the data generated by the image sensor can be transmitted outwards in a synchronous mode when the number of the pixels which are obviously changed in the front frame image and the back frame image is large, and the comprehensive information of the original images such as the texture, the brightness and the like of the images can be kept to a great extent.

Optionally, the outputting, by the image processing module, the digital signal of the target pixel point in the nth frame original image in an asynchronous transmission manner includes: and aiming at the target pixel point with the changed initial voltage signal, the image processing module outputs the digital signal of the target pixel point of the nth frame original image and the row and column address corresponding to the target pixel point in an asynchronous transmission mode. Therefore, the embodiment can realize that when the pixel points which are obviously changed in the front and back frame images are small, the data generated by the image sensor can be transmitted outwards in an asynchronous mode, and the transmission bandwidth is saved to a great extent.

Optionally, the set threshold satisfies the following formula:

wherein th denotes a set threshold value, len _L The bit width occupied by the pixel value is represented, H represents the number of pixels of each line of the original image, W represents the number of pixels of each column of the original image, and the length of information carried by each pixel point is len _L +1, 1 denotes Out _c The bit-width of (c) is,

indicating rounding up, the larger the image resolution, the smaller the th-setting threshold.

Optionally, when a ratio of the number of target pixels to the total number of pixels of the nth frame original image is less than or equal to a set threshold, outputting a digital signal of the target pixels in the nth frame original image in an asynchronous transmission manner, including:

the image processing module is used for extracting an interesting region from the nth frame original image according to the region boundary of a moving object corresponding to the event set and outputting digital signals of all pixel points of the interesting region in the nth frame original image in an asynchronous transmission mode; the interested region comprises digital signals of target pixel points in the nth frame original image. According to the method, the original image can be preprocessed based on event activities of the edge of the object, the Region of interest (ROI) is extracted and sent to the receiving party from the sensor, and the Region outside the ROI is not transmitted.

Optionally, the image sensor includes a pixel array constituted by an analog-to-digital conversion unit and a pixel unit circuit;

the pixel unit circuit comprises a photoelectric signal conversion unit, a voltage amplification unit, a first output unit, a second output unit and a comparison unit.

The photoelectric signal conversion unit is used for converting exposure information of pixel points in an image frame into initial voltage signals; the voltage amplifying unit is used for amplifying and outputting the initial voltage signal; the first output unit is connected to the voltage amplifying unit and used for receiving and storing a first voltage signal of a pixel point of an image frame in the (n-1) th frame; the second output unit is connected to the voltage amplifying unit and used for receiving and outputting a second voltage signal of the pixel point of the image frame at the nth frame; the comparison unit is respectively connected with the output end of the first output unit and the output end of the second output unit, and is used for receiving the first voltage signal from the first output unit and the second voltage signal from the second output unit, and comparing the first voltage signal with the second voltage signal to determine whether a change exists between the initial voltage signal of the pixel point in the image frame at the n-1 frame and the initial voltage signal of the pixel point in the image frame at the n-1 frame.

The pixel unit circuit provided by the invention has the beneficial effects that: the pixel unit circuit comprises two output units, wherein the second output unit stores the pixel signal of the nth frame, the first output unit stores the pixel signal of the (n-1) th frame, and the voltage signals of the two output units are simultaneously output to the comparison unit. The pixel structure is suitable for processing sampling information in parallel by lines, including subsequent voltage signal comparison and analog-to-digital conversion processes, and the operation speed can be greatly increased and the system operation bandwidth can be increased due to parallel operation of pixels in the same line; in addition, the comparison unit can determine whether the initial voltage signals of the pixel points have changes, which is helpful for judging whether to trigger analog-to-digital conversion.

Optionally, the photoelectric signal conversion unit comprises a photodiode; the voltage amplification unit comprises a first MOS tube, a second MOS tube and a third MOS tube; the source electrode or the drain electrode of the first MOS tube and the grid electrode of the second MOS tube are connected to the cathode of the photodiode, the source electrode or the drain electrode of the second MOS tube is connected to the source electrode or the drain electrode of the third MOS tube, and the drain electrode or the source electrode of the second MOS tube and the drain electrode or the source electrode of the first MOS tube are connected to a power supply voltage.

Optionally, the comparison unit comprises a comparator; the first output unit comprises a first switching transistor, a capacitor, a second switching transistor, a fourth MOS transistor and a fifth MOS transistor;

the capacitor is used for storing charges, one end of the capacitor is connected with the first switch transistor and the second switch transistor, and the other end of the capacitor is grounded; the grid electrode of the fifth MOS tube is connected with the second switching transistor, the source electrode or the drain electrode of the fifth MOS tube is connected with the source electrode or the drain electrode of the fourth MOS tube, and the drain electrode or the source electrode of the fifth MOS tube is connected with the first input end of the comparator; the drain electrode or the source electrode of the fourth MOS tube is grounded;

the first switching transistor and the second switching transistor are used for controlling the charging and discharging states of the capacitor; when the first switching transistor is closed and the second switching transistor is opened, the initial voltage signals of the pixel points of the image frame are stored to the capacitor; when the first switching transistor is switched off and the second switching transistor is switched on, the charges in the capacitor are transmitted to the fifth MOS transistor through the second switching transistor.

Optionally, the second output unit includes a sixth MOS transistor and a seventh MOS transistor; the grid electrode of the seventh MOS tube is connected to the source electrode or the drain electrode of the second MOS tube, the source electrode or the drain electrode of the seventh MOS tube is connected to the source electrode or the drain electrode of the sixth MOS tube, the drain electrode or the source electrode of the seventh MOS tube is connected to the second input end of the comparator, and the drain electrode or the source electrode of the sixth MOS tube is grounded.

Optionally, the image sensor further includes a latch, where the latch is configured to store an analog-to-digital conversion result and a comparison result of each pixel point, where the conversion result includes a digital signal of each pixel point of the n-1 th frame original image and a digital signal of a target pixel point of the nth frame target original image; the comparison result comprises a result identification that the initial voltage signal of the pixel point changes and a result identification that the initial voltage signal of the pixel point does not change.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

FIG. 1 is a schematic diagram of a vision system according to an embodiment of the present invention;

fig. 2A and fig. 2B are schematic diagrams of two image capturing scenes according to an embodiment of the present invention;

FIG. 3 is a schematic diagram of another image extraction method according to an embodiment of the present invention;

fig. 4 is a schematic circuit diagram of a pixel unit according to an embodiment of the present invention;

FIG. 5 is a schematic diagram of another specific pixel unit circuit structure according to an embodiment of the present invention;

FIG. 6 is a schematic flow chart of a timing control method for a pixel unit circuit according to an embodiment of the present invention;

fig. 7 is a schematic structural diagram of an image sensor according to an embodiment of the present invention;

FIG. 8 is a timing waveform diagram illustrating the circuit operation of a unit pixel according to an embodiment of the present invention.

Detailed Description

In order to make the contents of the present invention more clearly understood, the contents of the present invention are further explained below with reference to the attached drawings. The invention is of course not limited to this particular embodiment, and general alternatives known to those skilled in the art are also covered by the scope of the invention.

In the following detailed description of the embodiments of the present invention, in order to clearly illustrate the structure of the present invention and to facilitate explanation, the structure shown in the drawings is not drawn to a general scale and is partially enlarged, deformed and simplified, so that the present invention should not be construed as limited thereto.

An embodiment of the present invention provides a vision system, including: the image sensor 01, the event counting module 02 and the image processing module 03 may optionally further include a data encoding module 04. Wherein:

the image sensor 01 is used for acquiring an original image, when the initial voltage signals of the same pixel point in the nth frame original image and the nth-1 frame original image are changed, the analog-to-digital conversion unit is triggered to perform analog-to-digital conversion on the analog signals of the pixel point to obtain digital signals of the pixel point, and a trigger signal is sent to the event counting module, wherein n is a positive integer.

The event counting module 02 is configured to determine, within the exposure time of the nth frame original image, the number of target pixel points in which the initial voltage signal in the nth frame original image changes by counting the number of trigger signals from the image sensor. For example, the event counting module 02 may be an RS trigger or a counter, etc. When the event counting module 02 is an RS trigger, counting may be performed by receiving an event pulse from the image sensor.

The image processing module 03 is configured to output, to the data encoding module 04, digital signals of each pixel point of the nth frame original image in a synchronous transmission manner when a ratio of the number of target pixel points to the total number of pixel points of the nth frame original image is greater than a set threshold; and when the ratio of the number of target pixel points to the total number of pixel points of the nth frame original image is less than or equal to a set threshold, outputting 04 digital signals of the target pixel points in the nth frame original image to a data coding module in an asynchronous transmission mode.

It is to be noted that the set threshold in the above embodiment may satisfy the following formula:

wherein th represents a set threshold value, len _L The bit width occupied by the pixel value is represented, H represents the number of pixels of each line of the original image, W represents the number of pixels of each column of the original image, and the length of information carried by each pixel point is len _L +1, 1 denotes the bit width of the comparator comp,

For example, when the image resolution is recorded as H × W, H represents the number of pixels in each row of the original image, W represents the number of pixels in each column of the original image, the number of target pixels is recorded as event _ count, the total number of pixels in the original image of the nth frame is H × W, and th represents a set threshold, when the image resolution is recorded as H × W, H represents the number of pixels in each column of the original image, W represents a set threshold, and when the image resolution is recorded as event _ count, H represents the total number of pixels in the original image of the nth frame, th represents a set threshold, and when the total number of pixels in the original image of the nth frame is recorded as H × W

Then, the digital signals of the pixel points of the nth frame original image can be output to the data encoding module 04 in a synchronous transmission mode; when in use

And then, outputting the digital signal of the target pixel point in the original image of the nth frame 04 to the data encoding module in an asynchronous transmission mode.

In the synchronous mode, since a complete pixel matrix is transmitted, the address does not need to be encoded, the address of the changed pixel can be analyzed in a subsequent algorithm outside the sensor chip, and thus the length of the information carried by each pixel is len _L +1, where 1 is Out _c Is fixed (len), the total amount of data to be transmitted is fixed (len) _L +1)×H×W。

In asynchronous mode, the changing pixel address needs to be additionally encoded, so the length of each event is

Total amount of data is

When the event _ count increases to a certain amount, the total amount of data transmitted asynchronously exceeds the amount of data transmitted synchronously, at which time

The value of (d) is the value of th. The larger the output resolution of the image sensor, the smaller the th critical value for switching asynchronous transmission to synchronous transmission. When the image sensor is an event camera, the commonly used 640 × 480 (e.g., SONY IMX637) resolution, based on which th is calculated to be about 33%, the set threshold will be further reduced as the image resolution increases.

Alternatively, the image processing module 03 may be an image processor, and may also be a circuit for performing image processing.

In a possible implementation manner, the image processing module 03 outputs the digital signals of the pixels of the nth frame original image in a synchronous transmission manner, including: aiming at a target pixel point with a changed initial voltage signal, the image processing module outputs a digital signal of the target pixel point of the nth frame original image and an identifier that the initial voltage signal of the target pixel point has a change in a synchronous transmission mode; and aiming at the non-target pixel points of which the initial voltage signals are not changed, the image processing module outputs the digital signals of the non-target pixel points of the n-1 th frame of original image in a synchronous transmission mode. For example, the indication that the initial voltage signal of the pixel has a change may be 0 or 1, and if the comparison result is 0, it indicates that the pixel value has not changed significantly from the previous frame, and if the result is 1, it indicates that the pixel value has changed. The information output by the image processing module 03 in the synchronous transmission mode may be p ═ L, Out _c ) Denotes that p denotes a pixel, L is its value, Out _c Is the result of the state comparison. Illustratively, as shown in fig. 2A, (a) in fig. 2A has more target pixels where there is a change in the initial voltage signal than (b) in fig. 2A, and therefore, the digital signal of each pixel in (b) in fig. 2A and the indication of the change in the initial voltage signal of each pixel are output in a synchronous transmission manner. In this embodiment, since the analog-to-digital conversion unit does not perform analog-to-digital conversion on the analog signal of the pixel point where the initial voltage signal in the nth frame changes, the digital signal of the non-target pixel point of the n-1 st frame original image is used as the non-target of the nth frame original imageAnd marking the digital signals of the pixels. Therefore, when the number of pixels which are obviously changed in the front frame image and the rear frame image is large, the data generated by the image sensor can be transmitted outwards in a synchronous mode, and the comprehensive information of the original image such as image texture, brightness and the like can be kept to a large extent.

In another possible implementation, the outputting, by the image processing module 03, the digital signal of the target pixel point in the nth frame original image in an asynchronous transmission manner includes: for a target pixel point with a changed initial voltage signal, the image processing module 03 outputs a digital signal of the target pixel point of the nth frame original image and a row-column address corresponding to the target pixel point in an asynchronous transmission mode. For example, the asynchronous transmission means that only pixel information of pixel points with obviously changed brightness is output, and the image processing module 03 can transmit the pixel information in the form of an event stream. Each event is composed of the analog-to-digital conversion value of the pixel point and the corresponding row and column addresses, for example, the output result is represented by e ═ L, x, y, e represents an event, L is the pixel value of the event generation position, x is the column address of the event generation, and y is the row address. Referring to fig. 2B, (a) in fig. 2B has fewer target pixels with variations in the initial voltage signal than (B) in fig. 2B, and therefore, the digital signals of the respective target pixels in (B) in fig. 2B and the row and column addresses corresponding to the target pixels are output in an asynchronous transmission manner.

Optionally, in an asynchronous transmission mode, the image processing module 03 may extract, from the n-th frame original image, an event set generated by all target pixel points in a time period corresponding to the acquisition time of the n-th frame original image to the acquisition time of the n-1 th frame, and according to a region boundary of a moving object corresponding to the event set, an interested region from the n-th frame original image, and output a digital signal of each pixel point of the interested region in the n-th frame original image in an asynchronous transmission manner; the interested region comprises digital signals of target pixel points in the nth frame original image. That is to say, in the asynchronous transmission mode, in this embodiment, the image processing module may extract the region of interest of the original frame based on the edge event of the moving object, and the user may select to only connect to the original frameThe frame cut by the Region of Interest (ROI) is received, so that the data transmission quantity can be reduced, the effective information content in the frame can be improved, and the operation of a subsequent visual algorithm is facilitated. Specifically, the ROI positioning method is as follows: dividing the event sequence into a series of slices in a time dimension at small intervals, wherein each slice comprises an event set formed by aggregating asynchronous events generated by a moving object sensed by a sensor in the time, and the event set is marked as D ═ e { _t ＝(L _t ,x _t ,y _t )|t _start ≤t≤t _end In which e _t Representing an event generated at a time t, t _start And t _end Are the starting and ending points of the time slice.

Firstly, abnormal point detection is carried out on the event set, outlier events and noise in the event set are removed, and then the boundary of a region containing a moving object is searched in the event set:

thereby obtaining a circumscribed rectangle of the motion region, (x) ₁ ,y ₁ ) And (x) ₂ ,y ₂ ) The coordinates of the top left corner vertex and the bottom right corner vertex of the rectangle, and the area in the rectangle is the extracted region of interest. Illustratively, as shown in fig. 3, (a) column in fig. 3 shows 5 consecutive original images in a certain segment of surveillance video, and (b) column in fig. 3 shows the time interval between the current frame and the previous frameThe event set generated by all the changed pixels and the circumscribed rectangle corresponding to the event set, column (c) in fig. 3 shows the region of interest marked out after the rectangular frame is transferred to the original image, and it can be seen that most of the static background (i.e. the part outside the frame) of the original image is excluded.

Therefore, the vision system provided by the embodiment can support two modes of outputting the data of the CMOS image sensor, which are a synchronous mode and an asynchronous mode. When the initial voltage signal has more changed target pixel points, the digital signal of the whole frame image is output, which is beneficial to furthest retaining the comprehensive information in the image, including absolute brightness, color, contrast, saturation, texture and the like. When the target pixel point with the changed initial voltage signal is smaller, the pixel value of the changed pixel point is transmitted, and the data of the pixel point has strong space sparsity, so that the data redundancy can be effectively reduced.

In the above vision system, the image sensor includes a pixel array constituted by the analog-to-digital conversion unit 03 and the pixel unit circuit. Referring to fig. 4, the pixel unit circuit includes a photoelectric signal conversion unit 401, a voltage amplification unit 402, a first output unit 403, a second output unit 404, and a comparison unit 405. Wherein,

the photoelectric signal conversion unit 401 is configured to convert exposure information of a pixel in an image frame into an initial voltage signal.

And a voltage amplifying unit 402 for amplifying and outputting the initial voltage signal.

The first output unit 403 and the second output unit 404 are both connected to the voltage amplifying unit 402, and the first output unit 403 is used for receiving and storing a first voltage signal of a pixel point of an image frame at the time of the (n-1) th frame; the second output unit 404 is configured to receive and output a second voltage signal of a pixel point of the image frame at the nth frame.

A comparing unit 405, where the comparing unit 405 is respectively connected to an output end of the first output unit 403 and an output end of the second output unit 404, and is configured to receive the first voltage signal from the first output unit 403 and the second voltage signal from the second output unit 404, and compare the first voltage signal and the second voltage signal to determine whether there is a change between the initial voltage signal of the pixel in the image frame at the n-1 frame and the initial voltage signal of the pixel in the image frame at the n-1 frame.

The analog-to-digital conversion unit 406 is connected to an output terminal of the second output unit 404. The comparing unit 405 is further configured to: when the absolute value of the difference between the first voltage signal and the second voltage signal is not within the set range, it is determined that there is a change between the initial voltage signal of the pixel in the image frame at the n-1 frame and the initial voltage signal of the pixel in the image frame at the n frame, and a trigger signal is output, where the trigger signal is used to activate the analog-to-digital conversion unit 406 to perform analog-to-digital conversion.

It can be seen that the pixel unit circuit includes two output units, wherein the second output unit 404 stores and outputs the pixel signal of the nth frame, the first output unit 403 stores the pixel signal of the (n-1) th frame, and the voltage signals of the two output units are simultaneously output to the comparison unit 405. The pixel structure is suitable for processing sampling information in parallel by lines, including subsequent voltage signal comparison and analog-to-digital conversion processes, and the operation speed can be greatly increased and the system operation bandwidth can be increased due to parallel operation of pixels in the same line; in addition, the comparing unit 405 may determine whether the initial voltage signal of the pixel point changes, which is helpful for determining whether to trigger analog-to-digital conversion, when a pixel signal changes significantly, the comparing unit 405 triggers the analog-to-digital conversion unit 406 to perform analog-to-digital conversion, and when the pixel signal does not change significantly, the comparing unit 405 does not trigger the analog-to-digital conversion unit 406 to perform analog-to-digital conversion, so as to save power consumption on the premise of ensuring the CIS performance.

In a specific embodiment, as shown in fig. 5, the optical-to-electrical signal conversion unit 401 includes a Photodiode (PD); the voltage amplifying unit 402 includes a first MOS transistor M1, a second MOS transistor M2, and a third MOS transistor M3.

Specifically, the cathode of the photodiode PD is connected to the source or drain of the first MOS transistor M1, the anode of the photodiode PD is connected to the ground VSS, and the photodiode is configured to generate an electrical signal in response to an optical signal. The gate of the first MOS transistor M1 is connected to a driving circuit, the driving circuit inputs a reset signal (reset) to the first MOS transistor M1, and the drain or source of the first MOS transistor M1 is connected to a power supply voltage VDD for resetting the photodiode PD.

The gate of the second MOS transistor M2 is also connected to the cathode of the photodiode, the source or drain of the second MOS transistor M2 is connected to the power voltage VDD, and the drain or source is connected to one end of the third MOS transistor M3. The first MOS transistor and the second MOS transistor M2 are configured to amplify an initial voltage signal of a pixel of an image frame to the first node a. The second MOS transistor M2 may be a Source Follower (SF) amplifier transistor, and the third MOS transistor M3 may be an address switch.

The first output unit 403 is used for outputting the first voltage signal of the first node a in the (n-1) th frame. Referring to fig. 5, the first output unit 403 includes a first switching transistor S1, a capacitor C, a second switching transistor S2, a fourth MOS transistor M4 and a fifth MOS transistor M5. One end of a capacitor C is connected with the first switch transistor S1 and the second switch transistor S2, the other end is connected with the ground VSS, the capacitor C is used for storing charges, the first switch transistor S1 and the second switch transistor S2 are used for controlling the charging and discharging states of the capacitor, when the first switch transistor S1 is closed, the second switch transistor S2 is opened, and the charges at the first node are stored in the capacitor C; when the first switching transistor S1 is turned off and the second switching transistor S2 is turned on, the charge in the capacitor C is transferred to the fifth MOS transistor M5 through the second switching transistor S2. As seen from fig. 5, the gate of the fifth MOS transistor M5 is connected to the second switching transistor, one end of the fifth MOS transistor M5 is connected to one end of the fourth MOS transistor M4, and the other end of the fifth MOS transistor M5 is connected to the first Input1 of the comparator comp. The other end of the fourth MOS transistor M4 is grounded to VSS.

The second output unit 404 is configured to output a second voltage signal of the first node in an nth frame, where n is a positive integer. As shown in fig. 5, the second output unit includes a sixth MOS transistor M6 and a seventh MOS transistor M7. The gate of the seventh MOS transistor M7 is connected to the first node a, one end of the seventh MOS transistor M7 is connected to one end of the sixth MOS transistor M6, and the other end of the seventh MOS transistor M7 is connected to the second Input2 of the comparator comp. The other end of the sixth MOS transistor M6 is grounded to VSS.

In some optional embodiments, the pixel unit circuit is further connected to a driving circuit, the driving circuit is connected to the gate of the first MOS transistor M1, the gate of the third MOS transistor M3, the gate of the fourth MOS transistor M4, and the gates of the first switching transistor S1, the second switching transistor S2 and the sixth MOS transistor M6, and the driving circuit is configured to drive the first MOS transistor M1, the third MOS transistor M3, the fourth MOS transistor M4, the first switching transistor S1, the second switching transistor S2 and the sixth MOS transistor M6.

As shown in fig. 5, the comparing unit includes a comparator comp shared by the pixel unit circuits in the same column, and is used for outputting a trigger signal when the absolute value of the difference between the first voltage signal and the second voltage signal is not within a set range, and the trigger signal is used for activating the analog-to-digital converting unit to perform analog-to-digital conversion.

It should be noted that each transistor in the above description may be an NMOS transistor or a PMOS transistor. In the present application, one end of any one transistor is a source and the other end is a drain, or one end of any one transistor is a drain and the other end is a source.

In this embodiment, optionally, the source or the drain of the third MOS transistor M3 is connected to the source or the drain of the eighth MOS transistor M8, the drain or the source of the eighth MOS transistor M8 is grounded, the eighth MOS transistor M8 is shared by the pixel unit circuits in the same column, and the eighth MOS transistor M8 is used for providing a bias voltage.

Optionally, the source or the drain of the fifth MOS transistor M5 is connected to the source or the drain of a ninth MOS transistor M9, the drain or the source of the ninth MOS transistor M9 is grounded, the ninth MOS transistor M9 is shared by the pixel unit circuits in the same column, and the ninth MOS transistor M9 is configured to provide a bias voltage;

the drain or the source of the seventh MOS transistor M7 is connected to the source or the drain of a tenth MOS transistor M10, the drain or the source of the tenth MOS transistor M10 is grounded, the tenth MOS transistor M10 is shared by the pixel unit circuits in the same column, and the tenth MOS transistor M10 is used for providing a bias voltage. The eighth MOS transistor M8 inputs bias voltage, the gates of the ninth MOS transistor M9 and the tenth MOS transistor M10 input bias voltage, and the drains or sources of the ninth MOS transistor M9 and the tenth MOS transistor M10 are connected with the power voltage VDD.

It should be noted that the fourth MOS transistor M4 and the comparator comp may be shared by the pixel units in the same column, and the ninth MOS transistor M9 and the tenth MOS transistor M10 are shared by the pixel units in the same column.

In this embodiment, the pixel unit circuit includes two output units, wherein the second output unit stores the pixel signal of the nth frame, the first output unit stores the pixel signal of the (n-1) th frame, and the voltage signals of the two output units are output simultaneously. The pixel structure is suitable for processing sampling information in parallel by lines, including subsequent voltage signal comparison and analog-to-digital conversion processes, and the operation speed can be greatly increased and the system operation bandwidth can be increased due to parallel operation of pixels in the same line; in addition, when one pixel signal is changed remarkably, analog-to-digital conversion is triggered, and power consumption is saved.

Based on the above pixel unit circuit, the following further combines the timing control method in fig. 6, the image sensor structure diagram in fig. 7, and the timing flow chart in fig. 8 to show a flow chart of the timing control method of the pixel unit circuit, which specifically includes the following steps.

In S601, in the first period T1 from the time T1 for the n-1 th frame, the driving circuit controls the reset signal rst input to the first MOS transistor M1 of the pixel unit circuit 701 to be at the high level, turns on the first MOS transistor M1, and resets the photodiode PD.

S602, during a second time period T2 from the time T2, the driving circuit controls the reset signal rst input to the first MOS transistor M1 to switch from high level to low level, the first MOS transistor M1 is turned off, and the pixel unit circuit starts exposure.

S603, the first switch transistor S1 is controlled to be turned on, and in a third time period T3 beginning at time T3, the driving circuit controls the row control signal row to be switched to a high potential, the pixels in the corresponding row are selected, and the sampling signals of the pixels in the row are stored in the capacitor C.

In this step, when the row address selection logic and signal driving section 703 of the image sensor controls the reset signal rst to switch from a high level to a low level, the pixel unit circuit 701 starts integrating the sampled exposure information, and then the driving circuit controls the first switching transistor S1 to turn on, and when the row control signal row switches to a high level at time T3, the pixels of the corresponding row are selected, and the sampled signals of all the pixels of the row are stored on the capacitor C. In addition, a column address encoder 702 of the image sensor is used to implement column address encoding to generate X column addresses [ M-1:0], a row address encoder 704 is used to implement row address encoding to generate Y column addresses [ N-1:0], M and N being positive integers.

S604, in the nth frame, during the fourth period T4 from the time T4, the driving circuit controls the reset signal rst input to the first MOS transistor M1 to be high again, turns on the first MOS transistor M1, and resets the photodiode PD.

S605, in a fifth period T5 from the time T5, the driving circuit controls the reset signal rst Input to the first MOS transistor M1 to be switched from high level to low level, the first MOS transistor M1 is turned off, the pixel unit circuit starts exposure, the second voltage signal of the nth frame is output to the second Input terminal Input2 of the comparator comp through the second transmission unit, and the first voltage signal of the n-1 th frame stored in the capacitor C is output to the first Input terminal Input1 of the comparator comp through the first transmission unit.

In the step, after the pixel unit circuit is exposed, because the photodiode PD adopts a special semiconductor PN junction technology, electrons are accumulated on the surface of the photodiode PD by a photovoltaic effect under light, and the photovoltaic special effect refers to: the PN junction intrinsic layer can generate photogenerated electrons and photogenerated holes after absorbing photons, and the photogenerated electrons and the photogenerated holes can be separated in an electric field built in the PN junction, so that the phenomena of photovoltage and photogenerated current are formed.

In this embodiment, when the pixel unit circuit undergoes a photoelectric conversion and an integration effect for a period of time, the voltage at the sampling point is proportional to the illumination intensity, and reaches a specific value. In a sixth time period T6 beginning at time T6, the n-1 th frame signal stored in the first transmission unit is output to the first Input terminal Input1, when the row control signal is switched to high level, the pixel of the corresponding row is selected, and the control signal Input to the first switching transistor S1 is low level, the first switching transistor S1 is still in an off state, so the charge is not output to the second Input terminal Input2 of the comparator comp through the first transmission unit, but directly output to the second Input terminal Input of the analog-to-digital conversion unit through the second transmission unit, and the second voltage signal of the n-th frame is also output to the Input port of the analog-to-digital conversion unit.

S606, when the difference between the second voltage signal and the first voltage signal is not within the set range, the comparator comp triggers the analog-to-digital conversion unit to perform analog-to-digital conversion at time T7.

Specifically, the comparator comp determines a difference between a second voltage signal and a first voltage signal collected by the same pixel in two frames, determines whether an absolute value of the difference is within a set range, and if the absolute value exceeds the set range, it indicates that the pixel signal has a significant change, so that a trigger signal (change) output by the comparator is switched from a low level to a high level, and activates an Analog-to-Digital Conversion (ADC) unit to start an Analog-to-Digital Conversion process, wherein the gradient waveform generator 709 is connected to the ADC. And writing the conversion result into the output result latch FIFO of the corresponding column through an analog-to-digital conversion process in a certain period. In connection with fig. 4, the event generator 705 in fig. 7 corresponds to the comparator comp in fig. 5, which outputs the trigger signal (change) of fig. 5.

In this embodiment, when a row of pixels is selected, all the pixels in the row input the voltage signal sampled in the current frame and the voltage signal stored on the capacitor C in the previous frame to the comparator for analog signal comparison. When a pixel signal is judged to have a significant change (the judgment criterion is that the absolute value of the difference between the two exceeds a set range), the comparator result of the column corresponding to the pixel will output 1, the analog-to-digital conversion unit is enabled, the pixel signal is converted into a digital signal by the column analog-to-digital conversion unit 706(a/D) shared by the columns, and stored into the column analog-to-digital conversion structure latch 707 of the corresponding column, and the event state of the column is set to 1 (comparison result latch FIFO), and stored into the event comparison result latch 708 of the corresponding column. If the determination criterion is that the absolute value of the difference between the pixel signal and the analog-to-digital conversion unit is within the set range when the pixel signal is determined not to have significantly changed), the comparator of the column corresponding to the pixel outputs 0, the analog-to-digital conversion unit is not activated, and the a/D conversion is not performed on the pixel signal, and the state of the column is set to 0.

Alternatively, S607, after the analog-to-digital conversion of the pixels of the row is completed, at time T8, the conversion result and the comparison result stored in the latch are cleared.

That is, the system finally reads out all conversion results stored in the column analog-to-digital conversion result latch 707 quickly in a serial manner to be cleared. Correspondingly, all comparison results stored in the event result latch 708 are also read out empty in a serial manner in a synchronous and fast manner.

It should be noted that when the value stored in each cell of the latch FIFO is a first value (e.g., 0), the ADC value of the corresponding column is an invalid value. The ADC value for the corresponding column is valid only if the value stored by each cell of the latch FIFO is the second value (e.g. 1).

In this embodiment, the output of the image sensor may be output in a synchronous manner or in an asynchronous manner. By synchronized is meant that the output comprises the value of each pixel and the value of the corresponding comparator result, i.e. the correspondence between the pixel value and the corresponding comparison result. When the value of the comparator result is 0, the value of the pixel is the previously acquired value of the pixel and does not change. When the value of the comparator result is 1, the value of the pixel changes, and the value output by the ADC replaces the value of the pixel acquired in the previous frame.

The asynchronous output means that a value obtained by analog-to-digital conversion of a pixel is output together with an address < pixel value, (X column address, Y row address) of a row and a column corresponding to the pixel. The synchronization and the asynchronization refer to whether the image sensor as a sender of information and the computer as a receiver of information need to require a synchronization signal frame _ START (or LINE _ START) at the START of each frame (or each LINE).

In summary, the present invention is advantageous in that when a row of pixels is selected, all the pixels in the row input the voltage signal sampled in the current image frame and the voltage signal stored on the capacitor in the previous frame into the comparator for analog signal comparison. When a pixel signal is judged to be changed remarkably, the analog-to-digital conversion unit is triggered to perform analog-to-digital conversion, and the analog-to-digital conversion is not triggered under the condition that the pixel signal is not changed remarkably, so that the power consumption is saved on the premise of ensuring the CIS performance.

The above description is only for the preferred embodiment of the present invention, and the embodiment is not intended to limit the scope of the present invention, so that all the equivalent structural changes made by using the contents of the description and the drawings of the present invention should be included in the scope of the present invention. The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention.

Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A vision system comprising an image sensor, an event counting module and an image processing module;

the image sensor is used for acquiring an original image, when the initial voltage signals of the same pixel point in the nth frame original image and the nth-1 frame original image are changed, carrying out analog-to-digital conversion on the analog signals of the pixel point to obtain digital signals of the pixel point, and sending a trigger signal to the event counting module, wherein n is a positive integer;

the event counting module is used for determining the number of target pixel points with changed initial voltage signals in the nth frame original image by counting the number of trigger signals from the image sensor within the exposure time of the nth frame original image;

the image processing module is used for outputting digital signals of all pixel points of the nth frame original image in a synchronous transmission mode when the ratio of the number of target pixel points to the total number of the pixel points of the nth frame original image is greater than a set threshold; and when the ratio of the number of the target pixel points to the total number of the pixel points of the nth frame original image is less than or equal to the set threshold, outputting the digital signals of the target pixel points in the nth frame original image in an asynchronous transmission mode.

2. The vision system of claim 1, wherein the image processing module outputs digital signals of respective pixel points of the nth frame original image in a synchronous transmission manner, and comprises:

aiming at a target pixel point with a changed initial voltage signal, the image processing module outputs a digital signal of the target pixel point of the nth frame original image and an identifier that the initial voltage signal of the target pixel point has a change in a synchronous transmission mode;

and aiming at the non-target pixel points of which the initial voltage signals are not changed, the image processing module outputs the digital signals of the non-target pixel points of the n-1 th frame of original image in a synchronous transmission mode.

3. The vision system of claim 2, wherein the image processing module outputs the digital signal of the target pixel point in the n-th original image in an asynchronous transmission mode, comprising:

and aiming at the target pixel point with the changed initial voltage signal, the image processing module outputs the digital signal of the target pixel point of the nth frame original image and the row and column address corresponding to the target pixel point in an asynchronous transmission mode.

4. A visual system according to any one of claims 1 to 3, wherein the set threshold value satisfies the following formula:

wherein th denotes a set threshold value, len _L Indicating the bit width occupied by the pixel value, H indicating the number of pixels per row of the original image, W indicating the number of pixels per column of the original image,

5. The visual system of any one of claims 1 to 3, wherein when the ratio of the number of target pixels to the total number of pixels in the n-th original image is less than or equal to a predetermined threshold, outputting the digital signal of the target pixels in the n-th original image in an asynchronous transmission mode, comprises:

the image processing module is used for extracting an interesting region from the nth frame original image according to the region boundary of a moving object corresponding to the event set and outputting digital signals of all pixel points of the interesting region in the nth frame original image in an asynchronous transmission mode; the interested region comprises digital signals of target pixel points in the nth frame original image.

6. The vision system of claim 1, wherein the image sensor comprises a pixel array of analog-to-digital conversion units and pixel cell circuits;

the pixel unit circuit comprises a photoelectric signal conversion unit, a voltage amplification unit, a first output unit, a second output unit and a comparison unit;

the photoelectric signal conversion unit is used for converting exposure information of pixel points in an image frame into initial voltage signals;

the voltage amplifying unit is used for amplifying and outputting the initial voltage signal;

the first output unit is connected to the voltage amplifying unit and used for receiving and storing a first voltage signal of the pixel point in the image frame at the (n-1) th frame;

the second output unit is connected to the voltage amplifying unit and used for receiving and outputting a second voltage signal of the pixel point in the image frame at the nth frame;

a comparing unit, connected to an output terminal of the first output unit and an output terminal of the second output unit, respectively, and configured to receive the first voltage signal from the first output unit and the second voltage signal from the second output unit, and compare the first voltage signal and the second voltage signal to determine whether there is a change between an initial voltage signal of the pixel in an image frame at an n-1 frame and an initial voltage signal of the pixel in the image frame at an n-1 frame;

the comparison unit is further configured to: when the absolute value of the difference value between the first voltage signal and the second voltage signal is not in a set range, determining that the initial voltage signal of the pixel point in the image frame at the (n-1) th frame and the initial voltage signal of the pixel point in the image frame at the n-th frame are changed, and sending a trigger signal to the analog-to-digital conversion unit;

the analog-to-digital conversion unit is connected with the output end of the second output unit and is used for receiving a second voltage signal of the pixel point in the image frame at the nth frame; and when the trigger signal is received, starting to perform analog-to-digital conversion on the second voltage signal.

7. The vision system of claim 6, wherein said photoelectric signal conversion unit comprises a photodiode; the voltage amplification unit comprises a first MOS tube, a second MOS tube and a third MOS tube;

the source electrode or the drain electrode of the first MOS tube and the grid electrode of the second MOS tube are connected to the cathode of the photodiode, the source electrode or the drain electrode of the second MOS tube is connected to the source electrode or the drain electrode of the third MOS tube, and the drain electrode or the source electrode of the second MOS tube and the drain electrode or the source electrode of the first MOS tube are connected to a power supply voltage.

8. A visual system according to claim 7, wherein the comparison unit comprises a comparator;

the first output unit comprises a first switching transistor, a capacitor, a second switching transistor, a fourth MOS transistor and a fifth MOS transistor;

9. The vision system of claim 8, wherein the second output unit comprises a sixth MOS transistor and a seventh MOS transistor; the grid electrode of the seventh MOS tube is connected to the source electrode or the drain electrode of the second MOS tube, the source electrode or the drain electrode of the seventh MOS tube is connected to the source electrode or the drain electrode of the sixth MOS tube, the drain electrode or the source electrode of the seventh MOS tube is connected to the second input end of the comparator, and the drain electrode or the source electrode of the sixth MOS tube is grounded.

10. The vision system as claimed in claim 6, wherein said image sensor further comprises a latch for storing the analog-to-digital conversion result and the comparison result of each pixel point, said conversion result comprising the digital signal of each pixel point of the n-1 th frame of original image and the digital signal of the target pixel point of the n-th frame of target original image; the comparison result comprises a result identification that the initial voltage signal of the pixel point changes and a result identification that the initial voltage signal of the pixel point does not change.