CN113489925B - Focal plane detector reading circuit for realizing convolution calculation - Google Patents
Focal plane detector reading circuit for realizing convolution calculation Download PDFInfo
- Publication number
- CN113489925B CN113489925B CN202110607345.3A CN202110607345A CN113489925B CN 113489925 B CN113489925 B CN 113489925B CN 202110607345 A CN202110607345 A CN 202110607345A CN 113489925 B CN113489925 B CN 113489925B
- Authority
- CN
- China
- Prior art keywords
- sum
- convolution
- equal
- circuit
- difference circuit
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000004364 calculation method Methods 0.000 title claims abstract description 11
- 239000003990 capacitor Substances 0.000 claims abstract description 20
- 230000010354 integration Effects 0.000 claims abstract description 10
- 238000000034 method Methods 0.000 claims abstract description 8
- 238000003466 welding Methods 0.000 claims description 2
- 238000013528 artificial neural network Methods 0.000 abstract description 8
- 238000013135 deep learning Methods 0.000 description 5
- 230000001133 acceleration Effects 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 2
- 230000006835 compression Effects 0.000 description 2
- 238000007906 compression Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000011176 pooling Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 206010063385 Intellectualisation Diseases 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000003708 edge detection Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005265 energy consumption Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 238000004806 packaging method and process Methods 0.000 description 1
- 238000013139 quantization Methods 0.000 description 1
- 238000012549 training Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
- H04N25/70—SSIS architectures; Circuits associated therewith
-
- H—ELECTRICITY
- H01—ELECTRIC ELEMENTS
- H01L—SEMICONDUCTOR DEVICES NOT COVERED BY CLASS H10
- H01L27/00—Devices consisting of a plurality of semiconductor or other solid-state components formed in or on a common substrate
- H01L27/14—Devices consisting of a plurality of semiconductor or other solid-state components formed in or on a common substrate including semiconductor components sensitive to infrared radiation, light, electromagnetic radiation of shorter wavelength or corpuscular radiation and specially adapted either for the conversion of the energy of such radiation into electrical energy or for the control of electrical energy by such radiation
- H01L27/144—Devices controlled by radiation
- H01L27/146—Imager structures
- H01L27/14601—Structural or functional details thereof
- H01L27/14609—Pixel-elements with integrated switching, control, storage or amplification elements
Abstract
The invention discloses a focal plane detector reading circuit for realizing convolution calculation, which is characterized in that integration capacitors corresponding to photosensitive elements are used for taking different capacitance values and are combined with input resistors of a sum and difference circuit to realize the weight of convolution kernels, the integration voltages of corresponding pixel integration capacitors are summed by the sum and difference circuit according to the definition of the convolution kernels, and the output value of the sum and difference circuit is the multiplication and addition operation of pixel values and the weight of the convolution kernels, so that the convolution operation of images is realized. In the scheme, the integral capacitor is a part of the sensing device, is also a device for realizing a calculation function, and is an element for realizing the integration of sensing, and compared with the method of digitalizing an integral signal and then carrying out digital multiply-add operation, the scheme has higher operation speed and extremely low calculation power consumption overhead, thereby improving the energy efficiency of the neural network circuit.
Description
Technical Field
The invention relates to a design method of a reading circuit of an intelligent focal plane detector, in particular to a reading circuit of a focal plane detector for realizing convolution calculation, which realizes the convolution operation of a detected image, is suitable for the arrangement of an artificial intelligence algorithm based on deep learning in the focal plane detector and can achieve higher energy efficiency.
Background
The deep learning algorithm based on the neural network has a good application effect in many fields, so that the deep learning algorithm becomes a mainstream algorithm of artificial intelligence at present, image processing is an important application field of the deep learning algorithm, and the deep learning algorithm is widely applied to the aspects of image classification and positioning, face recognition and the like. The neural network training work and the reasoning work are mainly carried out at the cloud end, and the arrangement of the edge equipment mainly utilizes the trained network to carry out reasoning. Because most edge detection equipment is embedded systems, energy consumption becomes an important factor for successful deployment, and methods such as algorithms for achieving higher energy efficiency and network compression and hardware acceleration are widely adopted.
The focal plane detector is used as an important imaging sensor, a neural network is deployed in a focal plane reading circuit, detector intellectualization is realized, the application range of the detector can be greatly expanded, the convolutional neural network is mainly suitable for image processing at present, the network consists of a plurality of layers of convolution layers, pooling layers and subsequent output layers, particularly, the convolution operation amount of an input layer is maximum, if the conventional scheme is adopted by the on-chip integrated neural network, each photosensitive element signal needs to be firstly digitized, then convolution operation is carried out, and the convolution operation needs to occupy a large amount of calculation resources.
The sense integration is an important method for improving energy efficiency, signal charges are converted into voltage signals by adopting an integrating capacitor in the reading of signals in a focal plane reading circuit, so that the voltage signals are used as components of sense, and the corresponding relation between the voltage and the signals is determined by the integrating capacitor, so that the multiplication of the signals can be realized through the capacitance value, and the signal charges can be used as a device for realizing the sense integration, and the multiplication of the signals and a convolution kernel can be realized under the condition of hardly increasing extra time delay and power consumption. The input resistance in the sum and difference circuit can also influence the weight of convolution operation, and low-power-consumption operation of the neural network can be further realized by combining network compression and quantization of network parameters.
Disclosure of Invention
Aiming at the problems of large operation amount and large circuit power consumption in the traditional scheme of calculating after the signal of the photosensitive element is digitized, the invention aims to provide a read-out circuit design which utilizes the integral capacitance of the read-out circuit as a device for realizing convolution operation, and realizes the convolution operation of an image signal under the condition of not obviously increasing additional operation, thereby improving the energy efficiency of neural network deployment and achieving the aim of low-power-consumption neural network deployment.
The principle of the invention is shown in figure 1, the focal plane detector is a row and b columns, the total pixel number is a multiplied by b, DijIs the photosensitive element (i is more than or equal to 1 and less than or equal to a, j is more than or equal to 1 and less than or equal to b) of the ith row and the jth column, CijFor its integrating capacitance, RijFor its input resistance connected to the sum and difference circuit, DlmIs the photosensitive element (1 is more than or equal to l and less than or equal to a, 1 is more than or equal to m and less than or equal to b) in the mth column of the first row, ClmFor its integrating capacitance, RlmIs its input resistance connected to the sum and difference circuit. The method is characterized in that: the convolution operation of the image is realized by the integration capacitance of the reading circuit and the addition and difference circuit. Determining the convolution kernel size of the first layer of convolution layer to be nxn according to the required implementation algorithm, and setting kxy、kpq(1. ltoreq. x, y, p, q. ltoreq. n) are two weights of a convolution kernel, where k isxyPositive value, and D in convolution operationijCorresponds to the signal of, kpqNegative value, and D in convolution operationlmThe signals of (a) correspond. The photosensitive chip is electrically connected with the reading circuit through inverse welding, the photosensitive elements in the photosensitive chip are connected with the integrating capacitors in the reading circuit one by one, all the integrating capacitors in the traditional reading circuit are designed into a uniform capacitance value, and the light in the photosensitive chip is in the scheme of the inventionMinyuan DijAnd integrating capacitor C in readout circuitijConnected via an input resistor RijA photosensitive element D in the photosensitive chip connected with the positive electrode of the sum-difference circuitlmAnd integrating capacitor C in readout circuitlmConnected via an input resistor RlmConnected to the negative pole of the sum and difference circuit, where CijAnd RijIs given by a convolution kernel kxyDetermination of Cij×RijAnd k isxyIn inverse ratio, ClmAnd RlmIs given by a convolution kernel kpqDetermination of Clm×RlmAnd k ispqAnd inversely proportional, when the convolution kernel weight k is 0, the integrating capacitor is not connected with the input of the sum and difference circuit, signal charges are injected into the integrating capacitor within a given integration time and generate voltage, the voltage signal is input into the sum and difference circuit to generate an output signal, and the output of the sum and difference circuit designed according to the method is the multiplication and addition calculation of the image signal and the convolution kernel.
Because a large amount of A/D conversion is omitted and only relatively few sum and difference circuits are added, the convolution operation of the first layer of the image is realized under the condition of adding less extra power, and the operation efficiency is improved. After the first layer of convolution and pooling, the signal with reduced data size is input into a subsequent network for digitization and calculation of a deeper network.
Drawings
Fig. 1 is a schematic diagram of a convolution kernel and a graphic signal performing a multiply-add operation by using a read circuit integrating capacitor and a sum-difference circuit.
Detailed Description
The following detailed description is to be taken in conjunction with the accompanying drawings to better illustrate the structural and functional features of the present invention, and not to limit the scope of the invention.
According to the invention, the integrating capacitors of the unit circuits in the readout circuit are set to different capacitance values according to the requirements of an implementation algorithm, in addition, a sum and difference circuit for realizing convolution addition operation is added in the readout circuit, and the multiplication operation of signals and convolution kernels is realized by the input resistors of the integrating capacitors and the sum and difference circuit. The output of the sum and difference circuit is the result of the convolution operation of the image signal and the convolution kernel. The signal after dimensionality reduction of the first layer convolution layer is used as the input of a subsequent network layer, the subsequent network layer can be realized on a reading circuit chip or in an independent circuit, for example, the subsequent network circuit can be designed into a special circuit chip and a detector packaged in the same shell, namely a system-level packaging scheme, or the detector is packaged independently, the subsequent network realizes network acceleration through an FPGA or an embedded system, and higher flexibility can be achieved and the development cost is reduced.
Claims (1)
1. A focal plane detector reading circuit for realizing convolution calculation has a row, b columns and a>0，b>0; total number of pixels a × b, DijIs the photosensitive element in the ith row and the jth column, i is more than or equal to 1 and less than or equal to a; j is more than or equal to 1 and less than or equal to b and CijFor its integrating capacitance, RijFor its input resistance connected to the sum and difference circuit, DlmL is more than or equal to 1 and less than or equal to a of the photosensitive element in the mth row of the first row; m is more than or equal to 1 and less than or equal to b; clmFor its integrating capacitance, RlmAn input resistor connected to the sum and difference circuit; the method is characterized in that:
the convolution operation of the image is realized by the integration capacitance of the reading circuit and the addition and difference circuit. Determining the convolution kernel size of the first layer of convolution layer to be nxn, n according to the required implementation algorithm>0; let kxy、kpqIs two weight values of convolution kernel, x is more than or equal to 1, y, p, q is less than or equal to n, wherein k isxyPositive value, and D in convolution operationijCorresponds to the signal of, kpqNegative value, and D in convolution operationlmThe signals of (1) correspond; the photosensitive chip is electrically connected with the reading circuit through inverse welding, the photosensitive elements in the photosensitive chip are connected with the integrating capacitors in the reading circuit one by one, all the integrating capacitors in the traditional reading circuit are designed into a uniform capacitance value, and the photosensitive elements D in the photosensitive chip are connected with the integrating capacitors one by oneijAnd the readout circuitPartial capacitance CijConnected via an input resistor RijA photosensitive element D in the photosensitive chip connected with the positive electrode of the sum-difference circuitlmAnd integrating capacitor C in readout circuitlmConnected via an input resistor RlmConnected to the negative pole of the sum and difference circuit, where CijAnd RijIs given by a convolution kernel kxyDetermination of Cij×RijAnd k isxyIn inverse ratio, ClmAnd RlmIs given by a convolution kernel kpqDetermination of Clm×RlmAnd k ispqAnd inversely proportional, when the convolution kernel weight k is 0, the integrating capacitor is not connected with the input of the sum and difference circuit, signal charges are injected into the integrating capacitor within a given integration time and generate voltage, the voltage signal is input into the sum and difference circuit to generate an output signal, and the output of the sum and difference circuit designed according to the method is the multiplication and addition calculation of the image signal and the convolution kernel.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110607345.3A CN113489925B (en) | 2021-06-01 | 2021-06-01 | Focal plane detector reading circuit for realizing convolution calculation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110607345.3A CN113489925B (en) | 2021-06-01 | 2021-06-01 | Focal plane detector reading circuit for realizing convolution calculation |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113489925A CN113489925A (en) | 2021-10-08 |
| CN113489925B true CN113489925B (en) | 2022-07-08 |
Family
ID=77933927
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110607345.3A Active CN113489925B (en) | 2021-06-01 | 2021-06-01 | Focal plane detector reading circuit for realizing convolution calculation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113489925B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115295544B (en) * | 2022-09-29 | 2022-12-20 | 中国电子科技集团公司信息科学研究院 | Infrared sensing and calculation integrated chip and working method thereof |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102017147A (en) * | 2007-04-18 | 2011-04-13 | 因维萨热技术公司 | Materials systems and methods for optoelectronic devices |
| CN208766715U (en) * | 2018-07-26 | 2019-04-19 | 珠海市一微半导体有限公司 | The accelerating circuit of 3*3 convolution algorithm |
| CN111669527A (en) * | 2020-07-01 | 2020-09-15 | 浙江大学 | Convolution operation framework in CMOS image sensor |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7054041B2 (en) * | 2001-12-06 | 2006-05-30 | General Motors Corporation | Image sensor method and apparatus having addressable pixels and non-destructive readout |
-
2021
- 2021-06-01 CN CN202110607345.3A patent/CN113489925B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102017147A (en) * | 2007-04-18 | 2011-04-13 | 因维萨热技术公司 | Materials systems and methods for optoelectronic devices |
| CN208766715U (en) * | 2018-07-26 | 2019-04-19 | 珠海市一微半导体有限公司 | The accelerating circuit of 3*3 convolution algorithm |
| CN111669527A (en) * | 2020-07-01 | 2020-09-15 | 浙江大学 | Convolution operation framework in CMOS image sensor |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113489925A (en) | 2021-10-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Hsu et al. | A 0.5-V real-time computational CMOS image sensor with programmable kernel for feature extraction | |
| CN111801693B (en) | Neural network device, signal generation method, and program | |
| Hung et al. | Incorporating luminance, depth and color information by a fusion-based network for semantic segmentation | |
| US9721332B2 (en) | Spike domain convolution circuit | |
| CN110288510B (en) | Proximity sensor vision perception processing chip and Internet of things sensing device | |
| CN114067153B (en) | Image classification method and system based on parallel double-attention light-weight residual error network | |
| CN113489925B (en) | Focal plane detector reading circuit for realizing convolution calculation | |
| CN108200362A (en) | Bionical retina imaging circuit and sub-circuit based on space contrast degree | |
| US20190332929A1 (en) | Event driven mathematical engine and method | |
| CN108985314A (en) | Object detection method and equipment | |
| Hsu et al. | A 0.8 V intelligent vision sensor with tiny convolutional neural network and programmable weights using mixed-mode processing-in-sensor technique for image classification | |
| Abedin et al. | Mr-pipa: An integrated multilevel rram (hfo x)-based processing-in-pixel accelerator | |
| US6694049B1 (en) | Multimode invariant processor | |
| CN116012722A (en) | Remote sensing image scene classification method | |
| Datta et al. | Toward efficient hyperspectral image processing inside camera pixels | |
| Tabrizchi et al. | TizBin: A low-power image sensor with event and object detection using efficient processing-in-pixel schemes | |
| CN115004263A (en) | Face detection device and method and face unlocking system | |
| KR20200068560A (en) | Neuromorphic arithmetic device and operating method thereof | |
| Lee et al. | Input noise immunity of multilayer perceptrons | |
| Keshavarz | A FPGA implementation of neural/wavelet face detection system | |
| Ding et al. | A Spatial-Spectral-Temporal Attention Method for Hyperspectral Image Change Detection | |
| Li et al. | A precision-improved processing architecture of physical computing for energy-efficient SIFT feature extraction | |
| CN116402106B (en) | Neural network acceleration method, neural network accelerator, chip and electronic equipment | |
| Zhang et al. | Reducing the energy cost of inference via in-sensor information processing | |
| Mayr et al. | Pulsed multi-layered image filtering: a VLSI implementation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |