CN112261308A - Photosensitive integrated circuit with on-chip mode recognition - Google Patents
Photosensitive integrated circuit with on-chip mode recognition Download PDFInfo
- Publication number
- CN112261308A CN112261308A CN202011078404.4A CN202011078404A CN112261308A CN 112261308 A CN112261308 A CN 112261308A CN 202011078404 A CN202011078404 A CN 202011078404A CN 112261308 A CN112261308 A CN 112261308A
- Authority
- CN
- China
- Prior art keywords
- pattern recognition
- neural network
- integrated circuit
- pixels
- pixel
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/70—Circuitry for compensating brightness variation in the scene
- H04N23/73—Circuitry for compensating brightness variation in the scene by influencing the exposure time
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/67—Focus control based on electronic image sensor signals
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a photosensitive integrated circuit with on-chip mode identification, which comprises: an imaging unit two-dimensionally arranged with a plurality of pixels; the convolutional neural network unit is coupled and connected with at least one part of pixels through the programmable logic array; the convolution neural network unit processes image signals generated by at least one part of pixel exposure of the imaging unit according to the neural network model and outputs a pattern recognition result, so that the integration level of the image pattern recognition system is improved, the hardware cost of the system is reduced, and the operation efficiency of the system is improved.
Description
Technical Field
The invention relates to the technical field of integrated circuits, in particular to a photosensitive integrated circuit with on-chip mode identification.
Background
Artificial intelligence techniques based on convolutional neural networks have been applied in many different fields, including image processing. However, most of the artificial intelligence techniques in the prior art generally require exposure through a photosensitive chip to generate an image signal, an ISP processor processes the image signal to obtain original image data, an encoder outputs an encoded image, a local AI coprocessor or a GPU in the cloud performs pattern recognition processing on the image according to a pre-trained neural network model, and then outputs a pattern recognition result. In the processing process, image data needs to be transmitted among a plurality of chips or even among different network nodes, so that higher transmission delay is brought, the leakage of privacy images is possibly caused, and information safety hidden dangers are brought, so that the application range of the artificial intelligence technology is greatly limited, and the hardware cost is increased.
Disclosure of Invention
The invention aims to: a photosensitive integrated circuit with on-chip pattern recognition is provided, which can reduce the data transmission delay of image pattern recognition, reduce the hardware scale, prevent the leakage of privacy images and guarantee the information security.
In order to solve the technical problems, the invention is realized by the following technical scheme:
a photosensitive integrated circuit with on-die pattern recognition, comprising: an imaging unit two-dimensionally arranged with a plurality of pixels; the convolutional neural network unit is coupled and connected with at least one part of pixels through the programmable logic array; and the convolutional neural network unit processes an image signal generated by exposing at least one part of pixels of the imaging unit according to the neural network model and outputs a pattern recognition result.
Preferably, the convolutional neural network unit includes a convolutional operation circuit including an adder and a plurality of multipliers, and the programmable logic array includes a plurality of switch arrays, and the multipliers are coupled to at least a part of the pixels via the switch arrays.
Preferably, the integrated circuit further comprises a processor for configuring the programmable logic array according to the neural network model.
Preferably, the switch array comprises a decoder and a plurality of transistors, wherein the input end of the decoder is connected with the processor, the control end of each transistor is connected with the output end of the decoder, and the controlled end of each transistor is connected with the pixel and the corresponding multiplier.
Preferably, the imaging unit includes a plurality of pixel arrays connected with the corresponding multipliers via the corresponding switch arrays.
Preferably, the convolution step size of the neural network model is n (n is a positive integer greater than 2), and adjacent pixels in the pixel array are spaced by n-1 pixels.
Preferably, the integrated circuit further comprises a memory for storing the neural network model.
Preferably, the integrated circuit further comprises an input/output interface for outputting the pattern recognition result.
Preferably, the photosensitive unit and the convolution operation unit are located on different wafers in the same package.
Compared with the prior art, the invention has the beneficial effects that: the photosensitive integrated circuit with the on-chip mode identification processes image signals generated by at least part of pixel exposure of the imaging unit through the on-chip integrated convolution neural network unit according to the neural network model and outputs a mode identification result, so that the integration level of an image mode identification system is improved, the hardware cost of the system is reduced, the operation efficiency of the system is improved, meanwhile, the connection between pixels and the convolution neural network unit can be changed by configuring the programmable logic array according to the neural network model, and the flexibility and the adaptability of the system are improved.
Drawings
FIG. 1 is a schematic diagram of a photosensitive integrated circuit with on-chip pattern recognition according to an embodiment of the present invention;
FIG. 2 is a detailed structure diagram of the convolution operation circuit of FIG. 1;
FIG. 3 is a schematic view of the light sensing unit of FIG. 1;
FIG. 4 is a schematic structural diagram of the pixel array a in FIG. 2;
fig. 5 is a schematic structural diagram of the switch array a in fig. 2.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The following detailed description of the implementation of the present invention is made with reference to specific embodiments:
referring also to fig. 1-5, a photosensitive integrated circuit with on-chip pattern recognition includes: an imaging unit two-dimensionally arranged with a plurality of pixels; the convolutional neural network unit is coupled and connected with at least one part of pixels through the programmable logic array; and the convolutional neural network unit processes an image signal generated by exposing at least one part of pixels of the imaging unit according to the neural network model and outputs a pattern recognition result.
Specifically, each pixel includes a photosensitive device, and may further include an exposure control circuit, an analog-to-digital conversion circuit, and the like, and when a light image is focused on the imaging unit, the pixel exposure generates an image signal.
The convolutional neural network unit may include an arithmetic circuit for operating a neural network model, and performs a specific pattern recognition task, such as image classification, object recognition, etc., according to the neural network model. The convolutional neural network unit can directly read the image signal through the programmable logic array and process the image signal according to the neural network model, so as to obtain a pattern recognition result. Therefore, the image data can be prevented from being output to the outside of the chip for processing, and the integration level of the system is improved. On the other hand, when only the pattern recognition result is output and the image signal or the image data is not output, the image cannot be reconstructed from the outside, thereby avoiding leakage of the privacy image.
In some embodiments, the programmable logic array may include a plurality of programmable logic paths, the logic paths may connect the pixels and the convolutional neural network unit, and the programmable logic array is programmed to control on/off of the logic paths, so as to control how many pixels the convolutional neural network unit has access to, and thus control the size of the input image of the convolutional neural network unit, so that the integrated circuit may adjust the size of the input image according to different neural network models, thereby improving flexibility of the system.
In some embodiments, the convolutional neural network unit includes a convolutional operation circuit including an adder and a plurality of multipliers, i.e., multiplier a, multiplier b. The multipliers may calculate products of the weights and values of corresponding pixels of the input image signal, and the adder may add the calculation results of the multipliers to obtain a convolution operation result. The adder and multiplier of the convolution operation circuit may perform convolution processing on the image signal to realize a specific filtering function, such as smoothing, blurring, denoising, sharpening, edge extraction, etc., on the image, and may also be used as an input layer of a convolutional neural network. The number of multipliers can be customized according to the size of the convolution operator, for example, the size of the convolution operator is 3 × 3, and the number of multipliers is 9. The weights can also be customized according to the use of the convolution operator. The programmable logic array comprises a plurality of switch arrays, and the multiplier is coupled with at least one part of the pixels through the corresponding switch array.
It is worth noting that, in an operating state, the convolution of the image requires that the convolution operator moves in the image by a certain step size. For example, in the first convolution operation, the pixels 00 to 22 are respectively connected to the multipliers a to j, and the convolution operation unit performs convolution operation on the pixels 00 to 22; in the second convolution operation, the switch array changes the connection between the pixel and the multiplier, and connects the pixel separated by one step with the corresponding multiplier, such as pixel 03-pixel 25, so as to simulate the processing process that the convolution operator moves in the image according to the step in the convolution operation. Thus, convolution operation can be realized without providing a multiplier for each pixel, thereby reducing the circuit scale.
The imaging unit may include a plurality of pixel arrays connected with the corresponding multipliers via the corresponding switch arrays. This allows the input of the corresponding pixel array to be controlled by the switch array.
In some embodiments, the pixel array may be divided sequentially, i.e., the imaging unit is divided into a plurality of pixel regions of the same size, for example, in fig. 3, pixel array 0 includes pixels 00-02, pixels 10-12, pixels 20-22; pixel array 1 includes pixels 03-05, pixels 13-15, pixels 23-25; and so on. Therefore, when some pixel areas do not need to be processed, the corresponding pixel array can be disconnected from the multiplier through the switch array, so that the size of an input image is reduced, and the processing efficiency is improved. In other embodiments, the convolution step size of the neural network model is n (n is a positive integer greater than 2), and adjacent pixels in the pixel array are spaced by n-1 pixels. For example, the convolution step size is 3 in fig. 3, and pixel array 0 includes pixel 00, pixel 03, pixel 30, pixel 33, and so on; the pixel array 1 includes a pixel 01, a pixel 04, a pixel 31, a pixel 34, and the like; and so on.
In some embodiments, the integrated circuit further comprises a processor configured to configure the programmable logic array according to the neural network model. The processor may be connected to the programmable logic array, the convolutional neural network unit via a bus. The processor may be a processor of an ARM core or a processor of another core. The processor may extract parameters of the neural network model, such as convolution step sizes, and configure the programmable logic array according to the convolution step sizes. Specifically, in some embodiments, the switch array a includes a decoder a and a plurality of transistors, i.e., transistors a0-am, an input terminal of the decoder a is connected to the processor, a control terminal of the transistors a0-am is connected to an output terminal of the decoder a, and a controlled terminal of the transistors a0-am is connected to the pixel (pixel 00, pixel 03.., pixel ii) and the corresponding multiplier a. When the processor outputs a numerical value, the decoder outputs a corresponding code, so that the corresponding transistor is controlled to be switched on or switched off. For example, in the first convolution operation, if the processor output value is 1, the decoder corresponding output is 00001, and therefore the transistor a0 is turned on, and the other transistors are turned off, and therefore the pixel 00 is connected to the multiplier a, and in the second convolution operation, the processor output is 2, the decoder output is 000010, the transistor a1 is turned on, and the other transistors are turned off, and then the pixel 03 is connected to the multiplier. And by analogy, in the convolution operation process, one pixel is selected from each pixel array to participate in convolution operation, so that one convolution operation circuit finishes a plurality of times of convolution operation operations according to a certain step length, and the operation that a convolution operator slides in an image according to the step length is simulated. Therefore, it is not necessary to configure a plurality of convolution operation circuits, thereby reducing the circuit scale.
In some embodiments, the integrated circuit further comprises a memory for storing the neural network model. The memory may be coupled to the processor, and the processor may read the neural network model stored in the memory.
In some embodiments, the integrated circuit further comprises an input-output interface for outputting the pattern recognition result.
In some embodiments, the photosensing unit and the convolution operation unit are on different wafers within the same package. Since different circuits may need to be manufactured using different processes, the circuits on different chips may be coupled by wafer bonding or other processes. This allows a more flexible production process.
In summary, the present invention integrates the sensitization and the pattern recognition in the same chip package, directly completes the pattern recognition operation in the chip, and only outputs the recognition result, thereby speeding up the processing speed and avoiding the transmission delay outside the chip.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present invention, and not for limiting the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions in the embodiments of the present application.
Claims (9)
1. A photosensitive integrated circuit with on-chip pattern recognition, comprising: an imaging unit two-dimensionally arranged with a plurality of pixels; the convolutional neural network unit is coupled and connected with at least one part of pixels through the programmable logic array; and the convolutional neural network unit processes an image signal generated by exposing at least one part of pixels of the imaging unit according to the neural network model and outputs a pattern recognition result.
2. A photosensitive integrated circuit with on-die pattern recognition in accordance with claim 1, wherein the convolutional neural network unit comprises a convolutional arithmetic circuit comprising an adder and a plurality of multipliers, and the programmable logic array comprises a plurality of switch arrays, the multipliers being coupled to at least a portion of the pixels via the switch arrays.
3. A light sensing integrated circuit with on-die pattern recognition as recited in claim 2, further comprising a processor configured to configure the programmable logic array according to the neural network model.
4. The photosensitive integrated circuit with on-chip pattern recognition of claim 3, wherein the switch array comprises a decoder and a plurality of transistors, an input terminal of the decoder is connected with the processor, a control terminal of the transistor is connected with an output terminal of the decoder, and a controlled terminal of the transistor is connected with the pixel and the corresponding multiplier.
5. The photosensitive integrated circuit with on-chip pattern recognition of claim 2, wherein the imaging unit comprises a plurality of pixel arrays, the pixel arrays being connected with the corresponding multipliers via corresponding switch arrays.
6. A photosensitive integrated circuit with on-chip pattern recognition according to claim 5, wherein the convolution step size of the neural network model is n (n is a positive integer greater than 2), and adjacent pixels in the pixel array are spaced by n-1 pixels.
7. A photosensitive integrated circuit with on-chip pattern recognition as claimed in claim 1, further comprising a memory for storing the neural network model.
8. The photosensitive integrated circuit with on-chip pattern recognition of claim 1, further comprising an input-output interface for outputting the pattern recognition result.
9. A photosensitive integrated circuit with on-die pattern recognition as claimed in claims 1-8, wherein the photosensitive unit and the convolution operation unit are on different wafers in the same package.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011078404.4A CN112261308A (en) | 2020-10-10 | 2020-10-10 | Photosensitive integrated circuit with on-chip mode recognition |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011078404.4A CN112261308A (en) | 2020-10-10 | 2020-10-10 | Photosensitive integrated circuit with on-chip mode recognition |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112261308A true CN112261308A (en) | 2021-01-22 |
Family
ID=74241984
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011078404.4A Pending CN112261308A (en) | 2020-10-10 | 2020-10-10 | Photosensitive integrated circuit with on-chip mode recognition |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112261308A (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108268943A (en) * | 2017-01-04 | 2018-07-10 | 意法半导体股份有限公司 | Hardware accelerator engine |
| CN108304923A (en) * | 2017-12-06 | 2018-07-20 | 腾讯科技（深圳）有限公司 | Convolution algorithm processing method and Related product |
| CN109032781A (en) * | 2018-07-13 | 2018-12-18 | 重庆邮电大学 | A kind of FPGA parallel system of convolutional neural networks algorithm |
| CN109948777A (en) * | 2018-11-14 | 2019-06-28 | 深圳大学 | The implementation method of convolutional neural networks is realized based on the FPGA convolutional neural networks realized and based on FPGA |
| CN110991609A (en) * | 2019-11-27 | 2020-04-10 | 天津大学 | Line buffer for improving data transmission efficiency |
| CN111464764A (en) * | 2020-03-02 | 2020-07-28 | 上海集成电路研发中心有限公司 | Memristor-based image sensor and convolution operation method thereof |
| KR20200095163A (en) * | 2019-01-31 | 2020-08-10 | 한국기술교육대학교 산학협력단 | Conv-xp pruning apparatus of convolutional neural network suitable for an acceleration circuit |
-
2020
- 2020-10-10 CN CN202011078404.4A patent/CN112261308A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108268943A (en) * | 2017-01-04 | 2018-07-10 | 意法半导体股份有限公司 | Hardware accelerator engine |
| CN108304923A (en) * | 2017-12-06 | 2018-07-20 | 腾讯科技（深圳）有限公司 | Convolution algorithm processing method and Related product |
| CN109032781A (en) * | 2018-07-13 | 2018-12-18 | 重庆邮电大学 | A kind of FPGA parallel system of convolutional neural networks algorithm |
| CN109948777A (en) * | 2018-11-14 | 2019-06-28 | 深圳大学 | The implementation method of convolutional neural networks is realized based on the FPGA convolutional neural networks realized and based on FPGA |
| KR20200095163A (en) * | 2019-01-31 | 2020-08-10 | 한국기술교육대학교 산학협력단 | Conv-xp pruning apparatus of convolutional neural network suitable for an acceleration circuit |
| CN110991609A (en) * | 2019-11-27 | 2020-04-10 | 天津大学 | Line buffer for improving data transmission efficiency |
| CN111464764A (en) * | 2020-03-02 | 2020-07-28 | 上海集成电路研发中心有限公司 | Memristor-based image sensor and convolution operation method thereof |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Shi et al. | A 1000 fps vision chip based on a dynamically reconfigurable hybrid architecture comprising a PE array processor and self-organizing map neural network | |
| Serrano-Gotarredona et al. | AER image filtering architecture for vision-processing systems | |
| EP2235662B1 (en) | Vision sensors, systems, and methods | |
| US20150310311A1 (en) | Dynamically reconstructable multistage parallel single instruction multiple data array processing system | |
| CN112424798A (en) | Neural network circuit device, neural network processing method, and execution program of neural network | |
| CN109564684B (en) | Image semantic segmentation method, programmable logic circuit, system and electronic equipment | |
| Suarez et al. | Low-power CMOS vision sensor for Gaussian pyramid extraction | |
| CN110738317A (en) | FPGA-based deformable convolution network operation method, device and system | |
| CN112005251A (en) | Arithmetic processing device | |
| Nudd | Image understanding architectures | |
| Rodríguez Vázquez et al. | In the quest of vision-sensors-on-chip: Pre-processing sensors for data reduction | |
| CN112261308A (en) | Photosensitive integrated circuit with on-chip mode recognition | |
| CN115775214B (en) | Point cloud completion method and system based on multi-stage fractal combination | |
| Linan et al. | Architectural and basic circuit considerations for a flexible 128× 128 mixed-signal SIMD vision chip | |
| Jendernalik et al. | Analogue CMOS ASICs in image processing systems | |
| Ito et al. | A computational digital pixel sensor featuring block-readout architecture for on-chip image processing | |
| Elouardi et al. | Image processing vision systems: Standard image sensors versus retinas | |
| KR20220084845A (en) | Npu device performing convolution calculation based on the number of channels and method of thereof | |
| Komuro et al. | A new architecture of programmable digital vision chip | |
| Fernández et al. | On the emulation of large-neighborhood templates with binary CNN-based architectures | |
| Heyrman et al. | Smart camera design for intensive embedded computing | |
| Albo-Canals et al. | Embedded low-power low-cost Camera Sensor based on FPGA and its applications in mobile robots | |
| Fang | A system-on-chip design of a low-power smart vision system | |
| EP4290411A1 (en) | Semiconductor chip, device and system comprising the semiconductor chip, and method for manufacturing the semiconductor chip or the system | |
| Pinna et al. | Design of an integrated silicon connectionist retina |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210122 |
|
| RJ01 | Rejection of invention patent application after publication |