CN113411522A

CN113411522A - Reconfigurable artificial intelligence sensor chip architecture

Info

Publication number: CN113411522A
Application number: CN202010417357.5A
Authority: CN
Inventors: 徐辰
Original assignee: SmartSens Technology Shanghai Co Ltd
Current assignee: SmartSens Technology Shanghai Co Ltd
Priority date: 2020-03-16
Filing date: 2020-05-18
Publication date: 2021-09-17
Also published as: CN212305476U

Abstract

The invention provides a reconfigurable artificial intelligence sensor chip architecture, which comprises a first chip layer and a second chip layer, wherein the first chip layer is a pixel array layer and is used for collecting image data; the second chip layer is a processing circuit layer and comprises an image data processing unit, an AI algorithm unit and an output circuit, and the processing circuit processes interested data according to a reconfigurable mode and outputs corresponding image data. The chip architecture of the invention can also comprise a third chip layer which is arranged between the first chip layer and the second chip layer and comprises an image data storage unit for caching image data. According to the reconfigurable artificial intelligence sensor chip architecture design scheme provided by the invention, the image data acquired by the pixel array is processed in advance according to a reconfigurable mode and then output to the back end for operation, so that the bandwidth of data transmission can be effectively saved, the operation process of the back end is simplified, and the efficiency and the performance of an artificial intelligence system are improved.

Description

Reconfigurable artificial intelligence sensor chip architecture

Technical Field

The invention relates to the field of artificial intelligence system design and application, in particular to a reconfigurable artificial intelligence system sensor chip architecture design technology.

Background

With the development and application of Artificial Intelligence (AI), an image sensor becomes an indispensable component of an artificial intelligence system, which is one of important parts of the artificial intelligence system, and is mainly used for acquiring, identifying, analyzing, and judging images in an application environment, including objects, figures, surrounding environments, and the like. For example, the robot recognizes a target object in a scene, and the unmanned vehicle recognizes and analyzes and judges the surrounding environment during driving. At present, most artificial intelligence system application design architectures adopt that raw data (raw data) output by an image sensor and not processed is sent to a Cloud processor (Cloud processor) or an Edge processor (Edge processor), and the calculated data is fed back to a front end to control the operation of an artificial intelligence system. However, with the development of applications, a plurality of image sensor devices are usually included in an artificial intelligence system at the same time, and a bandwidth bottleneck occurs in the process of transmitting a large amount of unprocessed raw data generated by a plurality of image sensors to a cloud or an edge processing end, so that the real-time performance and the accuracy of the application of the artificial intelligence system are limited, and the problem of poor customer experience process is caused. For example, current unmanned vehicle systems typically employ multiple image sensors to collect, feed back, etc. information around the vehicle. With the increase of the number of image sensors adopted by the unmanned automobile and the improvement of the resolution, a large amount of transmission data can be generated, so that a data transmission bottleneck is generated between the image sensors and the main control equipment. And such a large amount of raw data cannot be calculated and analyzed by the master control device in real time, which limits the improvement of real-time performance and resolution of the whole artificial intelligence system. These problems may affect the driving speed of the unmanned vehicle system during the use process, and the timeliness and accuracy of the judgment of the surrounding environment, and may lead to the dangerous problem of the unmanned vehicle during the driving process in severe cases.

Based on the above problems, the present invention aims to adopt a chip architecture design scheme for performing advanced processing on the data of interest for a large amount of raw data output by a plurality of image sensors of an artificial intelligence system in the prior art, and to perform no processing or processing according to low requirements on unnecessary data information of output redundancy, so as to simplify the back-end operation of the system and improve and enhance the working efficiency and performance of the artificial intelligence system.

Disclosure of Invention

The invention aims to provide a reconfigurable artificial intelligence sensor chip architecture design, which comprises the following steps:

the pixel array of the image sensor is arranged on the first chip layer and used for acquiring image information;

the second chip is used for processing a circuit layer, and the processing circuit comprises an image data processing unit, an AI algorithm unit and an output circuit;

the first chip and the second chip are electrically connected in the chip in a semiconductor hybrid bonding (hybrid bond) process mode, and image data acquired by a pixel array of the first chip are connected in the chip and transmitted to the processing circuit for processing;

optionally, the chip architecture may further include a third chip disposed between the first chip and the second chip, where the third chip includes an image data storage unit for caching image data; the image data storage unit may be a dynamic random access memory DRAM;

the chip architecture has multiple reconfigurable implementation modes according to application:

optionally, the processing circuit performs operation processing on the edge information according to the application, or performs operation processing on the event trigger information;

optionally, the processing circuit processes image data of a region of interest (ROI) according to an application, and the output circuit outputs the image data of the region of interest, or may output image data of non-region of interest including different resolutions at the same time; the output interested area is image data with high resolution, and the non-interested area is image data with low resolution, so that the real-time and quick data transmission is realized, and the transmission bandwidth is saved;

optionally, a central processing unit of a Cloud processor (Cloud processor) or an Edge processor (Edge processor) is adopted to cooperate with the image data processing unit, the image data storage unit and the programmable logic unit FPGA to process the image data;

optionally, the pixel array adopts a BSI design structure, so that the sensitivity and the signal-to-noise ratio of the pixel output are improved.

The technical scheme of the reconfigurable artificial intelligence sensor chip architecture provided by the invention adopts a reconfigurable sensor chip structure design to implement the mode of preprocessing image data and then transmitting the image data to a back-end application, solves the problems that in the prior art, the transmission bandwidth is limited and the real-time performance and the accuracy of data transmission are influenced when a large amount of image data are transmitted to the back-end for operation processing, and can effectively improve the working efficiency and the performance of an artificial intelligence system.

Drawings

FIG. 1 is a block diagram of an image sensor in the prior art;

FIG. 2 is a schematic structural diagram of an AI image sensor chip according to a first embodiment of the invention;

FIG. 3 is a structural diagram of an AI image sensor chip according to a second embodiment of the invention; and

FIG. 4 is a schematic diagram of an artificial intelligence image sensor platform architecture of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described below with reference to a plurality of drawings provided by the present invention. Those skilled in the art will appreciate that the embodiments described herein are illustrative of some, but not exhaustive, of the embodiments in which the invention can be practiced. Other embodiments of the present invention are also within the scope of the present invention. Reference throughout this patent specification to "one example," "an embodiment," or "one application" means that a particular feature, structure, or characteristic described in connection with the example is included in at least one example of the present invention. Thus, the appearances of the phrases "in one example," "in an embodiment," or "in an application example" in various places throughout this specification are not necessarily all referring to the same example. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more examples.

The present invention will be described in detail below with reference to the drawings and embodiments of the present invention. Fig. 1 is a basic structure diagram of an image sensor system in the prior art, and as shown in fig. 1, an image sensor system 100 includes a pixel array 101, a readout circuit 102 and a control circuit 104 connected to the pixel array 101, an image data processing/storing module 103 connected to the readout circuit 102 for performing operation processing such as data processing and storage on an output of the pixel circuit, and a state/timing control module 105 connected to the readout circuit 102 and the control circuit 104 for implementing read control on the pixel array 101. The pixel array 101 includes a plurality of pixel cells arranged in rows (R1, R2, R3 … Ry) and columns (C1, C2, C3 … Cx), and pixel signals output from the pixel array 101 are output to the readout circuit 102 via column lines. In one embodiment, after each pixel unit acquires image data, the image data is read out according to the readout circuit 102 in which the state/timing control module 105 designates the readout mode, and then transferred to the image data processing/storing module 103. In a specific application, the readout circuit 102 may include an analog-to-digital conversion circuit (ADC), an amplification circuit, and others, and performs conversion from an analog signal to a digital signal and amplification of an output signal. In some embodiments, the state/timing control module 105 may include programming options to determine whether the read is read via a global exposure mode or a rolling exposure mode. The image data processing/storing module 103 may store only image data or image data applied or processed by an image effect. In one application example, readout circuitry 102 may read out image data one row at a time along readout column lines (as shown in FIG. 1), or may read out data in a number of other ways, such as by reading out multiple rows simultaneously. The operation of the control circuit 104 may be determined by the current settings of the state/timing control module 105. For example, the control circuit 104 generates a shutter signal for controlling image acquisition. In some applications, the shutter signal may be a global exposure signal such that all pixels of the pixel array 101 acquire their image data simultaneously through a single acquisition window.

Fig. 2 is a schematic structural diagram of an AI image sensor chip according to a first embodiment of the present invention, and as shown in the diagram, the AI image sensor chip structural design according to the first embodiment includes a first chip and a second chip, where the first chip includes a pixel array of an image sensor and is an image acquisition layer. In this embodiment, no limitation is imposed on the specific circuit design of the pixel unit in the pixel array, and the pixel arrays of various pixel unit circuit design structures may be disposed on the first chip. The pixel array in the first chip may be designed as a backside illumination (BSI) sensor pixel to improve the sensitivity and the signal-to-noise ratio, and improve the performance of the image sensor. The second chip is a processing circuit layer that includes other circuit designs of the sensor in addition to the pixel array. The second chip also comprises a circuit ADC, an amplifying circuit and the like for performing analog-digital signal conversion on the image data of the pixel array, and the second chip converts the analog image signal into a digital signal and amplifies the digital signal for subsequent data processing. The processing circuit layer adopts advanced logic circuits to increase logic density and improve operation capability. The first chip and the second chip are electrically connected internally through a semiconductor hybrid bonding process to form a complete chip form. The image data collected by the pixel array of the first chip is transmitted to the processing circuit layer of the second chip through the inside of the chip, and the data transmission speed and the efficiency are high.

In the scheme of the invention, the second chip is a processing circuit layer, the image data acquired by the pixel array is transmitted to the processing circuit through the inside of the chip, and the processing circuit carries out advanced processing on the acquired image data in a reconfigurable mode according to AI application. In order to reduce the data processing and transmission amount and not process the image data required by redundant non-back-end calculation, the scheme of the invention only processes the interested data so as to reduce the data processing and transmission amount. The output circuit transmits the processed interested data to a rear-end processor, such as an edge processor or a cloud processor, continues to perform operation processing, and controls the AI system according to the operation processing result. The implementation mode can effectively save the bandwidth of data transmission, reduce the data amount of back-end processing and calculation, simplify the back-end calculation process and further improve the efficiency and performance of the system. In the scheme of the invention, the interested data can contain data in various forms according to a reconfigurable mode.

In an application example of the present invention, the artificial intelligence system may be an unmanned vehicle system, and the unmanned vehicle system usually needs to be designed and adopted with a plurality of sensors, including image sensors (cameras), radars, laser scanners, GPS, and other devices. The data volume that a plurality of sensors gathered is very big at the normal in-process that traveles, and to the data that a plurality of cameras were gathered, the restriction that transmission bandwidth can exist to the process that these data transmission were handled to the rear end, and a large amount of data calculation processes can be very complicated simultaneously, and data calculation feedback process efficiency also can be lower, and this can bring the influence to the promptness and the accuracy of abnormal conditions processing in the system use. According to the technical scheme provided by the invention, according to a reconfigurable mode, the sensor chip can be designed to realize the processing process as edge operation or event trigger (event trigger) information operation, the processing circuit carries out advanced preprocessing on data, such as classification, marking, compression, enhancement and the like, and the execution device is driven to process after the calculation processing. The edge calculation is closer to the execution unit, and quick response can be realized. This way, the amount of data which is arithmetically processed by the processing circuit can be simplified, and unnecessary image data is not processed, that is, the processing circuit processes only data of interest. Or the data of the trigger event can be processed, for example, when the unmanned vehicle is in a driving process, only the image data of the emergency event of the person or the object appearing in the camera scene is processed, so that the data amount required for operation and processing of the processing circuit can be effectively reduced, the part of transmission data can be transmitted to the rear end at a very high frame rate, and the process of performing the rear-end processing is very fast and efficient. By adopting the technical implementation scheme of the invention, the data quantity required to be processed by the sensor chip is greatly reduced, the data transmission and feedback processes are timely, the timely data feedback is provided for the risk prevention and control and processing of the unmanned automobile system, and effective edge calculation is a very important component for ensuring the safety of the unmanned automobile.

In another application example of the present invention, the processing circuit of the sensor chip processes the image data of the region of interest according to a reconfigurable manner. For example, when the AI system is a face recognition application system, the processing circuit of the sensor chip processes, according to the acquired image information, the image data processing unit only processes, such as cropping, size definition, detection, tracking, etc., the image data of a face region of interest of the AI system, and the region of interest may be one or more image regions. The output circuit may output only the high-resolution image data of the face portion of interest. For image data which is not concerned by other systems around the face image, the processing circuit does not output the data, or outputs the data in a lower resolution image, and the output image data of the interested region is transmitted to a back-end processor through a network for analysis and processing. Because the processing circuit processes the image data only containing the region of interest, the data amount of the arithmetic processing is small, the image can be output at a high frame rate, and the efficiency of the data transmission process is very high. Meanwhile, the image definition of the face part of interest is high, the image quality is relatively high, and the accuracy and the efficiency of the application of the face recognition system can be effectively improved.

Fig. 3 is a schematic structural diagram of an AI image sensor chip according to a second embodiment of the present invention, and as shown in the diagram, the AI image sensor chip according to the second embodiment of the present invention includes a third chip in addition to the first chip layer and the second chip layer. The third chip is arranged between the first chip and the second chip and is electrically connected with the first chip and the second chip in the chip in a semiconductor hybrid bonding mode to form a complete chip structure. The third chip designs a DRAM for an image data storage unit, and the DRAM is used for caching image data, so that the transmission efficiency is improved. The invention scheme provided by the second embodiment is also applicable to various reconfigurable designs, and the second chip processing circuit in the embodiment is also implemented to transmit interested data to the back-end processor after being preprocessed according to different applications.

According to another application example of the invention, a central processing unit CPU of a cloud processor or an edge processor can be used in cooperation with an image data processing unit and an image data storage unit of a sensor chip and a programmable logic unit FPGA to process data of interest of an AI system, and the data is processed and cached and then output to a back-end processor for processing and control operation. The realization mode can effectively improve the efficiency of data calculation and transmission of the AI system, optimize the control process and further improve the performance of the AI system.

FIG. 4 is a schematic diagram of an artificial intelligence image sensor platform architecture according to the present invention. As shown in fig. 4, the left side is a schematic diagram of a structure including a plurality of sensor chips of the present invention, and a plurality of image sensors are usually included at the same time in an AI system application. The image sensor chip structure diagram depicted in the figures is depicted for illustrative purposes only and is not meant to be a limitation upon the description of the chip layer of the present invention. In combination with the above embodiments of the present invention, after the image data is collected by the plurality of image sensor chips, the interested data is processed according to different applications, the processed interested data is transmitted to the edge processor SoC or the cloud processor for further operation, and the AI system is further controlled according to the processing result. The data transmission can be realized in a wireless network transmission mode, the transmission bandwidth can be effectively reduced due to the small amount of transmitted data, the transmission efficiency is improved, and the application requirements of various AI systems can be met.

The various embodiments of the invention and the accompanying drawings are presented for illustrative purposes and various equivalent modifications are possible in different forms without departing from the broader spirit and scope of the invention. Modifications of the embodiments of the invention which are based on the above detailed description are considered to fall within the scope of the invention. The terms used in the following claims should not be construed to be limited to the specific embodiments disclosed in the specification and the claims. Rather, what is intended to be covered is to be construed as broadly as is set forth in the claims when interpreted in accordance with the breadth to which they are fairly set forth. The specification and drawings are to be regarded in an illustrative rather than a restrictive sense.

Claims

1. A reconfigurable artificial intelligence sensor chip architecture, the chip architecture comprising:

a first chip layer including a pixel array;

the second chip layer comprises a processing circuit, and the processing circuit comprises an image data processing unit, an AI algorithm unit and an output circuit;

the first chip layer and the second chip layer are electrically connected in a semiconductor hybrid bonding mode; the processing circuit processes the interested data according to a reconfigurable mode and outputs corresponding image data.

2. The reconfigurable artificial intelligence sensor chip architecture of claim 1, wherein the chip architecture includes a third chip layer disposed between the first chip layer and the second chip layer, the third chip layer including an image data storage unit that buffers image data.

3. The reconfigurable artificial intelligence sensor chip architecture of claim 1, wherein the processing circuitry operates on edge information or on event-triggered information in a reconfigurable manner.

4. The reconfigurable artificial intelligence sensor chip architecture of claim 1, wherein the processing circuitry processes image data of a region of interest in a reconfigurable manner.

5. The reconfigurable artificial intelligence sensor chip architecture of claim 4, wherein the output circuit outputs image data of a region of interest or image data of a region of non-interest including different resolutions.

6. The reconfigurable artificial intelligence sensor chip architecture of claim 5, wherein the regions of interest are high resolution image data and the regions of non-interest are low resolution image data.

7. The reconfigurable artificial intelligence sensor chip architecture of claim 1 or 2, wherein the image data processing unit, the image data storage unit, the FPGA unit and the back-end central processing unit cooperate to process data of interest according to a reconfigurable manner.

8. The reconfigurable artificial intelligence sensor chip architecture of claim 7, wherein the back-end is a cloud processor or an edge processor.

9. The reconfigurable artificial intelligence sensor chip architecture of claim 1, wherein the pixel array is designed for a back-lit configuration.

10. The reconfigurable artificial intelligence sensor chip architecture of claim 1, wherein the output circuitry outputs corresponding image data at a high frame rate.