CN107316014B

CN107316014B - Memory with image recognition function

Info

Publication number: CN107316014B
Application number: CN201710460367.5A
Authority: CN
Inventors: 张国飙
Original assignee: Hangzhou Haicun Information Technology Co Ltd
Current assignee: Hangzhou Haicun Information Technology Co Ltd
Priority date: 2016-03-07
Filing date: 2017-03-07
Publication date: 2020-11-10
Anticipated expiration: 2037-03-07
Also published as: CN107358100A; CN107316014A; CN111446247A; CN111446248B; CN107169404B; CN111446246B; CN111446249A; CN107169404A; CN111446250A; CN111384052B; CN111446248A; CN111446247B; CN111384051B; CN111384050A; CN107317803A; CN111627909A; CN107301222B; CN111384051A; CN111384052A; CN107392017A

Abstract

The invention provides a memory with an image recognition function. The memory includes a plurality of memory processing units, each unit including a mode processing circuit and at least one three-dimensional memory (3D-M) array. The 3D-M array is stacked above the mode processing circuitry and stores user image data and the like. The pattern processing circuit performs pattern recognition on the user image data stored in the 3D-M array based on the input image model.

Description

Memory with image recognition function

Technical Field

The present invention relates to the field of integrated circuits, and more particularly, to a memory with image recognition function.

Background

Pattern matching and pattern recognition refer to finding a pattern identical to or close to a retrieval pattern (pattern for retrieval) among target patterns (retrieved patterns). The pattern matching requires finding the same pattern, and the pattern recognition requires finding only the close pattern. Unless otherwise specified, the present specification does not distinguish between pattern matching and pattern recognition, and uses pattern processing to refer collectively to various operations performed on patterns.

Pattern processing (including pattern matching and pattern recognition) is widely used. Common pattern processing includes string matching, code matching, voice recognition, image recognition, and the like. String matching is widely used in the fields of big data analysis (such as financial data analysis, e-commerce data analysis, bioinformatics) and the like: searching for the search character string from big data (mostly a text database at present and containing a target character string), and performing statistical analysis. Code matching is widely used in the fields of anti-malware (anti-malware, such as network security and computer antivirus): and searching virus identification (virus signature) from the network data packet or checking whether the network data packet conforms to network specifications (network rules) so as to determine whether the network data packet is safe. Speech recognition matches speech signals collected by a speech sensor or stored in a speech archive to a library of acoustic models and a library of language models. Image recognition matches image signals collected by the image sensor, or stored in an image archive, to an image model library.

With the advent of the big data age, traditional schema libraries (including search schema libraries and target schema libraries) have become large databases (TB-level to PB-level, and even EB-level): the amount of data in the search pattern library (including all patterns used for searching) is already large, while the amount of data in the target pattern library (including all patterns retrieved, typically the user database) is much larger. The von Neumann architecture adopted by the current computer can not meet the requirement of the big data era on mode processing. In the von Neumann architecture, the processor for processing the patterns and the memory for storing the patterns are separate: the memory (e.g., hard disk, optical disk, magnetic tape, etc.) is used only to store pattern data, and cannot perform any pattern processing on it; all mode processing is performed by an external processor (e.g., CPU, GPU). As is well known, the bandwidth between the separate processor and the memory is limited, and it takes a long time for the light to read all the data from the pattern library, and then process and analyze them. Therefore, it takes a long time for pattern processing of a large pattern library.

One typical application of pattern processing is image recognition. One approach to image recognition is to perform pattern recognition on a user's image from an image model library. In recognition, the pattern processor compares the user image data with the image models in the image model library one-to-one to find the closest image model. In conventional computer architectures, user image data is stored in external memory (e.g., a hard disk). The external memory is a simple memory and does not have any image recognition function. When a new image model needs to be identified, all user image data in the hard disk of the computer needs to be read into the processor for image identification. Because the conventional image processors (such as CPU and GPU) have limited number of kernels, low image recognition parallelism, and long time-consuming reading user image data from the external memory, the conventional computer architecture is inefficient in processing image recognition.

Disclosure of Invention

The main object of the present invention is to improve the efficiency of image recognition.

Another object of the present invention is to provide a memory having an image recognition function efficiently.

To achieve these and other objects, the present invention provides a memory with an image recognition function: it can not only store pattern data but also perform pattern processing on it. The memory chip includes a plurality of memory processing units, each of which includes a pattern processing circuit and at least one 3D-M (3D-M) array storing at least one pattern including a target pattern and a search pattern. Vertical integration of the 3D-M array with the mode processing circuitry brings many advantages: since the 3D-M array does not occupy substrate area, it can be integrated on a pattern processing circuit, which can increase memory capacity and reduce chip area. More importantly, since the 3D-M array and the pattern processing circuitry are in the same chip and in close proximity, a large bandwidth electrical connection can be made between them. By using massively parallel computing (each memory chip may contain tens of thousands of memory processing units), the memory enables fast pattern processing for large pattern libraries.

Accordingly, the present invention provides a memory (200) with image recognition function, comprising: an input bus (110) for transmitting at least one image model; a semiconductor substrate (0) and a plurality of memory processing units (100aa-100mn) coupled to the input bus (110), each memory processing unit (100ij) containing at least a three-dimensional memory (3D-M) array (170) and a mode processing circuit (180), wherein: said 3D-M array (170) being stacked over the substrate (0), the 3D-M array (170) storing at least one user image data; said pattern processing circuitry (180) being located in the substrate (0), the pattern processing circuitry (180) performing pattern recognition on the user image data according to the image model; the 3D-M array (170) and the pattern processing circuit (180) are coupled by a plurality of contact via holes (1av, 3 av).

Drawings

FIG. 1 is a circuit block diagram of a distributed mode processor.

FIGS. 2A-2C are circuit block diagrams of three types of memory processing units;

FIG. 3A is a cross-sectional view of a three-dimensional writable memory (3D-W) and a 3D-W based memory processing unit; fig. 3B is a cross-sectional view of a three-dimensional printed memory (3D-P) and a 3D-P based memory processing unit.

Fig. 4 is a perspective view of a storage processing unit.

Fig. 5A to 5C are substrate circuit layout diagrams of three kinds of memory processing units.

It is noted that the figures are diagrammatic and not drawn to scale. Dimensions and structures of parts in the figures may be exaggerated or reduced for clarity and convenience. In different embodiments, alphabetic suffixes following numbers represent different instances of the same class of structure; the same numerical prefixes refer to the same or similar structures. In this specification, "/" denotes "and/or".

Detailed Description

Fig. 1 shows a memory 200 with image recognition function, which is a distributed mode processor chip. Processor 200 contains m n memory processing units 100aa-100 mn. These memory processing units 100aa to 100mn are each formed on the substrate 0. An input bus 110 is coupled to each memory processing unit and an output bus 120 is coupled to each memory processing unit. Note that distributed mode processor chip 200 may contain tens of thousands of memory processing units 100aa-100 mn. Such a large number of memory processing units 100aa-100mn may ensure massively parallel computations to achieve high-speed mode processing.

Each memory processing unit 100ij has a pattern processing circuit 170 and at least one 3D-M array 180 that stores at least one pattern. Fig. 2A to 2C are circuit block diagrams of three kinds of memory processing units 100 ij. In these embodiments, one mode processing circuit 180 serves a different number of 3D-M arrays 170. The mode processing circuit 180 of FIG. 2A serves a 3D-M array 170: the pattern data stored in the 3D-M array 170 is fed to the pattern processing circuitry 180 over the large bandwidth electrical connection 160 (see fig. 3A-4), performs pattern matching or pattern recognition with the input pattern data 110, and generates output pattern data 120. The pattern processing circuit 180 in FIG. 2B services four storage arrays 170A-170D: the pattern data stored in the 3D-M arrays 170A-170D is fed to the pattern processing circuit 180 via the large bandwidth electrical connections 160A-160D (see FIGS. 3A-4) and is pattern matched or pattern identified with the input pattern data 110. The pattern processing circuit 180 in FIG. 2C serves eight storage arrays 170A-170D and 170W-170Z: the pattern data stored in the 3D-M arrays 170A-170D and 170W-170Z is fed into the pattern processing circuit 180 through the large bandwidth electrical connections 160A-160D and 160W-160Z (see FIGS. 3A-4) and is pattern matched or pattern identified with the input pattern data 110. As can be seen from fig. 5A-5C later, the pattern processing circuit 180 serving more 3D-M arrays has a larger area and a stronger function.

Fig. 3A-3B show two typical 3D-ms. Fig. 3A is a cross-sectional view of a 3D-W based memory processing unit 100 ij. The 3D-W stored information is entered using electrical programming. A common 3D-W is 3D-XPoint. Other 3D-Ws include memristor, Resistive Random Access Memory (RRAM), Phase Change Memory (PCM), programmable addressing cell (PMC), conductive bridging random-access memory (CBRAM), and the like. The 3D-W is further divided into a three-dimensional one-time-programmable memory (abbreviated as 3D-OTP) and a three-dimensional multi-time-programmable memory (abbreviated as 3D-MTP) according to the programmable times. As the name implies, 3D-OTP can be programmed once and 3D-MTP can be programmed multiple times (including over-programming). The 3D-OTP process is mature, and can store a retrieval pattern library (such as a virus identification library, a network specification library, an acoustic model library, a language model library and the like), and pattern data in the pattern library is only added and is not modified. The 3D-MTP is a general purpose memory that can be used to store a library of target patterns, such as user data (including user codes) and the like.

The 3D-W based memory processing cell 100ij has a substrate circuit layer 0K formed on a substrate 0. A memory layer 16A is stacked over the substrate circuit 0K, and a memory layer 16B is stacked over the memory layer 16A. The substrate circuit layer 0K contains peripheral circuits of the memory layers 16A, 16B, which include transistors 0t and interconnection lines 0M. Each memory layer (e.g., 16A) has a plurality of first address lines (e.g., 2a, in the y-direction), a plurality of second address lines (e.g., 1a, in the x-direction), and a plurality of 3D-W memory elements (e.g., 1 aa). The memory layers 16A, 16B are coupled to the substrate 0 through contact via holes 1av, 3av, respectively.

3D-W memory cell 5aa contains a programming film 12 and a diode film 14. The programming film 12 may be an antifuse film (for 3D-OTP) or other multi-time programming film (for 3D-MTP). The diode membrane 14 has the following broad features: under the reading voltage, the resistance is small; when the applied voltage is less than the read voltage or in the opposite direction to the read voltage, the resistance is larger. The diode film may be a P-i-N diode or may be a metal oxide (e.g., TiO)₂) Diodes, etc.

Fig. 3B is a cross-sectional view of a 3D-P based memory processing unit 100 ij. The information stored in the 3D-P is recorded in a printing mode (printing method) in the factory production process. This information is permanently fixed and cannot be changed after shipment. The printing method may be photo-lithography (photo-lithography), nano-imprint method (nano-imprint), electron beam scanning exposure (e-beam lithography), DUV scanning exposure, laser scanning exposure (laser patterning), or the like. A common 3D-P has a three-dimensional mask-programmed read only memory (3D-MPROM), which is programmed to record data through a mask by photolithography. The 3D-P has a faster reading speed than the 3D-W, is suitable for storing fixed pattern libraries (such as an acoustic model library, a language model library and the like) and realizes high-performance pattern processing (such as natural language processing, real-time language translation and the like).

The 3D-P contains at least two memory elements 5aa, 6 aa. Memory cell 6aa is a low resistance memory cell and memory cell 5aa is a high resistance memory cell. The low resistance memory cell 6aa includes a diode film 14, and the high resistance memory cell 5aa includes a high resistance film 12 more than the low resistance memory cell 6 aa. As a simple example, the high-resistance film 12 may be a silicon dioxide film. The high-resistance film 12 is physically removed at the memory cell 6aa by imprinting in the process flow.

In the 3D-M (including 3D-W and 3D-P) of fig. 3A-3B, each storage layer contains a plurality of 3D-M arrays. A 3D-M array is a collection of all memory cells in a memory layer that share at least one address line. In one 3D-M array, all address lines are contiguous and do not share any address lines with a different 3D-M array. On the other hand, one 3D-M chip contains a plurality of 3D-M modules. Each 3D-M module contains all the storage layers in the 3D-M, the top storage layer of which contains only one 3D-M array, and the projection of the 3D-M array on the substrate determines the boundaries of the 3D-M module.

Fig. 4 discloses the structure of the memory processing unit 100ij from another point of view. The 3D-M array 170 of storage modes is stacked above the substrate 0 and the mode processing circuitry 180 of processing modes is located in the substrate 0 and is at least partially covered by the 3D-M array 170. Communication between the 3D-M array 170 and the pattern processing circuit 180 is achieved through a large bandwidth electrical connection 160 made up of a large number of contact via holes 1av, 3 av. Since the contact via holes 1av, 3av are numerous (may be tens of thousands) and have very short lengths (in the order of micrometers), the bandwidth of the on-chip electrical connections 160 is much higher than the communication bandwidth between chips.

Vertical integration of the 3D-M array 170 with the mode processing circuitry 180 brings about a number of advantages: since the 3D-M array 170 does not occupy substrate area 0, it can be integrated on the pattern processing circuit 180, which can increase memory capacity and reduce chip area. More importantly, since the 3D-M array 170 and the pattern processing circuit 180 are in the same chip 200 and in close proximity, a large bandwidth can be achieved between them. By employing massively parallel computations, distributed pattern processor 200 enables fast pattern processing for large pattern libraries.

Fig. 5A-5C disclose three specific implementations of a storage processing unit. The embodiment of fig. 5A corresponds to the memory processing unit 100ij of fig. 2A. The mode processing circuit 180 serves a 3D-M array 170 that is completely covered by the 3D-M array 170. In this embodiment, the 3D-M array 170 contains four peripheral circuits, including X decoders 15, 15 'and Y decoders (including readout circuits) 17, 17', with a mode processing circuit 180 located between the four peripheral circuits. In FIG. 5A, since the 3D-M array 170 is located above the substrate circuit 0K, not in the substrate circuit 0K, its projection onto the substrate 0 is represented here by a dashed line.

In the present embodiment, the mode processing circuit 180 is limited to four peripheral circuits, and the area thereof cannot exceed the area of the 3D-M array 170, so that the area thereof is small and the function thereof is limited. This embodiment is well suited to achieve simpler pattern processing (such as string matching and code matching). It is clear that more complex pattern processing (e.g., speech recognition, image recognition) requires larger circuitry, which requires more substrate area under the 3D-M array 170 in order to layout the pattern processing circuitry 180. Fig. 5B-5C disclose two mode processing circuits 180 having larger areas and more powerful functions.

The embodiment of fig. 5B corresponds to the memory processing unit 100ij of fig. 2B. In this embodiment, one mode processing circuit 180 services four 3D-M arrays 170A-170D. Each 3D-M array (e.g., 170A) has only two peripheral circuits (e.g., X-decoder 15A and Y-decoder 17A). Under these four 3D-M arrays 170A-170D, the substrate circuitry 0K can be laid out freely, forming a pattern processing circuit 180. It is apparent that the pattern processing circuit 180 in fig. 5B can be four times as large as that of fig. 5A, which can implement more complex pattern processing functions.

The embodiment of fig. 5C corresponds to the memory processing unit 100ij of fig. 2C. In this embodiment, one mode processing circuit 180 services eight 3D-M arrays 170A-170D and 170W-170Z. The eight 3D-M arrays are divided into two

groups

150A, 150B. Each set (e.g., 150A) includes four 3D-M arrays (e.g., 170A-170D). Under the first set 150A of four 3D-M arrays 170A-170D, the substrate circuitry can be laid out freely, forming a first mode processing circuit assembly A180A. Similarly, under the second set 150B of four 3D-M arrays 170W-170Z, the substrate circuitry can also be laid out freely, forming a second mode processing circuit component B180B. The first mode processing circuit component 180A and the second mode processing circuit component 180B constitute a mode processing circuit 180. In this embodiment, a gap (e.g., G) is left between adjacent peripheral circuits (e.g., between

adjacent X decoders

15A, 15C; between adjacent Y decoders 17A, 17B; between adjacent Y decoders 17C, 17D) to form routing channels 190Xa, 190Ya, 190Yb for communication between different mode

processing circuit elements

150A, 150B, or between different mode processing circuits. It is apparent that the mode processing circuit 180 in fig. 5C can be eight times larger than that of fig. 5A, which enables more complex mode processing functions.

In some embodiments of the present invention, mode processing circuit 180 need only perform a portion of the mode processing functions. For example, the pattern processing circuit 180 only needs to complete simple pattern processing (e.g., extraction and processing of simple features). The screened patterns are further sent to a more powerful external processor (e.g., CPU, GPU) via an output bus 120 to complete the final pattern processing. Since most patterns in the pattern library are filtered out by simple pattern processing, the output patterns occupy only a small portion of the pattern library, which can reduce the bandwidth pressure of the output bus 120.

In the distributed mode processor 200, the storage processing unit 100ij may adopt two mode processing modes — a processor-like mode and a memory-like mode. For the processor-like approach, the storage processing unit 100ij looks like a processor that can perform pattern processing on the external user data using its own search pattern library. Specifically, the 3D-M array 170 of storage processing units 100ij stores a search database; the input data 110 of the storage processing unit 100ij are user data (including user codes) which are generally generated in real time, such as network data packets; the storage processing unit 100ij performs pattern matching or pattern recognition on the user data 110 with the retrieval pattern library. Due to the large bandwidth connection 160 between the 3D-M array 170 and the pattern processor 180, this pattern processing is more efficient than conventional pattern processing, where the pattern library is retrieved and stored in a separate memory.

For the memory-like manner, the memory processing unit 100ij looks like a memory which is mainly used for storing user data and can perform mode processing by using a self-contained mode processing circuit to the outside. Specifically, user data is stored permanently in the 3D-M array 170 of storage processing units 100 ij; the input data 110 of the storage processing unit 100ij is retrieval mode data; the storage processing unit 100ij performs pattern matching or pattern recognition on the retrieval pattern data 110 and its user data. Note that the distributed mode processor chips 200 using the class memory method may be packaged into a memory card (e.g., SD card, TF card) and a solid state disk like a flash memory chip for storing massive user data (e.g., user data files). Since each memory processing unit 100ij in each distributed mode processor chip 200 is provided with a mode processing circuit 180, the mode processing circuit 180 only needs to process the data stored in the 3D-M array 170 in the memory processing unit 100 ij. Thus, regardless of the capacity of the memory card and the solid state drive, the time of pattern processing is close to the time of processing data stored in the 3D-M array 170 coupled thereto by the single pattern processing circuit 180. This great advantage is not imaginable for conventional processors.

In the class memory mode, the storage processing unit 100ij is the final storage device of the user data. This is in contrast to conventional processors that contain embedded memory: the embedded memory in a conventional processor only temporarily stores user data, and the final storage device of the user data is also an external memory (e.g., a hard disk, an optical disk, a magnetic tape, etc.). If user data is stored persistently in a legacy processor, the legacy processor can only serve these data, but not other data. That is, a large amount of user data requires the use of many processors. This approach is prohibitively expensive because conventional processors are very expensive. In contrast, in the memory processing unit 100ij of the present invention, the mode processing circuit 180 is integrated under the 3D-M array 170 and is formed simultaneously with the peripheral circuits (e.g., decoders) of the 3D-M array. Since the 3D-M originally has peripheral circuits formed thereon and the peripheral circuits occupy only a small area on the substrate 0 (see fig. 5A to 5C), most of the substrate area can be used to form the mode processing circuit 180, and the mode processing circuit 180 is free of charge for the 3D-M. Thus, a large number of nearly free mode processing circuits 180 may be formed on distributed mode processor chip 200, each mode processing circuit 180 serving a particular data (stored in 3D-M array 170 coupled thereto).

A brief introduction to the application of the distributed mode processor follows. As an example, distributed mode processor 200 is an anti-malware (anti-malware) processor that is used primarily for network security and computer disinfection. The network security can adopt a processor-like mode: the input data 110 to the distributed pattern processor 200 is a network packet, and the 3D-M array 170 stores a network specification library and a virus identification library, which are pattern matched by the pattern processing circuit 180. Computer sterilization can adopt a processor-like mode and a memory-like mode: for the processor-like approach, user data stored in the computer is transmitted as input data 110 to the distributed pattern processor 200, the 3D-M array 170 stores a library of virus signatures, and the pattern processing circuit 180 performs pattern matching on them; for memory-like approach, the virus identification is passed as input data 110 to the distributed pattern processor 200, the user data is stored in the 3D-M array 170, and the pattern processing circuit 180 performs pattern matching on them. In the processor-like approach, the 3D-M may be a 3D-OTP or a 3D-MTP that is used to store a library of network specifications and a library of virus identifications. In the class memory mode, the 3D-M is preferably a 3D-MTP, which stores a user database.

As another example, the distributed pattern processor 200 may be used for big data analysis (e.g., financial data analysis, e-commerce data analysis, bioinformatics). Big data analytics relates to unstructured data or semi-structured data. Conventional analysis methods using relational databases are ineffective for this purpose. Distributed mode processor 200 can improve big data analysis capabilities. To improve efficiency, it is preferable to use a memory-like approach: the big data is stored as an archive in the 3D-M array 170, and keywords for data analysis are sent as input pattern data 110 to the distributed pattern processor 200, which are pattern matched by the pattern processing circuit 180. In big data analysis, the 3D-M is preferably a 3D-MTP, which is used to store user data.

The distributed mode processor 200 may also be used for speech recognition and/or image recognition. In the recognition process, a class processor mode and a class memory mode may be employed. For the processor-like approach, user-generated speech/image data is provided as input data 110 to the distributed pattern processor 200, and the 3D-M array 170 stores various recognition model libraries (e.g., acoustic model libraries, language model libraries, image model libraries, etc.), which are then recognized by the pattern processor 180. For the memory-like approach, user-generated speech/image data is stored as an archive in the 3D-M array 170, and the speech signal or image signal to be searched is sent as input data 110 to the distributed pattern processor 200, after which the pattern processing circuit 180 performs recognition and search. In a processor-like approach, the 3D-M may be a 3D-P, 3D-OTP or 3D-MTP that stores a library of acoustic models, a library of language models, a library of image models, and the like. In the class memory approach, the 3D-M is preferably a 3D-MTP, which stores a voice/image archive.

It will be understood that changes in form and detail may be made therein without departing from the spirit and scope of the invention, and are not intended to impede the practice of the invention. The invention, therefore, is not to be restricted except in the spirit of the appended claims.

Claims

1. A memory (200) having an image recognition function, comprising:

an input bus (110) for transmitting an image model;

a plurality of memory processing units (100aa-100mn) coupled to the input bus (110), each memory processing unit (100ij) of the plurality of memory processing units (100aa-100mn) comprising a mode processing circuit (180) and at least one three-dimensional memory 3D-M array (170); wherein the 3D-M array (170) stores at least a portion of the image data; the pattern processing circuit (180) performs image recognition on the image data according to the image model;

the mode processing circuit (180) is positioned above a semiconductor substrate (0), and all transistors (0t) of the mode processing circuit (180) are positioned on the upper surface of the semiconductor substrate (0);

the 3D-M array (170) is located above the pattern processing circuitry (180), none of the memory cells (5aa) of the 3D-M array (170) being in contact with the semiconductor substrate (0);

the 3D-M array (170) and the mode processing circuitry (180) are coupled by a plurality of contact via holes (1av), the contact via holes (1av) being located entirely between the 3D-M array (170) and the semiconductor substrate (0) and not penetrating any semiconductor substrate;

the 3D-M array (170) at least partially overlaps the pattern processing circuit (180).

2. The memory (200) with image recognition function according to claim 1, further characterized in that: there is no semiconductor substrate between the mode processing circuitry (180) and the 3D-M array (170).

3. The memory (200) with image recognition function according to claim 1, further characterized in that: the 3D-M array (170) contains a plurality of vertically stacked memory cells (5 aa).

4. The memory (200) with image recognition function according to claim 1, further characterized in that: the 3D-M array is a three-dimensional writable storage 3D-W array.

5. The memory (200) with image recognition function according to claim 1, further characterized in that: the pattern processing circuit (180) performs only a partial pattern matching function; the memory (200) is coupled to an external processor via an output bus (120).

6. The memory (200) with image recognition function according to claim 1, further characterized in that: the memory (200) is part of a memory card or solid state drive.