TW202001692A - Framebuffer-less system and method of convolutional neural network - Google Patents

Abstract

A framebuffer-less system of convolutional neural network (CNN) includes a region of interest (ROI) unit that extracts features, according to which a region of interest in an input image frame is generated; a convolutional neural network (CNN) unit that processes the region of interest of the input image frame to detect an object; and a tracking unit that compares the features extracted at different times, according to which the CNN unit selectively processes the input image frame.

Description

Convolutional neural network system and method without frame buffer

The invention relates to a convolutional neural network (CNN), in particular to a convolutional neural network system without frame buffer.

Convolutional neural network (CNN) is a type of artificial neural network (artificial neural network), which can be used for machine learning. Convolutional neural networks can be applied to signal processing, such as image processing and computer vision.

The first figure shows a block diagram of a traditional convolutional neural network 900, revealed in "Reconfigurable Streaming Deep Convolutional Neural Accelerator for Resetable Streaming for Internet of Things" proposed by Li Du et al. Network Accelerator for Internet of Things”, August 2017, IEEE Transactions on Circuits and Systems I: Periodical paper, the content of which is considered part of this manual. The convolutional neural network 900 includes a buffer bank 91, which includes a static random access memory (SRAM) for storing intermediate data and a frame buffer 92 To exchange data, the frame buffer 92 includes dynamic random access memory (DRAM), such as double data rate synchronous dynamic random access memory (DDR DRAM), for storing the entire image frame for the convolutional neural network Road operation. The buffer group 91 is divided into two parts: the input layer and the output layer. The convolutional neural network 900 includes a column buffer 93 to remap the output of the buffer group 91 to a convolution unit (CU) engine array 94. The convolution unit engine array 94 includes complex convolution units to perform highly parallel convolution operations. The convolution unit engine array 94 includes a pre-fetch controller 941 for periodically obtaining parameters from a direct memory access (DMA) controller (not shown) and updating the convolution unit engine array The weight and bias value of 94. The convolutional neural network 900 also includes an accumulation buffer 95 with scratchpad memory for storing partial convolution results of the convolution unit engine array 94. The accumulation buffer 95 includes a max pool 951 to pool output layer data. The convolutional neural network 900 includes an instruction decoder 96 for storing commands pre-stored in the frame buffer 92.

As shown in the first picture of the traditional convolutional neural network system, the frame buffer contains dynamic random access memory (DRAM), such as double data rate synchronous dynamic random access memory (DDR DRAM), for storage The entire image frame is used for convolutional neural network operation. For example, an image frame with a resolution of 320x240 requires a frame buffer with a space of 320x240x8 bits. However, double data rate synchronous dynamic random access memory (DDR DRAM) is not suitable for low-power applications, such as wearable or Internet of Things (IoT) devices. Therefore, there is an urgent need to propose a novel convolutional neural network system for low power applications.

In view of the above, one of the objectives of the embodiments of the present invention is to provide a convolutional neural network system without frame buffer. In this embodiment, a simple system architecture can be used to perform a convolutional neural network operation on a high-resolution image frame.

According to an embodiment of the present invention, a frameless buffer convolutional neural network system includes a region of interest unit, a convolutional neural network unit, and a tracking unit. The region of interest unit extracts the features, and generates the region of interest based on the input image frame. The convolutional neural network unit processes the region of interest in the input image frame to detect objects. The tracking unit compares the features extracted at different times, so that the convolutional neural network unit selectively processes the input image frame accordingly.

Figure 2A shows a block diagram of a framebuffer-less convolutional neural network (CNN) system 100 according to an embodiment of the present invention, and Figure 2B shows a frameless buffer-less convolutional neural network (CNN) system 100. Flow chart of the CNN method 200.

In this embodiment, the frame buffer-free convolutional neural network system (hereinafter referred to as the system) 100 may include a region of interest (ROI) unit 11 for generating interest in the input image frame Area (step 21). Since the system 100 of this embodiment does not include a frame buffer, the region of interest unit 11 may use a scan line-based technology and a block-based mechanism to find the region of interest in the input image frame. Among them, the input image frame is divided into a plurality of image blocks, arranged in a matrix form, such as 4x6 image blocks.

In this embodiment, the region-of-interest unit 11 generates block-based features to determine whether each image block performs a convolutional neural network (CNN) operation. The third diagram shows a detailed block diagram of the region of interest unit 11 of the second diagram A. In this embodiment, the region of interest unit 11 may include a feature extractor 111, for example, to extract shallow features from the input image frame. In one example, the feature extractor 111 generates (shallow) features of a block based on a block-based histogram. In another example, the feature extractor 111 generates a (shallow) feature of the block according to frequency analysis.

The region of interest unit 11 may further include a classifier 112, such as a support vector machine (SVM), to determine whether each block of the input image frame performs a convolutional neural network operation. In this way, a decision map 12 can be generated, which includes a plurality of blocks representing the input image frame (which can be arranged in a matrix form). The fourth diagram A illustrates the decision diagram 12, which includes 4x6 blocks, where X indicates that the relevant block does not need to perform a convolutional neural network operation, C indicates that the related block needs to perform a convolutional neural network operation, and D indicates the relevant area The block detected an object (such as a dog). With this, the region of interest can be determined and convolutional neural network operations can be performed.

Referring to FIG. 2B, the system 100 may include a register 13, such as a static random access memory (SRAM), for storing (shallow) features generated by the feature extractor 111 (of the region of interest 11) (step twenty two). The fifth figure shows a detailed block diagram of the register 13 of the second figure A. In this embodiment, the register 13 may include two feature maps, that is, the first feature map 131A, for storing the features of the previous image frame (at the previous time t-1); And the second feature map 131B is used to store the features of the current image frame (at the current time t). The register 13 may further include a sliding window 132, which may be 40x40x8 bits in size, and is used to store a block of the input image frame.

Referring to FIG. 2A, the system 100 of this embodiment may include a convolutional neural network (CNN) unit 14 that receives and processes the region of interest of the input image frame generated by (region of interest unit 11) to detect Measure the object (step 23). Among them, the convolutional neural network unit 14 of this embodiment is only executed in the region of interest, rather than the conventional system with a frame buffer, which is executed on the entire input image frame.

The sixth figure shows a detailed block diagram of the convolutional neural network unit 14 of the second figure A. The convolutional neural network unit 14 may include a convolution unit 141, which includes a complex convolution engine for performing convolution operations. The convolutional neural network unit 14 may include an activation unit 142, which may perform an activation function when a predetermined feature is detected. The convolutional neural network unit 14 may further include a pooling unit 143 for performing down-sampling or pooling on the input image frame.

The system 100 of this embodiment may include a tracking unit 15 for comparing the first feature map 131A (of the previous image frame) with the second feature map 131B (of the current image frame), and then updating the decision map 12 (step 24 ). The tracking unit 15 analyzes the content change between the first feature map 131A and the second feature map 131B. The fourth diagram B illustrates another decision diagram 12, which is updated after the fourth diagram A. In this example, at the previous time, an object was detected in the blocks located in rows 5-6 and column 3 (as indicated by D in Figure 4A), but at the current time, the object disappeared (as X marked in Figure 4B). According to this, the convolutional neural network unit 14 does not need to perform convolutional neural network operations on blocks with no feature changes. In other words, the convolutional neural network unit 14 selectively performs convolutional neural network operations on blocks with characteristic changes. Therefore, the system 100 can greatly speed up operations.

Compared with the traditional convolutional neural network system, the operation of the convolutional neural network in the above embodiment can be greatly reduced (and accelerated). In addition, since the embodiment of the present invention does not require a frame buffer, this embodiment can be preferably applied to low-power applications, such as wearable or Internet of Things (IoT) devices. For image frames with a resolution of 320x240 and a (non-overlapping) sliding window size of 40x40, conventional systems with frame buffers require 8x6 sliding windows to perform convolutional neural network operations. Conversely, the system 100 of this embodiment requires only a few (less than 10) sliding windows to perform convolutional neural network operations.

The above are only the preferred embodiments of the present invention and are not intended to limit the scope of the patent application of the present invention; all other equivalent changes or modifications made without departing from the spirit of the invention should be included in the following Within the scope of patent application.

100‧‧‧Convolutional neural network system without frame buffer 11‧‧‧region of interest unit 111‧‧‧feature extractor 112‧‧‧ classifier 12‧‧‧decision diagram 13‧‧‧ scratch 131A‧‧‧first feature map 131B‧‧‧second feature map 132‧‧‧sliding window 14‧‧‧convolutional neural network unit 141‧‧‧convolution unit 142‧‧‧ excitation unit 143‧‧‧ pool Conversion unit 15‧‧‧Tracking unit 200‧‧‧Convolutional neural network method without frame buffer 21‧‧‧Generate region of interest 22 in input image frame 22‧‧‧Save feature in feature map 23‧‧ ‧Process the region of interest to detect objects 24‧‧‧Compare features and perform convolutional neural network operations in blocks with feature changes 900‧‧‧Convolutional neural network 91‧‧‧Buffer group 92‧‧‧ Frame buffer 93 ‧ ‧ ‧ line buffer 94 ‧ ‧ ‧ convolution unit engine array 941 ‧ ‧ ‧ prefetch controller 95 ‧ ‧ ‧ cumulative buffer 951 ‧ ‧ ‧ maximum pooling 96 ‧ ‧ ‧ instruction decoder

The first figure shows the block diagram of a traditional convolutional neural network. Figure 2A shows a block diagram of a frameless buffer convolutional neural network system according to an embodiment of the present invention. FIG. 2B shows a flowchart of a frameless buffer convolution neural network method according to an embodiment of the present invention. The third diagram shows a detailed block diagram of the region of interest unit in the second diagram A. The fourth diagram A illustrates the decision diagram, which includes 4x6 blocks. The fourth picture B illustrates another decision picture, which is updated after the fourth picture A. The fifth figure shows a detailed block diagram of the register in the second figure A. The sixth figure shows a detailed block diagram of the convolutional neural network unit of the second figure A.

100‧‧‧Convolutional neural network system without frame buffer

11‧‧‧ Region of Interest Unit

12‧‧‧Decision map

13‧‧‧register

14‧‧‧Convolutional Neural Network Unit

15‧‧‧Tracking unit

Claims (20)

A convolutional neural network system without frame buffer includes: a region-of-interest unit for extracting features to generate the region of interest of the input image frame; a convolutional neural network unit for processing the input The region of interest of the image frame is used to detect objects; and a tracking unit compares the features extracted at different times, so that the convolutional neural network unit selectively processes the input image frame accordingly. The convolutional neural network system without frame buffer according to item 1 of the patent application scope, wherein the region of interest unit uses a scan line-based technology and a block-based mechanism for the input image frame Find the region of interest, where the input image frame is divided into plural image blocks. The convolutional neural network system without frame buffer according to item 2 of the patent application scope, wherein the region-of-interest unit generates block-based features, thereby determining whether each image block performs a convolutional neural network operating. The convolutional neural network system without frame buffer according to item 2 of the patent application scope, wherein the region of interest unit includes: a feature extractor that extracts the feature from the input image frame; and a classifier, It is determined whether each image block performs a convolutional neural network operation, so a decision map is generated and the region of interest is determined accordingly. The convolutional neural network system without frame buffer according to item 4 of the patent application scope, wherein the feature extractor generates shallow features of the image block based on block-based histogram or frequency analysis. The convolutional neural network system without frame buffer according to item 4 of the patent application scope further includes a temporary memory for storing the feature. The convolutional neural network system without frame buffer according to item 6 of the patent application scope, wherein the temporary memory includes a first feature map for storing the features of the previous image frame; and a second feature map, Used to store the characteristics of the current image frame. The convolutional neural network system without frame buffer according to item 6 of the patent application scope, wherein the temporary memory includes a sliding window for storing a block of the input image frame. According to the convolutional neural network system without frame buffer described in item 7 of the patent application scope, wherein the tracking unit compares the first feature map and the second feature map to update the decision map accordingly. The convolutional neural network system without frame buffer according to item 1 of the patent application scope, wherein the convolutional neural network unit includes: a convolution unit, including a complex convolution engine, used to perform convolution operations in The region of interest; an excitation unit, which performs an excitation function when a preset feature is detected; and a pooling unit, which is used to perform a reduced sampling rate on the input image frame. A convolutional neural network method without frame buffer includes: extracting features and generating an interest region of an input image frame based thereon; performing a convolutional neural network operation on the interest region of the input image frame to Detecting objects; and comparing features extracted at different times to selectively process the input image frame. According to the convolutional neural network method without frame buffer described in item 11 of the patent application scope, wherein the generation of the region of interest uses a scanning line-based technology and a block-based mechanism, wherein the input image frame Divide into multiple image blocks. According to the convolutional neural network method without frame buffer described in item 12 of the patent scope, the step of generating the region of interest includes: generating block-based features, based on which it is determined whether each image block performs volume Integrate neural network operation. The convolutional neural network method without frame buffer according to item 12 of the patent application scope, wherein the step of generating the region of interest includes: extracting the feature from the input image frame; and determining each image by classification method Whether the block performs a convolutional neural network operation, so a decision graph is generated to determine the region of interest. The convolutional neural network method without frame buffer according to item 14 of the patent application scope, wherein the step of extracting the feature includes: generating shallow features of the image block based on block-based histogram or frequency analysis. The convolutional neural network method without frame buffer according to item 14 of the patent application scope further includes a step to temporarily store the feature. The convolutional neural network method without frame buffer according to item 16 of the patent application scope, wherein the step of temporarily storing the feature includes: generating a first feature map for storing the feature of the previous image frame; and generating The second feature map is used to store the features of the current image frame. According to the convolutional neural network method without frame buffer described in item 16 of the patent scope, the step of temporarily storing the feature includes: generating a sliding window for storing a block of the input image frame. The convolutional neural network method without frame buffer according to item 17 of the patent application scope, wherein the step of comparing the features includes: comparing the first feature map and the second feature map, and updating the decision map accordingly. The convolutional neural network method without frame buffer according to item 11 of the patent application scope, wherein the step of performing the operation of the convolutional neural network includes: using a complex convolution engine to perform a convolution operation on the sense Area of interest; when a preset feature is detected, an excitation function is performed; and a reduced sampling rate is performed on the input image frame.

Images