CN110717575A

CN110717575A - Frame buffer free convolutional neural network system and method

Info

Publication number: CN110717575A
Application number: CN201810767312.3A
Authority: CN
Inventors: 杨得炜
Original assignee: Himax Technologies Ltd
Current assignee: Himax Technologies Ltd
Priority date: 2018-07-13
Filing date: 2018-07-13
Publication date: 2020-01-21
Anticipated expiration: 2038-07-13
Also published as: CN110717575B

Abstract

The invention relates to a convolutional neural network system and a convolutional neural network method without a frame buffer. The frame buffer-free convolutional neural network system comprises: a region-of-interest unit for extracting features to generate a region of interest of the input image frame; a convolutional neural network unit processing a region of interest of an input image frame to detect an object; and a tracking unit for comparing the features extracted at different times so that the convolutional neural network unit selectively processes the input image frame accordingly.

Description

Frame buffer free convolutional neural network system and method

Technical Field

The present invention relates to a Convolutional Neural Network (CNN), and more particularly, to a frame buffer-less convolutional neural network system.

Background

A Convolutional Neural Network (CNN) is one of artificial neural networks (artificial neural networks) and can be used for machine learning (machine learning). Convolutional neural networks are applicable to signal processing, such as image processing and computer vision.

Fig. 1 shows a block diagram of a conventional convolutional Neural Network 900, disclosed in "a Reconfigurable Streaming reliable Neural Network Accelerator for Internet of Things" (a Reconfigurable Streaming reliable Neural Network Accelerator for Internet of Things) proposed by Li Du et al, 8 months 2017, Institute of Electrical and Electronics Engineers (IEEE) circuit and systems society I: periodic papers, the contents of which are considered part of this specification. The convolutional neural network 900 includes a buffer bank 91 comprising a single-port Static Random Access Memory (SRAM) for storing intermediate data (intermediate data) and exchanging data with a frame buffer 92, the frame buffer 92 comprising a Dynamic Random Access Memory (DRAM), such as a double data rate synchronous dynamic random access memory (DDR DRAM), for storing an entire image frame for operation of the convolutional neural network. The buffer group 91 is divided into two parts: an input layer and an output layer. The convolutional neural network 900 includes a column buffer 93 for remapping (remap) the output of the buffer group 91 to a Convolution Unit (CU) engine array 94. Convolution unit engine array 94 includes a plurality of convolution units to perform highly parallel convolution operations. The convolution unit engine array 94 includes a pre-fetch controller 941 for periodically fetching parameters from a Direct Memory Access (DMA) controller (not shown) and updating the weights and bias values of the convolution unit engine array 94. Convolutional neural network 900 also includes an accumulation (accumulation) buffer 95, having draft (scratch pad) memory, for storing the partial convolution results of convolutional unit engine array 94. The accumulation buffer 95 contains a maximum pooling (max pool)951 to pool the output layer data. The convolutional neural network 900 includes an instruction decoder 96 for storing commands pre-stored in a frame buffer 92.

As shown in FIG. 1, in a conventional convolutional neural network system, a frame buffer comprises a Dynamic Random Access Memory (DRAM), such as a double data rate synchronous dynamic random access memory (DDR DRAM), for storing an entire image frame for convolutional neural network operation. For example, a frame with a resolution of 320x240 requires a frame buffer with a space of 320x240x8 bits. However, double data rate synchronous dynamic random access memory (DDRDRAM) is not suitable for low power applications, such as wearable or internet of things (IoT) devices. Therefore, it is desirable to provide a novel convolutional neural network system suitable for low power applications.

Disclosure of Invention

In view of the foregoing, it is an object of the embodiments of the present invention to provide a convolutional neural network system without a frame buffer. The present embodiment can use a simple system architecture to perform the convolutional neural network operation on high resolution image frames.

According to an embodiment of the present invention, a frame buffer-less convolutional neural network system includes a region of interest unit, a convolutional neural network unit, and a tracking unit. The region of interest unit extracts features to generate a region of interest of the input image frame. The convolutional neural network unit processes a region of interest of the input image frame to detect an object. The tracking unit compares the extracted features at different times so that the convolutional neural network unit can selectively process the input image frame accordingly.

Brief Description of Drawings

Fig. 1 shows a block diagram of a conventional convolutional neural network.

FIG. 2A is a block diagram of a convolutional neural network system without a frame buffer according to an embodiment of the present invention.

FIG. 2B shows a flow diagram of a method for a frame buffer free convolutional neural network, in accordance with an embodiment of the present invention.

Fig. 3 is a block diagram showing a detailed structure of the roi unit of fig. 2A.

Fig. 4A illustrates a decision diagram, which includes 4x6 blocks.

FIG. 4B illustrates another decision diagram, which is updated after FIG. 4A.

FIG. 5 is a block diagram illustrating a detailed structure of the register of FIG. 2A.

FIG. 6 is a block diagram of the convolutional neural network unit of FIG. 2A.

Detailed Description

Fig. 2A shows a block diagram of a frame buffer-less Convolutional Neural Network (CNN) system 100 according to an embodiment of the present invention, and fig. 2B shows a flowchart of a frame buffer-less Convolutional Neural Network (CNN) method 200 according to an embodiment of the present invention.

In the present embodiment, the frame-buffer-less convolutional neural network system (hereinafter referred to as the system) 100 may include a region of interest (ROI) unit 11 for generating a region of interest in an input image frame (step 21). Since the system 100 of the present embodiment does not include a frame buffer, the roi unit 11 can employ a scan line based technique and a block based mechanism to find the roi in the input image frame. The input image frame is divided into a plurality of image blocks arranged in a matrix form, such as 4 × 6 image blocks.

In the present embodiment, the roi unit 11 generates block-based features to determine whether to perform a Convolutional Neural Network (CNN) operation on each image block. Fig. 3 shows a block diagram of the region of interest unit 11 of fig. 2A. In the present embodiment, the region of interest unit 11 may include a feature extractor 111, for example, for extracting a shallow feature (shallow feature) from the input image frame. In one example, the feature extractor 111 generates (shallow) features of the tile based on a histogram (histogram) of the tile. In another example, the feature extractor 111 generates (shallow) features of the block based on frequency analysis.

The region of interest unit 11 may also include a classifier 112, such as a Support Vector Machine (SVM), for determining whether each block of the input image frame performs convolutional neural network operations. Thereby, a decision map (decision map)12 can be generated, which includes a plurality of blocks (which can be arranged in a matrix form) representing the input image frame. Fig. 4A illustrates the decision of fig. 12, which includes 4X6 blocks, where X indicates that the relevant block does not need to perform the convolutional neural network operation, C indicates that the relevant block needs to perform the convolutional neural network operation, and D indicates that the relevant block detects an object (e.g., a dog). Therefore, the region of interest can be determined and the convolutional neural network operation can be executed.

Referring to fig. 2B, the system 100 may comprise a register 13, such as a Static Random Access Memory (SRAM), for storing the (shallow) features generated by the feature extractor 111 (of the region of interest unit 11) (step 22). FIG. 5 is a block diagram illustrating a detailed structure of the register 13 of FIG. 2A. In the present embodiment, the register 13 may comprise two feature maps (feature maps), i.e., the first feature map 131A, for storing the features of the previous image frame (at the previous time t-1); and a second feature map 131B for storing features of the current frame (at the current time t). The buffer 13 may also include a sliding window 132, which may be 40x40x8 bits in size, for storing blocks of the input image frame.

Referring to fig. 2A, the system 100 of the present embodiment may include a Convolutional Neural Network (CNN) unit 14, which receives and processes the region of interest of the generated input image frame (region of interest unit 11) to detect the object (step 23). The convolutional neural network unit 14 of the present embodiment is only performed in the region of interest, rather than the entire input image frame as in the conventional system with a frame buffer.

Fig. 6 shows a block diagram of the convolutional neural network unit 14 of fig. 2A. The convolutional neural network unit 14 may include a convolution unit 141 including a plurality of convolution engines (convolution engines) for performing convolution operations. The convolutional neural network unit 14 may include an excitation (activation) unit 142 that may perform an excitation function when a default feature is detected. The convolutional neural network unit 14 may also include a pooling (pooling) unit 143 to perform down-sampling or pooling on the input image frame.

The system 100 of the present embodiment may include a tracking unit 15 for comparing the first feature map 131A (of the previous frame) with the second feature map 131B (of the current frame), and then updating the decision map 12 (step 24). The tracking unit 15 analyzes the content change between the first feature map 131A and the second feature map 131B. FIG. 4B illustrates another decision FIG. 12, updated after FIG. 4A. In this example, at the previous time, the blocks in rows 5-6 and column 3 have detected an object (D as indicated in FIG. 4A), but at the current time, the object disappears (X as indicated in the fourth B). Accordingly, the convolutional neural network unit 14 does not need to perform convolutional neural network operations for the blocks without feature changes. In other words, the convolutional neural network unit 14 selectively performs convolutional neural network operations for blocks having characteristic variations. Thus, the system 100 may substantially speed up operation.

The convolutional neural network operation of the above embodiments can be substantially reduced (and accelerated) compared to conventional convolutional neural network systems. Furthermore, since embodiments of the present invention do not require a frame buffer, embodiments may be better suited for low power applications, such as wearable or internet of things (IoT) devices. For image frames with a resolution of 320x240 and a (non-overlapping) sliding window size of 40x40, a conventional system with a frame buffer requires an 8x6 sliding window to perform convolutional neural network operations. In contrast, the system 100 of the present embodiment requires only a few (less than 10) sliding windows to perform convolutional neural network operations.

The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention; it is intended that all such equivalent changes and modifications be included within the scope of the following claims without departing from the spirit of the invention.

[ description of reference ]

100 convolutional neural network system without frame buffer

11 region of interest unit

111 characteristic extractor

112 classifier

12 decision diagram

13 buffer

131A first characteristic diagram

131B second characteristic diagram

132 sliding window

14 convolutional neural network unit

141 convolution unit

142 excitation unit

143 pooling unit

15 tracking unit

200 method for frame buffer free convolutional neural network

21 generating a region of interest in an input image frame

22 store characteristic feature maps

23 processing the region of interest to detect the object

24 comparing features and performing convolutional neural network operations on blocks with feature variations

900 convolutional neural network

91 buffer group

92 frame buffer

93-row buffer

94 convolution unit engine array

941 prefetch controller

95 accumulation buffer

951 maximum pooling

96 instruction decoder

Claims

1. A frame buffer free convolutional neural network system, comprising:

a region-of-interest unit for extracting features to generate a region of interest of the input image frame;

a convolutional neural network unit processing a region of interest of the input image frame to detect an object; and

a tracking unit to compare the features extracted at different times so that the convolutional neural network unit selectively processes the input image frame accordingly.

2. The frame buffer free convolutional neural network system of claim 1, wherein the region of interest unit employs a scan line based technique and a block based mechanism for finding the region of interest in the input image frame, wherein the input image frame is divided into a plurality of image blocks.

3. The frame buffer-less convolutional neural network system of claim 2, wherein the region-of-interest unit generates block-based features to determine whether to perform convolutional neural network operations for each image block.

4. The framebuffer-less convolutional neural network system of claim 2, wherein the region-of-interest unit comprises:

a feature extractor for extracting the feature from the input image frame; and

the classifier determines whether each image block executes the operation of the convolutional neural network, thereby generating a determination map to determine the region of interest.

5. The frame buffer-less convolutional neural network system of claim 4, wherein the feature extractor generates shallow features of the image blocks according to a block-based histogram or frequency analysis.

6. The frame buffer-less convolutional neural network system of claim 4, further comprising a register for storing the feature.

7. The frame buffer-less convolutional neural network system of claim 6, wherein the buffer comprises: the first characteristic picture is used for storing the characteristics of the previous image frame; and a second feature map for storing the features of the current image frame.

8. The frame buffer-less convolutional neural network system of claim 6, wherein the register comprises a sliding window for storing blocks of the input image frame.

9. The frame-buffer-less convolutional neural network system of claim 7, wherein the tracking unit compares the first feature map with the second feature map to update the decision map.

10. The framebuffer-less convolutional neural network system of claim 1, wherein the convolutional neural network unit comprises:

a convolution unit including a plurality of convolution engines for performing convolution operations on the region of interest;

an excitation unit that performs an excitation function when the default feature is detected; and

a pooling unit for performing a down-sampling of the input image frame.

11. A method for a frame buffer free convolutional neural network, comprising:

extracting features to generate a region of interest of the input image frame;

executing a convolutional neural network operation on the region of interest of the input image frame to detect an object; and

the features extracted at different times are compared to selectively process the input image frame.

12. The method of claim 11, wherein the region of interest is generated using a scan line based technique and a block based mechanism, wherein the input image frame is divided into a plurality of image blocks.

13. The method for frame buffer free convolutional neural network of claim 12, wherein the step of generating the region of interest comprises:

block-based features are generated to determine whether each image block performs convolutional neural network operations.

14. The method for frame buffer free convolutional neural network of claim 12, wherein the step of generating the region of interest comprises:

extracting the feature from the input image frame; and

a classification method is used to determine whether each image block executes the operation of the convolutional neural network, so as to generate a determination map, and the interested region is determined accordingly.

15. The method for frame buffer-less convolutional neural network of claim 14, wherein the step of extracting the feature comprises:

shallow features of the image block are generated based on a block-based histogram or frequency analysis.

16. The method for frame buffer free convolutional neural network of claim 14, further comprising the step of staging the feature temporarily.

17. The method for frame buffer-less convolutional neural network of claim 16, wherein the step of staging the feature comprises:

generating a first feature map for storing the features of the previous image frame; and generating a second feature map for storing the features of the current image frame.

18. The method for frame buffer-less convolutional neural network of claim 16, wherein the step of staging the feature comprises:

generating a sliding window for storing the blocks of the input image frame.

19. The method for frame buffer-less convolutional neural network of claim 17, wherein the step of comparing the features comprises:

and comparing the first characteristic diagram with the second characteristic diagram to update the decision diagram.

20. The method for frame buffer free convolutional neural network of claim 11, wherein the step of performing the convolutional neural network operation comprises:

using a plurality of convolution engines to perform convolution operations on the region of interest;

executing an incentive function when a default feature is detected; and

a down-sampling rate is performed on the input image frame.