US20200065250A1 - Feature map caching method of convolutional neural network and system thereof - Google Patents
Feature map caching method of convolutional neural network and system thereof Download PDFInfo
- Publication number
- US20200065250A1 US20200065250A1 US16/543,645 US201916543645A US2020065250A1 US 20200065250 A1 US20200065250 A1 US 20200065250A1 US 201916543645 A US201916543645 A US 201916543645A US 2020065250 A1 US2020065250 A1 US 2020065250A1
- Authority
- US
- United States
- Prior art keywords
- cache
- layer operation
- tensors
- tensor
- neural network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
- G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
- G06F12/0866—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches for peripheral storage systems, e.g. disk cache
- G06F12/0871—Allocation or management of cache space
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F5/00—Methods or arrangements for data conversion without changing the order or content of the data handled
- G06F5/06—Methods or arrangements for data conversion without changing the order or content of the data handled for changing the speed of data flow, i.e. speed regularising or timing, e.g. delay lines, FIFO buffers; over- or underrun control therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
- G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
- G06F12/0875—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches with dedicated cache, e.g. instruction or stack
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
- G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
- G06F12/0893—Caches characterised by their organisation or structure
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
- G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2212/00—Indexing scheme relating to accessing, addressing or allocation within memory systems or architectures
- G06F2212/10—Providing a specific technical effect
- G06F2212/1028—Power efficiency
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2212/00—Indexing scheme relating to accessing, addressing or allocation within memory systems or architectures
- G06F2212/45—Caching of specific data in cache memory
- G06F2212/454—Vector or matrix data
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Definitions
- the present disclosure relates to a feature map caching method of convolutional neural network and system thereof. More particularly, the present disclosure relates to feature map caching method of convolutional neural network and system thereof which according to a convolutional neural network connection list.
- a first in first out (FIFO) flushing algorithm and a last recently used (LRU) flushing algorithm may flush a tensor in a cache which will be used in the future, and stored the tensor into a DRAM. Therefore, DRAM access times are increased and cause the DRAM bandwidth power consumption of the system to be increased.
- FIFO first in first out
- LRU last recently used
- a feature map caching method of a convolutional neural network includes a connection analyzing step and a plurality of layer operation steps.
- the connection analyzing step is for analyzing a network to establish a convolutional neural network connection list.
- the convolutional neural network connection list includes a plurality of tensors and a plurality of layer operation coefficients.
- Each of the tensors includes a plurality of feature maps.
- Each of the layer operation coefficients includes a step index, at least one input operand label and an output operand label.
- the step index as a processing order for the layer operation step.
- the input operand label is for selecting a part of the tensors, and the output operand label is for assigning one of the tensors.
- At least one of the layer operation steps is for flushing at least one of the tensors in a cache according to a distance between the at least one of the layer operation steps and a future layer operation step of the layer operation steps with the at least one input operand label that matches the at least one of the tensors, and the at least one layer operation steps is for performing a convolutional operation step according to the layer operation coefficients.
- the distance is calculated according to the convolutional neural network connection list.
- a system of the feature map caching method of the convolutional neural network includes a lookup controller, a Central Processing Unit (CPU), the cache and a Dynamic Random Access Memory (DRAM).
- the lookup controller is for performing the connection analyzing step so as to establish the convolutional neural network connection list.
- the Central Processing Unit is for performing the layer operation steps.
- the cache is electronically connected to the Central Processing Unit and performing the layer operation steps.
- the Dynamic Random Access Memory is electronically connected to the cache and performing a Dynamic Random Access Memory write-back step.
- FIG. 1 shows a flow chart of a feature map caching method of a convolutional neural network according to one embodiment of the present disclosure.
- FIG. 2 shows a flow chart of one of the layer operation steps of FIG. 1 .
- FIG. 3 shows a schematic diagram of one example of the feature map caching method of the convolutional neural network of FIG. 1 .
- FIG. 4 shows a schematic diagram of another example of the feature map caching method of the convolutional neural network of FIG. 1 .
- FIG. 5 shows a block diagram of a system of the feature map caching method of the convolutional neural network of FIG. 1 .
- FIG. 1 shows a flow chart of a feature map caching method s 100 of a convolutional neural network according to one embodiment of the present disclosure.
- FIG. 2 shows a flow chart of one of the layer operation steps s 120 of FIG. 1 .
- FIG. 3 shows a schematic diagram of one example of the feature map caching method s 100 of the convolutional neural network of FIG. 1 .
- the feature map caching method s 100 of the convolutional neural network includes a connection analyzing step s 110 and a plurality of layer operation steps s 120 .
- connection analyzing step s 110 is for analyzing a network to establish a convolutional neural network connection list.
- the convolutional neural network connection list includes a plurality of tensors and a plurality of layer operation coefficients.
- Each of the tensors includes a plurality of feature maps, and a number of the feature maps are a channel number of the tensor.
- Each of the layer operation coefficients includes a step index, an operation type and at least one input operand label and an output operand label.
- the step index as a processing order for a layer operation step.
- the operation type indicating a layer operation of the layer operation step is a convolution, an activation, or a batch normalization.
- the at least one input operand label is for selecting a part of the tensors as the input, and the output operand label is for assigning one of the tensors as the output.
- At least one of the layer operation steps s 120 is for flushing at least one of the tensors in a cache 430 (shown in FIG. 5 ) according to a distance between the at least one of the layer operation steps s 120 and a future layer operation step of the layer operation steps s 120 with the at least one input operand label that matches the at least one of the tensors, and a part of the layer operation steps s 120 is for performing a convolutional operation step s 126 according to the layer operation coefficients. The distance is calculated according to the convolutional neural network connection list.
- Table 1 lists a convolutional neural network connection list and a conventional cache operation script.
- Table 2 lists a convolutional neural network connection list and a cache operation script of the feature map caching method s 100 of the convolutional neural network corresponding to the FIG. 3 .
- a number of DRAM access times of the conventional method is nine, and a number of DRAM access times of the feature map caching method s 100 of the convolutional neural network is one. Therefore, the feature map caching method s 100 of the convolutional neural network can optimize the DRAM access times so as to reduce the DRAM bandwidth power consumption because the DRAM access times will consume most of the DRAM bandwidth power consumption in the system.
- the at least one of the layer operation steps s 120 flushes at least one of the tensors in the cache 430 according to the distance.
- the at least one of the tensors which needs to be flushed has a largest distance corresponding to each of the tensors in the cache 430 . Therefore, the DRAM access times can be optimized so as to reduce the DRAM bandwidth power consumption.
- connection analyzing step s 110 of the feature map caching method s 100 of the convolutional neural network analyzes the network so as to establish the convolutional neural network connection list of Table 2.
- the capacity of the cache 430 can store three tensors.
- a number of the tensors storing in the cache 430 is depended on the capacity of the cache 430 and the number of the feature maps.
- Each of the layer operation steps s 120 includes a cache checking step s 121 .
- the cache checking step s 121 is for checking the tensors in the cache 430 and a number of the tensors in the cache 430 .
- a content of each of the layer operation steps s 120 is denoted that the tensors storing in the cache 430 after performing each of the layer operation steps s 120 .
- a performing result of the cache checking step s 121 of the layer operation step 1 represents none tensor storing in the cache 430 .
- a performing result of the cache checking step s 121 of the layer operation step 2 is two tensors storing in the cache 430 and respective to the tensor A and the tensor B.
- the performing result of the cache checking step s 121 of the layer operation step 2 can correspond to the content of the cache operation script of the layer operation step 1 in Table 2 because the layer operation steps s 120 performs the cache checking step s 121 of the layer operation step 2 after performing the layer operation step 1 .
- the performing result of the cache checking step s 121 of the layer operation step 3 is corresponded to the content of the cache operation script of the layer operation step 2 in Table 2, and the number of the tensors in the cache 430 is three.
- a performing result of the cache checking step s 121 of each of a layer operation step 4 to a layer operation step 6 is same as above, and will not be described again herein.
- Each of the layer operation steps s 120 further includes a distance computing step s 122 .
- the distance computing step s 122 is for calculating the distance corresponding to each of the tensors in the cache 430 according to the convolutional neural network connection list and the cache operation script.
- the distance corresponding to each of the tensors in the cache 430 is corresponded by formula (1).
- dist t ( i ) ⁇ N , if all the j with i ⁇ j ⁇ N ⁇ ⁇ such that ⁇ ⁇ t ⁇ Tin j min i ⁇ j ⁇ N , t ⁇ Tin j ⁇ ( j - i ) , otherwise ⁇ . ( 1 )
- i is a step index of a layer operation step i and i is from 1 to N
- j is a step index of a layer operation step j
- N is a number of the layer operation steps
- t is one of the tensors in the cache
- Tin j is a set of the at least one input operand label of the layer operation step j.
- the distance computing step s 122 of each of the layer operation steps s 120 via the formula (1) to compute each of the distance corresponding to each of the tensors in the cache 430 so as to perform the flushing step s 123 of each of the layer operation steps s 120 .
- the step index of the layer operation step 1 is 1.
- the distance computing step s 122 of the layer operation step 1 is not performed because there is none tensor in the cache 430 .
- the step index of the layer operation step 3 is 3.
- the distance computing step s 122 of the layer operation step 3 calculates the distance corresponding to each of the tensor A, the tensor B and the tensor C because of the performing result of the cache checking step s 121 of the layer operation step 3 .
- the distance corresponding to each of the tensor A, the tensor B and the tensor C is calculated according to the convolutional neural network connection list of Table 2.
- the tensor A is an input operand label of the layer operation step 4 , and it denotes the future layer operation step of the tensor A is the layer operation step 4 and j corresponding to the tensor A is four.
- the distance computing step s 122 of the layer operation step 4 to layer operation step 6 is same as above, and will not be described again herein.
- Each of the layer operation steps s 120 further includes a flushing step s 123 .
- the flushing step s 123 is for flushing the at least one of the tensors in the cache 430 according to the distance corresponding to each of the tensors in the cache 430 and the convolutional neural network connection list.
- a distance corresponding to each of the at least one of the tensors in the cache 430 is greater than or equal to a distance corresponding to each of the tensors which is not flushed in the cache 430 .
- the flushing step s 123 flushes the at least one of the tensors in the cache 430 so as to avoid a capacity of the cache 430 exceeds after performing each of the layer operation step s 120 according to the convolutional neural network connection list.
- the layer operation step 1 and the layer operation step 2 do not perform the flushing step s 123 because the capacity of the cache 430 is not exceed after performing the layer operation step 1 and the layer operation step 2 .
- the cache checking step s 121 of the layer operation step 3 there are the tensor A, the tensor B and the tensor C in the cache 430 .
- the flushing step s 123 of layer operation step 3 flushes at least one of the tensors in the cache so as to avoid the capacity of the cache 430 exceed.
- the tensor B is flushed in the flushing step s 123 of the layer operation step 3 because the distance corresponding to the tensor B is greater than each of the distance corresponding to the tensor A and the tensor C.
- the flushing step s 123 of each of the layer operation step 4 to layer operation step 6 is same as above, and will not be described again herein.
- the at least one of the tensors which needs to be flushed in the cache 430 is selected by a first in first out (FIFO) flushing method.
- Each of the layer operation steps s 120 further includes a Dynamic Random Access Memory (DRAM) write-back step s 124 .
- the DRAM write-back step s 124 is for determining whether the at least one of the tensors in the cache 430 needs to be written back to a Dynamic Random Access Memory (DRAM) 440 (shown in FIG. 5 ) according to the convolutional neural network connection list.
- DRAM Dynamic Random Access Memory
- the feature map caching method s 100 of the convolutional neural network can optimize the DRAM access times so as to reduce the DRAM bandwidth power consumption.
- Each of the layer operation steps s 120 further includes a reading step s 125 .
- the reading step s 125 is for reading at least another one of the tensors from the DRAM 440 and storing into the cache 430 according to the convolutional neural network connection list.
- the tensor A is the input operand label of the layer operation step 1 .
- the performing result of the cache checking step s 121 of the layer operation step 1 the tensor A is not stored in the cache 430 .
- the tensor A is stored in the DRAM 440 .
- the reading step s 125 of the layer operation step 1 read the tensor A from the DRAM 440 and stores into the cache 430 so as to perform the convolutional operation step s 126 of the layer operation step 1 because the tensor A is the input operand label of the convolutional operation step s 126 of the layer operation step 1 .
- Each of the layer operation coefficients further includes a kernel.
- the kernel and the at least one input operand label are configured to perform the convolutional operation.
- the convolutional operation step s 126 is for calculating the output operand label by the convolutional operation with the at least one input operand label and the kernel.
- the input operand labels concatenate to each other along a channel dimension, and then the convolutional operation is performed. Please refer to FIG. 3 and Table 2, the convolutional operation step s 126 of layer operation step 1 , the tensor B is calculated by the convolutional operation with the tensor A and the kernel of the layer operation step 1 .
- a number of the at least one input operand label of the convolutional operation step s 126 is two.
- the convolutional operation step s 126 of layer operation step 4 concatenates the tensor A and the tensor D to each other along the channel dimension and then performs the convolutional operation with the kernel of the layer operation step 4 so as to calculate the tensor E.
- the convolutional operation step s 126 of the layer operation step 2 , the layer operation step 3 , the layer operation step 5 and the layer operation step 6 are same as above, and will not be described again herein.
- Each of the layer operation steps s 120 further includes a cache storing step s 127 .
- the cache storing step s 127 is for storing the output operand label into the cache 430 .
- the cache storing step s 127 of the layer operation step 1 stores the tensor B into the cache 430 . Therefore, the content of the cache 430 of the layer operation step 1 is the tensor A and the tensor B.
- the cache storing step s 127 of the layer operation step 2 to the layer operation step 6 are same as above, and will not be described again herein.
- the feature map caching method s 100 of the convolutional neural network via the connection analyzing step s 110 and the layer operation steps s 120 so as to optimize the DRAM access times and reduce the DRAM bandwidth power consumption.
- FIG. 4 shows a schematic diagram of another example of the feature map caching method s 100 of the convolutional neural network of FIG. 1 .
- Table 3 lists a convolutional neural network connection list and a cache operation script of FIG. 4 .
- the connection analyzing step s 110 of the feature map caching method s 100 of the convolutional neural network analyzes the network so as to establish the convolutional neural network connection list of Table 3.
- the capacity of the cache 430 is three tensors. However, the capacity of the cache 430 can be any positive integer, and the present disclosure is not limited thereto.
- the performing result of the cache checking step s 121 represents none tensor stored in the cache 430 . Therefore, the layer operation step 1 do not perform the distance computing step s 122 , the flushing step s 123 and the DRAM write-back step s 124 because there represents none tensor in the cache 430 .
- the reading step s 125 of the layer operation step 1 read the tensor A from the DRAM 440 and stores into the cache 430 so as to perform the convolutional operation step s 126 of the layer operation step 1 .
- the convolutional operation step s 126 of the layer operation step 1 calculates the tensor B by the convolution operation with the tensor A and the kernel of the layer operation step 1 .
- the cache storing step s 127 of the layer operation step 1 stores the tensor B into the cache 430 .
- the content of the layer operation step 1 is two tensors in the cache 430 and respective to the tensor A and the tensor B.
- the step index of the layer operation step 2 is 2.
- the performing result of the cache checking step s 121 represents two tensors storing in the cache 430 and respective to the tensor A and the tensor B.
- the distance corresponding to the tensor A and the tensor B are the difference between the step index of the layer operation step 2 and the step index of the future layer operation step of the tensor A and the tensor B respectively.
- the tensor A is the input operand label of the layer operation step 3 , it denotes the future layer operation step of the tensor A is the layer operation step 3 , and j corresponding to the tensor A is three.
- the flushing step s 123 of the layer operation step 2 is not performed because the capacity of the cache 430 does not exceed after performing the layer operation step 2 according to the convolutional neural network connection list.
- the DRAM write-back step s 124 of the layer operation step 2 is not performed because the flushing step s 123 without flushes any tensor from the cache 430 .
- the reading step s 125 of the layer operation step 2 is not performed because the input operand label of the convolution operation step s 126 is the tensor B, and the tensor B is stored in the cache 430 already.
- the convolutional operation step s 126 of the layer operation step 2 calculates the tensor C by the convolution operation with the tensor B and the kernel of the layer operation step 2 .
- the cache storing step s 127 of the layer operation step 2 stores the tensor C into the cache 430 .
- the content of the layer operation step 2 is three tensors in the cache 430 and respective to the tensor A, the tensor B and the tensor C.
- the step index of the layer operation step 3 is 3.
- the performing result of the cache checking step s 121 represents three tensors stored in the cache 430 and respective to the tensor A, the tensor B and the tensor C.
- the distance computing step s 122 of the layer operation step 3 the distances corresponding to the tensor A, the tensor B and the tensor C are the difference between the step index of the layer operation step 3 and the step index of the future layer operation step of the tensor A, the tensor B and the tensor C respectively.
- the tensor A and the tensor C are the input operand labels of the layer operation step 3 , it denotes j corresponding to each of the tensor A and the tensor C is three respectively.
- the flushing step s 123 flushes the tensor B so as to avoid the cache 430 exceeds after performing the layer operation step 3 .
- the distance corresponding to the tensor B is less than N, the tensor B is written into the DRAM 440 at the DRAM write-back step s 124 .
- the reading step s 125 is not performed because the input operand label of the convolution operation step s 126 is stored in the cache 430 already.
- the number of the input operand label is 2, and the input operand labels are the tensor A and the tensor C respectively.
- the convolutional operation step s 126 of the layer operation step 3 concatenates the tensor A and the tensor C to each other along the channel dimension and then performs the convolutional operation with the kernel of the layer operation step 3 so as to calculate the tensor D.
- the cache storing step s 127 stores the tensor D into the cache 430 .
- the content of the layer operation step 3 is three tensors in the cache 430 and respective to the tensor A, the tensor C and the tensor D.
- the step index of the layer operation step 4 is 4.
- the performing result of the cache checking step s 121 represents three tensors stored in the cache 430 and respective to the tensor A, the tensor C and the tensor D.
- the distance computing step s 122 the distance corresponding to the tensor A, the tensor C and the tensor D are the difference between the step index of the layer operation step 4 and the and the step index of the future layer operation step of the tensor A, the tensor C and the tensor D respectively.
- the tensor D is the input operand label of the layer operation step 4
- it denotes j corresponding to the tensor D is four.
- the tensor A and the tensor C do not the input operand label of the layer operation step 4 to the layer operation step 7 , it denotes the distance corresponding to the tensor A and the tensor C are equal to N respectively.
- dist A (4) N.
- dist C (4) N.
- dist D (4) 0.
- the distance corresponding to the tensor A and the distance corresponding to the tensor C are same, and both of them are greater than the distance corresponding to the tensor D.
- the tensor A is stored into the cache 430 earlier than the tensor C, and the flushing step s 123 flushes the tensor A according to the FIFO flush method.
- the tensor A is not written into the DRAM 440 at the DRAM write-back step s 124 .
- the reading step s 125 is not performed because the input operand label of the convolution operation step s 126 is stored in the cache 430 already.
- the convolutional operation step s 126 calculates the tensor E by the convolution operation with the tensor D and the kernel of the layer operation step 4 .
- the cache storing step s 127 stores the tensor E into the cache 430 .
- the content of the layer operation step 4 is three tensors in the cache 430 and respective to the tensor C, the tensor D and the tensor E.
- the step index of the layer operation step 5 is 5.
- the performing result of the cache checking step s 121 represents three tensors stored in the cache 430 and respective to the tensor C, the tensor D and the tensor E.
- the distance computing step s 122 the distance corresponding to the tensor C, the tensor D and the tensor E are the difference between the step index of the layer operation step 5 and the and the step index of the future layer operation step of the tensor C, the tensor D and the tensor E respectively.
- the tensor E is the input operand label of the layer operation step 5 , it denotes j corresponding to the tensor E is five.
- the tensor D is the input operand label of the layer operation step 6 , it denotes j corresponding to the tensor D is six.
- the reading step s 125 is not performed because the input operand label of the convolutional operation step s 126 is stored in the cache 430 already.
- the convolutional operation step s 126 calculates the tensor F by the convolution operation with the tensor E and the kernel of the layer operation step 5 .
- the cache storing step s 127 stores the tensor F into the cache 430 .
- the content of the layer operation step 5 are three tensors in the cache 430 and respective to the tensor D, the tensor E and the tensor F.
- the step index of the layer operation step 6 is 6.
- the performing result of the cache checking step s 121 represents three tensors stored in the cache 430 and respective to the tensor D, the tensor E and the tensor F.
- the distance computing step s 122 the distance corresponding to the tensor D, the tensor E and the tensor F are the difference between the step index of the layer operation step 6 and the step index of the future layer operation step of the tensor D, the tensor E and the tensor F respectively.
- the tensor D and the tensor F are the input operand labels of the layer operation step 6 , it denotes j corresponding to each of the tensor D and the tensor F is six.
- the tensor E is not the input operand label of each of the layer operation step 6 and layer operation step 7 , it denotes the distance corresponding to the tensor E is N.
- dist D (6) 0.
- dist E (6) N.
- dist F (6) 0.
- the flushing step s 123 of the layer operation step 6 flushes the tensor E because the distance corresponding to the tensor E is greater than each of the distances corresponding to the tensor D and the tensor F.
- the DRAM write-back step s 124 is not performed because the distance corresponding to the tensor E is not less than N.
- the reading step s 125 is not performed because the input operand label of the convolutional operation step s 126 of the layer operation step 6 is stored in the cache 430 already. Because the number of the input operand label is two and respective to the tensor D and the tensor F, the convolutional operation step s 126 of the layer operation step 6 concatenates the tensor D and the tensor F to each other along the channel dimension and then performs the convolutional operation with the kernel of the layer operation step 6 so as to calculate the tensor G.
- the cache storing step s 127 stores the tensor G into the cache 430 .
- the content of the layer operation step 6 is three tensors in the cache 430 and respective to the tensor D, the tensor F and the tensor G.
- the feature map caching method s 100 of the convolutional neural network starts to perform the layer operation step 7 .
- the step index of the layer operation step 7 is 7.
- the performing result of the cache checking step s 121 represents three tensors stored in the cache 430 and respective to the tensor D, the tensor F and the tensor G.
- the distance computing step s 122 the distance corresponding to the tensor D, the tensor F and the tensor G are the difference between the step index of the layer operation step 7 and the step index of the future layer operation step of the tensor D, the tensor F and the tensor G respectively.
- the tensor D and the tensor F do not the input operand labels of the layer operation step 7 , it denotes the distance corresponding to the tensor D and the tensor F are equal to N respectively.
- the input operand labels of the convolutional operation step s 126 of layer operation step 7 further includes the tensor B which is not stored in the cache 430 . Therefore, the flushing step s 123 should flush two tensors in the cache 430 so as to store the tensor B and the output operand label of the layer operation step 7 for avoiding the cache 430 exceed after performing the layer operation step 7 .
- the flushing step s 123 flushes the tensor D and the tensor F in the cache 430 because each of the distance corresponding to the tensor D and the tensor F is greater than the distance corresponding to the tensor G.
- the DRAM write-back step s 124 is not performed because each of the distances corresponding to the tensor D and the tensor E are equal to N.
- the reading step s 125 read the tensor B from the DRAM 440 and stored in the cache 430 . The tensor B is written into the DRAM 440 at the layer operation step 3 .
- the convolutional operation step s 126 concatenates the tensor B and the tensor G to each other along the channel dimension and then performs the convolutional operation with the kernel of the layer operation step 7 so as to calculate the tensor H.
- the cache storing step s 127 stores the tensor H into the cache 430 .
- the content of the layer operation step 7 is three tensors in the cache 430 and respective to the tensor B, the tensor G and the tensor H. Therefore, the feature map caching method s 100 of the convolutional neural network via the connection analyzing step s 110 and the layer operation steps s 120 so as to optimize the DRAM access times and reduce the DRAM bandwidth power consumption.
- FIG. 5 shows a block diagram of a system 400 of the feature map caching method s 100 of convolutional neural network of FIG. 1 .
- the system 400 of the feature map caching method s 100 of the convolutional neural network includes a lookup controller 410 , a Central Processing Unit (CPU) 420 , the cache 430 and the Dynamic Random Access Memory (DRAM) 440 .
- CPU Central Processing Unit
- DRAM Dynamic Random Access Memory
- the lookup controller 410 is for performing a connection analyzing step s 110 so as to establish the convolutional neural network connection list.
- the Central Processing Unit (CPU) 420 is for performing the layer operation steps s 120 .
- the cache 430 is electronically connected to the CPU 420 and performing the layer operation steps s 120 .
- the DRAM 440 is electronically connected to the cache 430 and performing the DRAM write-back step s 124 .
- the system 400 via the lookup controller 410 so as to analyze the network and establishes the convolutional neural network connection list.
- the convolutional neural network connection list includes a plurality of tensors and a plurality of layer operation coefficients.
- Each of the layer operation coefficients includes the at least one input operand label and the output operand label.
- the input operand label is a part of the tensors
- the output operand label is one of the tensors.
- the CPU 420 performs the layer operation steps s 120 according to the convolutional neural network connection list.
- the CPU 420 calculates the distance corresponding to each of the tensors in the cache 430 according to the formula (1).
- the flushing step s 123 of each of the layer operation steps s 120 the CPU 420 determines whether the cache 430 flushes the at least one tensor in the cache 430 according to the convolutional neural network connection list.
- the CPU 420 determines to flush the at least one tensor in the cache 430 .
- the CPU 420 produces a flushing command.
- the cache 430 flushes the at least one tensor in the cache 430 according to the flushing command.
- DRAM write-back step s 124 the CPU 420 determines whether the at least one of the tensors in the cache 430 needs to be written into the DRAM 440 according to the convolutional neural network connection list.
- the CPU 420 determines to write the at least one of the tensors into the DRAM 440 , the CPU 420 produces a DRAM storing command.
- the at least one of the tensors which needs to be flushed is written into the DRAM 440 , according to the DRAM storing command.
- the CPU 420 determines whether the cache 430 read and store the at least one of the tensors which is stored in the DRAM 440 according to the convolutional neural network connection list.
- the CPU 420 determines to read the at least one of the tensors from the DRAM 440 , the CPU 420 produces a reading command.
- the cache 430 read and stores the at least one of the tensors from the DRAM 440 according to the reading command.
- convolutional operation step s 126 the CPU 420 calculates the output operand label of each of the layer operation step s 120 according to the convolutional neural network connection list.
- cache storing step s 127 the CPU 420 produces a cache storing command.
- the cache 430 stores the output operand label of each of the layer operation steps s 120 , according to the cache storing command.
- the system 400 further includes a cache controller 450 .
- the cache controller 450 is electronically connected to the cache 430 and the CPU 420 for configuring to send the flushing command and the cache storing command to the cache 430 so as to perform the flushing step s 123 and the cache storing step s 127 .
- the cache controller 450 receives the flushing command, the DRAM storing command, the reading command and the cache storing command from the CPU 420 and transmits to the cache 430 so as to perform the layer operation steps s 120 .
- the system 400 via the lookup controller 410 to perform the connection analyzing step s 110 so as to establish the convolutional neural network connection list.
- the system 400 via the CPU 420 to perform the layer operation steps s 120 according to the convolutional neural network connection list so as to optimize the DRAM access times. Therefore, the DRAM bandwidth power consumption of the system 400 can be reduced.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- General Health & Medical Sciences (AREA)
- Evolutionary Computation (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Artificial Intelligence (AREA)
- Neurology (AREA)
- Memory System Of A Hierarchy Structure (AREA)
- Image Analysis (AREA)
Abstract
Description
- This application claims priority to U.S. Provisional Application Ser. No. 62/720,335, filed Aug. 21, 2018, which is herein incorporated by reference.
- The present disclosure relates to a feature map caching method of convolutional neural network and system thereof. More particularly, the present disclosure relates to feature map caching method of convolutional neural network and system thereof which according to a convolutional neural network connection list.
- Because a first in first out (FIFO) flushing algorithm and a last recently used (LRU) flushing algorithm may flush a tensor in a cache which will be used in the future, and stored the tensor into a DRAM. Therefore, DRAM access times are increased and cause the DRAM bandwidth power consumption of the system to be increased.
- Hence, how to optimize the DRAM access times to reduce the DRAM bandwidth power consumption is the most important issue.
- According to one aspect of the present disclosure, a feature map caching method of a convolutional neural network includes a connection analyzing step and a plurality of layer operation steps. The connection analyzing step is for analyzing a network to establish a convolutional neural network connection list. The convolutional neural network connection list includes a plurality of tensors and a plurality of layer operation coefficients. Each of the tensors includes a plurality of feature maps. Each of the layer operation coefficients includes a step index, at least one input operand label and an output operand label. The step index as a processing order for the layer operation step. The input operand label is for selecting a part of the tensors, and the output operand label is for assigning one of the tensors. At least one of the layer operation steps is for flushing at least one of the tensors in a cache according to a distance between the at least one of the layer operation steps and a future layer operation step of the layer operation steps with the at least one input operand label that matches the at least one of the tensors, and the at least one layer operation steps is for performing a convolutional operation step according to the layer operation coefficients. The distance is calculated according to the convolutional neural network connection list.
- According to another aspect of the present disclosure, a system of the feature map caching method of the convolutional neural network includes a lookup controller, a Central Processing Unit (CPU), the cache and a Dynamic Random Access Memory (DRAM). The lookup controller is for performing the connection analyzing step so as to establish the convolutional neural network connection list. The Central Processing Unit is for performing the layer operation steps. The cache is electronically connected to the Central Processing Unit and performing the layer operation steps. The Dynamic Random Access Memory is electronically connected to the cache and performing a Dynamic Random Access Memory write-back step.
- The present disclosure can be more fully understood by reading the following detailed description of the embodiment, with reference made to the accompanying drawings as follows:
-
FIG. 1 shows a flow chart of a feature map caching method of a convolutional neural network according to one embodiment of the present disclosure. -
FIG. 2 shows a flow chart of one of the layer operation steps ofFIG. 1 . -
FIG. 3 shows a schematic diagram of one example of the feature map caching method of the convolutional neural network ofFIG. 1 . -
FIG. 4 shows a schematic diagram of another example of the feature map caching method of the convolutional neural network ofFIG. 1 . -
FIG. 5 shows a block diagram of a system of the feature map caching method of the convolutional neural network ofFIG. 1 . - The embodiment will be described with the drawings. For clarity, some practical details will be described below. However, it should be noted that the present disclosure should not be limited by the practical details, that is, in some embodiment, the practical details is unnecessary. In addition, for simplifying the drawings, some conventional structures and elements will be simply illustrated, and repeated elements may be represented by the same labels.
-
FIG. 1 shows a flow chart of a feature map caching method s100 of a convolutional neural network according to one embodiment of the present disclosure.FIG. 2 shows a flow chart of one of the layer operation steps s120 ofFIG. 1 .FIG. 3 shows a schematic diagram of one example of the feature map caching method s100 of the convolutional neural network ofFIG. 1 . InFIG. 1 ,FIG. 2 andFIG. 3 , the feature map caching method s100 of the convolutional neural network includes a connection analyzing step s110 and a plurality of layer operation steps s120. - In detail, the connection analyzing step s110 is for analyzing a network to establish a convolutional neural network connection list. The convolutional neural network connection list includes a plurality of tensors and a plurality of layer operation coefficients. Each of the tensors includes a plurality of feature maps, and a number of the feature maps are a channel number of the tensor. Each of the layer operation coefficients includes a step index, an operation type and at least one input operand label and an output operand label. The step index as a processing order for a layer operation step. The operation type indicating a layer operation of the layer operation step is a convolution, an activation, or a batch normalization. The at least one input operand label is for selecting a part of the tensors as the input, and the output operand label is for assigning one of the tensors as the output. At least one of the layer operation steps s120 is for flushing at least one of the tensors in a cache 430 (shown in
FIG. 5 ) according to a distance between the at least one of the layer operation steps s120 and a future layer operation step of the layer operation steps s120 with the at least one input operand label that matches the at least one of the tensors, and a part of the layer operation steps s120 is for performing a convolutional operation step s126 according to the layer operation coefficients. The distance is calculated according to the convolutional neural network connection list. Table 1 lists a convolutional neural network connection list and a conventional cache operation script. Table 2 lists a convolutional neural network connection list and a cache operation script of the feature map caching method s100 of the convolutional neural network corresponding to theFIG. 3 . In Table 1 and Table 2, a number of DRAM access times of the conventional method is nine, and a number of DRAM access times of the feature map caching method s100 of the convolutional neural network is one. Therefore, the feature map caching method s100 of the convolutional neural network can optimize the DRAM access times so as to reduce the DRAM bandwidth power consumption because the DRAM access times will consume most of the DRAM bandwidth power consumption in the system. -
TABLE 1 convolutional neural network connection list input output operand operand Conventional cache operation script layer operation steps label label Content Flush Read Write layer operation step 1 A B A, B — A — layer operation step 2 B C A, B, C — — — layer operation step 3 C D B, C, D A — A layer operation step 4 D, A E D, A, E B, C A B, C layer operation step 5 E F A, E, F D — D layer operation step 6 F, D G F, B, G A, E D A, E DRAM access times — — — — 3 6 -
TABLE 2 convolutional neural network connection list input output operand operand cache operation script layer operation steps label label Content Flush Read Write layer operation step 1 A B A, B — A — layer operation step 2 B C A, B, C — — — layer operation step 3 C D A, C, D B — — layer operation step 4 D, A E A, D, E C — — layer operation step 5 E F D, E, F A — — layer operation step 6 F, D G D, F, G E — — DRAM access times — — — — 1 0 - The at least one of the layer operation steps s120 flushes at least one of the tensors in the
cache 430 according to the distance. The at least one of the tensors which needs to be flushed has a largest distance corresponding to each of the tensors in thecache 430. Therefore, the DRAM access times can be optimized so as to reduce the DRAM bandwidth power consumption. - The connection analyzing step s110 of the feature map caching method s100 of the convolutional neural network analyzes the network so as to establish the convolutional neural network connection list of Table 2. In order to describe clearly, in Table 2, the capacity of the
cache 430 can store three tensors. However, in real application, a number of the tensors storing in thecache 430 is depended on the capacity of thecache 430 and the number of the feature maps. - Please refer to
FIG. 2 ,FIG. 3 and Table 2. Each of the layer operation steps s120 includes a cache checking step s121. The cache checking step s121 is for checking the tensors in thecache 430 and a number of the tensors in thecache 430. In Table 2, a content of each of the layer operation steps s120 is denoted that the tensors storing in thecache 430 after performing each of the layer operation steps s120. A performing result of the cache checking step s121 of the layer operation step 1 represents none tensor storing in thecache 430. A performing result of the cache checking step s121 of the layer operation step 2 is two tensors storing in thecache 430 and respective to the tensor A and the tensor B. In other words, the performing result of the cache checking step s121 of the layer operation step 2 can correspond to the content of the cache operation script of the layer operation step 1 in Table 2 because the layer operation steps s120 performs the cache checking step s121 of the layer operation step 2 after performing the layer operation step 1. Hence, the performing result of the cache checking step s121 of the layer operation step 3 is corresponded to the content of the cache operation script of the layer operation step 2 in Table 2, and the number of the tensors in thecache 430 is three. A performing result of the cache checking step s121 of each of a layer operation step 4 to a layer operation step 6 is same as above, and will not be described again herein. - Each of the layer operation steps s120 further includes a distance computing step s122. The distance computing step s122 is for calculating the distance corresponding to each of the tensors in the
cache 430 according to the convolutional neural network connection list and the cache operation script. The distance corresponding to each of the tensors in thecache 430 is corresponded by formula (1). -
- wherein i is a step index of a layer operation step i and i is from 1 to N, j is a step index of a layer operation step j, N is a number of the layer operation steps, and t is one of the tensors in the cache, and Tinj is a set of the at least one input operand label of the layer operation step j. When layer operation step j does not exist in the convolutional neural network connection list, distt (i)=N. Therefore, the distance computing step s122 of each of the layer operation steps s120 via the formula (1) to compute each of the distance corresponding to each of the tensors in the
cache 430 so as to perform the flushing step s123 of each of the layer operation steps s120. - Please refer to
FIG. 3 and Table 2, the number of the layer operation steps s120 is six, and it denotes N=6. In the layer operation step 1, the step index of the layer operation step 1 is 1. The distance computing step s122 of the layer operation step 1 is not performed because there is none tensor in thecache 430. In the layer operation step 3, the step index of the layer operation step 3 is 3. The distance computing step s122 of the layer operation step 3 calculates the distance corresponding to each of the tensor A, the tensor B and the tensor C because of the performing result of the cache checking step s121 of the layer operation step 3. In detail, the distance corresponding to each of the tensor A, the tensor B and the tensor C is calculated according to the convolutional neural network connection list of Table 2. In other words, according to the formula (1), the distance corresponding to the tensor A, the tensor B and the tensor C are the difference between the step index of the layer operation step 3 and the step index of the future layer operation step of the tensor A, the tensor B and the tensor C respectively. Therefore, in the layer operation step 3, the step index corresponding to each of the tensor A, the tensor B and the tensor C is the step index of the layer operation step 3, it denotes i=3. Further, according to the convolutional neural network connection list of Table 2, the tensor A is an input operand label of the layer operation step 4, and it denotes the future layer operation step of the tensor A is the layer operation step 4 and j corresponding to the tensor A is four. The tensor C is an input operand label of the layer operation step 3, and it denotes the future layer operation step of the tensor C is the layer operation step 3 and j corresponding to the tensor C is three. Therefore, when t is tensor A, distA (3)=1. when t is tensor C, distC (3)=0. The tensor B is not one of the input operand label of the layer operation step 3 to the layer operation step 6, it denotes j corresponding to the tensor B does not exist in the convolutional neural network connection list of Table 2, and distB (3)=N. The distance computing step s122 of the layer operation step 4 to layer operation step 6 is same as above, and will not be described again herein. - Each of the layer operation steps s120 further includes a flushing step s123. The flushing step s123 is for flushing the at least one of the tensors in the
cache 430 according to the distance corresponding to each of the tensors in thecache 430 and the convolutional neural network connection list. A distance corresponding to each of the at least one of the tensors in thecache 430 is greater than or equal to a distance corresponding to each of the tensors which is not flushed in thecache 430. Therefore, the flushing step s123 flushes the at least one of the tensors in thecache 430 so as to avoid a capacity of thecache 430 exceeds after performing each of the layer operation step s120 according to the convolutional neural network connection list. Please refer to Table 2, the layer operation step 1 and the layer operation step 2 do not perform the flushing step s123 because the capacity of thecache 430 is not exceed after performing the layer operation step 1 and the layer operation step 2. In the cache checking step s121 of the layer operation step 3, there are the tensor A, the tensor B and the tensor C in thecache 430. In layer operation step 3, if the flushing step s123 is not performed, the capacity of thecache 430 will be exceeded after performing the layer operation step 3 because the capacity of thecache 430 is three tensors. The flushing step s123 of layer operation step 3 flushes at least one of the tensors in the cache so as to avoid the capacity of thecache 430 exceed. According to the preforming result of the distance computing step s122 of the layer operation step 3, the distances corresponding to the tensor A, the tensor B and the tensor C are distA (3)=1, distB (3)=N=6 and distC (3)=0 respectively. Therefore, the tensor B is flushed in the flushing step s123 of the layer operation step 3 because the distance corresponding to the tensor B is greater than each of the distance corresponding to the tensor A and the tensor C. The flushing step s123 of each of the layer operation step 4 to layer operation step 6 is same as above, and will not be described again herein. Further, when a part of the distance corresponding to each of the at least one of the tensors which needs to be flushed in thecache 430 is equal to another part of the distance corresponding to each of the tensors which is not flushed in thecache 430, the at least one of the tensors which needs to be flushed in thecache 430 is selected by a first in first out (FIFO) flushing method. - Each of the layer operation steps s120 further includes a Dynamic Random Access Memory (DRAM) write-back step s124. The DRAM write-back step s124 is for determining whether the at least one of the tensors in the
cache 430 needs to be written back to a Dynamic Random Access Memory (DRAM) 440 (shown inFIG. 5 ) according to the convolutional neural network connection list. When the distance corresponding to each of the at least one of the tensors which needs to be flushed in thecache 430 is less than N, the at least one of the tensors which needs to be flushed from thecache 430 is written into theDRAM 440. When the distance corresponding to each of the at least one of the tensors which needs to be flushed from thecache 430 is equal to N, the at least one of the tensors which needs to be flushed from thecache 430 is flushed without written into theDRAM 440. Therefore, the feature map caching method s100 of the convolutional neural network can optimize the DRAM access times so as to reduce the DRAM bandwidth power consumption. - Each of the layer operation steps s120 further includes a reading step s125. The reading step s125 is for reading at least another one of the tensors from the
DRAM 440 and storing into thecache 430 according to the convolutional neural network connection list. Please refer toFIG. 3 and Table 2, the tensor A is the input operand label of the layer operation step 1. However, the performing result of the cache checking step s121 of the layer operation step 1, the tensor A is not stored in thecache 430. The tensor A is stored in theDRAM 440. Therefore, the reading step s125 of the layer operation step 1 read the tensor A from theDRAM 440 and stores into thecache 430 so as to perform the convolutional operation step s126 of the layer operation step 1 because the tensor A is the input operand label of the convolutional operation step s126 of the layer operation step 1. - Each of the layer operation coefficients further includes a kernel. The kernel and the at least one input operand label are configured to perform the convolutional operation. The convolutional operation step s126 is for calculating the output operand label by the convolutional operation with the at least one input operand label and the kernel. When a number of the input operand label is plural, the input operand labels concatenate to each other along a channel dimension, and then the convolutional operation is performed. Please refer to
FIG. 3 and Table 2, the convolutional operation step s126 of layer operation step 1, the tensor B is calculated by the convolutional operation with the tensor A and the kernel of the layer operation step 1. In the layer operation step 4, a number of the at least one input operand label of the convolutional operation step s126 is two. The convolutional operation step s126 of layer operation step 4 concatenates the tensor A and the tensor D to each other along the channel dimension and then performs the convolutional operation with the kernel of the layer operation step 4 so as to calculate the tensor E. The convolutional operation step s126 of the layer operation step 2, the layer operation step 3, the layer operation step 5 and the layer operation step 6 are same as above, and will not be described again herein. - Each of the layer operation steps s120 further includes a cache storing step s127. The cache storing step s127 is for storing the output operand label into the
cache 430. Please refer toFIG. 3 and Table 2, the cache storing step s127 of the layer operation step 1 stores the tensor B into thecache 430. Therefore, the content of thecache 430 of the layer operation step 1 is the tensor A and the tensor B. The cache storing step s127 of the layer operation step 2 to the layer operation step 6 are same as above, and will not be described again herein. - Gives as above, the feature map caching method s100 of the convolutional neural network via the connection analyzing step s110 and the layer operation steps s120 so as to optimize the DRAM access times and reduce the DRAM bandwidth power consumption.
- Please refer to
FIG. 1 ,FIG. 2 ,FIG. 4 and Table 3,FIG. 4 shows a schematic diagram of another example of the feature map caching method s100 of the convolutional neural network ofFIG. 1 . Table 3 lists a convolutional neural network connection list and a cache operation script ofFIG. 4 . The connection analyzing step s110 of the feature map caching method s100 of the convolutional neural network analyzes the network so as to establish the convolutional neural network connection list of Table 3. The number of the layer operation steps s120 is seven, it denotes N=7. Further, the capacity of thecache 430 is three tensors. However, the capacity of thecache 430 can be any positive integer, and the present disclosure is not limited thereto. -
TABLE 3 convolutional neural network connection list input output operand operand cache operation script layer operation steps label label Content Flush Read Write layer operation step 1 A B A, B — A — layer operation step 2 B C A, B, C — — — layer operation step 3 A, C D A, C, D B — B layer operation step 4 D E C, D, E A — — layer operation step 5 E F D, E, F C — — layer operation step 6 F, D G D, F, G E — — layer operation step 7 G, B H B, G, H D, F B — - In the layer operation step 1, the performing result of the cache checking step s121 represents none tensor stored in the
cache 430. Therefore, the layer operation step 1 do not perform the distance computing step s122, the flushing step s123 and the DRAM write-back step s124 because there represents none tensor in thecache 430. The reading step s125 of the layer operation step 1 read the tensor A from theDRAM 440 and stores into thecache 430 so as to perform the convolutional operation step s126 of the layer operation step 1. The convolutional operation step s126 of the layer operation step 1 calculates the tensor B by the convolution operation with the tensor A and the kernel of the layer operation step 1. The cache storing step s127 of the layer operation step 1 stores the tensor B into thecache 430. Hence, the content of the layer operation step 1 is two tensors in thecache 430 and respective to the tensor A and the tensor B. After performing the cache storing step s127 of the layer operation step 1, the feature map caching method s100 of the convolutional neural network starts to perform the layer operation step 2. - In the layer operation step 2, the step index of the layer operation step 2 is 2. The performing result of the cache checking step s121 represents two tensors storing in the
cache 430 and respective to the tensor A and the tensor B. According to the formula (1), the distance corresponding to the tensor A and the tensor B are the difference between the step index of the layer operation step 2 and the step index of the future layer operation step of the tensor A and the tensor B respectively. In the layer operation step 2, i corresponding to each of the tensor A and the tensor B is the step index of the layer operation step 2, it denotes i=2. According to the convolutional neural network connection list of Table 3, the tensor A is the input operand label of the layer operation step 3, it denotes the future layer operation step of the tensor A is the layer operation step 3, and j corresponding to the tensor A is three. The tensor B is the input operand label of the layer operation step 2, it denotes the future layer operation step of the tensor B is the layer operation step 2, and j corresponding to the tensor B is two. Therefore, when t is the tensor A, distA (2)=1. When t is the tensor B, distB (2)=0. The flushing step s123 of the layer operation step 2 is not performed because the capacity of thecache 430 does not exceed after performing the layer operation step 2 according to the convolutional neural network connection list. The DRAM write-back step s124 of the layer operation step 2 is not performed because the flushing step s123 without flushes any tensor from thecache 430. The reading step s125 of the layer operation step 2 is not performed because the input operand label of the convolution operation step s126 is the tensor B, and the tensor B is stored in thecache 430 already. The convolutional operation step s126 of the layer operation step 2 calculates the tensor C by the convolution operation with the tensor B and the kernel of the layer operation step 2. The cache storing step s127 of the layer operation step 2 stores the tensor C into thecache 430. Hence, the content of the layer operation step 2 is three tensors in thecache 430 and respective to the tensor A, the tensor B and the tensor C. After performing the cache storing step s127 of the layer operation step 2, the feature map caching method s100 of the convolutional neural network starts to perform the layer operation step 3. - In the layer operation step 3, the step index of the layer operation step 3 is 3. The performing result of the cache checking step s121 represents three tensors stored in the
cache 430 and respective to the tensor A, the tensor B and the tensor C. In the distance computing step s122 of the layer operation step 3, the distances corresponding to the tensor A, the tensor B and the tensor C are the difference between the step index of the layer operation step 3 and the step index of the future layer operation step of the tensor A, the tensor B and the tensor C respectively. In the layer operation step 3, i corresponding to each of the tensor A, the tensor B and the tensor C is the step index of the layer operation step 3, it denotes i=3. According to the convolutional neural network connection list and the cache operation script of Table 3, the tensor A and the tensor C are the input operand labels of the layer operation step 3, it denotes j corresponding to each of the tensor A and the tensor C is three respectively. The tensor B is the input operand label of the layer operation step 7, it denotes j corresponding to the tensor B is seven. Therefore, when t is the tensor A, distA (3)=0. When t is the tensor B, distB (3)=4. When t is the tensor C, distC (3)=0. The tensor B is the tensor which needs to be flushed in thecache 430, because of the largest distance. Therefore, the flushing step s123 flushes the tensor B so as to avoid thecache 430 exceeds after performing the layer operation step 3. However, the distance corresponding to the tensor B is less than N, the tensor B is written into theDRAM 440 at the DRAM write-back step s124. The reading step s125 is not performed because the input operand label of the convolution operation step s126 is stored in thecache 430 already. The number of the input operand label is 2, and the input operand labels are the tensor A and the tensor C respectively. The convolutional operation step s126 of the layer operation step 3 concatenates the tensor A and the tensor C to each other along the channel dimension and then performs the convolutional operation with the kernel of the layer operation step 3 so as to calculate the tensor D. The cache storing step s127 stores the tensor D into thecache 430. Hence, the content of the layer operation step 3 is three tensors in thecache 430 and respective to the tensor A, the tensor C and the tensor D. After performing the layer operation step 3, the feature map caching method s100 of the convolutional neural network starts to perform the layer operation step 4. - In the layer operation step 4, the step index of the layer operation step 4 is 4. The performing result of the cache checking step s121 represents three tensors stored in the
cache 430 and respective to the tensor A, the tensor C and the tensor D. In the distance computing step s122, the distance corresponding to the tensor A, the tensor C and the tensor D are the difference between the step index of the layer operation step 4 and the and the step index of the future layer operation step of the tensor A, the tensor C and the tensor D respectively. In the layer operation step 4, i corresponding to each of the tensor A, the tensor C and the tensor D are the step index of the layer operation step 4, it denotes i=4. According to the convolutional neural network connection list and the cache operation script of Table 3, the tensor D is the input operand label of the layer operation step 4, it denotes j corresponding to the tensor D is four. The tensor A and the tensor C do not the input operand label of the layer operation step 4 to the layer operation step 7, it denotes the distance corresponding to the tensor A and the tensor C are equal to N respectively. Therefore, when t is the tensor A, distA (4)=N. When t is the tensor C, distC (4)=N. When t is the tensor D, distD (4)=0. Following the distance computing step s122, the distance corresponding to the tensor A and the distance corresponding to the tensor C are same, and both of them are greater than the distance corresponding to the tensor D. However, the tensor A is stored into thecache 430 earlier than the tensor C, and the flushing step s123 flushes the tensor A according to the FIFO flush method. Because the distance corresponding to the tensor A is equal to N, the tensor A is not written into theDRAM 440 at the DRAM write-back step s124. The reading step s125 is not performed because the input operand label of the convolution operation step s126 is stored in thecache 430 already. The convolutional operation step s126 calculates the tensor E by the convolution operation with the tensor D and the kernel of the layer operation step 4. The cache storing step s127 stores the tensor E into thecache 430. Hence, the content of the layer operation step 4 is three tensors in thecache 430 and respective to the tensor C, the tensor D and the tensor E. After performing the layer operation step 4, the feature map caching method s100 of the convolutional neural network starts to perform the layer operation step 5. - In the layer operation step 5, the step index of the layer operation step 5 is 5. The performing result of the cache checking step s121 represents three tensors stored in the
cache 430 and respective to the tensor C, the tensor D and the tensor E. In the distance computing step s122, the distance corresponding to the tensor C, the tensor D and the tensor E are the difference between the step index of the layer operation step 5 and the and the step index of the future layer operation step of the tensor C, the tensor D and the tensor E respectively. In the layer operation step 5, i corresponding to each of the tensor C, the tensor D and the tensor E is the step index of the layer operation step 5, it denotes i=5. According to the convolutional neural network connection list and the cache operation script of Table 3, the tensor E is the input operand label of the layer operation step 5, it denotes j corresponding to the tensor E is five. The tensor D is the input operand label of the layer operation step 6, it denotes j corresponding to the tensor D is six. The tensor C do not the input operand label of the layer operation step 5 to the layer operation step 7, it denotes the distance corresponding to the tensor C is N. Therefore, when t is the tensor C, distC (5)=N. When t is the tensor D, distD (5)=1. When t is the tensor E, distE (5)=0. Because the tensor C has the largest distance, the flushing step s123 flushes the tensor C so as to avoid thecache 430 exceed after performing the layer operation step 5. The DRAM write-back step s124 is not performed because the distance corresponding to the tensor C is equal to N. The reading step s125 is not performed because the input operand label of the convolutional operation step s126 is stored in thecache 430 already. The convolutional operation step s126 calculates the tensor F by the convolution operation with the tensor E and the kernel of the layer operation step 5. The cache storing step s127 stores the tensor F into thecache 430. Hence, the content of the layer operation step 5 are three tensors in thecache 430 and respective to the tensor D, the tensor E and the tensor F. After performing the layer operation step 5, the feature map caching method s100 of the convolutional neural network starts to perform the layer operation step 6. - In the layer operation step 6, the step index of the layer operation step 6 is 6. The performing result of the cache checking step s121 represents three tensors stored in the
cache 430 and respective to the tensor D, the tensor E and the tensor F. In the distance computing step s122, the distance corresponding to the tensor D, the tensor E and the tensor F are the difference between the step index of the layer operation step 6 and the step index of the future layer operation step of the tensor D, the tensor E and the tensor F respectively. In the layer operation step 6, i corresponding to each of the tensor D, the tensor E and the tensor F is the step index of the layer operation step 6, it denotes i=6. According to the convolutional neural network connection list and the cache operation script of Table 3, the tensor D and the tensor F are the input operand labels of the layer operation step 6, it denotes j corresponding to each of the tensor D and the tensor F is six. The tensor E is not the input operand label of each of the layer operation step 6 and layer operation step 7, it denotes the distance corresponding to the tensor E is N. Therefore, when t is the tensor D, distD (6)=0. When t is the tensor E, distE (6)=N. When t is the tensor F, distF (6)=0. The flushing step s123 of the layer operation step 6 flushes the tensor E because the distance corresponding to the tensor E is greater than each of the distances corresponding to the tensor D and the tensor F. The DRAM write-back step s124 is not performed because the distance corresponding to the tensor E is not less than N. The reading step s125 is not performed because the input operand label of the convolutional operation step s126 of the layer operation step 6 is stored in thecache 430 already. Because the number of the input operand label is two and respective to the tensor D and the tensor F, the convolutional operation step s126 of the layer operation step 6 concatenates the tensor D and the tensor F to each other along the channel dimension and then performs the convolutional operation with the kernel of the layer operation step 6 so as to calculate the tensor G. The cache storing step s127 stores the tensor G into thecache 430. Hence, the content of the layer operation step 6 is three tensors in thecache 430 and respective to the tensor D, the tensor F and the tensor G. After performing the layer operation step 6, the feature map caching method s100 of the convolutional neural network starts to perform the layer operation step 7. - In the layer operation step 7, the step index of the layer operation step 7 is 7. The performing result of the cache checking step s121 represents three tensors stored in the
cache 430 and respective to the tensor D, the tensor F and the tensor G. In the distance computing step s122, the distance corresponding to the tensor D, the tensor F and the tensor G are the difference between the step index of the layer operation step 7 and the step index of the future layer operation step of the tensor D, the tensor F and the tensor G respectively. In the layer operation step 7, i corresponding to each of the tensor D, the tensor F and the tensor G is the step index of the layer operation step 7, it denotes i=7. According to the convolutional neural network connection list and the cache operation script of Table 3, the tensor D and the tensor F do not the input operand labels of the layer operation step 7, it denotes the distance corresponding to the tensor D and the tensor F are equal to N respectively. The tensor G is one of the input operand labels of the layer operation step 7, it denotes j corresponding to the tensor D is seven. Therefore, when t is the tensor D, distD (7)=N. When t is the tensor F, distF (7)=N. When t is the tensor G, distG (7)=0. However the input operand labels of the convolutional operation step s126 of layer operation step 7 further includes the tensor B which is not stored in thecache 430. Therefore, the flushing step s123 should flush two tensors in thecache 430 so as to store the tensor B and the output operand label of the layer operation step 7 for avoiding thecache 430 exceed after performing the layer operation step 7. The flushing step s123 flushes the tensor D and the tensor F in thecache 430 because each of the distance corresponding to the tensor D and the tensor F is greater than the distance corresponding to the tensor G. The DRAM write-back step s124 is not performed because each of the distances corresponding to the tensor D and the tensor E are equal to N. The reading step s125 read the tensor B from theDRAM 440 and stored in thecache 430. The tensor B is written into theDRAM 440 at the layer operation step 3. Because the number of the input operand label is two and respective to the tensor B and the tensor G, the convolutional operation step s126 concatenates the tensor B and the tensor G to each other along the channel dimension and then performs the convolutional operation with the kernel of the layer operation step 7 so as to calculate the tensor H. The cache storing step s127 stores the tensor H into thecache 430. Hence, the content of the layer operation step 7 is three tensors in thecache 430 and respective to the tensor B, the tensor G and the tensor H. Therefore, the feature map caching method s100 of the convolutional neural network via the connection analyzing step s110 and the layer operation steps s120 so as to optimize the DRAM access times and reduce the DRAM bandwidth power consumption. -
FIG. 5 shows a block diagram of asystem 400 of the feature map caching method s100 of convolutional neural network ofFIG. 1 . InFIG. 5 , thesystem 400 of the feature map caching method s100 of the convolutional neural network includes alookup controller 410, a Central Processing Unit (CPU) 420, thecache 430 and the Dynamic Random Access Memory (DRAM) 440. - In detail, the
lookup controller 410 is for performing a connection analyzing step s110 so as to establish the convolutional neural network connection list. The Central Processing Unit (CPU) 420 is for performing the layer operation steps s120. Thecache 430 is electronically connected to theCPU 420 and performing the layer operation steps s120. TheDRAM 440 is electronically connected to thecache 430 and performing the DRAM write-back step s124. In other words, thesystem 400 via thelookup controller 410 so as to analyze the network and establishes the convolutional neural network connection list. The convolutional neural network connection list includes a plurality of tensors and a plurality of layer operation coefficients. Each of the layer operation coefficients includes the at least one input operand label and the output operand label. The input operand label is a part of the tensors, and the output operand label is one of the tensors. TheCPU 420 performs the layer operation steps s120 according to the convolutional neural network connection list. In the distance computing step s122 of each of the layer operation steps s120, theCPU 420 calculates the distance corresponding to each of the tensors in thecache 430 according to the formula (1). In the flushing step s123 of each of the layer operation steps s120, theCPU 420 determines whether thecache 430 flushes the at least one tensor in thecache 430 according to the convolutional neural network connection list. When theCPU 420 determines to flush the at least one tensor in thecache 430, theCPU 420 produces a flushing command. Thecache 430 flushes the at least one tensor in thecache 430 according to the flushing command. In DRAM write-back step s124, theCPU 420 determines whether the at least one of the tensors in thecache 430 needs to be written into theDRAM 440 according to the convolutional neural network connection list. When theCPU 420 determines to write the at least one of the tensors into theDRAM 440, theCPU 420 produces a DRAM storing command. The at least one of the tensors which needs to be flushed is written into theDRAM 440, according to the DRAM storing command. In reading step s125, theCPU 420 determines whether thecache 430 read and store the at least one of the tensors which is stored in theDRAM 440 according to the convolutional neural network connection list. When theCPU 420 determines to read the at least one of the tensors from theDRAM 440, theCPU 420 produces a reading command. Thecache 430 read and stores the at least one of the tensors from theDRAM 440 according to the reading command. In convolutional operation step s126, theCPU 420 calculates the output operand label of each of the layer operation step s120 according to the convolutional neural network connection list. In cache storing step s127, theCPU 420 produces a cache storing command. Thecache 430 stores the output operand label of each of the layer operation steps s120, according to the cache storing command. - The
system 400 further includes acache controller 450. Thecache controller 450 is electronically connected to thecache 430 and theCPU 420 for configuring to send the flushing command and the cache storing command to thecache 430 so as to perform the flushing step s123 and the cache storing step s127. In detail, thecache controller 450 receives the flushing command, the DRAM storing command, the reading command and the cache storing command from theCPU 420 and transmits to thecache 430 so as to perform the layer operation steps s120. - The details of the distance computing step s122 and the flushing step s123 of each of the layer operation steps s120 are same as above, and will not be described again herein.
- Give as above, the
system 400 via thelookup controller 410 to perform the connection analyzing step s110 so as to establish the convolutional neural network connection list. Thesystem 400 via theCPU 420 to perform the layer operation steps s120 according to the convolutional neural network connection list so as to optimize the DRAM access times. Therefore, the DRAM bandwidth power consumption of thesystem 400 can be reduced. - Although the present disclosure has been described in considerable detail with reference to certain embodiments thereof, other embodiments are possible. Therefore, the spirit and scope of the appended claims should not be limited to the description of the embodiments contained herein.
- It will be apparent to those skilled in the art that various modifications and variations can be made to the structure of the present disclosure without departing from the scope or spirit of the disclosure. In view of the foregoing, it is intended that the present disclosure cover modifications and variations of this disclosure provided they fall within the scope of the following claims.
Claims (16)
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/543,645 US11782839B2 (en) | 2018-08-21 | 2019-08-19 | Feature map caching method of convolutional neural network and system thereof |
TW108129882A TWI718634B (en) | 2018-08-21 | 2019-08-21 | Feature map caching method of convolutional neural network and system thereof |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201862720335P | 2018-08-21 | 2018-08-21 | |
US16/543,645 US11782839B2 (en) | 2018-08-21 | 2019-08-19 | Feature map caching method of convolutional neural network and system thereof |
Publications (2)
Publication Number | Publication Date |
---|---|
US20200065250A1 true US20200065250A1 (en) | 2020-02-27 |
US11782839B2 US11782839B2 (en) | 2023-10-10 |
Family
ID=69584631
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/543,645 Active 2042-07-04 US11782839B2 (en) | 2018-08-21 | 2019-08-19 | Feature map caching method of convolutional neural network and system thereof |
Country Status (2)
Country | Link |
---|---|
US (1) | US11782839B2 (en) |
TW (1) | TWI718634B (en) |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20180253635A1 (en) * | 2017-03-03 | 2018-09-06 | Samsung Electronics Co, Ltd. | Neural network devices and methods of operating the same |
US20180341495A1 (en) * | 2017-05-26 | 2018-11-29 | Purdue Research Foundation | Hardware Accelerator for Convolutional Neural Networks and Method of Operation Thereof |
Family Cites Families (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10776690B2 (en) * | 2015-10-08 | 2020-09-15 | Via Alliance Semiconductor Co., Ltd. | Neural network unit with plurality of selectable output functions |
US9665799B1 (en) * | 2016-01-29 | 2017-05-30 | Fotonation Limited | Convolutional neural network |
TWI634490B (en) * | 2016-11-14 | 2018-09-01 | 美商耐能股份有限公司 | Convolution operation device and convolution operation method |
TWI634436B (en) * | 2016-11-14 | 2018-09-01 | 耐能股份有限公司 | Buffer device and convolution operation device and method |
-
2019
- 2019-08-19 US US16/543,645 patent/US11782839B2/en active Active
- 2019-08-21 TW TW108129882A patent/TWI718634B/en active
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20180253635A1 (en) * | 2017-03-03 | 2018-09-06 | Samsung Electronics Co, Ltd. | Neural network devices and methods of operating the same |
US20180341495A1 (en) * | 2017-05-26 | 2018-11-29 | Purdue Research Foundation | Hardware Accelerator for Convolutional Neural Networks and Method of Operation Thereof |
Also Published As
Publication number | Publication date |
---|---|
US11782839B2 (en) | 2023-10-10 |
TWI718634B (en) | 2021-02-11 |
TW202024925A (en) | 2020-07-01 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
Nguyen et al. | An approximate memory architecture for a reduction of refresh power consumption in deep learning applications | |
CN105550125B (en) | Hybrid memory architecture | |
US7558920B2 (en) | Apparatus and method for partitioning a shared cache of a chip multi-processor | |
KR102130578B1 (en) | Semiconductor device | |
US12111764B2 (en) | Forward caching memory systems and methods | |
CN108647155B (en) | Deep learning-based multi-level cache sharing method and device | |
US11467968B2 (en) | Memory-adaptive processing method for convolutional neural network | |
US12061554B2 (en) | Asynchronous forward caching memory systems and methods | |
CN108520296B (en) | Deep learning chip-based dynamic cache allocation method and device | |
WO2023041002A1 (en) | Near memory computing accelerator, dual in-line memory module and computing device | |
US10175893B2 (en) | Predictive scheduler for memory rank switching | |
US11782839B2 (en) | Feature map caching method of convolutional neural network and system thereof | |
CN110490312B (en) | Pooling calculation method and circuit | |
CN111783984A (en) | Neural network operation method, device, equipment and storage medium | |
US20230281129A1 (en) | Information processing apparatus and memory access control method | |
US20220230055A1 (en) | Computing circuit and data processing method based on convolutional neural network and computer readable storage medium | |
US10942860B2 (en) | Computing system and method using bit counter | |
CN115543869A (en) | Multi-way set connection cache memory and access method thereof, and computer equipment | |
US11494430B2 (en) | Data storage apparatus and data prediction method thereof | |
US11157174B2 (en) | Hybrid first-fit K-choice insertions for hash tables, hash sets, approximate set membership data structures, and caches | |
US10754576B2 (en) | Performance evaluation apparatus and performance evaluation method | |
US20100122039A1 (en) | Memory Systems and Accessing Methods | |
Li et al. | Optimizing Data Layout for Training Deep Neural Networks | |
US20230168809A1 (en) | Intelligence processor device and method for reducing memory bandwidth | |
CN117742652B (en) | Memory bandwidth optimization device and method based on dynamic window statistics |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: NATIONAL TSING HUA UNIVERSITY, TAIWAN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:CHAO, PING;KAO, CHAO-YANG;LIN, YOUN-LONG;REEL/FRAME:050094/0097 Effective date: 20190524 |
|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY |
|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO SMALL (ORIGINAL EVENT CODE: SMAL); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
AS | Assignment |
Owner name: NEUCHIPS CORPORATION, TAIWAN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:NATIONAL TSING HUA UNIVERSITY;REEL/FRAME:052472/0473 Effective date: 20200323 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |