KR20180101978A - Neural network device, Neural network processor and method of operating neural network processor - Google Patents

Abstract

Provided are a method for selectively performing a convolution operation on an input feature map and a weight map based on input feature information indicating whether each of a plurality of input features of the input feature map has a non-zero value and weight information indicating whether each of a plurality of weights of the weight map has a non-zero value, and a processor therefor. Accordingly, the present invention can reduce an operation amount and an operation time in the convolution operation.

Description

Technical Field [0001] The present invention relates to a neural network processor, a neural network processor, and a neural network device,

The present disclosure relates to a neural network processor, a method of operating a neural network processor, and a neural network device.

A neural network refers to a computational architecture that models a biological brain. Recently, with the development of neural network technology, various kinds of electronic systems have been actively studied for analyzing input data and extracting valid information using a neural network device.

Neural network devices require large amounts of computation for complex input data. There is a need for a technique capable of efficiently processing neural network operations in order to analyze high-quality input in real time and extract information.

A neural network processor, a method of operating a neural network processor, and a neural network apparatus. The technical problem to be solved by this embodiment is not limited to the above-mentioned technical problems, and other technical problems can be deduced from the following embodiments.

In accordance with one aspect, the neural network processor includes input feature information that indicates whether each of a plurality of input features of an input feature map has a non-zero value, Acquiring weight information indicating whether each of the plurality of weights has a nonzero value, calculating, based on the obtained input feature information and weight information, a plurality of input features and an input feature And a fetch controller for determining a weight; And a data operation circuit that performs a convolution operation on the determined weight and the input feature to generate an output feature map.

The data operation circuit may also selectively perform a convolution operation only for a weight and an input feature determined among a plurality of input features and a plurality of weights.

The fetch controller also performs an operation on input feature information and weight information to detect input features and weights having nonzero values and the data operation circuit performs a convolution operation on the detected input features and weights .

Further, the input feature information includes a feature having a zero value represented by 0, a feature having a non-zero value includes an input feature vector represented by 1, and the weight information is expressed by a weight having a zero value as 0 , And a weight having a non-zero value may include a weight vector denoted by 1.

In addition, when the determined input features and weights are the first input feature and the first weight, and the second input feature and the second weight, the data operation circuit, in the current cycle, converts the first input feature and the first weight into input features From the map and weight map to perform the convolution operation, and in the next cycle, perform the convolution operation by reading the second input feature and the second weight from the input feature map and the weight map.

According to another aspect, a method of operation of a neural network processor includes input feature information indicating whether each of a plurality of input features of an input feature map has a non-zero value, Obtaining weight information indicating whether each of the weights has a non-zero value; Determining input features and weights to be convoluted among the plurality of input features and the plurality of weights based on the obtained input feature information and weight information; And performing a convolution operation on the determined weight and the input feature to generate an output feature map.

According to another aspect, a neural network device comprises: a processor array including a plurality of neural network processors; A memory for storing an input feature map and a plurality of weight maps; And a controller for assigning the input feature map and the plurality of weight maps to the processor array, wherein the controller is configured to assign the plurality of weight maps to the plurality of weight groups based on the ratio of the weights having non-zero values in the weight map Group, and assign each of the plurality of weight groups to the processor array.

The controller may also group the plurality of weight maps into a plurality of weight groups so that the weight ratios with non-zero values of each of the weight maps included in the weight group become equal.

The controller may also group the plurality of neural network processors into a plurality of processor groups and sequentially assign each of the plurality of weight groups to each of the plurality of processor groups.

The controller also includes input feature information that indicates whether each of the input features of the input feature map has a nonzero value and weight information that indicates whether each of the weights of the plurality of weight maps has a non- And the processor array may perform a convolution operation on the input feature map and the plurality of weight maps based on the input feature information and weight information to generate an output feature map.

According to another aspect, there is provided a computer-readable recording medium having recorded thereon a program for implementing a method of operating a neural network processor.

According to these embodiments, the neural network processor can selectively perform convolution operations on input features and weights having a nonzero value, thereby reducing the amount of computation and computation time in convolution computation on input features and weights .

In addition, according to the embodiments, the neural network apparatus sequentially allocates each of the plurality of weight groups grouped based on the ratio of the weights having the non-zero value to the processor array, thereby improving the convolution operation speed of the processor array .

1 shows an example of a neural network structure.
2 is a block diagram illustrating a neural network processor in accordance with one embodiment.
3 shows a specific embodiment in which the neural network processor performs selective convolution.
4 is a diagram for explaining a method of operating a neural network processor according to an embodiment of the present invention.
5 shows an embodiment of a neural network device.
6 shows an embodiment in which a neural network device groups a plurality of weight maps into a plurality of weight groups.
7A and 7B show an embodiment in which a neural network device processes an input feature map and a weight map.
8 is a diagram for explaining a method of operating a neural network device according to an embodiment.
9 is a block diagram illustrating an electronic system according to an embodiment of the present disclosure.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Exemplary embodiments will now be described in detail with reference to the accompanying drawings. It is to be understood that the following embodiments are for the purpose of describing the technical contents, but do not limit or limit the scope of the rights. Those skilled in the art can easily deduce from the detailed description and examples that the scope of the present invention falls within the scope of the right.

As used herein, the terms " comprising " or " comprising " and the like should not be construed as necessarily including the various elements or steps described in the specification, May not be included, or may be interpreted to include additional components or steps.

In addition, terms including ordinals such as 'first' or 'second' used in this specification can be used to describe various elements, but the elements should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another.

The present invention relates to a method and apparatus for processing a texture called a cubemap, and a detailed description thereof will be omitted with respect to matters widely known to those skilled in the art to which the following embodiments belong.

1 shows an example of a neural network structure.

1 shows a structure of a convolutional neural network as an example of a neural network structure. 1, a convolutional layer 10 is shown in the convolutional neural network, but the convolutional layer 10 is followed by a pooling layer, a pulley connected a fully connected layer, and the like.

In the convolution layer 10, the first feature map FM1 may be an input feature map and the second feature map FM2 may be an output feature map. The feature map means data in which various features of input data are expressed. The feature maps FM1 and FM2 may have a two-dimensional matrix or a three-dimensional matrix shape. Feature maps (FM1, FM2) having such a multidimensional matrix shape may be referred to as feature tensors. Also, the input feature map may be referred to as activation. Feature maps FM1 and FM2 have a width W (or a column), a height H (or a row), and a depth D, which correspond to the x, y, and z axes Can respond. At this time, the depth D may be referred to as a channel number.

In the convolution layer 10, a convolution operation on the first feature map FM1 and the weight map WM may be performed, and as a result a second feature map FM2 may be generated. The weight map WM may filter the first feature map FM1 and may be referred to as a filter or kernel. The depth of the weight map WM, that is, the number of channels is equal to the depth of the first feature map FM1, that is, the number of channels, and the same channel of the weight map WM and the first feature map FM1 can be convoluted have. The weight map WM is shifted in such a manner as to traverse the first input feature map FM1 as a sliding window. The amount to be shifted may be referred to as "stride length" or "stride ". During each shift, each of the weights contained in the weight map WM may be multiplied and added with all the feature values in the overlapping region of the first feature map FM1. As the first feature map FM1 and the weight map WM are convolved, one channel of the second feature map FM2 can be generated. Although one weight map WM is shown in FIG. 1, a plurality of weight maps may be convolved with the first feature map FM1 to generate a plurality of channels of the second feature map FM2 . In other words, the number of channels of the second feature map FM2 may correspond to the number of weight maps.

In addition, the second feature map FM2 of the convolution layer 10 may be an input feature map of another layer. For example, the second feature map FM2 may be an input feature map of a pooling layer.

2 is a block diagram illustrating a neural network processor in accordance with one embodiment.

The neural network processor 100 may be implemented with hardware circuits. For example, the neural network processor 100 may be implemented as an integrated circuit. The neural network processor 100 includes a central processing unit (CPU), a multi-core CPU, an array processor, a vector processor, a digital signal processor (DSP), a field- But are not limited to, at least one of Programmable Logic Array (PLA), Application Specific Integrated Circuit (ASIC), programmable logic circuitry, Video Processing Unit (VPU), and Graphics Processing Unit Do not.

The neural network processor 100 may include a fetch controller 112, and a data operation circuit 114. The neural network processor 100 shown in FIG. 2 is only shown in the components associated with this embodiment. Accordingly, it will be understood by those skilled in the art that other general-purpose components other than the components shown in FIG. 2 may be further included.

The fetch controller 112 obtains input feature information that indicates whether each of a plurality of input features of the input feature map has a zero value and weight information that indicates whether each of the plurality of weights of the weight map has a zero value can do. According to one embodiment, the fetch controller 112 may receive input feature information and weight information from the outside. Further, according to another embodiment, the fetch controller 112 may generate input feature information from the input feature map and generate weight information from the weight map.

The fetch controller 112 may determine input features and weights to be convoluted among the plurality of input features and the plurality of weights based on the input feature information and weight information. According to one embodiment, the fetch controller 112 uses the input feature information and weight information to generate input features and weights having equally non-zero values at locations that correspond to each other among a plurality of input features and a plurality of weights And the detected input features and weights may be determined as input features and weights to be convoluted. For example, if the input feature information and the weight information each indicate a feature or weight having a zero value and a feature or weight having a non-zero value is a bit vector denoted by 1, then the fetch controller 112 An AND operation on input feature information and weight information may be performed to determine input features and weights to be convoluted. According to one embodiment, the fetch controller 112 may comprise an arithmetic logic circuit.

Data operation circuit 114 may perform a convolution operation on input features and weights determined by fetch controller 112. [ In other words, the data operation circuitry 114 selectively performs a convolution operation only on input features and weights determined by the fetch controller 112, among a plurality of input features of the input feature map and a plurality of weights of the weight map . Specifically, the data operation circuit 114 may selectively perform a convolution operation only on input features and weights having the same non-zero value at positions corresponding to each other among a plurality of input features and a plurality of weights. For example, the data operation circuit 114 may perform a convolution operation by multiplying the input feature value by the weight value. According to one embodiment, the data operation circuit 114 may be implemented as an arithmetic operation circuit. In addition, data operation circuitry 114 may perform convolution operations on input features and weights to generate output feature values. Thus, the data processing circuitry 114 may perform optional convolution on the input feature map and weight map, based on the input features and weights having a nonzero value at locations that correspond to each other, thereby creating an output feature map can do.

According to one embodiment, the neural network processor 100 may further include an internal memory. The internal memory may be a cache memory of the neural network processor 100. The internal memory may be a static random access memory (SRAM). However, the present invention is not limited thereto, and the internal memory may be implemented as a simple buffer of the neural network processor 100, a cache memory, or another kind of memory of the neural network processor 100. The internal memory may store data generated according to operation operations performed in the data operation circuit 114, for example, output feature values, output feature maps, or various types of data generated during the operation.

The neural network processor 100 may store the output feature map generated by the data operation circuit 114 in an internal memory or output it to the outside.

Thus, according to the present disclosure, the neural network processor 100 can selectively perform convolution operations on input features and weights having nonzero values, which can omit meaningless operations that do not affect the output features have. Thus, the neural network processor 100 can reduce the amount of computation and computation time in convolution computation for input features and weights.

3 shows a specific embodiment in which the neural network processor performs selective convolution.

The fetch controller 112 may obtain an input feature vector that indicates whether each of a plurality of input features of the input feature map has a zero value and a weight vector that indicates whether each of the plurality of weights of the weight map has a zero value have. The input feature vector is a bit-vector denoted as 1 if the value of each of the input features is a nonzero value and denoted as 0 if it is a zero value, and the weight vector is also a weight vector if the value of each of the weights is non- 1, and zero if it is zero. In other words, as shown in FIG. 3, the input feature vector is represented by 1 since the 0th input feature, the 3 rd input feature, and the 4th input feature of the 5 input features have non-zero values, The feature and the second input feature are zero vectors because they have zero values. In addition, the weight vector is represented by 1 because the zero weight, the first weight, and the third weight of the five input weights have a non-zero value, and the second and fourth weights have a zero value, It is a vector to be displayed. Although the input feature map and the input feature vector for the five input features and the weight map and weight vector for the five weights are shown in FIG. 3, the number is not limited thereto. In addition, the input feature map and weight map shown in Fig. 3 may be an input feature map and a weight map corresponding to a portion of the entire input feature map and the entire weight map.

The fetch controller 112 may determine the input features and weights to be convoluted through an AND operation on the input feature vector and the weight vector. Specifically, the fetch controller 112 performs an AND operation on the input feature vector and weight vector to obtain input features and weights having equally non-zero values at positions corresponding to each other among a plurality of input features and a plurality of weights Can be detected. 3, the fetch controller 112 may perform an AND operation on the input feature vector and the weight vector, and the zeroth input feature and weight whose result is 1, and the third input feature and weight Can be detected. Thus, the fetch controller 112 may determine the detected input features and weights as input features and weights to be convoluted.

Data operation circuit 114 may perform a convolution operation on input features and weights determined by fetch controller 112. [ The data operation circuit 114 may sequentially read the input features and weights from the input feature map and the weight map to perform the convolution operation. In other words, the data operation circuit 114 performs the convolution operation by reading the n-th input feature of the input feature map and the n-th weight of the weight map in the current cycle, It is possible to perform the convolution operation by reading the n + 1th weight of the feature and weight map. At this time, the data operation circuit 114 can perform a convolution operation on the input features and weights determined by the fetch controller 112, and the convolution operation on input features and weights not determined by the fetch controller 112 Can be omitted. As shown in FIG. 3, the data operation circuit 114 performs a convolution operation on the 0th input feature and the 0th weight in the current cycle, and in the next cycle, the first input feature and the first weight, and the second input The convolution operation for the third input feature and the third weight can be performed while omitting the convolution for the feature and the second weight. In other words, the data operation circuit 114 may perform a convolution operation only on input features and weights having a non-zero value at positions corresponding to each other in the input feature map and the weight map.

Data operation circuit 114 may perform a convolution operation on input features and weights determined by fetch controller 112 to generate an output feature map. In addition, the data operation circuit 114 can generate the output feature map by accumulating the convolution operation result of FIG. 3 on the output feature map generated through the convolution operation on the weight map and other input feature maps.

4 is a diagram for explaining a method of operating a neural network processor according to an embodiment of the present invention.

The method shown in FIG. 4 may be performed by each component of the neural network processor 100 of FIG. 2, and redundant description is omitted.

At step S410, the neural network processor 100 determines whether each of a plurality of input features of the input feature map has input feature information that indicates whether each of the input features has a non-zero value, Weight information indicating whether or not each of the weights has a non-zero value can be obtained. According to one embodiment, the neural network processor 100 may receive input feature information and weight information from the outside. Further, in accordance with another embodiment, neural network processor 100 may generate input feature information from an input feature map and generate weight information from a weight map.

At step s420, the neural network processor 100 may determine input features and weights to be convoluted among the plurality of input features and the plurality of weights based on the input feature information and weight information obtained at s410 . According to one embodiment, the neural network processor 100 may perform an operation on input feature information and weight information to detect input features and weights having non-zero values and to perform a convolution operation on the detected input features and weights Can be determined by the input features and weights to be performed.

At step s430, neural network processor 100 may perform a convolution operation on the weight and input features determined at S420 to generate an output feature map. In other words, the neural network processor 100 may selectively perform a convolution operation only on the input features and weights determined in S420, among the plurality of weights of the input features and the weight map of the input feature map. Specifically, the neural network processor 100 may selectively perform a convolution operation only on input features and weights having the same non-zero value at positions corresponding to each other among a plurality of input features and a plurality of weights. In addition, the neural network processor 100 may perform convolution operations on input features and weights to produce output feature values. Thus, the neural network processor 100 can perform selective convolution on the input feature map and weight map, based on input features and weights having non-zero values at locations that correspond to each other, resulting in an output feature map can do.

5 shows an embodiment of a neural network device.

The neural network device 1000 may include a controller 1010, a processor array 1020, and a memory 1030. The components 1010, 1020, 1030 of the neural network device 1000 may communicate via the system bus. In an embodiment, the neural network device 1000 may be implemented as a single semiconductor chip, for example, as a system-on-chip (SoC). However, the present invention is not limited thereto, and the neural network device 1000 may be implemented by a plurality of semiconductor chips.

The controller 1010 may be implemented as a central processing unit (CPU), a microprocessor, or the like, and may control the overall operation of the neural network device 1000. The controller 1010 can control the operation of the processor array 1020 and the memory 1030. For example, the controller 1010 may set and manage parameters such that the processor array 1020 can execute layers of the neural network normally. Also, according to one embodiment, the controller 1010 may include a rectifier linear unit (ReLU) module.

The processor array 1020 may include a plurality of neural network processors. In addition, the processor array 1020 may be implemented as a plurality of neural network processors in the form of arrays. According to one embodiment, each of the plurality of neural network processors included in the processor array 1020 may be the neural network processor 100 of FIGS. 2 and 3. In addition, a plurality of neural network processors may be implemented to operate in parallel simultaneously. Further, each of the plurality of neural network processors may operate independently. For example, each neural network processor may be implemented as a core circuit, each of which may execute instructions.

The memory 1030 may be implemented as a random access memory (RAM), for example, a dynamic random access memory (DRAM), an SRAM, or the like. The memory 1030 may store various programs and data. According to one embodiment, the memory 1030 may store weight maps or input feature maps provided from an external device, such as a server or external memory.

The controller 1010 may assign the input feature map and the weight map stored in the memory 1030 to the processor array 1020. [

In addition, the controller 1010 may generate, from the input feature map, input feature information that indicates whether each of a plurality of input features of the input feature map has a non-zero value. Further, the controller 1010 can generate, from the weight map, weight information indicating whether or not each of the plurality of weights of the weight map has a non-zero value. In addition, the controller 1010 may provide input feature information and weight information to the processor array 1020. Further, according to another embodiment, the controller 1010 may receive input feature information and weight information from the outside.

Each neural network processor of the processor array 1020 may perform a convolution operation on the assigned input feature map and weight map to generate an output feature map. In addition, the processor array 1020 can perform a convolution operation on the input feature map and the weight map based on the input feature information and the weight information to generate an output feature map. Specifically, the processor array 1020 may selectively perform convolution on input features and weights having a non-zero value to generate an output feature map.

The controller 1020 may divide the input feature map according to a spatial dimension. For example, the controller 1020 may partition the input feature map based on the size of the weight map. Controller 1020 may then allocate the partitioned input feature maps to processor array 1020. [

The controller 1010 may group a plurality of neural network processors of the processor array 1020 into a plurality of processor groups. In other words, the controller 1010 may group a predetermined number of neural network processors into one processor group, and consequently determine a plurality of processor groups. For example, if there are 100 multiple neural network processors, the controller 1010 may group 10 neural network processors into one processor group, resulting in 10 processor groups.

The controller 1020 may assign each of the partitioned input feature maps to each processor group of the processor array 1020. [ Further, the controller 1020 may assign a plurality of weight maps to each processor group of the processor array 1020. Thus, each of the neural network processors included in the processor group of the processor array 1020 can be assigned the same input feature map, and can be assigned different weight maps.

According to one embodiment, the memory 1030 may buffer the weight maps corresponding to the layer to be executed by the processor array 1020. When a calculation is performed using the weight map in the processor array 1020, the used weight map may be output from the external memory and stored in the internal memory of the neural network processor of the processor array 1020. [ The memory 1030 may temporarily store the weight maps before the weight maps output from the external memory are provided to the internal memory of the neural network processor of the processor array 1020. [ Further, according to one embodiment, the memory 1030 may temporarily store an output feature map output from the processor array 1020. [

According to one embodiment, the controller 1010 may group a plurality of weight maps into a plurality of weight groups, based on the ratio of weights having non-zero values in the weight map. Specifically, the controller 1010 may group the plurality of weight maps into a plurality of weight groups so that the weight ratios of non-zero values of each of the weight maps included in the weight group become equal to each other. For example, the controller 1010 can arrange a plurality of weight maps in ascending order based on the ratio of weights having a non-zero value among all the weights of the weight map, A plurality of weight groups can be determined by grouping them in the set number unit. In accordance with one embodiment, controller 1010 may group a plurality of weight maps into a plurality of weight map groups, based on the number of neural network processors included in one processor group of processor array 1020. For example, if the number of weighted maps is 381 and the number of neural network processors included in one group of the processor array 1020 is 40, the controller 1010 can group the plurality of weight maps into 10 groups have. In other words, the controller 1010 can determine one weight map group including nine weight map groups including the 40 weight maps and 21 weight maps.

The controller 1010 may assign a plurality of weight groups to the processor array 1020. Specifically, the controller 1010 may assign each of a plurality of weight groups to each processor group of the processor array 1020 sequentially. For example, the controller 1010 may assign the weight maps included in the zeroth weight group of the plurality of weight groups to each of the processor groups of the processor array 1020. [ After the convolution by the processor array 1020 is completed for the zeroth weight group, the controller 1020 sends the weight maps contained in the first one of the plurality of weight groups to the processor groups 1020 of the processor array 1020 . ≪ / RTI >

Thus, according to the present disclosure, the neural network device 1000 sequentially assigns each of the plurality of weight groups grouped based on the ratio of the weights having a non-zero value to the processor array 1020, ) Can be improved. In other words, since the non-zero value weight ratios of the weight maps allocated to each processor group of the processor array 1020 are equal, the convolution operation speed between the neural network processors in the processor group can be equalized, The arithmetic operation speed of the array 1020 can be improved.

6 shows an embodiment in which a neural network device groups a plurality of weight maps into a plurality of weight groups.

According to one embodiment, graph 610 shows the ratio of nonzero value weights each of the plurality of weight maps stored in memory 1030 has. The controller 1010 can detect the ratio of the weights having the non-zero value for each of the plurality of weight maps stored in the memory 1030 as shown in the graph 610 and calculate the ratio of the weights having the non- , The graph 620, and the like. The graph 620 shows that a plurality of weight maps are arranged in ascending order based on the ratio of weights having a non-zero value. The controller 1010 may group the aligned plurality of weight maps into a plurality of weight map groups. For example, the controller 1010 may group the plurality of aligned weight maps in order of a predetermined number of units, as shown in the graph 620, to determine ten weight map groups. In other words, the controller 1010 can determine the weight map groups 0 to 9. Therefore, the ratio of the non-zero value weight of the weight maps included in each of the ten weight map groups can be equalized.

7A and 7B show an embodiment in which a neural network device processes an input feature map and a weight map.

The controller 1010 may divide the input feature map having width W, height H and channel C according to a spatial dimension and divide the input feature maps sequentially into processor arrays 1020). ≪ / RTI > 7b, the controller 1010 may assign the partitioned input feature maps to each processor group of the processor array 1020 in a zig-zag direction.

Referring to FIG. 7A, the controller 1010 may allocate each of the divided input feature maps A0 and A1 to each of the zeroth processor group and the first processor group of the processor array 1020. FIG.

Further, the controller 1010 may assign a plurality of weight groups to each processor group of the processor array 1020 sequentially. For example, the controller 1010 may assign the weight maps K0 and K1 of the zero-weight group to each of the zeroth processor group and the first processor group of the processor array 1020. [ In addition, controller 1010 may assign a first weight group to processor array 1020 when processor array 1020 completes a convolution operation on all input feature maps partitioned with the zero weight group.

The processor group of the processor array 1020 may perform a convolution operation on the assigned input feature map and weight map to generate an output feature map. For example, a zero-neural network processor of the zeroth processor group may perform a convolution operation on input feature map A0 and weight map K0 to generate an output feature map Psum0. In other words, the 0 < th > neural network processor may generate an output feature map (Psum0) for the input feature map A0 corresponding to a portion of the entire input feature map. In addition, the first neural network processor of the zeroth processor group may perform a convolution operation on the input feature map A0 and the weight map K1 to generate the output feature map Psum1. Similarly, the 0 < th > neural network processor of the first processor group may perform a convolution operation on the input feature map A1 and the weight map K0 to generate the output feature map Psum0. In addition, the first neural network processor of the first processor group may perform a convolution operation on the input feature map Al and the weight map K1 to generate the output feature map Psuml.

Next, referring to FIG. 7B, after the convolution operation for the divided input feature maps A0 and A1 is completed, the controller 1010 transfers the other divided input feature maps A2 and A3 to the zeroth processor group of the processor array 1020 and the 1 processor group. The zeroth processor group of processor array 1020 may then perform a convolution operation on the pre-allocated weight maps K0 and K1 and the input feature map A2 to generate output feature maps Psum0 and Psum1. In addition, the first processor group of the processor array 1020 may perform a convolution operation on the pre-allocated weight maps K0 and K1 and the input feature map A3 to generate the output feature maps Psum0 and Psum1.

8 is a diagram for explaining a method of operating a neural network device according to an embodiment.

The method shown in Fig. 8 can be performed by each component of the neural network apparatus 1000 of Fig. 5, and redundant explanations are omitted.

In step s810, the neural network device 1000 may group the plurality of weight maps into a plurality of weight groups, based on the ratio of the weights having non-zero values in the weight map. Specifically, the neural network apparatus 1000 can group a plurality of weight maps into a plurality of weight groups so that the weight ratios of non-zero values of each of the weight maps included in the weight group are similar to each other.

In addition, the neural network device 1000 may group a plurality of neural network processors into a plurality of processor groups. In other words, the neural network apparatus 1000 can group a predetermined number of neural network processors into one processor group, and consequently determine a plurality of processor groups.

In step s820, the neural network device 1000 may assign each of a plurality of weight groups and an input feature map to a plurality of neural network processors. Specifically, the neural network apparatus 1000 can sequentially assign each of a plurality of weight groups to a plurality of processor groups.

Thus, each of the plurality of processor groups may perform a convolution operation on the assigned weight group and the input feature map to generate an output feature map.

9 is a block diagram illustrating an electronic system according to an embodiment of the present disclosure.

The electronic system 1 according to the embodiment of the present disclosure analyzes inputted data in real time based on a neural network, extracts valid information, makes a situation determination based on the extracted information, To control the configurations of the electronic device. For example, the electronic system 1 may be a robotic device such as a drone, an Advanced Drivers Assistance System (ADAS), a smart TV, a smart phone, a medical device, a mobile device, an image display device, Internet of Things) devices, and the like, and it can be mounted on one of various kinds of electronic devices.

1, the electronic system 1 includes a central processing unit (CPU) 110, a random access memory (RAM) 120, a neural network device 130, a memory 140, a sensor module 150, Communication module 160 may be included. The electronic system 1 may further include an input / output module, a security module, a power control device, and the like. Some of the components of the electronic system 1 (CPU 110, RAM 120, neural network device 130, memory 140, sensor module 150 and communication module 160) Can be mounted on one semiconductor chip. In addition, the neural network device 130 may correspond to the neural network device 1000 of FIG.

The CPU 110 controls the overall operation of the electronic system 1. The CPU 110 may include one processor core or a plurality of processor cores (Multi-Core). CPU 110 may process or execute programs and / or data stored in memory 140. In one embodiment, the CPU 110 may control the functions of the neural network device 130 by executing programs stored in the memory 140.

The RAM 120 may temporarily store programs, data, or instructions. For example, the programs and / or data stored in the memory 140 may be temporarily stored in the RAM 120 according to the control of the CPU 110 or the boot code. The RAM 120 may be implemented as a memory such as DRAM (Dynamic RAM) or SRAM (Static RAM).

The neural network device 130 may perform an operation of the neural network based on the received input data, and may generate an information signal based on the result of the operation. Neural networks may include, but are not limited to, Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Deep Belief Networks, and Restricted Boltzman Machines.

The information signal may include one of various kinds of recognition signals such as a speech recognition signal, an object recognition signal, an image recognition signal, a biometric information recognition signal, and the like. For example, the neural network device 130 may receive frame data included in a video stream as input data, and generate a recognition signal for an object included in the image represented by the frame data from the frame data. However, the present invention is not limited thereto. Depending on the type or function of the electronic device on which the electronic system 1 is mounted, the neural network device 130 can receive various kinds of input data, can do.

The memory 140 is a storage area for storing data, and can store an OS (Operating System), various programs, and various data. In an embodiment, the memory 140 may store intermediate results, such as an output feature map, generated in the operation of the neural network device 130 in the form of an output feature list or an output feature matrix. In an embodiment, the memory 140 may store a compressed output feature map. The memory 140 may also store various parameters used in the neural network device 130, such as a weight map or a weight list.

The memory 140 may be a DRAM, but is not limited thereto. The memory 140 may include at least one of a volatile memory or a nonvolatile memory. The non-volatile memory may be a ROM, a PROM, an EPROM, an EEPROM, a flash memory, a phase-change RAM (PRAM), a magnetic RAM (MRAM) RRAM (Resistive RAM), FRAM (Ferroelectric RAM), and the like. The volatile memory includes a dynamic RAM (SRAM), a synchronous DRAM (SDRAM), a phase-change RAM (PRAM), a magnetic RAM (MRAM), a resistive RAM (RRAM), a ferroelectric RAM . In the embodiment, the memory 14 may be a hard disk drive (HDD), a solid state drive (SSD), a compact flash (CF), a secure digital (SD), a micro secure digital secure digital), xD (extreme digital), or a Memory Stick.

The sensor module 150 can collect information around the electronic device on which the electronic system 1 is mounted. The sensor module 150 may sense or receive a signal (e.g., a video signal, a voice signal, a magnetic signal, a biological signal, a touch signal, etc.) from the outside of the electronic device, and convert the sensed or received signal into data. To this end, the sensor module 150 may include a sensing device such as a microphone, an imaging device, an image sensor, a light detection and ranging (LIDAR) sensor, an ultrasonic sensor, an infrared sensor, a biosensor, Or the like.

 The sensor module 150 may provide the converted data to the neural network device 130 as input data. For example, the sensor module 150 can include an image sensor, which captures the external environment of the electronic device to produce a video stream, and provides a continuous data frame of the video stream to the neural network device 130 as input data Can be provided in order. However, the present invention is not limited to this, and the sensor module 150 may provide various kinds of data to the neural network device 130.

The communication module 160 may have various wired or wireless interfaces capable of communicating with external devices. For example, the communication module 160 may be a wireless local area network (LAN) such as a wired local area network (LAN), a wireless local area network (WLAN) such as Wi-fi (Wireless Fidelity), a wireless personal communication network (WPAN), Wireless USB (Universal Serial Bus), Zigbee, NFC (Near Field Communication), RFID (Radio Frequency Identification), PLC (Power Line Communication) And a communication interface connectable to a mobile cellular network, such as Long Term Evolution (LTE).

In an embodiment, the communication module 160 may receive a weight map from an external server. The external server can perform training based on a large amount of learning data and provide a weight map to the electronic system 1 including the training weight. The received weight map may be stored in the memory 1400.

The apparatus according to the above embodiments may include a processor, a memory for storing and executing program data, a permanent storage such as a disk drive, a communication port for communicating with an external device, a touch panel, a key, The same user interface device, and the like. Methods implemented with software modules or algorithms may be stored on a computer readable recording medium as computer readable codes or program instructions executable on the processor. Here, the computer-readable recording medium may be a magnetic storage medium such as a read-only memory (ROM), a random-access memory (RAM), a floppy disk, a hard disk, ), And a DVD (Digital Versatile Disc). The computer-readable recording medium may be distributed over networked computer systems so that computer readable code can be stored and executed in a distributed manner. The medium is readable by a computer, stored in a memory, and executable on a processor.

This embodiment may be represented by functional block configurations and various processing steps. These functional blocks may be implemented in a wide variety of hardware and / or software configurations that perform particular functions. For example, embodiments may include integrated circuit components such as memory, processing, logic, look-up tables, etc., that may perform various functions by control of one or more microprocessors or other control devices Can be employed. Similar to how components may be implemented with software programming or software components, the present embodiments may be implemented in a variety of ways, including C, C ++, Java (" Java), an assembler, and the like. Functional aspects may be implemented with algorithms running on one or more processors. In addition, the present embodiment can employ conventional techniques for electronic environment setting, signal processing, and / or data processing. Terms such as "mechanism", "element", "means", "configuration" may be used broadly and are not limited to mechanical and physical configurations. The term may include the meaning of a series of routines of software in conjunction with a processor or the like.

The specific implementations described in this embodiment are illustrative and do not in any way limit the scope of the invention. For brevity of description, descriptions of conventional electronic configurations, control systems, software, and other functional aspects of such systems may be omitted. Also, the connections or connecting members of the lines between the components shown in the figures are illustrative of functional connections and / or physical or circuit connections, which may be replaced or additionally provided by a variety of functional connections, physical Connection, or circuit connections.

In this specification (particularly in the claims), the use of the terms "above" and similar indication words may refer to both singular and plural. In addition, when a range is described, it includes the individual values belonging to the above range (unless there is a description to the contrary), and the individual values constituting the above range are described in the detailed description. Finally, if there is no explicit description or contradiction to the steps constituting the method, the steps may be performed in an appropriate order. It is not necessarily limited to the description order of the above steps. The use of all examples or exemplary terms (e. G., The like) is merely intended to be illustrative of technical ideas and is not to be limited in scope by the examples or the illustrative terminology, except as by the appended claims. It will also be appreciated by those skilled in the art that various modifications, combinations, and alterations may be made depending on design criteria and factors within the scope of the appended claims or equivalents thereof.

Claims (15)

A neural network processor comprising:
Input feature information indicating whether each of a plurality of input features of an input feature map has a non-zero value and a plurality of weights of a weight map each having a non- And determining a weight and an input feature to be convoluted among the plurality of input features and the plurality of weights based on the obtained input feature information and weight information, (fetch controller); And
And a data operation circuit that performs a convolution operation on the determined weight and the input feature to produce an output feature map. The method according to claim 1,
The data operation circuit comprising:
And selectively perform a convolution operation only on the determined weight and input feature among the plurality of input features and the plurality of weights. The method according to claim 1,
The fetch controller includes:
Performing an operation on the input feature information and the weight information to detect input features and weights having non-zero values,
The data operation circuit comprising:
And performs a convolution operation on the detected input features and weights. The method according to claim 1,
Wherein the input feature information comprises:
A zero-valued feature is represented by 0, a non-zero-valued feature includes an input feature vector denoted by 1,
The weight information includes:
A weight having a zero value is represented by 0, and a weight having a non-zero value includes a weight vector denoted by 1. The method according to claim 1,
If the determined input features and weights are a first input feature and a first weight, and a second input feature and a second weight,
The data operation circuit comprising:
In the current cycle, a first input feature and a first weight are read from the input feature map and the weight map to perform a convolution operation,
And in a subsequent cycle, reads a second input feature and a second weight from the input feature map and the weight map to perform a convolution operation. A method of operating a neural network processor,
Input feature information indicating whether each of a plurality of input features of an input feature map has a non-zero value and whether or not each of the plurality of weights of the weight map has a non- Obtaining weight information;
Determining input features and weights to be convoluted among the plurality of input features and the plurality of weights based on the obtained input feature information and weight information; And
And performing a convolution operation on the determined weight and the input feature to generate an output feature map. The method according to claim 6,
Wherein the generating comprises:
And selectively performing a convolution operation only on the determined weight and input feature among the plurality of input features and the plurality of weights. The method according to claim 6,
Wherein the determining comprises:
Performing an operation on the input feature information and the weight information to detect input features and weights having non-zero values,
Wherein the generating comprises:
And performing a convolution operation on the detected input features and weights. The method according to claim 6,
Wherein the input feature information comprises:
A zero-valued feature is represented by 0, a non-zero-valued feature includes an input feature vector denoted by 1,
The weight information includes:
Wherein a weight having a zero value is represented by 0, and a weight having a non-zero value includes a weight vector denoted by 1. The method according to claim 1,
If the determined input features and weights are a first input feature and a first weight, and a second input feature and a second weight,
Wherein the generating comprises:
Reading a first input feature and a first weight from the input feature map and the weight map in a current cycle to perform a convolution operation; And
And in the next cycle, reading a second input feature and a second weight from the input feature map and the weight map and performing a convolution operation. In a neural network device,
A processor array including a plurality of neural network processors;
A memory for storing an input feature map and a plurality of weight maps; And
And a controller for assigning the input feature map and the plurality of weight maps to the processor array,
The controller comprising:
Grouping the plurality of weight maps into a plurality of weight groups and assigning each of the plurality of weight groups to the processor array based on a ratio of a weight having a non-zero value in the weight map. 12. The method of claim 11,
The controller comprising:
And groups the plurality of weight maps into a plurality of weight groups so that the ratio of each of the weight maps included in the weight group is equalized. 12. The method of claim 11,
The controller comprising:
Group the plurality of neural network processors into a plurality of processor groups and sequentially assign each of the plurality of weight groups to each of the plurality of processor groups. 12. The method of claim 11,
The controller comprising:
Input feature information indicating whether each of the input features of the input feature map has a nonzero value and weight information indicating whether each of the weights of the plurality of weight maps has a nonzero value is provided to the processor array ,
The processor array comprising:
And performs a convolution operation on the input feature map and the plurality of weight maps based on the input feature information and the weight information to generate an output feature map. A computer-readable recording medium storing a program for causing a computer to execute the method according to any one of claims 6 to 10.

Images