JP6714297B2 - Processing device and inference processing method - Google Patents

Description

The present invention relates to, for example, a processing device that executes inference processing such as image recognition.

Conventionally, in order to perform inference processing such as image recognition, a computer equipped with a convolutional neural network (CNN: Convolutional Neural Network) has been used. When implementing CNN in a computer, the learned data including the coefficient used in CNN obtained in advance by deep learning etc. is stored in the main memory of the computer in an undistorted (uncompressed) state, or simply The vector is quantized and stored.

For example, the learned data may be provided to the computer via the network. The learned data has a relatively large data size, such as 200 MB, in the undistorted state. Therefore, when the data is transmitted via the network, the network is burdened and the transmission cost increases. is there.

On the other hand, for example, a technique of compressing and transmitting learned data has been proposed (for example, see Non-Patent Document 1).

As a technique for effectively utilizing the memory space in a neuro computer different from the Neumann computer, a main memory unit, a compression unit, and a decompression unit are added to a conventional neuro computer, and 16-bit weight value input data is divided into four stages. There is known a technique of performing data compression by classifying the data into a plurality of bits and assigning them to the lower bits in descending order of appearance frequency (for example, refer to Patent Document 1).

JP-A-6-176000

Deep-Compression-AlexNet, Internet (URL:https://github.com/songhan/Deep-Compression-AlexNet)

For example, according to the technique of Non-Patent Document 1, it becomes possible to transfer learned data in a network at a relatively high speed. However, when performing inference processing in a computer, all of the learned data used for inference processing is expanded in the memory as a distortion-free state, and a large amount of memory is consumed.

The present invention has been made in view of the above circumstances, and an object thereof is to provide a technique capable of reducing the memory capacity required for inference processing.

In order to achieve the above object, a processing device according to a first aspect is a processing device that executes inference processing on processing target data, and performs partial processing in each of a plurality of processing layers that configure inference processing. Stream data that obtains stream data in which processing coefficient information corresponding to processing coefficients used for partial processing in a plurality of processing layers that configure the inference processing and the plurality of partial processing units to be executed are arranged in the order of execution of the processing layers An acquisition unit and a processing coefficient supply unit that extracts processing coefficient information from the stream data and supplies the processing coefficient corresponding to the processing coefficient information to the partial processing unit that uses the processing coefficient corresponding to the processing coefficient information. ..

In the above processing device, at least a part of the processing coefficient information in the stream data is a compression processing coefficient obtained by compressing the processing coefficient, and the processing coefficient supply unit expands the compression processing coefficient and partially processes the expanded processing coefficient. It may be supplied to the department.

Further, in the above processing device, the stream data includes partial processing specifying information for specifying each partial processing in the plurality of processing layers, and a partial processing unit for executing the partial processing is executed based on the partial processing specifying information of the stream data. You may make it further have a partial process part construction part to build.

Further, in the above processing device, the stream data may include compression information indicating a compression method of the processing coefficient in the compression processing coefficient, and the processing coefficient supply unit may expand the compression processing coefficient based on the compression information. ..

Further, in the above processing device, the processing coefficient includes filter data including a value corresponding to each area of the two-dimensional plane, and the compression processing coefficient obtained by compressing the filter data is compression filter data obtained by predictively encoding the filter data. Yes, the compression information includes prediction direction information indicating the prediction direction used when predictively encoding the filter data, and the processing coefficient supply unit expands the compression filter data based on the prediction direction indicated by the prediction direction information. You may do so.

Further, in the above processing device, the plurality of partial processing units may include a convolution processing unit that performs a convolution process, and the processing coefficient may be data of a plurality of filters used in the convolution process.

Further, the above processing device may include an image processing processor suitable for executing image processing, and the convolution processing unit may be configured using the image processing processor.

Further, in the above processing device, the plurality of partial processing units include a total combination processing unit that performs processing by using all the processing results of the partial processing units of the immediately preceding processing layer as input, and the processing coefficient is input to the total combination processing unit. It may be a weighting coefficient for each of the processed results.

Further, in the above processing device, the stream data may be received from an external device via a communication network.

Further, in order to achieve the above object, the processing device according to the second aspect is a processing device that executes inference processing on data to be processed, and is a part of each of a plurality of processing layers forming the inference processing. A plurality of partial processing units that execute processing, a memory unit that stores compression processing coefficients obtained by compressing processing coefficients used in at least one or more partial processing units, and a first partial processing in the inference processing, A read control unit that reads out a compression processing coefficient corresponding to a processing coefficient used by the first partial processing unit that executes the first partial processing from the memory unit, expands the read compression processing coefficient, and expands the expanded processing coefficient. A decompression processing unit that is passed to the one-part processing unit.

Further, in order to achieve the above object, the inference processing method according to the third aspect is an inference processing method by a processing device that executes inference processing on data to be processed, and the processing device is provided with an inference processing. A plurality of partial processing units that execute respective partial processes in the plurality of constituent processing layers are constructed, and the processing coefficient information corresponding to the processing coefficients used in the partial processing in the plurality of processing layers that constitute the inference processing is provided as the processing. Acquires stream data arranged according to the execution order of layers, extracts processing coefficient information in order from the beginning of the stream data, and outputs processing coefficient based on the processing coefficient information to the partial processing unit that uses the processing coefficient corresponding to the processing coefficient information. To supply.

In order to achieve the above object, the inference processing method according to the fourth aspect is an inference processing method by a processing device that executes inference processing on data to be processed, and the processing device configures the inference processing. A plurality of partial processing units that execute respective partial processes in a plurality of processing layers, and a memory unit that stores compression processing coefficients obtained by compressing processing coefficients used in at least one or more partial processing units. When executing the first partial processing in the inference processing, the compression processing coefficient corresponding to the processing coefficient used by the first partial processing unit that executes the first partial processing is read from the memory unit, and the read compression processing coefficient is read. Is extended, and the extended processing coefficient is passed to the first partial processing unit.

According to the present invention, it is possible to reduce the memory capacity required for inference processing.

FIG. 1 is a functional configuration diagram of a processing device according to the first embodiment. FIG. 2 is a diagram illustrating the format of stream data according to the first embodiment. FIG. 3 is a diagram for explaining the formats of the stream header part and the layer header part according to the first embodiment. FIG. 4 is a diagram for explaining the format of the layer unique information section according to the first embodiment. FIG. 5 is a hardware configuration diagram of the processing device according to the first embodiment. FIG. 6 is a functional configuration diagram of the processing device according to the second embodiment. FIG. 7 is a functional configuration diagram of the processing device according to the third embodiment.

Some embodiments will be described with reference to the drawings. It should be noted that the embodiments described below do not limit the invention according to the claims, and all of the elements and combinations described in the embodiments are essential to the solution means of the invention. Not necessarily.

First, the processing apparatus according to the first embodiment will be described.

FIG. 1 is a functional configuration diagram of a processing device according to the first embodiment.

The processing device 100 includes a processing control unit 10, one or more processing units (an example of partial processing units) 11 (11-1 to 11-n), and one or more decoders 12 (12-1, 12-3, 12). -N-1 etc.). Here, the processing control unit 10 corresponds to the stream data acquisition unit and the partial processing unit construction unit. The processing control unit 10 and the decoder 12 correspond to the processing coefficient supply unit.

A convolutional neural network (CNN) that executes inference processing is installed in the processing device 100. The CNN is composed of a plurality of processing layers. In the example of FIG. 1, the CNN includes layers 1 to N. The processing (partial processing) in each layer is executed by each processing unit 11. The CNN executes, for example, an inference process that infers what the image data to be processed represents (for example, what is included in a person, a dog, a cat, etc.) and outputs the inference result. The number of layers in the CNN and the type of partial processing executed in each layer can be set arbitrarily. In the present embodiment, the number of layers and the partial processing executed in each layer can be changed according to the contents of stream data 50 described later.

The type of partial processing in the layers forming the CNN illustrated in FIG. 1 is an example, but in the example illustrated in FIG. 1, the processing unit 11 includes a convolution processing unit 11-1 that executes partial processing of Layer 1, The Relu processing unit 11-2 that executes the partial processing of the layer 2, the convolution processing unit 11-3 that executes the partial processing of the layer 3, and the total combination processing unit 11-n-1 that executes the partial processing of the layer N-1. , The SoftMax processing unit 11-n for executing the partial processing of the layer N.

The convolution processing units 11-1 and 11-3 perform convolution processing on the input image data using each of a plurality of filter data (an example of processing coefficients). The Relu processing unit 11-2 executes a process (referred to as a Relu process) of applying Relu (Rectified Linear Unit Rectifier) to each image generated in the immediately previous layer. The all-combining processing unit 11-n-1 executes a process of combining all the plurality of results from the immediately preceding layer as inputs and using a plurality of weighting coefficients (an example of processing coefficients).

The decoder 12 expands the compressed data (compression processing coefficient: compression filter data, compression weight data) passed from the processing control unit 10, and passes the data (processing coefficient) obtained by the expansion to the processing unit 11.

The processing control unit 10 acquires the stream data 50, and according to the stream data 50, constructs the processing unit 11 in the CNN, and controls the decoder 12 to expand the compressed data in the stream data 50. The stream data 50 may be extracted from, for example, a main memory 102 (see FIG. 5) or an auxiliary storage device 106 (see FIG. 5) described later, or may be received from an external device via the communication I/F 105 and a network. .. When receiving from the external device via the network, the processing unit 11 is constructed from the previous layer before the reception of all the stream data 50 is completed, and the inference processing in the CNN is started. It is possible to complete the inference process earlier than when the inference process is started after receiving all the necessary data.

Before describing the operation of the processing control unit 10 in detail, the format of the stream data 50 will be described in detail.

FIG. 2 is a diagram illustrating the format of stream data according to the first embodiment. FIG. 3 is a diagram for explaining the formats of the stream header part and the layer header part according to the first embodiment. FIG. 4 is a diagram for explaining the format of the layer unique information section according to the first embodiment.

The stream data 50 includes a stream header part 51 and one or more layer parts 52 corresponding to the respective layers forming the CNN. The layer units 52 are arranged in the order according to the execution order of the layers in the CNN. The layer section 52 includes a layer header section 61, a layer unique information section 62, and one or more channel sections 63. Depending on the type of the CNN layer corresponding to the layer section 52, the layer unique information section 62 and the channel section 63 may not exist. The channel section 63 includes one or more channel data 71.

As shown in FIG. 3A, the stream header section 51 includes an identifier, a version, a header size, a stream size, a layer number, an output vector size, a CNN input image vertical width, a CNN input image horizontal width, a CNN input image format, and a CNN. It includes fields such as the number of channels and flags.

An identifier for identifying the stream data 50 is stored in the identifier field. The version field stores the version to which the stream data 50 corresponds. The size of the stream header section 51 (for example, in units of 8 bytes) is stored in the header size field. The size of the stream data 50 (for example, in units of 8 bytes) is stored in the stream size field. The number of layers of the CNN corresponding to the stream data 50 is stored in the number of layers field. The output vector size field stores the size of the vector output as a result of CNN. The CNN input image vertical width field stores the vertical width (the number of vertical pixels) of the image data input to the CNN. The CNN input image width field stores the width (number of horizontal pixels) of the image data input to the CNN. The CNN input image format field stores information indicating the format (image format) of image data input to the CNN. The information indicating the image format includes, for example, "FIXED8" indicating an 8-bit integer type, half-precision floating point type (FP16), and the like. The number of input channels in the CNN (for example, the number of image data input as one processing target) is stored in the CNN input channel number field. Various information is contained in the flag field.

As shown in FIG. 3B, the layer header section 61 includes fields such as an identifier, a header size, a layer size, and a layer type. An identifier for identifying a layer is stored in the identifier field. The header size field stores the size of the layer header portion 61 (for example, in units of 8 bytes). The layer size field stores the size of the layer unit 52 (for example, in units of 8 bytes). In the layer type field, information (layer type information) indicating the type of layers forming the CNN is stored. As the layer type information, for example, “1” indicating a Convolution layer for performing a Convolution process, “2” indicating a Pooling layer for performing a Pooling process, and “3” indicating a Relu layer for performing a Relu process. , "4" indicating a FullConnection layer for performing a FullConnection process, and "5" indicating a SoftMax layer for performing a SoftMax process.

The layer unique information part 62 exists, for example, when the layer type of the layer type field of the corresponding layer head part 61 is a Convolution layer, a Pooling layer, and a FullConnection layer, and the layer types are Relu layer and SoftMax layer. In some cases, it doesn't exist.

When the layer type is the Convolution layer, the layer unique information section 62 includes fields for the number of input channels, the number of output channels, stride, kernel size, padding, and weight compression method, as shown in FIG. 4A.

The number of channels input to this layer is stored in the input channel number field. The number of output channels field stores the number of channels output from this layer. The stride field stores the amount (stride) of moving the convolution filter in the convolution process. The kernel size field stores the kernel size of the convolution filter used in the convolution process. The padding field stores the amount of padding added around the input data. The weight compression method field stores information (compression information) indicating the compression method for the filter data (filter data) used for this layer. In the present embodiment, the compression information is “0” in the case of no compression, “1” in the case of Pruning encoding, and “2” in the case of Weight Sharing encoding, and the prediction encoding. In the case of, it is set to "3".

When the layer type is the Pooling layer, the layer unique information section 62 includes fields for the number of input channels, the number of output channels, stride, and kernel size, as shown in FIG. 4B.

The number of channels input to this layer is stored in the input channel number field. The number of output channels field stores the number of channels output from this layer. The stride field stores an amount (stride) for moving the processing range in the pooling processing. The kernel size field stores the kernel size for pooling processing.

When the layer type is the total combined layer, the layer unique information section 62 includes fields for the number of input channels, the number of output channels, and the weight compression method, as shown in FIG.

The number of channels input to this layer is stored in the input channel number field. The number of output channels field stores the number of channels output from this layer. The weight compression method field stores information (compression information) indicating the compression method for the weight of the data of each input channel, which is used for all the combined layers. In the present embodiment, the compression information is “0” in the case of no compression, “1” in the case of Pruning encoding, and “2” in the case of Weight Sharing encoding, and the prediction encoding. In the case of, it is set to "3".

The channel part 63 exists when the layer type of the layer type field of the corresponding layer head part 61 is, for example, a Convolution layer or a Full Connection layer.

In the channel data 71 of the channel part 63, data compressed by the compression method corresponding to the compression information stored in the weight compression method field of the layer specific information part 62 corresponding to the channel part 63 (processing coefficient information, compression processing coefficient). : For example, in the case of the convolutional layer, the compressed data of the filter data (processing coefficient), and in the case of the fully connected layer, the compression data of the weight (processing coefficient) for each input channel is stored. The filter data may be compressed by predictive coding, for example. The predictive encoding of the filter data is performed by, for example, a predictive filter that is predicted based on a reference filter data (for example, immediately preceding filter data) and a prediction direction based on a pixel at a predetermined position (for example, an upper end or a left end). This is a method of compressing data by encoding a difference value with data by predetermined encoding (for example, Pruning encoding and Weight Sharing encoding). When the data is compressed by predictive coding as described above, information indicating the prediction direction used when predictively coding the channel data 71 is provided at the beginning of each channel data 71 (prediction direction information). ) Is stored.

Returning to FIG. 1, the processing operation of the processing control unit 10 will be described.

The processing control unit 10 reads the stream data 50 in order from the beginning, and when reading the layer unit 52, the processing unit 11 corresponding to the layer type stored in the layer type field of the layer header unit 61 of the layer unit 52. To build. For example, when the layer type corresponds to Convolution, a convolution processing unit is constructed, when it corresponds to Pooling, a Pooling processing unit is constructed, and when it corresponds to Relu, a Relu processing unit is constructed. Then, in the case of being compatible with all combinations, the all-combining processing unit is constructed, and in the case of being compatible with SoftMax, the SoftMax processing unit is constructed. Further, if the layer unique information section 62 exists next to the layer header section 61, the processing control section 10 sets the processing section 11 to be constructed according to the contents of each field of the layer unique information section 62.

Further, when the layer unique information section 62 of the layer section 52 has a weight compression method field (that is, when the layer is a convolutional layer or a fully combined layer), the processing control section 10 changes the weight compression method field. In accordance with the compression information, the decoder 12 is generated (or the decoder 12 is operated so as to execute the decompression process for the channel data 71 of the channel unit 63 and pass the data after the decompression process to the processing unit 11 that executes the process of the layer. Control). Thereby, the processing control unit 10 can pass necessary data to the processing unit 11. Here, when the prediction direction information of the predictive coding is included at the beginning of the channel data 71, the decoder 12 expands the predictively coded data based on the predictive direction information.

Next, the hardware configuration of the processing device 100 will be described in detail.

FIG. 5 is a hardware configuration diagram of the processing device according to the first embodiment.

The processing device 100 includes, for example, a CPU (Central Processing Unit) 101, a main memory 102, a GPU (Graphics Processing Unit) 103, a reader/writer 104, a communication interface (communication I/F) 105, and an auxiliary storage device 106. And a input/output interface (input/output I/F) 107, a display device 108, and an input device 109. The CPU 101, main memory 102, GPU 103, reader/writer 104, communication I/F 105, auxiliary storage device 106, input/output I/F 107, and display device 108 are connected via a bus 110.

The CPU 101 centrally controls the entire processing device 100. The CPU 101 executes various processes by reading the program stored in the auxiliary storage device 106 into the main memory 102 and executing the program. In the present embodiment, the CPU 101 configures, for example, the processing control unit 10, the Relu processing unit 11-2, the SoftMax processing unit 11n, and the decoder 12 by executing the processing program stored in the auxiliary storage device 106.

The main memory 102 is, for example, a RAM, a ROM, or the like, and stores programs (processing programs and the like) executed by the CPU 101 and various types of information. The auxiliary storage device 106 is, for example, a non-temporary storage device (nonvolatile storage device) such as an HDD (Hard Disk Drive) and an SSD (Solid State Drive), and stores programs executed by the CPU 101 and various information. .. The various information includes, for example, stream data as shown in FIG. 2 and processing target data such as image data. When the processing target data is image data, it may be image data captured by a camera (not shown) of the processing device 100, for example.

The GPU 103 is, for example, a processor suitable for executing a specific process such as image processing, and is suitable for executing processes performed in parallel, for example. In this embodiment, the GPU 103 executes a predetermined process according to an instruction from the CPU 101. In the present embodiment, the GPU 103 constitutes, for example, the convolution processing unit 11-1, the total combination processing unit 11-n-1 and the like.

The reader/writer 104 can attach and detach the recording medium 111, and reads data from the recording medium 111 and writes data to the recording medium 111. Examples of the recording medium 111 include a non-transitory recording medium (nonvolatile recording medium) such as an SD memory card, an FD (floppy disk: registered trademark), a CD, a DVD, a BD (registered trademark), and a flash memory. In the present embodiment, the processing program may be stored in the recording medium 111 and read by the reader/writer 104 to be used. Alternatively, the stream data may be stored in the recording medium 111, and the read/write unit 104 may read the stream data for use. Alternatively, the data to be processed may be stored in the recording medium 111, and the reader/writer 104 may read and use the data.

The communication I/F 105 is connected to a network (communication network) and transmits/receives data to/from another device (external device) connected to the network. For example, the stream data, the processing target data, and the like may be received from another device connected to the network via the communication I/F 105.

The input/output I/F 107 is connected to an input device 109 such as a mouse and a keyboard, and receives an operation input by the user from the input device 109.

The display device 108 is, for example, a display device such as a liquid crystal display, and displays and outputs various information.

Next, the processing operation of the processing apparatus 100 shown in FIG. 1 will be described.

The processing control unit 10 of the processing device 100 starts acquisition of predetermined stream data 50 used to execute the inference processing. When the processing control unit 10 receives the layer unit 52 corresponding to the convolution processing unit 11-1 of the layer 1 of the CNN of the acquired stream data 50, the processing control unit 10 generates the convolution processing unit 11-1 and outputs the decoder 12-1. To generate. Next, the processing control unit 10 sequentially supplies the compression filter data included in each channel data 71 of the layer unit 52 to the decoder 12-1. The timing at which the processing control unit 10 supplies the compressed filter data to the decoder 12-1 may be, for example, the timing according to the request for the filter data from the convolution processing unit 11-1.

On the other hand, the processing control unit 10 inputs the image data, which is the processing target data, to the convolution processing unit 11-1 of Layer 1. The convolution processing unit 11-1 performs convolution processing on the input image data using each filter data supplied from the decoder 12-1, and obtains all execution results using each filter data. The data is output so as to be passed to the processing unit 11 (Relu processing unit 11-2 in the example of FIG. 1) of the layer 2 in the subsequent stage. Here, when the execution result can be directly passed to the processing unit 11 of the layer 2, it may be directly passed, and when it cannot be directly passed, the processing unit of the layer 2 may be passed. 11 may be temporarily stored in a predetermined readable buffer.

When the processing control unit 10 receives the layer unit 52 corresponding to the Relu processing unit 11-2 of the CNN layer 2 of the stream data 50, the processing control unit 10 generates the Relu processing unit 11-2 of the CNN layer 2. The Relu processing unit 11-2 receives each of the images of the respective processing results by the convolution processing unit 11-1 as input, executes Relu processing, and outputs so as to pass it to the layer 3 in the subsequent stage.

Similarly, when the processing control unit 10 receives the layer unit 52 corresponding to the next layer of the stream data 50, the processing unit 11 (convolution processing unit 11-3, total combination processing unit 11-n-) for each layer. 1, SoftMax processing unit 11n, etc.) and, when necessary, a decoder 12 (12-3, 12-n-1 etc.) for expanding and supplying data to the processing unit 11 is generated. By executing such a series of processes, the result of the inference process (inference result) is output from the final layer of the CNN. The inference result may be output to the main memory 102 or the like. The processing device 100 may display the inference result on the display device 108 or execute the process based on the inference result.

According to the processing device 100 according to the first embodiment described above, each layer of the CNN can be appropriately constructed according to the stream data 50, and the necessary data can be acquired from the stream data 50 by the partial processing unit that executes the processing of the layer. Can be supplied. Therefore, it is necessary to previously store all the coefficients used in the processing units of all the layers of the CNN in the main memory 102, or store the information of the entire CNN layer configuration in the main memory 102. Disappear. As a result, the capacity required for the main memory 102 can be reduced. Further, since each layer of the CNN can be appropriately constructed according to the stream data 50, by changing the content of the stream data 50, it is possible to easily construct the CNN that executes a desired inference process. Further, since the channel data in the stream data 50 can be compressed data, the data amount of the stream data 50 can be reduced, and the processing time for reading the stream data 50, transmission, etc. can be reduced. be able to.

Next, a processing device according to the second embodiment will be described. In the second embodiment, the same parts as those in the processing device according to the first embodiment are designated by the same reference numerals.

FIG. 6 is a functional configuration diagram of the processing device according to the second embodiment.

The processing device 100A according to the second embodiment includes one or more processing units (partial processing units) 11 (11-1 to 11-n) and one or more decoders 13 (13-1, 13-3, 13-n). −1 etc.), a FIFO (First IN, First Out) 14, a construction unit 15, and a memory unit 16. The stream data according to the present embodiment is data excluding data regarding the CNN configuration in the stream data 50 according to the first embodiment, for example, the channel unit 63 including data (processing coefficient information) set in the processing unit 11. It is data including only. Here, the processing device 100A may be configured by the hardware illustrated in FIG. 5, and for example, the construction unit 15, the Relu processing unit 11-2, the total combination processing unit 11-n-1, the SoftMax processing unit 11n, and the The decoder 13 may be configured by the CPU 101. Further, the FIFO 14 may be composed of the CPU 101 and the main memory 102. The memory unit 16 may be composed of the main memory 102. The convolution processing units 11-1, 11-3 and the like may be configured by the GPU 103.

The memory unit 16 stores information (neural network structure information) 16a indicating the structure of each layer of CNN. The neural network structure information 16a is information that can identify what processing units, decoders, FIFOs, etc., each layer has in the CNN, and what kind of connection they have.

The construction unit 15 creates each processing unit 11, each decoder 13, and each FIFO 14 that configure the CNN based on the neural network structure information 16a of the memory unit 16 and sets the connection relationship among them.

The decoder 13 is arranged corresponding to each processing unit 11 that uses the data of the channel unit 63 in the stream data, acquires the stream data from the upstream side, and uses the data used by the processing unit 11 corresponding to itself. The data of the channel section 63 including the data is extracted, and the data is decompressed and supplied to the corresponding processing section 11. Further, the decoder 13 passes the data of the channel unit 63 and the subsequent data storing the data of the processing unit 11 corresponding to itself in the stream data to the FIFO 14 connected to the downstream side as it is.

The FIFO 14 stores the data of the stream data input from the decoder 13 on the upstream side, and transfers the data input first to the decoder 13 on the downstream side.

According to the processing device 100A according to the second embodiment described above, by transmitting the processing coefficient information used by the processing unit 11 as stream data, the partial processing in each processing unit 11 can be executed. Therefore, it is not necessary to store in the main memory 102 all the processing coefficients used by the processing units 11 of all the layers of the CNN in advance. Thereby, the required capacity of the main memory 102 can be suppressed. In addition, since the processing coefficient information stored in the channel unit 63 in the stream data is compressed data, the data amount of the stream data can be reduced, and the processing time for reading and transmitting the stream data can be reduced. It can be reduced.

Next, a processing device according to the third embodiment will be described. In the third embodiment, the same parts as those in the processing devices according to the first and second embodiments are designated by the same reference numerals.

FIG. 7 is a functional configuration diagram of the processing device according to the third embodiment.

The processing device 100B according to the third embodiment includes one or more processing units (partial processing units) 11 (11-1 to 11-n) and one or more decoders (decompression processing units) 12 (12-1, 12-). 3, 12-n-1), the read control unit 17, and the memory unit 16. Here, the processing device 100B may be configured by the hardware shown in FIG. 5, and for example, the read control unit 17, the Relu processing unit 11-2, the all-combining processing unit 11-n-1, the SoftMax processing unit 11n, The decoder 12 and the decoder 12 may be configured by the CPU 101. The convolution processing units 11-1, 11-3 and the like may be configured by the GPU 103. The memory unit 16 may be composed of the main memory 102.

The memory unit 16 stores neural network structure information 16a, compression filter data 16b as an example of compression coefficient information, and compression weight data 16c as an example of compression coefficient information. The compression filter data 16b is data obtained by compressing the filter data used by all the convolution processing units 11-1 and 11-3 of the CNN, and the compression weight data 16c is the total combination processing unit 11-n-1. This is data obtained by compressing a plurality of weights used in.

The read control unit 17 receives a request from the CNN processing unit 11 for a processing coefficient required to execute a process, and compresses the requested processing coefficient to be used by the processing unit 11 (compressed coefficient data (compression). The filter data 16b or a part of the compression weight data 16c) is acquired from the memory unit 16 and the compression coefficient data is transmitted to the decoder 12 connected to the processing unit 11. The processing unit 11 may request the processing coefficient necessary for executing the processing, for example, immediately before the processing unit 11 executes the processing, or the processing of a relatively close layer is executed. It may be performed when there is.

According to the processing device 100B of the third embodiment described above, since the coefficient data used by the processing unit 11 is compressed and stored in the main memory 102, it is possible to suppress the capacity required in the main memory 102. Further, since the processing units 11 of the layers that are relatively close to each other request the processing coefficients, the number of the processing units 11 requesting the processing coefficients can be suppressed, and the load on the CPU 101 and the main memory 102. The used capacity can be relatively reduced.

It should be noted that the present invention is not limited to the above-described embodiment, and can be appropriately modified and carried out without departing from the spirit of the present invention.

For example, in each of the above-described embodiments, an example in which the present invention is applied to a processing device including a CNN that performs image recognition has been described. However, the present invention is a neural network that executes inference processing other than image recognition. The present invention can also be applied to a processing device configured with.

Further, in the first and second embodiments, the stream data 50 is configured to include processing coefficients such as filter data used by the processing unit 11 after being compressed, but the present invention is not limited to this. Alternatively, at least a part of these processing coefficients may be included in the stream data 50 without being compressed. Even in this case, since it is not necessary to store the entire stream data 50 in the main memory 102, the capacity required for the main memory 102 can be suppressed and it can be used for other processing in the main memory 102. The capacity can be increased appropriately.

Further, in the second embodiment, the FIFO 14 is provided between the decoder 13 and the decoder 13, but the present invention is not limited to this, and the decoder 14 may be provided with the function of the FIFO 14. Good.

Further, in each of the above-described embodiments, at least a part of the functional unit configured by the CPU 101 or the GPU 103 executing a program is configured by another processor or a hardware circuit that executes a specific process. Good. For example, the processing unit 11 (for example, the convolution processing units 11-1, 11-3, etc.) configured by the GPU 103 may be configured by the CPU 101, or may be configured by another hardware circuit. .. Further, the decoder 13 configured by the CPU 101 may be configured by another hardware circuit.

10... Processing control unit, 11... Processing unit, 12, 13... Decoder, 14... FIFO, 15... Construction unit, 16... Memory unit, 17... Read control unit, 50... Stream data, 100, 100A, 100B... Processing device , 101... CPU, 102... Main memory, 103... GPU

Claims (7)

A processing device that performs inference processing on data to be processed,
A plurality of partial processing units that execute respective partial processing in a plurality of processing layers that form the inference processing;
A stream data acquisition unit that acquires stream data in which processing coefficient information corresponding to processing coefficients used in partial processing in a plurality of processing layers that form the inference processing is arranged in accordance with an execution order of the processing layer,
A processing coefficient supply unit that extracts the processing coefficient information from the stream data and supplies a processing coefficient corresponding to the processing coefficient information to a partial processing unit that uses the processing coefficient corresponding to the processing coefficient information;
A processing device comprising. At least a part of the processing coefficient information in the stream data is a compression processing coefficient obtained by compressing the processing coefficient,
The processing device according to claim 1, wherein the processing coefficient supply unit expands the compression processing coefficient and supplies the expanded processing coefficient to the partial processing unit. The stream data includes partial processing specifying information for specifying each partial processing in the plurality of processing layers,
The processing device according to claim 1, further comprising a partial processing unit construction unit that constructs a partial processing unit that executes the partial processing based on the partial processing specifying information of the stream data. The stream data includes compression information indicating a compression method of the processing coefficient in the compression processing coefficient,
The processing device according to claim 2, wherein the processing coefficient supply unit expands the compression processing coefficient based on the compression information. The processing coefficient includes filter data including values corresponding to respective regions of a two-dimensional plane, and the compression processing coefficient obtained by compressing the filter data is compression filter data obtained by predictively encoding the filter data,
The compression information includes prediction direction information indicating a prediction direction used when predictively encoding the filter data,
The processing device according to claim 4, wherein the processing coefficient supply unit expands the compression filter data based on a prediction direction indicated by the prediction direction information. The processing device according to any one of claims 1 to 5, wherein the stream data is received from an external device via a communication network. An inference processing method by a processing device that executes inference processing on data to be processed,
In the processing device, a plurality of partial processing units that execute respective partial processing in a plurality of processing layers that constitute the inference processing are constructed,
Obtaining stream data in which processing coefficient information corresponding to processing coefficients used in partial processing in a plurality of processing layers that form the inference processing is arranged in accordance with the execution order of the processing layers,
The processing coefficient information is taken out in order from the beginning of the stream data, and the processing coefficient based on the processing coefficient information is supplied to the partial processing unit that uses the processing coefficient corresponding to the processing coefficient information.
Inference processing method.