JP7299770B2 - Arithmetic processing device and arithmetic processing method - Google Patents

Description

The present invention relates to a technique of inputting input data into a hierarchical neural network and performing arithmetic processing in each layer of the hierarchical neural network to calculate a feature plane in the layer.

A hierarchical computation method called a Convolutional Neural Network (hereinafter abbreviated as CNN) is attracting attention as a method that enables robust pattern recognition against variations in recognition targets. For example, Non-Patent Document 1 discloses various application examples and implementation examples.

FIG. 9 is a diagram showing an example of a simple CNN network configuration. An input layer 901 corresponds to image data of a predetermined size when image data is processed by CNN. 903 a to 903 d indicate characteristic surfaces of the first layer 908 , 905 a to 905 d indicate characteristic surfaces of the second layer 909 , and 907 indicate characteristic surfaces of the third layer 910 . A feature plane is a data plane corresponding to a processing result of a predetermined feature extraction operation (convolution operation and non-linear processing). A feature plane corresponds to a feature extraction result for recognizing a predetermined target in a higher hierarchy, and is a processing result for image data, so the processing result is also represented by a plane. Feature planes 903 a to 903 d are generated by convolution calculation and nonlinear processing corresponding to the input layer 901 . For example, the feature plane 903a is calculated by a two-dimensional convolution calculation schematically shown as 9021a and nonlinear transformation of the calculation result. For example, a convolution operation with a kernel (coefficient matrix) size of columnSize×rowSize is processed by a sum-of-products operation as shown in the following equation (1).

input(x, y) : reference pixel value at coordinates (x, y) output(x, y) : calculation result at coordinates (x, y) weight(column, row) : calculation of output (x, y) columnSize, rowSize : Convolution kernel size L : Number of feature maps in the previous layer In processing by CNN, multiple convolution kernels are scanned pixel by pixel, and product-sum operations are repeated, and the final product-sum A feature plane is calculated by nonlinearly transforming the result. Note that when calculating the characteristic plane 903a, the number of convolution kernels is one because the number of connections with the previous layer is one. Here, 9021b, 9021c, and 9021d are convolution kernels used when calculating feature planes 903b, 903c, and 903d, respectively.

FIG. 10 is a diagram illustrating an example of calculating the characteristic plane 905a. Characteristic plane 905a is connected to characteristic planes 903a to 903d in first layer 908, which is the previous layer of second layer 909 to which characteristic plane 905a belongs. When calculating the data of the characteristic plane 905a, the convolution processing unit 1001 performs a filtering operation using a kernel schematically indicated by 9041a on the characteristic plane 903a. Hold. Convolution operation processing unit 1001 performs convolution operation of kernels indicated by 9042 a , 9043 a , and 9044 a on characteristic planes 903 b , 903 c , and 904 c , respectively, and accumulates the result of the filter operation in cumulative adder 1002 . After completing the four types of convolution calculations, the cumulative adder 1002 performs cumulative addition of the four types of convolution calculations. A nonlinear transformation processing unit 1003 performs nonlinear transformation processing using a logistic function or a hyperbolic tangent function (tanh function) on the result of the cumulative addition.

By performing the above processing while scanning the entire image pixel by pixel, the feature plane 905a is calculated. Similarly, the feature plane 905b is calculated by convolution operation of four kernels indicated by 9041b/9042b/9043b/9044b with respect to the feature plane of the first layer 908 which is the previous layer, cumulative addition, and non-linear processing. This also applies to the characteristic surfaces 905c and 905d. Furthermore, a feature plane 907 is calculated using convolution operations of four kernels indicated by 9061, 9062, 9063, and 9064 for the feature planes 905a to 905d of the second layer 909, which is the previous layer. It is assumed that each kernel coefficient is determined in advance by learning using a general method such as perceptron learning or backpropagation learning.

When CNN processing hardware that performs computation using CNN is installed in an embedded system to perform network processing, the CNN processing hardware performs computation using input data and weighting factors for each layer. Then, the CNN processing hardware uses the result of the calculation as the input data of the next layer, repeats the calculation with the weight coefficient of the next layer, and obtains the final pattern recognition result.

Processing by CNN requires an enormous number of multiply-accumulate operations in order to repeat a large number of convolutions, so CNN processing hardware is required to perform processing at high speed.

Patent No. 5368687 Japanese Patent Laid-Open No. 61-62187

Yann LeCun, Koray Kavukvuoglu and Clement Farabet: Convolutional Networks and Applications in Vision, Proc. International Symposium on Circuits and Systems (ISCAS'10), IEEE, 2010,

In the technique of Patent Document 1, an SRAM is provided inside the CNN processing hardware and used as an intermediate buffer for storing some or all of the feature planes 903a-d and 905a-d of the intermediate hierarchy. A convolution operation is performed on the data read from the intermediate buffer, and the processing result obtained by the convolution operation is stored in the intermediate buffer. By configuring the intermediate buffer with SRAM, the data access time is reduced and the processing efficiency is improved.

Furthermore, in order to reduce the data access time, Patent Document 2 and the like disclose a technique for increasing the speed by storing multiple lines required for a general filter operation in separate memories and simultaneously reading them out.

Thus, in order to efficiently perform sum-of-products operations, the CNN processing hardware is internally provided with a plurality of SRAMs so that a plurality of data can be read and written simultaneously with a short access time, and a plurality of lines are stored in separate memories. stored in and processed.

On the other hand, the pattern recognition results output by the CNN processing hardware are post-processed to identify the position coordinates of the detection target. Post-processing includes, for example, correction processing for increasing the accuracy of detection positions, extraction of detection positions, and processing for merging duplicate discrimination results. It is possible to improve the accuracy of the determination result. Post-processing requires flexibility, so processing by a general-purpose CPU is suitable. In general, the computation result output by the CNN processing hardware is transferred to a shared memory used by the general-purpose CPU using a DMAC or the like, and processed by the general-purpose CPU. Therefore, it takes time to transfer data to the shared memory, which lowers the processing performance. In addition, since data is transferred via the bus when transferring to the shared memory, there is a problem that the bus band becomes tight. Furthermore, there is the problem that the memory of the CNN processing hardware and the shared memory are required separately, which increases the cost.

In the present invention, when input data is input to a hierarchical neural network and arithmetic processing is performed in each layer of the hierarchical neural network, the processing speed can be increased without lowering the processing performance, and the cost can be reduced. We provide technology to

One aspect of the present invention is an arithmetic processing device that inputs input data to a hierarchical neural network and performs arithmetic processing in each layer of the hierarchical neural network,
a calculation unit that calculates the characteristic surface in each layer by referring to the characteristic surface in the layer before the layer ;
a feature plane holding unit having a plurality of memories that hold the feature planes calculated and referred to by the calculation unit;
Based on network information , which is information about each layer on which the arithmetic processing is performed, the characteristic plane calculated by the arithmetic unit is arranged and written in the plurality of memories, and is referred to by the arithmetic unit from the plurality of memories. a memory access manager that reads the feature plane ;
The characteristic surface holding unit is address-mapped in a memory space, and based on the network information, calculates an address of a pixel value of the characteristic surface that is address-mapped in the memory space, and uses the address to calculate the characteristic surface holding unit. and a processor for reading and processing pixel values from .

According to the configuration of the present invention, there is no need to transfer data to the shared memory, and processing can be speeded up without lowering processing performance. Furthermore, there is no need to separately provide a shared memory, and costs can be reduced.

FIG. 3 is a block diagram showing a hardware configuration example of a recognition processing unit 801; 4 is a flowchart showing the overall operation of the image processing system; 10 is a flowchart showing details of processing in step S206. FIG. 10 is a diagram showing an example of an allocation method for allocating the characteristic planes of the first layer 908 and the second layer 909 to the memory of the characteristic plane holding unit 1021; FIG. 5 is a diagram showing an example of a method of allocating addresses of characteristic planes of each memory allocated according to the allocation method shown in FIG. 4; 4 is a flowchart of processing in step S307. FIG. 4 is a diagram showing part of a memory map of the CPU 104; FIG. 2 is a block diagram showing a hardware configuration example of an image processing system; The figure which shows the network configuration example of simple CNN. FIG. 9 is a diagram for explaining an example of calculating a characteristic plane 905a; 4 is a block diagram showing a configuration example of a recognition processing unit 801; FIG.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. It should be noted that the following embodiments do not limit the invention according to the scope of claims. Although multiple features are described in the embodiments, not all of these multiple features are essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant description is omitted.

[First embodiment]
First, a hardware configuration example of an image processing system using a recognition processing device as an arithmetic processing device according to the present embodiment will be described with reference to the block diagram of FIG. The image processing system according to this embodiment has a function of detecting a specific object area from input image data.

The image input unit 800 acquires image data as input data by performing imaging. The image data may be image data of each frame in a moving image, or may be still image data. The image input unit 800 includes an optical system, a photoelectric conversion device such as a CCD (Charge-Coupled Devices) or a CMOS (Complementary Metal Oxide Semiconductor) sensor, a driver circuit for controlling the sensor, an AD converter, a signal processing circuit for performing various image corrections, and a It is composed of a frame buffer or the like.

A recognition processing unit 801 includes a recognition processing device as an arithmetic processing device according to the present embodiment, and detects a region of a specific object from image data from the image input unit 800 . A DMAC (Direct Memory Access Controller) 805 controls data transfer between each processing unit on the image bus 802 and each processing unit on the CPU bus 809 .

Bridge 803 provides a bridging function of image bus 802 and CPU bus 809 .

A preprocessing unit 804 performs various preprocessing for effectively performing recognition processing by the recognition processing unit 801 . Specifically, the preprocessing unit 804 performs image conversion processing such as color conversion processing/contrast correction processing on the image data acquired by the image input unit 800 by hardware.

The CPU 806 executes various processes using computer programs and data stored in the ROM 807 and RAM 808 . As a result, the CPU 806 controls the operation of the entire image processing system, and executes or controls each process described later as what the image processing system performs.

A ROM (Read Only Memory) 807 stores computer programs, data, and the like including instructions that define the operation of the CPU 806 . Information handled as known information by the image processing system in the following description is stored in the ROM 807 . The data stored in the ROM 807 also includes a data set for operating the recognition processing unit 801 (operation parameters and weighting coefficients according to CNN, which is an example of a hierarchical neural network). This data set is input to the recognition processing unit 801 via the DMAC 805 .

A RAM 808 has an area for storing computer programs and data loaded from the ROM 807, an area for holding image data acquired by the image input unit 800, an area for temporarily holding data to be processed, and the like. have The RAM 808 also has a work area used when the CPU 806 executes various processes. Thus, RAM 808 can provide various areas as appropriate.

The image data acquired by the image input unit 800 is preprocessed by the preprocessing unit 804 , temporarily stored in the RAM 808 , and transferred (inputted) to the recognition processing unit 801 by the DMAC 805 . The recognition processing unit 801 detects a region of a specific object in the image represented by the image data by performing predetermined discrimination processing on the input preprocessed image data on a pixel-by-pixel basis. Information related to the area detected by the recognition processing unit 801 (information defining the area in the image, the image in the area, etc.) is transferred to the RAM 808 by the DMAC 805 and stored in the RAM 808 .

Next, a hardware configuration example of the recognition processing unit 801 will be described using the block diagram of FIG. The control unit 101 controls the entire recognition processing unit 801 . The control unit 101 has a network information holding unit 1011 therein, and network information is set (stored) in the network information holding unit 1011 . The DMAC 805 transfers the network information previously stored in the ROM 807 to the network information holding unit 1011 and sets it. The network information includes the number of layers to be processed, the number of layers, information on feature planes for each layer (width and height of feature planes, number of feature planes), information on connections between layers (horizontal size of convolution kernel, vertical size).

The control unit 101 also includes an enable register as a register for operation control, and the CPU 806 instructs the start of processing. The CPU 806 instructs the control unit 101 to start processing for multiple layers, and the control unit 101 instructs the feature plane storage unit 102 and the convolution calculation unit 103 to start processing for each layer multiple times. The control unit 101 transmits the network information set (held) in the network information holding unit 1011 as a control parameter together with a control signal indicating an instruction to start processing.

When post-processing is applied to the feature plane of the generated hierarchy, when the feature plane storage unit 102 and the convolution calculation unit 103 complete the processing for each layer, the control unit 101 stores the network information set in the network information holding unit 1011. It is transmitted to the CPU 104 as a control parameter. Then, the control unit 101 issues an instruction to start processing.

The feature plane storage unit 102 is for storing feature planes, and includes a feature plane holding unit 1021 and a memory access management unit 1022 that controls reading and writing of data in the feature plane holding unit 1021 .

The characteristic plane holding unit 1021 has (N+1) dual-port SRAMs (memory 0 to memory N) (where N is an integer equal to or greater than 1). In the following description, it is assumed that N=5 as an example. That is, in the following description, it is assumed that the characteristic plane holding unit 1021 has six dual-port SRAMs (memory 0 to memory 5). Also, the data width of the SRAM is assumed to be 4 bytes.

Memory access management unit 1022 controls one port on one side of the dual port SRAM interface. Based on the control parameters (layer network information) received from the control unit 101, the memory access management unit 1022 transfers the feature plane generated for each layer, which is the calculation result of the convolution calculation unit 103, to the six dual-port SRAMs. Decide whether to place it in and write it. Also, the memory access management unit 1022 reads the reference feature plane and transmits it to the convolution operation unit 103 . When memory access management unit 1022 receives all characteristic planes of the hierarchy to be generated from convolution operation unit 103, memory access management unit 1022 notifies control unit 101 of completion.

The convolution operation unit 103 performs a convolution operation, receives a weighting factor corresponding to the filter kernel size and a reference feature plane, and outputs a feature plane as a calculation result. The DMAC 805 transfers weighting coefficients stored in advance in the ROM 807 to the convolution unit 103 . The convolution calculation unit 103 receives the filter kernel size as a control parameter from the control unit 101, and when instructed to start processing, performs calculation processing. Notify completion.

The CPU 104 performs post-processing of the feature plane. The CPU 104 is the bus master of the image bus 802, and the memory of the characteristic surface holding unit 1021 is address-mapped in the memory space via the memory control unit 105, which will be described later. Upon receiving a processing start instruction from the control unit 101, it acquires and processes data based on the network information of the layer to be processed.

The memory control unit 105 controls reading and writing of data with respect to the dual port SRAM of the characteristic plane holding unit 1021 , and controls one port on one side of the six dual port SRAM interfaces of the characteristic plane holding unit 1021 . The memory control unit 105 is a bus slave of the image bus 802, and receives a request from the bus master to read/write data from/to the memory.

Next, the overall operation of the image processing system will be described according to the flowchart of FIG. In step S<b>202 , the CPU 806 acquires image data (input data) via the image input unit 800 . In step S<b>203 , the preprocessing unit 804 preprocesses the image data acquired in step S<b>202 and stores it in the RAM 808 .

In step S204, the CPU 806 sets the start position address of the data set of the layer to be processed stored in the ROM 807 in the DMAC 805 and starts it. The number of layers to be processed may be one or plural. As a result, the DMAC 805 reads and transfers the data set of the layer to be processed from the set start address.

After the transfer is completed, in step S205, the CPU 806 again sets the start position address of the preprocessed data in the RAM 808 in the DMAC 805 and activates it. As a result, the DMAC 805 reads and transfers the preprocessed data from the set start address.

In step S206, the CPU 806 activates the recognition processing unit 801, so the recognition processing unit 801 executes processing. When the processing of the recognition processing unit 801 is completed, the results of the processing (detection results and intermediate hierarchical feature data) are stored in the RAM 808 again.

In step S207, the CPU 806 determines whether or not the processing of all hierarchies has been completed. As a result of this determination, if the processing of all layers has been completed, the processing according to the flowchart of FIG. 2 ends. On the other hand, if there remains a hierarchy for which processing has not been completed yet, the process proceeds to step S204, and the processing after step S204 is performed for the hierarchy for which processing has not been completed yet.

In a small-scale neural network for embedded devices, it is possible to process all layers at once. In this case, in step S207, it is determined that "processing of all layers has been completed", and the process proceeds to the flow chart of FIG. Following processing ends. On the other hand, since a large-scale neural network cannot process all layers at once, the neural network is processed by time division. In this case, in step S207, if there remains a hierarchy that has not been subjected to time-divisional processing, the process proceeds to step S204, and the remaining hierarchy is processed. In the processing of step S205 from the second time onward, the processing result of the recognition processing unit 801 stored in the RAM 808 is transferred as data to be processed.

Next, the details of the processing in step S206 will be described with reference to the flowchart of FIG. A control unit 101 controls the whole. When the processing of step S205 is completed, the information about the layer to be processed by the recognition processing unit 801 is set in the network information holding unit 1011. FIG.

In step S302, the control unit 101 transmits the information held in the network information holding unit 1011 to the feature plane storage unit 102 and the convolution calculation unit 103 as control parameters for each layer, and issues a processing start instruction.

In step S303, the feature plane storage unit 102 determines whether or not the processing is for the input layer. As a result of this determination, if the process is for the input layer, the process proceeds to step S304, and if the process is for a layer other than the input layer, the process proceeds to step S305.

In step S304, the feature plane storage unit 102 outputs the image data as a reference feature plane. On the other hand, in step S305, the feature plane storage unit 102 reads the feature plane of the previous layer from the feature plane storage unit 1021 and outputs it as a reference feature plane.

In step S<b>306 , the convolution calculation unit 103 performs convolution calculation based on the reference feature plane and the weighting factor, and transmits the calculation result to the feature plane storage unit 102 . In step S307, the memory access management unit 1022 determines the memory layout of the generated feature planes, and stores the feature planes in the feature plane storage unit 1021 according to the determined memory layout. Details of the memory arrangement will be described later with reference to FIGS.

Next, in step S308, the feature plane storage unit 102 determines whether generation of all feature planes has been completed. If the result of this determination is that all feature planes have been generated, the process proceeds to step S309. On the other hand, if there remains a feature plane for which generation has not been completed yet, the process proceeds to step S303, and the processing from step S303 onward is performed for the feature plane for which generation has not been completed.

In step S<b>309 , the feature plane storage unit 102 and the convolution calculation unit 103 send a completion notification to the control unit 101 . Upon receiving the completion notification, the control unit 101 determines whether or not the processing of the final layer has been completed.

As a result of this determination, if the processing of the final layer has been completed, the process proceeds to step S310. On the other hand, if the processing of the final layer has not been completed yet, the process proceeds to step S302, and the control unit 101 instructs the processing of the next layer.

In step S310, when the processing of the final layer is completed, the control unit 101 gives the network information such as the number of the final layer to the CPU 104, and issues a processing start instruction. Based on the network information, the CPU 104 reads out the characteristic surface of the final layer, identifies the position coordinates, and terminates the process. Details of the processing in step S310 (steps S3101 to S3104) will be described later.

Next, memory allocation controlled by the memory access management unit 1022 will be described with reference to FIGS. FIG. 4 is a diagram showing an example of an allocation method for allocating the characteristic planes of the first hierarchy 908 and the second hierarchy 909 in FIG.

The feature planes in the feature plane storage unit 102 are interleaved in a plurality of memories on a hierarchical basis. Further, the feature planes in the feature plane storage unit 102 are interleaved in a plurality of memories in units of lines.

Pixel values of characteristic planes 903a to 903d of the first layer 908 and characteristic planes 905a to 905d of the second layer 909 are shown on the left side of FIG. The pixel value of the characteristic plane 903a at the position (x, y) when the position of the upper left corner of the characteristic plane 903a is (0, 0) is expressed as a(x, y). The pixel value of the characteristic plane 903b at the position (x, y) when the position of the upper left corner of the characteristic plane 903b is (0, 0) is expressed as b(x, y). The pixel value of the characteristic plane 903c at the position (x, y) when the position of the upper left corner of the characteristic plane 903c is (0, 0) is expressed as c(x, y). The pixel value of the characteristic plane 903c at the position (x, y) when the position of the upper left corner of the characteristic plane 903c is (0, 0) is expressed as c(x, y). The pixel value of the characteristic plane 905a at the position (x, y) when the position of the upper left corner of the characteristic plane 905a is (0, 0) is expressed as a(x, y). The pixel value of the characteristic plane 905b at the position (x, y) when the position of the upper left corner of the characteristic plane 905b is (0, 0) is expressed as b(x, y). The pixel value of the characteristic plane 905c at the position (x, y) when the position of the upper left corner of the characteristic plane 905c is (0, 0) is expressed as c(x, y). The pixel value of the characteristic plane 905d at the position (x, y) when the position of the upper left corner of the characteristic plane 905d is (0, 0) is expressed as d(x, y).

In FIG. 4, the horizontal size of the feature surface of the first layer 908 is 32 pixels and the vertical size is 32 pixels, and the horizontal size of the feature surface of the second layer 909 is 16 pixels and the vertical size is 16 pixels. is 16 pixels. The pixel value data for one coordinate is assumed to be 1 byte. The right side of FIG. 4 shows an example of how each feature plane is allocated in memory.

The memory group of the characteristic plane holding unit 1021 is divided into two groups, one group being the first memory group and the other group being the second memory group. In this embodiment, since the characteristic plane holding unit 1021 has six dual-port SRAMs (memory 0 to memory 5), the memory 0 to memory 2 are the first memory group, and the memory 3 to memory 5 are the second memory group. and Characteristic planes are alternately arranged in the order of the first memory group and the second memory group for each layer. As a result, the input feature plane and the output feature plane can be arranged in separate memories, reading and writing can be performed simultaneously, and high-speed processing can be performed. Furthermore, one feature plane is assigned to a different memory for each line (a group of pixels having the same Y coordinate is called a line). That is, in the case of this embodiment, three consecutive lines of characteristic planes are arranged in different memories. As a result, when the size of the convolution kernel is 3×3 or less, the pixels of the reference feature plane to be input to the convolution operation unit 103 can be read out simultaneously, enabling high-speed processing. When the convolution kernel size is larger than 3×3, it can be read in 2 cycles.

According to the above description, as shown on the right side of FIG. 4, memory 0, memory 1, and memory 2 are alternately assigned to the characteristic planes 903a to 903d of the first layer 908 for each line. Since the memory allocated to the last line of the characteristic plane 903a is memory 1, the next memory, memory 2, is allocated to the first line of the characteristic plane 903b. For efficient use of memory, memory is allocated when crossing feature planes in the same way as for consecutive lines. Similarly, memory 1, memory 2, and memory 0 are alternately assigned to each line of the characteristic plane 903c, and memory 0, memory 1, and memory 2 are alternately assigned to each line of the characteristic plane 903d.

Memory 3, memory 4, and memory 5 are alternately assigned to the characteristic planes 905a to 905d of the second layer 909 for each line. Since the memory assigned to the last line of the feature plane 905a is memory 3, the next memory, memory 4, is assigned to the first line of the feature plane 905b. For efficient use of memory, memory is allocated when crossing feature planes in the same way as for consecutive lines. Similarly, memory 3, memory 4, and memory 5 are alternately assigned to each line on the characteristic surface 905c, and memory 3, memory 4, and memory 5 are alternately assigned to each line of the characteristic surface 905d.

FIG. 5 is a diagram showing an example of a method of allocating addresses of characteristic planes of each memory allocated according to the allocating method shown in FIG. Since the memory has a data width of 4 bytes, 4 pixels of data are stored at one address. It is assumed that pixels with smaller x-coordinates are packed into lower bytes.

In the memory 0, the pixel values of the pixels on the first line of the characteristic surface 903a are stored at addresses 0 to 0x7, the pixel values of the pixels on the fourth line are stored at addresses 0x8 to 0xf, and the pixel values of the pixels on the seventh line are stored at addresses 0x10 to 0x17. . . and pixel values of 32 pixels with a 3-line offset are stored. The data size of the pixel values of these 32 pixels is 32 bytes.

After storing the pixel values for 11 lines of the characteristic plane 903a, the pixel values of the pixels of the second line of the characteristic plane 903b are stored from the next address 0x58. Similarly, pixel values are stored for the characteristic planes 903c and 903d. In the memory 1, 32 pixels are stored sequentially from the second line data of 903a at addresses 0 to 0x7 with an offset of 3 lines, and pixel values of characteristic planes 903b to 903d are also stored. The pixel values of the characteristic planes 903a to 903d are also stored in the memory 2 in the same manner. The memories 3 to 5 similarly store the pixel values (16 bytes) of 16 pixels of the characteristic planes 905a to 905d of the second layer 908 with a 3-line offset.

Next, in step S307 described above, the memory in which the memory access management unit 1022 stores the pixel value at the coordinates (x, y) of the feature plane with the feature plane number n and the method for determining the address are described in FIG. Description will be made according to the flow chart.

In step S302, the control unit 101 stores the layer number of the layer to be processed and the layer to be generated, the horizontal size w of the feature plane, the vertical size h of the feature plane, and the number of feature planes as control parameters for each layer. It transmits to the section 103 . It is assumed that the convolution operation unit 103 outputs data of all feature planes in raster order. When the convolution calculation unit 103 outputs, the processing according to the flowchart of FIG. 6 is started.

In step S602, the memory access management unit 1022 obtains the serial number L of all the feature plane lines with the feature plane number n and the coordinates (x, y) by calculating L=(n−1)×h+y.

In step S<b>603 , the memory access management unit 1022 determines the storage destination memory number mem by calculating the remainder of the serial number L according to the number of memories in the feature plane holding unit 1021 . That is, the memory access management unit 1022 obtains the storage destination memory number mem by calculating mem=mod (L, (number of memories/2)).

If mem=0 (when the remainder calculation result is 0), the process proceeds to step S604. In step S604, the memory access management unit 1022 determines whether the layer number of the layer to be generated is an odd number. If the number is odd, the process proceeds to step S605; The memory access management unit 1022 stores the pixel value in the memory 0 in step S605, and stores the pixel value in the memory 3 in step S606.

If mem=1 (when the remainder calculation result is 1), the process proceeds to step S607. In step S607, the memory access management unit 1022 determines whether or not the layer number of the layer to be generated is an odd number. If the number is odd, the process proceeds to step S608. The memory access management unit 1022 stores the pixel value in the memory 1 in step S608, and stores the pixel value in the memory 4 in step S609.

If mem=2 (when the remainder calculation result is 2), the process proceeds to step S610. In step S610, the memory access management unit 1022 determines whether the layer number of the layer to be generated is an odd number. If the number is odd, the process proceeds to step S611. The memory access management unit 1022 stores the pixel value in the memory 2 in step S611, and the memory access management unit 1022 stores the pixel value in the memory 5 in step S612.

By determining where to store pixel values in this way, when generating even-numbered layers from odd-numbered layers, the memory for reading the reference feature plane and the memory for writing the generated feature plane are separate. Similarly, when odd-numbered layers are generated from even-numbered layers, the memory for reading the reference feature plane and the memory for writing the generated feature plane are separate. In addition, the generated feature planes are stored in different memories line by line in the order of the serial number of the lines, so that a plurality of lines can be read at the same time when they are read as the reference feature planes.

Then, in step S613, the memory access management unit 1022 calculates a memory address. The address A0 storing the top pixel of one line is obtained by calculating A0=L/(number of memories/2)×w/4, and the address A1 is obtained by calculating A0+x/4.

Next, details of the processing in step S310 will be described. FIG. 7 is a diagram showing part of the memory map of the CPU 104. As shown in FIG. Each of memories 0 to 5 is a memory of 8 KB and allocated to a continuous area of 48 KB from a specific base address. The head address of memory 0 is incremented by address 0, and the head addresses after memory 1 are incremented by 0x2000.

The memory control unit 105 selects memory chip select according to the address of the access request. For example, an access request from address 0 to 0x1FFF selects memory 0 chip select, and an access request from 0x2000 to 0x3FFF selects memory 1 chip select. Also, the address of the memory is assumed to be [13:2] of the address of the access request.

In step S3101, CPU 104 acquires network information. The control unit 101 delivers the layer number of the final layer, the horizontal size w of the characteristic plane, the vertical size h, and the number of characteristic planes as control parameters, and the CPU 104 refers to these control parameters.

Next, in step S3102, the processing according to the flowchart of FIG. 6 is executed to read the pixel value of the feature plane with the feature plane number n and the coordinates (x, y), and the storage memory and address A1 are specified. Here, the processing according to the flowchart of FIG. 6 may be performed by the CPU 104 or by the memory access management unit 1022 .

Then, in step S3103, the CPU 104 acquires an address on the memory map by performing the following conversion processing from the storage memory specified in step S3102 and the address A1.

When the storage memory is memory 0, address=A1×4
When the storage memory is memory 1, address=A1×4+0x2000
When the storage memory is memory 2, address=A1×4+0x4000
When the storage memory is memory 3, address = A1x4+0x6000
When the storage memory is memory 4, address=A1×4+0x8000
When the storage memory is memory 5, address=A1×4+0xa000
Then, in step S3104, the CPU 104 issues an access request to the memory control unit 105 using the converted address, acquires the pixel values of the characteristic plane read in response to the access request, and specifies the position coordinates.

Thus, according to this embodiment, it is possible to eliminate the time required to transfer data to the shared memory and reduce the bus bandwidth. Also, the memory of the hardware that performs CNN processing (CNN processing hardware) is the address-mapped memory of the CPU. Therefore, when the CNN processing hardware does not operate, it can be used as a work memory for the CPU, and there is no need to separately provide a work memory, thereby reducing costs.

[Second embodiment]
Differences from the first embodiment will be described below, and the same as the first embodiment is assumed unless otherwise specified. A configuration example of the recognition processing unit 801 according to this embodiment will be described using the block diagram of FIG. 11 .

In the present embodiment, it is assumed that the characteristic planes of the layer to be processed are successively stored in raster order in a continuous area of 48 KB on the memory map, starting from the leading address 0. FIG. That is, according to the horizontal size w and vertical size h of the feature plane, the address A2 on the memory map of the feature plane data of the feature plane number n and the coordinates (x, y) is determined by the following formula.

A2 = w x h x n (feature surface offset) + y x w (line offset) + x
The memory control unit 1051 uses the access request address as it is as the address signal of the SRAM under the configuration that there is one SRAM interface (one chip select). The SRAM interface is connected to memory access manager 1024 . In step S310, the memory access management unit 1024 is notified of the SRAM access to A2 as it is.

The characteristic plane holding unit 1023 is assumed to be a single-port SRAM. When there is an access request from the SRAM interface of the memory control unit 1051, the memory access management unit 1024 obtains the address, the horizontal size w of the characteristic plane, the vertical size h, and the hierarchy number, the characteristic plane number n, and the coordinates (x, y). identify. Then, according to the flow chart of FIG. 6, the storage destination memory and address are determined, and the data is read from the memory. Read data is output to the memory control unit 1051 .

As described above, in this embodiment, the CPU 104 that post-processes the calculation result output by the CNN processing hardware issues an access request to a specific address on the memory map according to the horizontal size and vertical size of the feature plane. . The same effect as in the first embodiment can be obtained by the memory access management unit 1024 directly reading from the interface of the memory that stores the characteristic planes of the intermediate layer and the final layer. Note that the CPU 806 and RAM 808 outside the recognition processing unit 801 may be used as the general-purpose CPU and the address-mapped memory, respectively.

In each of the above-described embodiments, the case where all the functional units shown in FIGS. 1 and 11 are implemented by hardware has been described. However, some of the functional units shown in FIGS. 1 and 11 may be implemented by software (computer programs). In this case, the computer program is stored in a memory such as the ROM 807, and a processor such as the CPU 806 executes the computer program to implement the functions of the corresponding functional units.

It should be noted that the specific numerical values used in the above description are used for specific description, and are not intended to limit each of the above embodiments to these numerical values. It should be noted that part or all of the embodiments described above may be appropriately combined. Moreover, you may selectively use a part or all of each embodiment demonstrated above.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the claims are appended to make public the scope of the invention.

101: Control Unit 102: Feature Plane Storage Unit 1021: Feature Plane Holding Unit 1022: Memory Access Management Unit 103: Convolution Calculator 104: CPU 105: Memory Control Unit

Claims (8)

An arithmetic processing device that inputs input data to a hierarchical neural network and performs arithmetic processing in each layer of the hierarchical neural network,
a calculation unit that calculates the characteristic surface in each layer by referring to the characteristic surface in the layer before the layer ;
a feature plane holding unit having a plurality of memories that hold the feature planes calculated and referred to by the calculation unit;
Based on network information, which is information about each layer on which the arithmetic processing is performed , the characteristic planes calculated by the arithmetic unit are arranged and written in the plurality of memories, and are referred to by the arithmetic unit from the plurality of memories. a memory access manager that reads the feature plane ;
The characteristic surface holding unit is address-mapped in a memory space, and based on the network information, calculates an address of a pixel value of the characteristic surface that is address-mapped in the memory space, and uses the address to calculate the characteristic surface holding unit. and a processor that reads out pixel values from and processes them. 2. The arithmetic processing unit according to claim 1, wherein said network information includes the width and height of characteristic planes and the number of characteristic planes for each layer. 2. The arithmetic processing unit according to claim 1, wherein said memory stores feature planes of intermediate layers or final layers among said respective hierarchies, and said processor processes feature planes of intermediate layers or final layers. 2. The arithmetic processing unit according to claim 1, wherein said processor calculates said address based on said network information and information of said memory access management unit. 2. The arithmetic processing unit according to claim 1, wherein the memory access management unit interleaves and arranges the characteristic planes of the characteristic plane holding unit in a plurality of memories on a hierarchical basis. 2. The arithmetic processing unit according to claim 1, wherein the memory access management unit interleaves and arranges the characteristic planes of the characteristic plane holding unit in a plurality of memories in units of lines. 2. The arithmetic processing unit according to claim 1, wherein said memory access management unit calculates a storage destination of said memory based on an access request address of said processor, reads out data, and returns the data. An arithmetic processing method performed by an arithmetic processing device that inputs input data to a hierarchical neural network and performs arithmetic processing in each layer of the hierarchical neural network,
The arithmetic processing unit is
a calculation unit that calculates the characteristic surface in each layer by referring to the characteristic surface in the layer before the layer ;
a feature plane holding unit having a plurality of memories that hold the feature planes calculated and referred to by the calculation unit;
a memory access management unit that manages reading and writing with respect to the memory;
a processor that accesses the feature surface holder;
The memory access management unit arranges and writes the characteristic planes calculated by the calculation unit in the plurality of memories based on network information, which is information about each layer in which the calculation processing is performed, and writes reading a feature plane referenced by the computing unit;
The feature plane holding unit is address-mapped in a memory space, and the processor calculates an address of a pixel value of the feature plane address-mapped in the memory space based on the network information, and uses the address. An arithmetic processing method, comprising: reading pixel values from a feature plane holding unit and processing the pixel values.

Images