JP7420100B2 - Processing device, processing method, and program - Google Patents

Description

The present invention relates to a processing device, a processing method, and a program for use in a convolutional neural network.

Conventionally, in the field of machine learning, images and videos have been recognized using a model called a convolutional neural network (CNN). For example, in image recognition, a convolution layer and a pooling layer are used to transform an input image, gradually reducing the amount of data, and finally outputting probability values for each category.

Here, in the convolution layer of the CNN, filtering (filter processing; convolution processing) is performed on each local region (for example, a 3×3 cell region) in the input data. In the convolution process, data of the same cell of input data may be read from the main memory (RAM) many times (for example, as many times as the number of cells (coefficients) of the filter). Moreover, tens to hundreds of filters are used in one convolutional layer. For this reason, the number of times data is read from the main memory is extremely large, and this has become a bottleneck that prevents speeding up of CNN processing.

In Patent Document 1, in order to efficiently implement a convolutional layer of a CNN, input data across a plurality of channels is rearranged.

International Publication No. 2018/067603

However, in Patent Document 1, the input data is simply divided and rearranged, so for example, in a cell located at the division boundary in the original input data, the adjacency relationship changes significantly after the rearrangement. . Therefore, when performing filter processing on cells located at division boundaries in original input data, it is necessary to refer to cells at various positions in rearranged data. For this reason, a large number of processes are required to read data after rearranging it from the main memory to the registers, making it impossible to realize efficient convolutional layer processing.

Therefore, an object of the present invention is to efficiently realize processing of convolutional layers of a convolutional neural network.

In order to achieve the above object, the present invention employs the following configuration.

That is, a processing device according to one aspect of the present invention is a processing device that performs convolution processing on input data of a convolution layer of a convolutional neural network, the input data being two-dimensional data having data in each cell, arranging means for selecting a plurality of areas from the input data, concatenating the data of the plurality of areas and arranging the data at consecutive addresses on a main memory as arrangement data; processing means for performing convolution processing using a filter on a cell, and the size of each of the plurality of regions in the row direction is a size corresponding to an integral multiple of the number of cells that can be read out at once by the processor. The processing device is characterized in that each of the plurality of regions has data that overlaps columns in other regions by a number of columns equal to the size of the filter minus 1.

In this way, if the size of each of the multiple areas in the row direction is an integral multiple of the number of cells (word size) that the processor can read at once, it is possible to read data from a predetermined number of cells at once. It is possible to prevent a processor capable of reading data from reading only data of less than a predetermined number of cells from a target area. In other words, the read capacity of the processor can be utilized to the fullest. Furthermore, since there is an overlapping region in the arrangement data, other cells necessary for the filtering process are gathered around the cell to be filtered. Therefore, the processor can perform filter processing on the convolutional layer while reducing the number of reads from the main memory. Therefore, the processing of the convolutional layer of the convolutional neural network can be efficiently realized. Note that, here, the size of the filter is the size of n cells in the case of a filter of n cells×n cells.

In the above processing device, the arrangement means may connect (connect) the data of the plurality of areas in a column direction on the main memory and arrange the data as the arrangement data. According to this, cells (data) necessary for performing filter processing regarding the area are aggregated in one area out of the plurality of areas. Therefore, the processor can perform filter processing on the convolutional layer while reducing the number of reads from the main memory. Therefore, the processing of the convolutional layer of the convolutional neural network can be efficiently realized.

In the above processing device, when arranging the arrangement data, the arrangement means may arrange a row having the filter data between two adjacent regions among the plurality of regions. According to this, for example, it is not necessary to always store filter data for the number of channels in a plurality of registers in the processor, so that the registers can be used more effectively for filter processing.

In the processing device, each of the plurality of regions may further include data that overlaps with rows in other regions by a number of rows equal to the size of the filter minus 1. Since the rows overlap, when performing filter processing, it is possible to perform filter processing without referring to other regions among the plurality of regions. Therefore, the convolutional layer can be processed without re-reading the rows that have been read out once in the arrangement data, so that the convolutional layer processing can be made more efficient.

In the above processing device, the first memory address of each row of the arrangement data may be a memory address that is an integral multiple of the number of memory addresses that the processor can read out at once. According to this, the memory address at the beginning of each row of arrangement data can be matched with the memory address at the beginning of the memory block of the main memory. Therefore, accessing unnecessary memory blocks can be suppressed, so that reading processing from the main memory of the processor becomes more efficient.

In the above processing device, the input data is data spanning a plurality of channels, and the arrangement means determines a priority order for the plurality of regions in the order of column, row, and channel, and The data may be connected in a column direction on the main memory according to the priority order and arranged as the arrangement data. According to such an arrangement, when performing filter processing, it is possible to continuously perform the processing on regions in the same column and the same row that differ only in channels. Therefore, while the result of filter processing performed on one channel is stored in a register as an intermediate result, filter processing on another channel can be performed using another register. In other words, it is possible to suppress the occurrence of the process of reading and writing the intermediate results of one channel to the main memory, so that the processing of the convolution layer that requires summing the intermediate results of all channels can be efficiently executed.

In the above processing device, when performing the convolution process, the processing means is configured to generate a data block having a number of rows equal to the size of the filter, and a first plurality of rows that are continuous in the column direction in the arrangement data. reading the data block from the arrangement data and storing it in a register, shifting the data block in each row of the first plurality of rows in the row direction, and multiplying the data block by the value of one cell of the filter. You may do so. According to this, arithmetic processing can be applied to multiple pieces of data included in one data block at once, and as a result, filter processing can be executed on multiple local areas at once. . Therefore, the processing of the convolutional layer of the convolutional neural network can be efficiently realized.

In the above processing device, when performing the convolution process, the processing means shifts the data blocks of each row of the first plurality of rows by one cell in the row direction to the first plurality of data blocks. obtain each data block of each row of the rows up to the data block shifted in the row direction by a number of cells equal to the size of the filter minus 1, and the data block of each row of the first plurality of rows. and multiplying each of the obtained data blocks by the value indicated by the cell of the filter corresponding to the row to which the data block corresponds among the first plurality of rows and the amount by which the data block is shifted. By doing so, the data blocks of the second plurality of rows may be obtained, and the values of cells at corresponding positions in the data blocks of each row of the second plurality of rows may be summed.

The present invention may be understood as a device having at least a part of the above means, or as an electronic device, a control system, an information processing system, an information processing device, a processing system, or a data arrangement device. Further, the present invention may be understood as a control method, a processing method, and an arrangement method that include at least a part of the above processing. Further, the present invention can also be understood as a program for realizing such a method and a recording medium (storage medium) on which the program is recorded non-temporarily. Note that each of the above means and processes can be combined to the extent possible to constitute the present invention.

According to the present invention, processing of convolutional layers of a convolutional neural network can be efficiently realized.

FIG. 1A is a simple configuration diagram of a processing device, and FIG. 1B is an internal configuration diagram of a processor. FIG. 2 is a diagram illustrating processing of a convolutional layer. FIG. 3 is a flowchart of data arrangement processing. FIG. 4A is a diagram showing input data, and FIG. 4B is a diagram showing layout data. FIG. 5 is a diagram illustrating the priority order of selected areas. FIG. 6 is a diagram illustrating filter processing when data arrangement processing is not performed. FIG. 7 is a diagram illustrating filter processing.

Embodiments for carrying out the present invention will be described below with reference to the drawings.

First, a convolutional neural network (CNN) will be explained. CNN includes convolution layers and pooling layers. The amount of input data such as images is reduced by repeatedly performing a set of processes in the convolution layer and the pooling layer. Finally, for example, a value of the probability that the image is a predetermined object (for example, a person, a face, an animal such as a dog) is output as output data.

In the convolution layer, for example, as shown in FIG. 2, a matrix showing pixel values in a two-dimensional form for each of the R, G, and B channels of each pixel of an image is input as input data. Then, in the convolution layer, filters 211 to 213 called kernels are multiplied (applied) to local regions 201 to 203 (for example, 3×3 regions) of each channel of the input image. Thus, typically as many filters as there are channels or more are needed to apply to the input data. Then, by summing the value obtained by applying the filter 211 to the local area 201, the value obtained by applying the filter 212 to the local area 202, and the value obtained by applying the filter 213 to the local area 203, the value obtained by applying the filter 211 to the local area 201 is summed. An output 220 (output value V22) corresponding to the center cell) is obtained. Here, for example, the result of applying the filter 211 to the local area 201 is obtained by summing the results of multiplying the numerical values at corresponding positions in the 3×3 area in the local area 201 and the filter 211, using equation 1. It can be calculated as shown below. Note that hereinafter, the process of generating output data having the output 220 based on input data (local region) will be referred to as "convolution process." Further, a process that is a part of the "convolution process" and that applies a filter to a local region is called a "filter process."

The pooling layer processes and outputs information for each local region of the matrix processed by the convolutional layer. For example, the maximum value, average value, etc. in a 2×2 area are output as output values corresponding to that local area.

<Application example>
In the following, a processing device 1 will be described that performs convolution processing (filter processing) on input data, which is two-dimensional data having data in each cell, of a convolution layer of a convolution neural network. The processing device 1 selects a plurality of areas (selected areas) from input data, concatenates the data of the plurality of selected areas, and arranges the data at consecutive addresses on the main memory as placement data. Then, the processing device 1 uses a processor to read out the arrangement data and performs convolution processing. At this time, the processing device 1 sets the size of each of the plurality of selected areas in the row direction to a size corresponding to an integral multiple of the number of cells (word size) that can be read out at once by the processor. Furthermore, the processing device 1 divides the plurality of selection regions so that each of the plurality of selection regions has data (superimposed region) that overlaps columns of other selection regions by the number of columns obtained by subtracting 1 from the filter size. select.

If the size of each of the multiple selection areas in the row direction is an integral multiple of the word size, a processor that can read data from a predetermined number of cells at once can read data from less than the predetermined number of cells from the target selection area. This prevents only data from being read. In other words, the read capacity of the processor can be utilized to the fullest. Further, since there is an overlapping region in the arrangement data, other cells necessary for the filtering process are gathered around the cell to be filtered. Therefore, the processor can execute the convolution process while reducing the number of reads from the main memory. Therefore, processing of the convolutional layer of the convolutional neural network can be efficiently realized.

<Embodiment 1>
The configuration of the processing device 1 according to the first embodiment will be described below with reference to FIGS. 1A and 1B. FIG. 1A is a simple configuration diagram of the processing device 1. As shown in FIG. The processing device 1 may be any processing device (processing terminal) such as a PC, a server, or a smartphone. The processing device 1 includes a processor 10, a storage device 20, an input/output device 30, and a bus 40.

A processor 10 (CPU; central processing unit) controls each component (device) in the processing device 1 . For example, control is performed using data stored in the storage device 20 in accordance with user instructions input to the input/output device 30. Processor 10 has a plurality of registers 11 (generally 16 or 32 registers 11).

Each of the plurality of registers 11 is a storage circuit that temporarily stores data. Each of the plurality of registers 11 can read and write data faster than the storage device 20 (main memory 21). Therefore, the processor 10 executes various processes while temporarily storing data in the plurality of registers 11. Note that the plurality of registers 11 include a plurality of dedicated registers whose uses such as arithmetic operations are specified, and a plurality of general-purpose registers whose uses are not specified. Each of the plurality of registers 11 can store, for example, eight sets of 16-bit data. Therefore, for example, each of the plurality of registers 11 can store data of eight cells of input data at once.

The storage device 20 stores (records) data for the processor 10 to process. The storage device 20 includes a hard disk, RAM (main memory 21; Random Access
ROM (Read Only Memory) that stores data non-temporarily. The ROM stores, for example, a bios of an OS (operation system) and a program for the processor 10 to operate.

The main memory 21 temporarily stores data such as input data and filters. In the main memory 21, memory cells that store 1-bit information are spread out two-dimensionally (in the row and column directions). Note that the main memory 21 may be removable from the processing device 1, and in this case, the processing device 1 can be regarded as a memory control device. Note that, as described above, since the number of filters is equal to the number of channels of input data (or more), the main memory 21 stores filter data for the channels of input data.

The main memory 21 also stores data in units of predetermined blocks (hereinafter referred to as memory blocks). Therefore, for example, if the size of one memory block is 4 Bytes, the first memory address of each memory block will be an integral multiple of 4 Bytes. On the other hand, the processor 10 can read data at once from the main memory 21 in units of words (the number of cells or bits that can be read out at once by the processor 10). Therefore, if the data that the processor 10 requests to read spans two memory blocks in the main memory 21, the processor 10 needs to access the memory twice. Therefore, when the processor 10 accesses the main memory 21, reducing the number of memory blocks to be accessed achieves high-speed memory access. Note that although the processor 10 can read data at once from a plurality of cells in one row, it cannot read data at once from a plurality of cells in one column.

The input/output device 30 includes an input device that receives instructions (operations) from a user, and an output device that outputs data processed by the processor 10. Input devices include, for example, a mouse, a keyboard, a touch panel, a button, a dial, a microphone (voice input device), an attitude detection device (gyro sensor), a temperature sensor, and the like. The output device includes, for example, a display (display device), a speaker (sound output device), a printer, and the like.

The bus 40 is a path for communication between the processor 10, the storage device 20, and the input/output device 30. The processor 10 can acquire data from the storage device 20 via the bus 40, execute processing using the data, and store the processed data in the storage device 20.

(Internal configuration of processor)
Next, the internal configuration of the processor 10 will be explained with reference to FIG. 1B. FIG. 1B is an internal configuration diagram of the processor 10 used for rearranging input data, convolution processing (filter processing), and the like. The processor 10 includes an acquisition section 101, an arrangement section 102, and a processing section 103. These configurations may be realized by a dedicated logic circuit, or may be realized in software by the processor 10 executing a program.

The acquisition unit 101 acquires CNN input data (see FIG. 2) from the main memory 21 or the pooling layer. Here, for example, the input data includes data across a plurality of channels (for example, channel R, channel B, and channel B each indicating a pixel value of RBG). In each channel, cells each containing data are spread out in two dimensions. Note that the input data does not need to have multiple channels, and may have only a single channel.

The placement unit 102 selects a plurality of areas of a predetermined size as selection areas in the input data. Further, the arrangement unit 102 arranges the data in the selected area so as to connect (connect) the data in a predetermined order (priority order), and stores the data in the main memory 21 as arrangement data. As a result, arrangement data obtained by rearranging the input data is stored in the main memory 21. Then, the processing unit 103 reads out the arrangement data stored in the main memory 21 into the register 11, and executes convolution processing (filter processing) in the convolution layer.

[Data placement processing]
Below, detailed processing of the data arrangement processing mentioned above will be explained using the flowchart of FIG. 3. The processing in the flowchart of FIG. 3 is realized by the processor 10 executing a program stored in the ROM.

In step S1001, the acquisition unit 101 acquires input data used in the convolution layer of the CNN.

In step S1002, the placement unit 102 selects a plurality of areas from the input data as selection areas. Here, the arrangement unit 102 arranges the plurality of selection areas so that the size in the row direction of each of the plurality of selection areas is an integral multiple of the size of the above-mentioned word (the number of cells that can be read out at once by the processor 10). Select a selection area. Further, the arrangement unit 102 selects the plurality of selection regions so that each of the plurality of selection regions has data that overlaps with columns of other selection regions by the number of columns obtained by subtracting 1 from the size of the filter. Here, the filter size is the size of three cells in the case of a filter having 3×3 cells. Note that each of the plurality of selection regions may have data that overlaps with rows of other selection regions by the number of rows obtained by subtracting 1 from the size of the filter.

In step S1003, the placement unit 102 determines a priority order representing the placement order from all selected selection areas. Specifically, the arrangement unit 102 determines the priority order in all selection areas in the order of column, row, and channel.

For example, assume that the placement unit 102 selects selection regions CH_R11, CH_R21, CH_R31, CH_G11, CH_G21, CH_G31, CH_B11, CH_B21, and CH_B31 as shown in FIG. 4A from the input data. In this case, since rows are prioritized over channels, the placement unit 102 determines the order of selection areas CH_R11, CH_G11, CH_B11, CH_R21, CH_G21, CH_B21, CH_R31, CH_G31, and CH_B31 as the priority order (FIG. 4B reference).

Further, for example, it is assumed that selection regions CH_R11, CH_R12, CH_R13, CH_R21, CH_R22, CH_R23, CH_R31, CH_R32, and CH_R33 as shown in FIG. 5 are selected from the input data. In this case, since columns are prioritized over rows, the arrangement unit 102 determines the order of the selected regions CH_R11, CH_R21, CH_R31, CH_R12, CH_R22, CH_R32, CH_R13, CH_R23, and CH_R33 as the priority order.

In step S1004, the arrangement unit 102 generates arrangement data by concatenating the data of the selected areas according to the priority order. For example, as shown in FIG. 4A, when the selection area is selected from the input data, the arrangement unit 102 connects the selection area CH_R11 to the selection area CH_G11, CH_B11, and CH_R21 in the column direction according to the priority order. Arrange them as you like. The arrangement unit 102 generates arrangement data by arranging the plurality of selection areas selected in this way.

In step S1005, the placement unit 102 stores the placement data in the main memory 21. Note that the arrangement unit 102 preferably arranges the arrangement data so that the first memory address of each row of the arrangement data is a memory address that is an integral multiple of the number of memory addresses that the processor 10 can read out at once. Here, the number of memory addresses that the processor 10 can read out at once is generally equal to or an integral multiple of the number of memory addresses included in one memory block of the main memory 21. Therefore, the memory address at the beginning of each row of the arrangement data can be made to match the memory address at the beginning of the memory block of the main memory 21. Therefore, when the processor 10 reads data at once from word-sized cells of the main memory 21, the number of memory blocks to be read can be reduced, so the number of memory accesses can be reduced.

As in this embodiment, if the size in the row direction of each of a plurality of selected areas is an integer multiple of the word size, when the processor 10 reads one row of one selected area, the word size from the selected area is There is no need to read out cells in units of word size and then read out only the remaining cells. Therefore, unnecessary reading of the main memory by the processor 10 can be reduced, so that processing in the convolution layer becomes more efficient.

Note that, as described above, the processor 10 can read data at once from a plurality of cells arranged in one row, but cannot read data at once from a plurality of cells arranged in one column. Therefore, the size of each of the plurality of selection areas in the column direction may be any size as long as it is larger than the filter size. Here, if the number of registers 11 is sufficiently large, by increasing the size of each of the plurality of selection areas in the column direction, switching of the selection area to be filtered can be reduced. Therefore, the processing of reading data into the register 11 and releasing the data in the register 11 can be reduced. On the other hand, when the number of registers 11 is small, the calculation result (intermediate result) of filter processing for one selected area can be stored in the register 11 by reducing the size of each of the multiple selected areas in the column direction. I can afford to keep it. Therefore, it is possible to suppress the occurrence of the process of temporarily writing the intermediate result to the main memory 21 and then reading the intermediate result from the main memory 21. Therefore, it is preferable that the processor 10 determines the size of each of the plurality of selection areas in the column direction based on the number of registers 11 and the like. Further, the processor 10 performs convolution layer processing by experimentally changing the column direction size of each of the plurality of selection regions, and selects the one that can be processed fastest as a result of the final plurality of selection regions. The size of each selected area in the column direction may also be used.

Furthermore, in this embodiment, each of the plurality of selection areas has data that overlaps with columns of other selection areas by the number of columns that is the size of the filter minus 1. Note that each of the plurality of selection regions may have data that overlaps with rows of other selection regions by the number of rows obtained by subtracting 1 from the size of the filter. Here, when filtering is performed, for example, on a cell at the edge of a selected area without overlapping data between selected areas, it is also necessary to use data of cells surrounding the edge cell in the input data. Therefore, in such a case, the processor 10 needs to read the data of the surrounding cells from another selected area (another memory block of the main memory 21). On the other hand, if data overlaps as in the present embodiment, filter processing can be performed using the data of cells in the overlapped portion, so the processor 10 does not need to read data from cells in other selected areas. . Therefore, unnecessary reading of the main memory 21 by the processor 10 can be reduced, making convolution processing (processing in the convolution layer) more efficient.

[Effect of placing according to priority order]
Further, below, the effect of determining the priority order for all selection areas in the order of column, row, and channel, and arranging the selection areas according to the priority order will be explained.

(If not placed according to priority order)
First, an example of a process when the processor 10 reads data from the main memory 21 when the data is not arranged according to the priority order will be described. When data arrangement processing is not performed, processing in a convolution layer is performed on input data spanning multiple channels as shown in FIG. 4A. In this case, the processor 10 sequentially stores data in a plurality of registers 11 starting with the data in the first column of the first row of channel R, and performs filter processing based on the data stored in the registers 11. . Here, for example, if the size of the filter is 3×3, three rows of word-sized blocks BLK1 to BLK3 (data blocks) are stored in the register 11 in order from the upper left, as shown in FIG. Thereby, the processor 10 can calculate the output value of the filter process by calculation using the three blocks and the filter. After that, the processor 10 moves to the next column (to the right) of the blocks BLK1 to BLK3, stores the three blocks BLK4 to BLK6 in the register 11, and calculates the output value by multiplying them by the filter. Note that, for example, when performing filter processing on a local area spanning blocks BLK1 to BLK3 and blocks BLK4 to BLK6, the processor 10 stores six blocks BLK1 to BLK6 in the register 11, and stores these six blocks BLK1 to BLK6 in the register 11. Perform filter processing using block data. That is, when performing filter processing on one local region, six registers (twice the filter size) may be required to store the data of the obtained block.

Then, by repeating such processing, when the filter processing using the data in the first three rows is completed, the same processing is performed on the data in the second to fourth rows. At this time, since the number of registers 11 is limited, the data regarding the block used in the initial filtering process is discarded from the registers 11. According to this, when the processor 10 attempts to perform filter processing on a local area centered on the cell of block BLK3 in the third row, it is necessary to read blocks BLK2 and BLK3 again. Therefore, if the arrangement is not performed according to the priority order of this embodiment, there is a possibility that the data of one cell is read from a plurality of main memories 21 equal in number to the size of the filter.

(When arranging according to priority order)
Next, a process when the processor 10 reads data from the main memory 21 when performing the data arrangement process according to the present embodiment will be described. In this embodiment, the input data shown in FIG. 4A is rearranged into arrangement data as shown in FIG. 4B, and the processor 10 executes filter processing using the arrangement data.

Here, for example, assume that the row size of each selection area is the same as the word size. In this case, the processor 10 reads out the arrangement data row by row in the column direction (in row order) and stores them in the registers 11, respectively. Then, filter processing is executed using the data stored in the register 11. It is assumed that each selection area has data that overlaps with other selection areas by the number of rows and columns that are equal to the filter size minus 1 in the column and row directions. In this case, if the processor 10 reads the arrangement data in row order, there is no need to read out the rows previously read from the arrangement data again and use them for filter processing. In other words, the process of reading data from the register 11 by the processor 10 can be made continuous (simplified).

Then, as explained using FIG. 2, the output value of the convolution layer is calculated by summing the results (intermediate results) of filtering each of the plurality of channels. In order to cope with this, in the arrangement data, selection areas at corresponding positions (corresponding columns and rows) in a plurality of channels are arranged so as to be continuously connected. According to this, the processor 10 can, for example, sequentially filter the selected region of channel R, the corresponding selected region of channel G, and the selected region of channel B. Therefore, while the result of filtering the channel R selection area is held in the register 11, it is more likely that the channel G selection area and the channel B selection area can be filtered. Therefore, for example, the processor 10 stores the result of filtering the channel R selection area in the main memory 21, and after filtering the channel G and B selection areas, selects the channel R from the main memory 21. The process of reading out the results of filtering the area becomes unnecessary. Therefore, processing in the convolutional layer can be made more efficient.

[About filter processing according to this embodiment]
Here, with reference to FIG. 7, the filter processing according to this embodiment will be specifically described. Below, we will use three rows of blocks 710, 720, 730 (three rows of data blocks) that are continuous in the column direction of the selection area and a filter 750 to calculate the values of multiple cells (R22 to R27) included in block 720. The filtering process for local regions centered on each cell (having a cell) will be explained. It is assumed that each of the blocks 710, 720, and 730 has eight cells of the same word size and is temporarily stored in the register 11. Further, it is assumed that the size of the filter 750 is the size of three cells (3 cells x 3 cells).

First, the processing unit 103 of the processor 10 acquires, from the block 710, a block 710, a block 711 obtained by shifting the block 710 by one cell in the left direction (row direction), and a block 712 obtained by shifting the block 710 by two cells in the left direction. do. In other words, the processing unit 103 acquires each of the blocks from the target block to a block obtained by shifting the target block to the left by the cell value of the filter size minus 1. Then, the processing unit 103 obtains a block 715 by collectively multiplying the value of each cell of the block 710 by the value a of the cell in the first row and first column of the filter 750. Similarly, the processing unit 103 obtains a block 716 in which the value of each cell of the block 711 is multiplied by the value b of the cell in the first row and second column of the filter 750, and multiplies the value of each cell of the block 710 by A block 717 is obtained by collectively multiplying the values of the cells in the first row and third column. That is, blocks 710 to 712 corresponding to the first row of the three rows of blocks are multiplied by the value of the cell in the first row of the filter 750. At this time, each of the blocks 710 to 712 is multiplied by the value of the cell in the column (position) in the filter that corresponds to the amount of shift of the block.

By similarly performing such processing on blocks 720 and 730, processing unit 103 obtains blocks 720 to 722 and 730 to 732, which are obtained by shifting blocks 720 and 730, as shown in FIG. The processing unit 103 also inputs the value of the corresponding cell in the filter 750 to each of the blocks 720 to 722 and 730 to 732 (the block (row) to which the target block corresponds among the three blocks 710, 720, and 730). , and the shift amount of the target block by the value of the corresponding cell) to obtain blocks 725, 726, 727, 735, 736, and 737.

Then, the processing unit 103 sums up the values of cells in the same position (column) of blocks 715 to 717, 725 to 727, and 735 to 737, thereby outputting an output for a local area centered on the cell having R22 to R27. Values v22 to v27 can be calculated.

Note that when calculating the output values v22 to v27, the calculation order does not need to be as described above, and an optimized calculation order may be used. For example, the processing unit 103 adds up the values of cells at the same position in blocks 715 to 717, stores that value in one register 11, obtains blocks 720 to 723 based on block 720, and further The processing may be performed in the order in which blocks 725 to 727 are acquired. According to this, the data stored in at least some of the three registers 11 that stored the values of blocks 715 to 717 can be released (discarded). Therefore, filter processing can be performed with a small number of registers 11.

By performing filter processing in this manner, it is possible to perform filter processing on a plurality of cells (six cells in this embodiment) at once. Therefore, the number of processes can be significantly reduced compared to the case where filter processing is performed on each cell one by one. Note that such batch processing can be realized, for example, by using a processing method called SIMD (Single Instruction Multiple Data), which allows the same processing to be performed on multiple pieces of data with a single instruction.

In this embodiment, in the layout data of the main memory, the size of each of the plurality of selection areas in the row direction is a size corresponding to an integral multiple of the number of cells (word size) that the processor can read at once. It is. According to this, it is possible to prevent a processor that can read data from a predetermined number of cells at once from reading data from fewer than the predetermined number of cells from the selected area. In other words, the read capacity of the processor can be utilized to the fullest.

Furthermore, each of the plurality of selection regions has data that overlaps columns (and rows) of other selection regions by a number of columns (and rows) that is the size of the filter minus one. According to this, the cell to be filtered is surrounded by other cells necessary for the filtering. Therefore, the need to perform unnecessary reading from the main memory can be reduced, so that the processor can perform filter processing on the convolutional layer while reducing the number of times of reading from the main memory.

Therefore, the processing of the convolutional layer of the convolutional neural network can be efficiently realized.

Note that in step S1004, when connecting the data of two selected areas, the arrangement unit 102 may insert filter data between the two selected areas. That is, in the arrangement data, the arrangement unit 102 may arrange a row having filter data between two adjacent selection areas. At this time, data of the filter corresponding to the selected area is inserted in order to perform filter processing using the filter on the selected area that is connected after the filter among the two selected areas. According to this, for example, it is no longer necessary for a plurality of registers 11 to always store filter data for the number of channels, so that the registers 11 can be used more effectively for filter processing.

Note that the interpretation of the claims is not limited only by the matters described in the embodiments. The interpretation of the claims includes the range described in such a way that a person skilled in the art can recognize that the problem to be solved by the invention can be solved, taking into consideration the common general knowledge at the time of filing.

(Additional note 1)
A processing device (1) that performs convolution processing on input data of a convolution layer of a convolutional neural network, the input data being two-dimensional data having data in each cell,
arranging means (102) for selecting a plurality of areas from the input data, concatenating the data of the plurality of areas and arranging the data at consecutive addresses on the main memory (21) as arrangement data;
processing means (103) for reading out the arrangement data by a processor (10) and performing convolution processing on the arrangement data using a filter;
has
The size in the row direction of each of the plurality of areas is a size corresponding to an integral multiple of the number of cells that can be read out at once by the processor (10),
Each of the plurality of regions has data that overlaps columns of other regions by a number of columns that is the size of the filter minus 1.
A processing device (1) characterized by:

(Additional note 2)
A processing method executed by a processor (10) of a processing device (1) that performs convolution processing on input data of a convolution layer of a convolutional neural network, the input data being two-dimensional data having data in each cell,
a placement step (S1005) of selecting a plurality of areas from the input data, concatenating the data of the plurality of areas and placing it as placement data at consecutive addresses on the main memory (21);
a processing step of reading the placement data by a processor (10) and performing convolution processing using a filter on the placement data;
has
The size in the row direction of each of the plurality of areas is a size corresponding to an integral multiple of the number of cells that can be read out at once by the processor (10),
Each of the plurality of regions has data that overlaps columns of other regions by a number of columns that is the size of the filter minus 1.
A processing method characterized by:

(Appendix 3)
A program for executing a processing method by a processor (10) of a processing device (1) that performs convolution processing on input data of a convolution layer of a convolutional neural network, which is input data that is two-dimensional data having data in each cell. And,
The processing method includes:
a placement step (S1005) of selecting a plurality of areas from the input data, concatenating the data of the plurality of areas and placing it as placement data at consecutive addresses on the main memory (21);
a processing step of reading the placement data by a processor (10) and performing convolution processing using a filter on the placement data;
has
The size in the row direction of each of the plurality of areas is a size corresponding to an integral multiple of the number of cells that can be read out at once by the processor (10),
Each of the plurality of regions has data that overlaps columns of other regions by a number of columns that is the size of the filter minus 1.
A program characterized by:

1: Processing device, 10: Processor, 11: Register,
20: storage device, 21: main memory, 30: input/output device, 40: bus,
101: Acquisition unit, 102: Arrangement unit, 103: Processing unit

Claims (10)

A processing device that performs convolution processing on input data of a convolution layer of a convolutional neural network, the input data being two-dimensional data having data in each cell,
arranging means for selecting a plurality of areas from the input data, concatenating data in the plurality of areas and arranging the data at consecutive addresses on a main memory as arrangement data;
processing means for reading the arrangement data by a processor and performing convolution processing on the arrangement data using a filter;
has
The size in the row direction of each of the plurality of areas is a size corresponding to an integral multiple of the number of cells that can be read out at once by the processor,
Each of the plurality of regions has data that overlaps columns of other regions by a number of columns that is the size of the filter minus 1.
A processing device characterized by: The arrangement means connects the data of the plurality of areas in a column direction on the main memory and arranges the data as the arrangement data.
The processing device according to claim 1, characterized in that: When arranging the arrangement data, the arrangement means places a row having the filter data between two adjacent regions of the plurality of regions;
The processing device according to claim 2, characterized in that: Each of the plurality of regions further has data that overlaps with rows in other regions by a number of rows equal to the size of the filter minus 1.
The processing device according to any one of claims 1 to 3, characterized in that: The first memory address of each row of the arrangement data is a memory address that is an integral multiple of the number of memory addresses that the processor can read out at once.
The processing device according to any one of claims 1 to 4. The input data is data spanning multiple channels,
The arrangement means determines a priority order for the plurality of areas in the order of column, row, and channel, and connects the data of the plurality of areas in the column direction on the main memory according to the priority order to create the arrangement data. to be placed as,
The processing device according to any one of claims 1 to 5. When performing the convolution process, the processing means:
reading data blocks of a first plurality of rows of rows corresponding to the size of the filter and which are continuous in the column direction in the arrangement data from the arrangement data and storing them in a register;
performing a process of shifting in the row direction and a process of multiplying the data block in each row of the first plurality of rows by the value of one cell of the filter;
The processing device according to any one of claims 1 to 6, characterized in that: When performing the convolution process, the processing means:
A data block obtained by shifting the data block in each row of the first plurality of rows by one cell in the row direction, and a data block in each row of the first plurality of rows having a number of cells equal to the size of the filter minus 1. obtaining each data block up to the data block shifted in the row direction by the amount and the data block of each row of the first plurality of rows;
Multiplying each of the obtained data blocks by a value indicated by a cell of the filter corresponding to a row to which the data block corresponds among the first plurality of rows and an amount by which the data block is shifted. Obtain a second multi-row data block by
summing the values of cells at corresponding positions in the data blocks of each row of the second plurality of rows;
The processing device according to claim 7, characterized in that: A processing method executed by a processor of a processing device that performs convolution processing on input data of a convolution layer of a convolutional neural network, the input data being two-dimensional data having data in each cell,
a placement step of selecting a plurality of areas from the input data, concatenating the data of the plurality of areas and placing the data at consecutive addresses on the main memory as placement data;
a processing step of reading the placement data by a processor and performing convolution processing using a filter on the placement data;
has
The size in the row direction of each of the plurality of areas is a size corresponding to an integral multiple of the number of cells that can be read out at once by the processor,
Each of the plurality of regions has data that overlaps columns of other regions by a number of columns that is the size of the filter minus 1.
A processing method characterized by: A program for executing a processing method by a processor of a processing device that performs convolution processing on input data of a convolution layer of a convolutional neural network, the input data being two-dimensional data having data in each cell,
The processing method includes:
a placement step of selecting a plurality of areas from the input data, concatenating the data of the plurality of areas and placing the data at consecutive addresses on the main memory as placement data;
a processing step of reading the placement data by a processor and performing convolution processing using a filter on the placement data;
has
The size in the row direction of each of the plurality of areas is a size corresponding to an integral multiple of the number of cells that can be read out at once by the processor,
Each of the plurality of regions has data that overlaps columns of other regions by a number of columns that is the size of the filter minus 1.
A program characterized by:

Images