JP6964969B2 - Arithmetic processing unit, arithmetic processing method and program - Google Patents

Description

The present invention relates to an arithmetic processing unit, an arithmetic processing method, and a program used for pattern recognition and the like.

Multi-layer neural networks called deep nets (also called deep neural networks or deep learning) have received a great deal of attention in recent years. The deep net does not refer to a specific calculation method, but generally, a hierarchy in which the processing result of a certain hierarchy is input to the processing of the subsequent hierarchy with respect to the input data (for example, image data). Refers to those that perform general arithmetic processing. In particular, in the field of image identification, a deep net composed of a convolution layer that performs a convolution filter calculation and a coupling layer that performs a coupling calculation is becoming mainstream. Convolutional Neural Networks (hereinafter abbreviated as CNN) is a typical method for realizing the deep net, and a method based on CNN will be described below.

FIG. 3 is a diagram showing an example of a convolution filter operation. FIG. 3 shows a case where a filter operation in which the kernel size of the filter kernel 302 is 3 × 3 is performed on the image 301 to be processed. In such a case, the convolution filter calculation result is calculated by the product-sum calculation process shown in the following equation.

ここで、「ｄ_ｉ，ｊ」は座標（ｉ，ｊ）での処理対象画像画素値を示し、「ｆ_ｉ，ｊ」は座標（ｉ，ｊ）でのフィルタ演算結果を示す。また、「ｗ_ｓ，ｔ」は座標（ｉ＋ｓ−１，ｊ＋ｔ−１）の処理対象画像画素値に適用するフィルタカーネルの値（フィルタ係数パラメータ）を示し、「ｃｏｌｕｍｎＳｉｚｅ」及び「ｒｏｗＳｉｚｅ」はフィルタカーネルサイズを示す。フィルタカーネル３０２を処理対象画像中でスキャンさせつつ、上記の演算を行うことで、畳込みフィルタ演算の出力結果を得ることができる。

Here, "di _{, j} " indicates the image pixel value to be processed at the coordinates (i, j), and "fi _{, j} " indicates the filter calculation result at the coordinates (i, j). Further, _{"w s, t"} indicates the coordinates (i + s-1, j + t-1) of the processing target image pixel value filter kernel values that apply to (the filter coefficient parameters), "columnSize" and "rowSize" filter kernel Indicates the size. By performing the above calculation while scanning the filter kernel 302 in the image to be processed, the output result of the convolution filter calculation can be obtained.

Features are generated from the non-linear transformation processing represented by the convolution filter calculation and the sigmoid transformation. By repeating the operation of generating this feature amount hierarchically on the input image, a feature surface expressing the features of the image can be obtained. That is, the two-dimensional feature amount generated by repeating the convolution filter calculation for the entire input image becomes the feature surface.

A typical deep net is a structure in which a convolution filter operation is used for feature amount extraction processing from an image, and a matrix product calculation represented by perceptron is used for identification processing using the extracted feature amount. .. This feature extraction process is often a multi-layer process in which the convolution filter operation is repeated many times, and the identification process may also use a fully coupled multi-layer perceptron. This configuration is a very common configuration for a deep net that has been actively studied in recent years.

Here, an example of deep net calculation will be described with reference to FIG. In FIG. 4, feature data is extracted from the input image 401 as an input layer by a convolution filter calculation, and after the feature amount of the feature surface 407 is obtained, the feature amount of the feature surface 407 is identified. Is performed to obtain the identification result 414. The convolution filter calculation is repeated many times until the feature surface 407 is obtained from the input image 401. Further, the feature amount of the feature surface 407 is subjected to the perceptron treatment of full binding a plurality of times, and the final identification result 414 is obtained.

First, the first half of the convolution filter operation will be described. In FIG. 4, the input image 401 shows image data of a predetermined size raster-scanned with respect to the image data. The feature planes 403a to 403c show the feature planes of the first stage layer 408. As described above, the feature plane is a data plane showing the calculation result of a predetermined feature extraction filter (convolution filter calculation and non-linear processing). Since it is the calculation result for the raster-scanned image data, the calculation result is also represented by a surface. The feature planes 403a to 403c are generated by the convolution filter calculation and the non-linear processing on the input image 401. For example, the feature plane 403a is obtained by a convolution filter calculation using the filter kernel 4021a and a non-linear transformation of the calculation result. The filter kernel 4021b and the filter kernel 4021c in FIG. 4 are filter kernels used when generating the feature surface 403b and the feature surface 403c, respectively. The structure having a convolution filter calculation relationship for generating each feature surface described above is called a hierarchical connection relationship.

Next, an operation for generating the feature surface 405a of the second-stage layer 409 will be described.

The feature surface 405a is coupled to three feature surfaces 403a to 403c, which are a part of the feature surfaces of the layer 408 in the previous stage. Therefore, when calculating the data of the feature surface 405a, the convolution filter calculation using the kernel shown by the filter kernel 4041a is performed on the feature surface 403a, and the result is retained. Similarly, the feature planes 403b and 403c are subjected to convolution filter operations of the filter kernels 4042a and 4043a, respectively, and these results are retained. After the completion of these three types of filter operations, the retained results are added and a non-linear conversion process is performed. By processing the above processing on the entire image, the feature surface 405a is generated.

Similarly, when the feature surface 405b is generated, three convolution filter operations are performed by the filter kernels 4041b, 4042b, and 4043b on the feature surfaces 403a to 403c of the previous layer 408. Further, when the feature surface 407 of the third layer 410 is generated, two convolution filter operations are performed by the filter kernels 4061 and 4062 for the feature surfaces 405a to 405b of the previous layer 409.

Next, the latter half of the perceptron treatment will be described. In FIG. 4, it is a two-layer perceptron. The perceptron is a non-linear transformation of the weighted sum for each element of the input feature. Therefore, the intermediate processing result 413 can be obtained by performing a matrix product operation on the feature amount of the feature surface 407 and performing a non-linear transformation on the result. If the same process is repeated, the final identification result 414 can be obtained.

The hierarchical convolution filter operation in the convolution layer is an operation based on a coupling relationship in which a large number of feature planes on the reference side are referred to and a large number of feature planes on the output side are calculated. Many, processing time is long. As shown in Non-Patent Document 1, it is possible to reduce the number of convolution filter operations and the filter kernel by coarsening the coupling between layers.

Efficient processing including parallel processing that calculates multiple feature planes in parallel using multiple arithmetic circuits in order to process while improving and maintaining performance as the hierarchical structure of the deep net increases in scale. It is necessary to go.

However, since there are many coupling relationships between the feature planes between each layer, if a plurality of feature planes calculated by a plurality of arithmetic circuits in parallel are simply divided and assigned to a plurality of arithmetic circuits, each arithmetic circuit is used. In the memory of, there are many feature planes and kernel filters on the reference side that are retained in duplicate. Due to the duplicated data, the access time to the memory becomes long, and the high-speed arithmetic processing by the parallel processing of a plurality of arithmetic circuits is hindered.

In order to solve this problem, in Patent Document 1, a plurality of independent feature planes of a coupling relationship are calculated after dividing the coupling relationship in advance between a plurality of layers and learning independently from the divided coupling relationship. We are proposing a structure that assigns to the arithmetic circuit of the above and performs parallelization processing.

WO20141058565A

Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner. "Gradient-Based Learning Applied to Document Recognition," in procedings of the IEEE, 86 (11): 2278-2324, November 1998

However, since the method of Patent Document 1 is premised on learning after determining the structure of the hierarchical connection relationship in advance, there is a limitation of machine learning, and the patent document refers to the connection relationship learned by the conventional method of Non-Patent Document 1. Method 1 cannot be applied. Therefore, in the method of Patent Document 1, there is an influence on the accuracy of image identification and object detection due to the small degree of freedom of learning. That is, the structure in anticipation of parallelization in advance of the method of Patent Document 1 cannot be applied to the configuration of a coupling relationship such as a coarse hierarchical coupling relationship as shown in Non-Patent Document 1, and therefore, a plurality of arithmetic circuits are used. Parallelization processing cannot be executed at high speed.

The present invention has been made in view of the above problems, and even if the structure of the hierarchical connection relationship is not determined in advance , the operations in a predetermined layer are appropriately assigned to a plurality of arithmetic circuits to be parallelized. An object of the present invention is to provide an arithmetic processing unit that performs processing at high speed. Another object of the present invention is to provide a calculation processing method and a program thereof.

In order to solve the above problems, the arithmetic processing unit according to the present invention has the following configuration. That is,
Multiple arithmetic circuits that perform hierarchical arithmetic processing in parallel,
A plurality of operations in a predetermined layer in the arithmetic process, and each of the plurality of operations that refers to at least a part of the operation results in the layers prior to the predetermined layer is assigned to the plurality of arithmetic circuits. Has an allocation means and
The allocation means
An evaluation value corresponding to the amount of overlapping data between the plurality of arithmetic circuits related to the arithmetic result in the layer prior to the predetermined layer to be referred to at the time of calculation is calculated for the allocation candidate.
The evaluation value is an allocation candidate satisfying a predetermined criterion, and the duplication of the calculation result in the hierarchy before the predetermined hierarchy referred to in the calculation among the plurality of arithmetic circuits is at least from the first allocation candidate. It is characterized in that a small number of second allocation candidates are determined as allocations of the plurality of operations to the plurality of arithmetic circuits.

According to the present invention, it is possible to provide an arithmetic processing unit that performs parallel processing at high speed by appropriately assigning arithmetic operations in a predetermined layer of hierarchical arithmetic processing to a plurality of arithmetic circuits.

This is a hardware configuration of the arithmetic processing unit of the first embodiment. (A) It is a figure which shows the schematic structure of the arithmetic circuit. (B) It is a figure which shows the structure of the control part of the arithmetic circuit. It is a schematic diagram of a convolution filter operation. It is a schematic diagram of the relationship between continuous deep net hierarchies. This is an example of division of a hierarchical connection relationship. This is another example of division of a hierarchical connection relationship. It is a flowchart explaining the operation of the arithmetic processing unit of 1st Embodiment. It is a flowchart of the whole process based on the division method between layers. It is a flowchart which shows the division method between layers. It is a figure explaining the exchange of the characteristic plane allocation between layers. It is a figure explaining the exchange of the characteristic plane allocation between layers. It is a schematic diagram which shows the graph generation of the 2nd Embodiment. It is an overall flowchart based on the graph cut of the 2nd Embodiment. It is a schematic diagram which shows the dynamic variation of the hierarchical connection in 4th Embodiment. It is a figure which shows the structure of the cloud server system of 5th Embodiment.

(First Embodiment)
An object of the present embodiment is to optimize the division of the hierarchical connection relationship in order to efficiently and at high speed parallel process the feature planes according to the configuration of the hierarchical connection relationship. Hereinafter, the details of the present embodiment will be described with reference to the drawings.

FIG. 1 shows a configuration example of the arithmetic processing unit of the present embodiment. This arithmetic processing unit has a pattern recognition function of detecting and recognizing a specific object from the input image data.

The arithmetic processing apparatus is composed of an image input module 1000, arithmetic circuits 1002-1 to 1002-n, RAM1001-1 to 1001-n, I / F1011, RAM1009, DMAC1006, CPU1007, ROM1008, and the like. The image input module 1000 is composed of an optical system, a photoelectric conversion device such as a CCD or CMOS sensor, a driver circuit for controlling the sensor, an AD converter, a signal processing circuit for controlling various image corrections, a frame buffer, and the like.

The RAM (Random Access Memory) 1001-1 to 1001-n are used as arithmetic work buffers of the arithmetic circuits 1002-1 to 1002-n, respectively.

Further, in the parallel arithmetic processing apparatus of the present embodiment, there are a plurality of arithmetic circuits 1002-1 to 1002-n and RAM 1001-1 to 1001-n for performing parallel processing.

As shown in the hierarchical connection relationship of FIG. 4, the plurality of feature planes on the reference side for calculating the plurality of feature planes on the output side and the respective filter kernels are once passed through the I / F 1011 from the outside of the arithmetic processing unit. , Stored in RAM 1009. The allocation module 1012 for dividing various data related to the hierarchical coupling relationship held in the RAM 1009 and storing them in the RAM 1001-1 to 1001-n, which is the memory of each arithmetic circuit 1002-1 to 1002-n, is used. Create split information for hierarchical join relationships. Here, the hierarchical connection relationship includes the structure of the connection relationship of the hierarchical feature surfaces, the convolution filter coefficient required for the convolution filter calculation for each connection relationship, and information on a plurality of feature surfaces.

The arithmetic circuits 1002-1 to 1002-n are CNN processing units that process the convolution filter arithmetic assigned based on the division information of the hierarchical coupling relationship according to the present embodiment.

The DMAC (Direct Memory Access Controller) 1006 controls data transfer between each processing module on the image bus 1003 and the CPU bus 1010. The bridge 1004 provides a bridge function between the image bus 1003 and the CPU bus 1010. The pre-processing module 1005 performs various pre-processing for effectively performing the pattern recognition processing by the CNN processing. Specifically, image data conversion processing such as color conversion processing / contrast correction processing is processed by hardware. The preprocessing module 1005 does not have to be included in the arithmetic processing unit of the present embodiment. The CPU 1007 controls the operation of the entire device. The ROM (Read Only Memory) 1008 stores instructions and parameter data that define the operation of the CPU 1007. The RAM 1009 is a memory required for the operation of the CPU 1007. The CPU 1007 can also access the RAM 1001 on the image bus 1003 via the bridge 1004. The division information of the hierarchical connection relationship is not limited to being generated by the allocation module 1012, and may be acquired from an external device through the I / F 1011. For example, when the division information of the hierarchical connection relationship is generated in the external PC and held in the RAM 1009 through the I / F 1011, the division information is held in the RAM 1009 and the parallel processing can be performed using the division information.

Since the configurations and operations of the arithmetic circuits 1002-1 to 1002-n of the arithmetic processing unit of the present embodiment are the same, the internal configuration and operation of a typical arithmetic circuit 1002 will be described with reference to FIG. 2A. .. Based on the division information, the control unit 601 stores the necessary data / convolution filter coefficient of the feature surface on the reference side in the RAM 1001. Since RAM 1001-1 to 1001-n, which is a memory corresponding to each arithmetic circuit 1002-1 to 1002-n of the arithmetic processing unit of the present embodiment, is also the same, RAM 1001 indicates one of them. The control unit 601 reads out the data and the convolution filter coefficient of the feature surface on the reference side held in the RAM 1001 and supplies the data to the convolution calculation unit 602. The convolution calculation unit 602 performs a convolution operation on the characteristic surface on the reference side and the convolution filter coefficient based on the operation represented by the equation (1), and outputs the calculation result. The control unit 601 outputs the calculation result to the RAM 1009.

FIG. 2B is a diagram illustrating a detailed configuration of the control unit 601 of FIG. 2A. The sequence control unit 1201 inputs and outputs various control signals 1204 that control the operation of the arithmetic circuit according to the information set in the register group 1202. Similarly, the sequence control unit 1201 calculates the control signal 1206 for controlling the memory control unit 1205. The sequence control unit 1201 is composed of a sequencer including a binary counter, a Johnson counter, and the like. The register group 1202 is composed of a plurality of register sets, and for example, information on a feature surface on the reference side, information on a feature surface to be calculated, information on a kernel, information on a feature surface held by dividing a hierarchy, and the like are recorded. A predetermined value of the register group 1202 is written in advance from the CPU 1007 via the bridge 1004 and the image bus 1003.

The memory control unit 1205 supplies the data 1207 of the feature surface on the reference side and the convolution filter coefficient data 1208 from the RAM to the convolution calculation unit 602 based on the control signal 1206 from the sequence control unit 1201. The calculation result 1209 is acquired from the convolution calculation unit 602. Here, the reference-side feature plane data and the convolution filter coefficient data that are supplied are based on the division information of the hierarchical connection relationship.

FIG. 5 shows an example of initial division of the connection relationship between the reference side layer 123 and the output side layer 124. The coupling relationship of the convolution filter operations between the feature planes by different filter kernels is represented by arrows 113 to 122, and the feature planes 101 to 106 in the hierarchy 123 are set as the reference side feature planes, and the feature planes 107 to 107 in the hierarchy 124. It is connected to 112 by the arrows 113 to 122. Here, arrows 113 to 122 indicate the coupling relationship by the filter kernel and indicate that the convolution filter operation is performed by the filter kernel. For example, the arrow 113 indicates that the feature surface 101 on the reference side to the feature surface 107 on the output side are generated by the convolution filter calculation, and indicates that the convolution filter calculation is performed using the filter kernel 113.

From the viewpoint of the layer 124, which is the layer on the output side, the feature surface 107 is generated by the filter kernel 113 with the feature surface 101 as the feature surface on the reference side, and the feature surface 108 is generated by the filter kernel 115 with the feature surface 102 on the reference side. Generated as a feature plane. The feature plane 109 is generated by the filter kernels 116 and 119 with the feature planes 103 and 105 as the reference side feature planes, respectively, and the feature plane 110 is generated by the filter kernels 114 and 121 with the feature planes 101 and 106 as the reference side feature planes, respectively. Is generated as. The feature plane 111 is generated by the filter kernels 117 and 118 with the feature planes 102 and 104 as the reference side feature planes, respectively, and the feature plane 112 is generated by the filter kernels 120 and 122 with the feature planes 103 and 106 as the reference side feature planes, respectively. Generated as data. In order for one arithmetic circuit to sequentially calculate the feature plane 107, the feature plane 108, and the feature plane 109, the feature plane 101, the feature plane 102, the feature plane 103, and the feature plane 105 are each featured from the memory of the calculation circuit. It is necessary to read the filter kernel corresponding to the surface sequentially.

Next, with reference to FIG. 5, the outline of the initial division of the hierarchical connection relationship for parallelization and the result of the initial division will be described. The initial division is executed by the allocation module 1012 based on the division conditions. The division conditions include the specification of the number of divisions based on the number of usable arithmetic circuits and the maximum amount of data held in the memory of the arithmetic circuits. Further, in order to distribute the processing load, the allocation module 1012 reduces the difference in the amount of calculation of each arithmetic circuit based on the number of calculated feature planes and the number of reference feature planes. Performs initial division processing. By the processing of the initial division of the allocation module 1012, one candidate of the characteristic surface assigned to each arithmetic circuit can be obtained as the characteristic surface on the output side calculated by each arithmetic circuit.

FIG. 5 shows the result of the initial division of the connection relationship between the layers, and shows how the feature plane generated by the calculation is divided into the division 125 and the division 126. In the division 125, the feature planes 107, 108, 109 are used as one unit, and the convolution filter calculation for calculating the feature planes 107, 108, 109 is assigned to the calculation circuit 1002-1. In the division 126, the feature planes 110, 111, 112 are used as one unit, and the convolution filter calculation for calculating the feature planes 110, 111, 112 is assigned to the calculation circuit 1002-2.

Here, it is assumed that the feature plane and the filter coefficient on the reference side required to generate each feature plane are held and operated by the RAM 1001 of the assigned arithmetic circuit 1002. For example, the reference side feature planes 101, 102, 103, 105 for generating the feature planes 107, 108, 109 shown in the division 125 of FIG. 5, and the convolution filters shown by the arrows 113, 115, 116, 119, respectively. The coefficient is held in RAM 1001-1. Further, the convolution filter coefficients shown by the feature planes 101, 102, 103, 106 on the reference side for generating the feature planes 110, 111, 112 shown in the division 126 and the arrows 114, 117, 120, 118, 122, respectively. Is held in RAM 1001-2.

Here, paying attention to the retained data, the feature surfaces 101, 102, and 103 on the reference side need to be duplicated and retained by the RAM 1001 of the arithmetic circuit of each allocation destination, so that the memory required for parallel arithmetic is required. The area and data transfer amount increase.

Next, FIG. 6 shows an example of the division result different from that of FIG. 5 with respect to the same hierarchical connection relationship as in FIG. The division result shown in FIG. 6 is another candidate for the characteristic surface assigned to each arithmetic circuit as the characteristic surface on the output side calculated by each arithmetic circuit by the processing of the allocation module 1012 of the present embodiment. The allocation module 1012 selects an allocation from the candidates shown in FIG. 5 and the candidates shown in FIG. The selection of allocation will be described later, but here, first, the division result shown in FIG. 6 will be described.

The division results shown in FIG. 6 are division 201 and division 202. The arithmetic processing for generating the characteristic surfaces 107, 108, 111 shown in the division 201 is assigned to the arithmetic circuit 1002-1, and the arithmetic processing for generating the characteristic surfaces 109, 110, 112 shown in the division 202 is assigned to the arithmetic circuit. It is assigned to 1002-2.

RAM1001-1 for each of the reference-side feature planes 101, 102, 104 for generating the feature planes 107, 108, 111 shown in the division 201 and the convolution filter coefficients indicated by the arrows 113, 115, 117, 118, respectively. Hold with. Further, the feature planes 101, 103, 105, 106 on the reference side for generating the feature planes 109, 110, 112 shown in the division 202, and the convolution filter coefficients shown by the arrows 114, 116, 119 to 122, respectively. Is held by RAM 1001-2.

Here, paying attention to the retained data of the RAM 1001-1 and the RAM 1001-2, it is necessary to duplicately retain only the characteristic surface 101 on the reference side.

Comparing the divisions 125 and 126 of FIG. 5 with the divisions 201 and 202 of FIG. 6, the RAM 1001-1 and the RAM 1001 are the same, although the processing content and the amount of calculation for calculating the characteristic surface of the layer 124 on the output side are the same. The number of feature planes to be retained in duplicate at -2 is different. In this way, the amount of duplication of data to be held in the memory changes depending on the allocation unit for performing parallel processing, so that the efficiency of parallel processing is greatly affected.

FIG. 7 is a flowchart illustrating an operation for the parallel arithmetic unit of the present embodiment to perform pattern recognition. Hereinafter, it is assumed that the flowchart is realized by the CPU 1007 executing the control program.

In step S1101, the CPU 1007 executes various initialization processes prior to the start of the recognition process. The CPU 1007 transfers the filter coefficient required for the CNN processing operation of the arithmetic circuit from the ROM 1008 to the RAM 1001, and sets various registers for defining the operation of the arithmetic circuit 1002, that is, the hierarchical coupling relationship. Specifically, predetermined values are set in a plurality of registers existing in the control unit 601 of the arithmetic circuit 1002. Similarly, the CPU 1007 writes a value necessary for operation to a register such as the preprocessing module 1005.

Next, in step S1102, the allocation module 1012 determines the division of the hierarchical structure when calculating each feature surface, and generates the division information of the hierarchical connection relationship. Here, the division of the hierarchical structure is determined according to conditions such as the number of arithmetic circuits operating in parallel, and a specific division method will be described later.

A series of object recognition operations are started after step S1101 for performing the initialization process and step S1102 for dividing the hierarchical structure. First, in step S1103, the image input module 1000 converts the signal output by the image sensor into image data and stores it in a frame buffer built in the image input module 1000, although not shown in frame units. When the storage of the image data in the frame buffer is completed, the preprocessing module 1005 starts the image conversion process based on the predetermined signal. In step S1104, the preprocessing module 1005 extracts the luminance data from the image data on the frame buffer and performs the contrast correction process.

The luminance data is extracted by generating the luminance data from the RGB image data by a general linear conversion process. The contrast correction method also applies a generally known contrast correction process to enhance the contrast. The preprocessing module 1005 stores the luminance data after the contrast correction processing as a detection image in the RAM 1001 corresponding to the arithmetic circuit 1002 to which the parallel processing is distributed, based on the division information of the hierarchical connection relationship. When the preprocessing for one frame of image data is completed, the preprocessing module 1005 enables a completion signal (not shown).

In step S1105, the arithmetic circuit 1002 is activated based on the completion signal enabled by the preprocessing module 1005, and starts the object detection process based on the CNN process. The process in step S1105 is an operation based on the division information of the hierarchical connection relationship generated in step S1102. In step S1106, when the calculation of the characteristic surface of the final layer is completed, the arithmetic circuit 1002 generates a completion interrupt to the CPU 1007.

In step S1107, when the CPU 1007 receives the completion interrupt indicating the end of processing of the arithmetic circuit 1002, the CPU 1007 analyzes the characteristic surface of the final layer and determines the position and attributes of the object in the image. When the analysis process of step S1107 is completed, the process proceeds to step S1108 to determine the division of the hierarchical structure when calculating each feature plane at which the processing for the image of the next frame continues, and generate the division information of the hierarchical connection relationship. do.

Next, using FIG. 8, in step S1102, the allocation module 1012 divides the hierarchical structure of CNN processing so that the calculation of the plurality of feature planes is processed in parallel by the plurality of arithmetic circuits. The method of generating the division information of is described. In this embodiment, a calculation method based on simulated annealing (simulated annealing method) will be described. First, the division condition is acquired in step S801. The division condition referred to here is a specification of the number of divisions based on the number of usable arithmetic circuits, a condition based on the amount of data that needs to be held or transferred per arithmetic circuit, and the like. Next, in step S802, any layer is selected from the hierarchical connection relationship. That is, the pair of the input layer and the output layer that are directly connected is selected. The division of the connection relationship between the selected hierarchies is determined in step S803. Then, in step S804, the processes up to step S803 are repeated until the process is completed between all the layers.

FIG. 9 is a flowchart showing a specific process of determining the division between the selected hierarchies in step 803.

First, in step S901, the allocation module 1012 determines the coupling division between the initial layers, as shown in FIG. 5, based on the number of arithmetic circuits to which a plurality of convolution filter operations can be assigned in the CNN process. .. Next, the division of the initial hierarchical connection relationship is analyzed. As a result, with respect to the coupling relationship 127 between the layers shown in FIG. 5, as shown in FIG. 10, the allocation module 1012 calculates the characteristic surfaces 107 to 112 on the output side, and allocates 1306 to the arithmetic circuit 1 and the arithmetic circuit. Allocation to 2 It turned out that it was assigned like 1315. At this time, the data 1301 necessary for calculating the characteristic surfaces 107 to 109 on the output side with respect to the allocation 1306 needs to be held in a RAM (not shown) corresponding to the arithmetic circuit 1. Specifically, in order to calculate the feature surface 107 on the output side, the feature surface 101 on the reference side and the convolution filter kernel 1302, and in order to calculate the feature surface 108 on the output side, the feature surface 102 on the reference side and the convolution surface 102 are convoluted. The filter kernel 1303 is held in the RAM of the arithmetic circuit 1. Further, in order to calculate the feature surface 109 on the output side, the feature surface 103 on the reference side and the convolution filter kernel 1304, and the feature surface 105 on the reference side and the convolution filter kernel 1305 are held in the RAM of the arithmetic circuit 1.

Further, regarding the allocation 1315 to the arithmetic circuit 2, the data 1308 necessary for calculating the characteristic surfaces 110 to 112 on the output side needs to be held in a RAM (not shown) corresponding to the arithmetic circuit 2. Specifically, in order to calculate the feature surface 110 on the output side, the feature surface 101 on the reference side and the convolution filter kernel 1309, and the feature surface 106 on the reference side and the convolution filter kernel 1313 are held in the RAM of the arithmetic circuit 2. NS. Similarly, in order to calculate the feature surface 111 on the output side, the feature surface 102 on the reference side and the convolution filter kernel 1310, and the feature surface 104 on the reference side and the convolution filter kernel 1312 are held in the RAM of the arithmetic circuit 2. .. Further, in order to calculate the feature surface 112 on the output side, the feature surface 103 on the reference side and the convolution filter kernel 1311 and the feature surface 106 on the reference side and the convolution filter kernel 1314 are held in the RAM of the arithmetic circuit 2.

Here, looking at the data that needs to be held in the respective RAMs due to the allocation 1306 to the arithmetic circuit 1 and the allocation 1315 to the arithmetic circuit 2, the data of the feature planes 101, 102, and 103 on the reference side are duplicated. It is the data to be retained. In the example of FIG. 5, the number of characteristic surfaces assigned to each arithmetic circuit is the same so as to be well-balanced, but the present invention is not limited to this, and the characteristic surfaces assigned to each arithmetic circuit are not limited to this. The number may be different.

Next, in step S902, two characteristic surfaces on the output side assigned to different arithmetic circuits between the hierarchical processes are picked up. For example, the feature surface 109 and the feature surface 111 on the output side are picked up from the allocation 1306 to the arithmetic circuit 1 and the allocation 1315 to the arithmetic circuit 2 in FIG. 10, respectively, and the picked up feature surface allocation is performed in step S903. Exchange. In the exchange of feature surface allocation, the coupling relationship is divided so that the calculation of the feature surface 109 assigned to the calculation circuit 1 is assigned to the calculation circuit 2 and the calculation of the feature surface 111 assigned to the calculation circuit 2 is assigned to the calculation circuit 1. This is the process of changing information.

The data held in the RAM of each arithmetic circuit after exchanging the characteristic planes on the output side assigned to the two arithmetic circuits will be described with reference to FIG. The allocation of the characteristic surface on the output side to be calculated for the arithmetic circuit 1 and the arithmetic circuit 2 after the exchange is as shown in the allocation 1402 and the allocation 1405. When the coupling relationship after the exchange is analyzed, the data required to create the feature planes 107, 108, and 111 on the output side with respect to the allocation 1402 is the data 1401. That is, in order to calculate the feature surface 107 on the output side, the feature surface 101 on the reference side and the convolution filter kernel 1302, and in order to calculate the feature surface 108 on the output side, the feature surface 102 on the reference side and the convolution filter kernel It is necessary to hold 1303 in the RAM of the arithmetic circuit 1. Further, in order to calculate the feature surface 111 on the output side, it is necessary to hold the feature surface 102 on the reference side and the convolution filter kernel 1310, and the feature surface 104 on the reference side and the convolution filter kernel 1312 in the RAM of the arithmetic circuit 1. be.

Regarding the allocation 1405, the data required to generate the feature planes 110, 109, 112 on the output side is the data 1404. That is, in order to calculate the feature surface 110 on the output side, it is necessary to hold the feature surface 101 on the reference side and the filter kernel 1309, and the feature surface 106 on the reference side and the filter kernel 1313 in the RAM of the arithmetic circuit 2. Further, in order to calculate the feature surface 109 on the output side, it is necessary to hold the feature surface 103 on the reference side and the filter kernel 1304, and the feature surface 105 on the reference side and the filter kernel 1305 in the RAM of the arithmetic circuit 2. Further, in order to calculate the feature surface 112 on the output side, it is necessary to hold the feature surface 103 on the reference side and the filter kernel 1311, and the feature surface 106 on the reference side and the filter kernel 1314 in the RAM of the arithmetic circuit 2.

Here, looking at the data that needs to be held in the RAMs of the respective arithmetic circuits in the allocation 1402 and the allocation 1405, only the characteristic surface 101 on the reference side is duplicated and retained. Compared with the allocation 1306 and the allocation 1315 shown in FIG. 10, when the number of data required to be held in the RAM is compared by the number of feature planes, it is possible to reduce the number from 9 planes to 7 planes.

Next, the evaluation value is calculated in step S904. Here, the evaluation value is composed of a data amount of data duplicated and held in the RAM of each arithmetic circuit and a penalty value. The penalty value is determined based on the division condition acquired in step S801. The division condition is, for example, the number of arithmetic circuits or the amount of arithmetic processing that can be allocated (divided) per arithmetic circuit. The calculation processing amount of the calculation circuit is calculated based on conditions such as the characteristic surface of the reference side, the size of the filter kernel, and the number of calculation cycles. Further, the division condition may be determined in consideration of the allowable number of characteristic surfaces to be duplicated and held in the RAM of each arithmetic circuit, the processing load distribution condition of each arithmetic circuit, and the like. In the present embodiment, for the sake of simplification of the description, an example of the division condition will be described using the total value of the feature plane size on the reference side and the size of the filter kernel as the amount of arithmetic processing that can be processed per arithmetic circuit.

First, in calculating the evaluation value, when the number of divisions is a and the amount of data to be retained in duplicate between the allocation l and the allocation k is n (l, k), the data to be retained in duplicate in all allocations. The total amount s is

と表すことができる。

It can be expressed as.

Next, the penalty value will be described. And x _i the reference side wherein surface total size required to process the assignment i, when the total size of the filter kernel and w _i, the sum t _i of necessary data size in the arithmetic circuit for each one is t _i = can be expressed as x _i + _{w i.} As described above, in the present embodiment deals with the total value of the feature surface size and the size of the filter kernel can be processed reference side per one operation circuit as a split condition, when the condition value th, each t _i is th comparing whether less, if it exceeds th give C penalty value p _i for each assignment.

つまり、全割当てでのペナルティ値の合計は

In other words, the total penalty value for all allocations is

以上より評価値ｆは以下のように表すことができる。

From the above, the evaluation value f can be expressed as follows.

f = s + P
In step S905, based on the evaluation value calculated in step S904, if the evaluation value is better than that before the allocation, the change of the allocation is adopted, and if not, the allocation is returned to the allocation before the exchange. Here, as the criterion for changing the allocation, not only the evaluation value is good or bad before and after the change, but also the adoption selection may be made with a threshold value having a range such as changing if the evaluation value is n% or more good.

Steps from step S902 until constraint conditions such as the number of repetitions and time preset in step S906, the number of feature faces duplicated and held in the RAM of each arithmetic circuit, and the duplicate data amount reduction rate by exchanging feature faces are satisfied. The process up to S905 is repeated. For example, the processes from step S902 to step S905 are repeated until the number of feature planes that are duplicated and held in the RAM of each arithmetic circuit becomes a predetermined number or less. Moreover, when it is based on simulated annealing, it is also possible to follow the convergence condition. Further, the present embodiment is not limited to simulated annealing, and may have an evolutionary algorithm represented by a genetic algorithm or the like.

According to the method of the present embodiment, it is possible to optimally divide (assign) a plurality of convolution filter operations to each arithmetic circuit according to the existing hierarchical coupling relationship. By performing the optimum division, the amount of data duplicated and held in the memory of each arithmetic circuit is reduced, the time for data transfer and data reading is shortened, and efficient parallel processing becomes possible. Further, it can be said that this embodiment has high flexibility because it can be applied to a learned hierarchical connection relationship.

Although the case of CNN processing has been described in this embodiment, the method of this embodiment can be applied to other hierarchical processing such as Restricted Boltzmann Machines and Recurrent Neural Network.

Further, in the present embodiment, an example of hierarchical arithmetic processing for a feature surface which is two-dimensional feature data has been described, but for one-dimensional feature data such as voice data and three-dimensional feature data including time changes. It can also be applied to hierarchical arithmetic processing such as CNN processing.

(Second Embodiment)
Next, a second embodiment of the present invention will be described. Since the hardware configuration of this embodiment is the same as that of the first embodiment, the description thereof will be omitted.

In the present embodiment, when the allocation module 1012 divides the connection relationship between layers, the connection relationship between layers is regarded as a graph, and the graph is cut based on the maximum flow min-cut theorem to form a connection between layers. Decide on a split. The division method of this embodiment will be described with reference to FIGS. 12 and 13. FIG. 13 represents another embodiment of the inter-layer division determination step S803 on the flowchart of FIG. The graph in the present embodiment represents a hierarchical connection relationship. Further, in the following, when the hierarchical connection relationship is treated as a graph, the feature surface is used as the connection point of the graph, and the connection relationship of the feature surfaces is used as an effective graph from the input layer to the output layer.

In step S1602, the hierarchical connection relationship to be divided is selected. Since this process is based on the assumption that the graph will be divided repeatedly in a divide-and-rule manner, the graph will be iteratively processed until all the conditions are met. That is, the graph that does not meet the conditions is selected in this step.

Next, in step S1603, the selected graph is shaped to perform graph cutting. In explaining this step, a hierarchical connection relationship as shown in FIG. 5 will be described as an example. In FIG. 12, with respect to the hierarchical relationship of FIG. 5, each feature surface 101-112 is regarded as a coupling point, and s point 1501 and t point 1514 are added as transmission points and reception points, respectively. Further, the s point and all the connection points are connected in the direction of the connection point, and the t point and the connection point are connected in the direction of the t point. Here, arrows 113-122 represent the coupling relationship between the layer 123 of the feature surface on the reference side and the layer 124 of the feature surface on the output side, and the size and kernel of the feature surface on the reference side for each arrow. The total value of the filter coefficients is given as a weight. Here, the weight of the arrow is explained as the total value of the size of the feature plane and the coefficient of the kernel filter, but the present invention is not limited to this. For arrows 1502-1513 and arrows 1515-1526, the values are sufficiently large with respect to the weights of arrows 113-122.

In step S1604, the graph is cut by using a classical graph cut method such as the maximum flow minimum cut for the graphed relationship between the layers created in the previous step. The graph cut does not cut between arrows 1502-1513 and arrows 1515-1526 with sufficiently large weights, but cuts the input layer and output layer between arrows 113-122 with the least weighted coupling relationship. ..

In step S1605, it is confirmed whether the cut graph satisfies the division condition, and steps S1602-S1604 are repeatedly executed until the condition is satisfied. The division conditions described below can be obtained in step S801 of FIG. Here, the division condition is a predetermined value of the amount of data required for processing in all the cut divisions, as in the first embodiment. Normally, the data required for processing is held in the RAM of the arithmetic circuit. Therefore, check whether the amount of data held in the RAM exceeds the predetermined value, and if it does not fit in the predetermined value, display the graph. On the other hand, the process of cutting again is performed. When specifying the number of cut graphs, a plurality of cut graphs are randomly selected and merged, and the merged graph is cut again. By repeating this, it is possible to repeat the process so as to reach the specified number. By the processing of the present embodiment, the division result shown in FIG. 6 can be obtained as in the first embodiment. The division method of the present embodiment can minimize the amount of data to be duplicated and held in the RAM of each arithmetic circuit. Further, the amount of data held in the RAM of the arithmetic circuit can be prevented from exceeding a predetermined value.

(Third Embodiment)
Next, a third embodiment of the present invention will be described. In the present embodiment, a method of dynamically dividing the hierarchical connection relationship and performing the convolution filter calculation in the device based on a predetermined condition will be described. Since the hardware configuration of this embodiment is the same as that of the first embodiment, the description thereof will be omitted. In the present embodiment, the allocation module 1012 confirms the operating state of the arithmetic circuits 1002-1 to 1002-n at a predetermined timing.

The allocation module 1012 is dynamically divided according to the operating state of each arithmetic circuit 1002-1 to 1002-n. The allocation module 1012 outputs the division result of the coupling relationship, and based on this, the CPU 1007 transfers the feature surface and the filter kernel on the reference side to the RAMs of the respective arithmetic circuits 1002-1 to 1002-n at any time.

As a result, for example, as a result of checking the operating state of the arithmetic circuits at a certain timing, if M arithmetic circuits have completed processing among the N arithmetic circuits, the M arithmetic circuits are not processed. CNN operations can be assigned. As a result, even if the operating state of the arithmetic circuit fluctuates, it is possible to efficiently perform parallel processing by appropriately and optimally dividing the operation state.

(Fourth Embodiment)
Next, a fourth embodiment of the present invention will be described. This embodiment is also effective when the hierarchical connection relationship is dynamically changed. FIG. 14 shows how the hierarchical connection relationship is dynamically changing.

The normal CNN process is performed on the hierarchical combination shown in FIG. However, depending on internal factors such as imaging conditions and intermediate results of feature extraction, and physical factors such as computing resources and power consumption in the device, the structure and convolution of the coupling relationship shown by the broken lines 501, 502, and 503 in FIG. 14 It can also be applied to changes in filter coefficients and dynamic changes in feature planes.

Assuming that fluctuations occur depending on the imaging status and the intermediate result of feature extraction, when the probability that the identification target is a person is higher than a predetermined value from the intermediate result of the recognition process, the part indicated by the broken lines 501 to 503 is the feature amount related to the animal. If is the main, the part indicated by the broken line is not processed.

Further, depending on the priority such as the calculation resource and the power consumption in the device, the portion indicated by the broken lines 501 to 503 may not be processed within a range that does not significantly affect the result of the recognition process.

In this case, the CPU 1007 notifies the allocation module 1012 of the changed hierarchical connection relationship, and the allocation module 1012 divides the changed connection relationship. The CPU 1007 dynamically executes parallel processing by supplying data to the RAM, which is the memory of each arithmetic circuit, based on the dynamically divided coupling relationship of the allocation module 1012.

(Fifth Embodiment)
Next, a fifth embodiment of the present invention will be described. In the embodiments described so far, the embodiments have been for embedded devices, but in the present embodiment, the embodiments in the cloud server system will be described. FIG. 15 is a diagram showing a cloud server system.

The control PC 1701 controls all the CNN processing PCs 1705 to 1713 via the communication network 1702 to 1704. The control referred to here is the allocation of the feature plane data / weight kernel on the reference side to the CNN processing PCs 1705 to 1713 in order to perform parallel processing based on the division information of the hierarchy, and the distributed data processing control based on this allocation. be.

Specifically, the control PC 1701 divides the hierarchical structure by the method described with reference to FIGS. 7 and 8, and determines the allocation of characteristic surfaces and the like to each CNN processing PC. Similar to the first embodiment, the method of dividing the hierarchical structure is determined so as to reduce the amount of data that needs to be duplicated and held in each CNN processing PC, so that the amount of data supplied to the CNN processing PC 1705-1713 can be increased. There are few parallel operations, and parallel operations are performed efficiently.

Based on the division information determined by the control PC 1701, each CNN processing PC 1705-1713 performs a convolution operation and outputs the operation result to the control PC. As a result, parallel processing can be efficiently performed not only in the embedded device but also in the cloud server system.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

1000 Image Input Module 1001-1 to 1001-n RAM
1002-1 to 1002-n arithmetic circuit 1012 allocation module 1007 CPU
1008 ROM
1009 RAM

Claims (13)

Multiple arithmetic circuits that perform hierarchical arithmetic processing in parallel,
A plurality of operations in a predetermined layer in the arithmetic process, and each of the plurality of operations that refers to at least a part of the operation results in the layers prior to the predetermined layer is assigned to the plurality of arithmetic circuits. Has an allocation means and
The allocation means
An evaluation value corresponding to the amount of overlapping data between the plurality of arithmetic circuits related to the arithmetic result in the layer prior to the predetermined layer to be referred to at the time of calculation is calculated for the allocation candidate.
The evaluation value is an allocation candidate satisfying a predetermined criterion, and the duplication of the calculation result in the hierarchy before the predetermined hierarchy referred to in the calculation among the plurality of arithmetic circuits is at least from the first allocation candidate. An arithmetic processing unit, characterized in that a small number of second allocation candidates are determined as allocations of the plurality of operations to the plurality of arithmetic circuits. The arithmetic processing unit according to claim 1, wherein each of the plurality of arithmetic circuits has a storage means for storing arithmetic results in a hierarchy prior to the predetermined hierarchical reference when performing arithmetic. The arithmetic processing unit according to claim 1 or 2, wherein the arithmetic processing is an arithmetic processing by a convolutional neural network. The arithmetic processing unit according to any one of claims 1 to 3, wherein the allocation means allocates each of the plurality of operations to the plurality of arithmetic circuits based on a simulated annealing method. One of claims 1 to 4, wherein the allocating means allocates each of the plurality of operations to the plurality of arithmetic circuits so that the difference in the amount of operations between the plurality of arithmetic circuits is small. The arithmetic processing unit described in 1. The arithmetic processing according to claim 2, wherein the allocating means allocates each of the plurality of operations to the plurality of arithmetic circuits so that the amount of data stored in the storage means does not exceed a predetermined value. Device. The allocation means performs each of the plurality of operations so that the duplication of the calculation results in the layers prior to the predetermined layer referred to in the calculation among the plurality of calculation circuits is less than a predetermined number. The arithmetic processing apparatus according to any one of claims 1 to 6, wherein the arithmetic processing apparatus is assigned to an arithmetic circuit. Further, it has a first confirmation means for confirming the operating state of the plurality of arithmetic circuits.
Any of claims 1 to 7, wherein the allocation means changes the allocation of the plurality of operations to the plurality of arithmetic circuits based on the operating state confirmed by the first confirmation means. The arithmetic processing unit according to item 1. The arithmetic processing unit according to any one of claims 1 to 8, wherein the arithmetic circuit performs a convolution operation between an arithmetic result in a layer prior to the predetermined layer and a filter kernel. The allocation means indicates that the evaluation value of the second allocation candidate is duplicated and held between the plurality of arithmetic circuits is a predetermined number or less, or the allocation means in the first allocation candidate. The second is that the reduction rate of the amount of data duplicated and held between the plurality of arithmetic circuits in the second allocation candidate is equal to or greater than a predetermined value with respect to the amount of data duplicated and held among the plurality of arithmetic circuits. When the evaluation value of the allocation candidate of is indicated, the second allocation candidate is determined as the allocation of the plurality of operations to the plurality of arithmetic circuits, according to any one of claims 1 to 9. The arithmetic processing device described. The evaluation value according to any one of claims 1 to 10, wherein the evaluation value has a penalty value according to the number of the plurality of arithmetic circuits and the amount of arithmetic processing that can be assigned to the plurality of arithmetic circuits. Arithmetic processing unit. A plurality of operations in a predetermined layer in the arithmetic processing for a plurality of arithmetic circuits that perform hierarchical arithmetic processing in parallel, and at least a part of a plurality of operation results in a layer prior to the predetermined layer. In the allocation of each of the plurality of operations with reference to
An evaluation value corresponding to the amount of overlapping data between the plurality of arithmetic circuits related to the arithmetic result in the layer prior to the predetermined layer to be referred to at the time of calculation is calculated for the allocation candidate.
The evaluation value is an allocation candidate satisfying a predetermined criterion, and the duplication of the calculation result in the hierarchy before the predetermined hierarchy referred to in the calculation among the plurality of arithmetic circuits is at least from the first allocation candidate. An arithmetic processing method characterized in that a small number of second allocation candidates are determined as allocations of the plurality of operations to the plurality of arithmetic circuits. A program for operating a computer as the arithmetic processing unit according to any one of claims 1 to 11.

Images