JP2021099596A - Information processing apparatus - Google Patents

Abstract

To provide an information processing apparatus which can execute processing using a neural network and other processing in parallel without increasing arithmetic processing capability.SOLUTION: In an information processing apparatus 1, an operation unit 11 executes operations of a neural network 20 including a plurality of intermediate layers and operations of a prescribed program different from the neural network. An operation object selection unit 12 selects an intermediate layer where an operation should be performed, from the plurality of intermediate layers on the basis of a structure of the neural network 20, operation amount data indicative of respective operation amounts of the plurality of intermediate layers, and load of the operation unit 11. An operation instruction unit 13 instructs the operation unit 11 to perform the operation of the intermediate layer selected by the operation object selection unit 12.SELECTED DRAWING: Figure 1

Description

The present invention relates to an information processing device that executes operations related to a neural network.

Research is underway to mount artificial intelligence on vehicles such as automobiles. For example, Patent Document 1 discloses a vehicle electronic control device that constructs an artificial intelligence model based on a running state of a vehicle and executes processing using the constructed artificial intelligence model. In Patent Document 1, a neural network is used as an artificial intelligence model.

JP-A-2018-190045

The amount of calculation of the neural network increases according to the scale of the neural network. When an information processing device such as a vehicle electronic control device executes processing using a relatively large-scale neural network, the load on the information processing device becomes high. When the load on the information processing device becomes high, the information processing device may not be able to perform other processing other than the processing using the neural network.

In order to execute processing using a neural network and other processing in parallel, it is conceivable to adopt an information processing device having high arithmetic processing power. However, the adoption of an information processing device having high computing power causes an increase in cost.

In view of the above problems, an object of the present invention is to provide an information processing apparatus capable of executing processing using a neural network and other processing in parallel without increasing the arithmetic processing capacity of _Hlk27417505. To do.

In order to solve the above problems, the first invention is an information processing apparatus, which includes a calculation unit, a calculation target selection unit, and a calculation instruction unit. The arithmetic unit executes an arithmetic of a neural network including a plurality of intermediate layers and an arithmetic of a predetermined program different from the neural network. The calculation target selection unit selects an intermediate layer to be calculated from a plurality of intermediate layers based on the structure of the neural network, the calculation amount data indicating the calculation amount in each of the plurality of intermediate layers, and the load of the calculation unit. To do. The calculation instruction unit instructs the calculation unit to perform the calculation of the intermediate layer selected by the calculation target selection unit.

According to the first invention, it is possible to execute a process using a neural network and other processes in parallel without increasing the arithmetic processing capacity.

The second invention is the first invention, and in the calculation target selection unit, the total of the load of the calculation unit generated by the calculation of the neural network and the load of the calculation unit generated by the predetermined program calculation is calculated in advance. Select the intermediate layer to be calculated so as not to exceed the set upper limit of the load of the calculation unit.

According to the second invention, it is possible to prevent the calculation of the neural network from becoming longer than planned because the upper limit of the load of the calculation unit is suppressed during the calculation of the neural network.

The third invention is the first or second invention, in which the plurality of intermediate layers includes a first intermediate layer and a second intermediate layer. The second intermediate layer is independent of the calculation result of the first intermediate layer. The calculation target selection unit is the sum of the load of the calculation unit generated by the calculation of the first intermediate layer, the load of the calculation unit generated by the calculation of the second intermediate layer, and the load of the calculation unit generated by the calculation of a predetermined program. When the upper limit of the load of the calculation unit is not exceeded, the first intermediate layer and the second intermediate layer are selected as the intermediate layers to be calculated.

According to the third invention, since the calculation of two or more intermediate layers and other processing can be executed in parallel, the completion of the calculation of the neural network can be accelerated.

The fourth invention is any one of the first to third inventions, and the arithmetic unit calculates the first neural network including the first intermediate layer and the second neural network including the second intermediate layer. .. In the calculation target selection unit, the layer identification unit is a load of the calculation unit generated by the calculation of the first intermediate layer, a load of the calculation unit generated by the calculation of the second intermediate layer, and a calculation unit generated by the calculation of a predetermined program. When the total with the load does not exceed the upper limit of the load of the calculation unit, the first intermediate layer and the second intermediate layer are specified as the intermediate layers to be calculated.

According to the fourth invention, the two neural networks and other processes can be executed in parallel.

A fifth invention is a control method of an information processing apparatus including an arithmetic unit that executes an operation of a neural network including a plurality of intermediate layers and an operation of a predetermined program different from the neural network. b) Provide with steps. a) In the step, the intermediate layer to be calculated is selected from the plurality of intermediate layers based on the structure of the neural network, the calculation amount data indicating the calculation amount in each of the plurality of intermediate layers, and the load of the calculation unit. .. b) The step instructs the calculation unit to perform the calculation of the selected intermediate layer.

The fifth invention is used in the first invention.

The present invention can provide an information processing apparatus capable of executing processing using a neural network and other processing in parallel without increasing the arithmetic processing capacity.

It is a functional block diagram which shows the structure of the information processing system which concerns on 1st Embodiment of this invention. It is a figure which shows the hardware configuration of the information processing apparatus shown in FIG. It is the schematic which shows an example of the structure of the neural network shown in FIG. It is a figure which shows an example of the arithmetic amount data 52 shown in FIG. It is a figure explaining the calculation method of the arithmetic amount of the convolution layer shown in FIG. It is a graph which shows an example of the time change of the general load excluding the load generated by the calculation of the neural network among the loads of the information processing apparatus shown in FIG. It is a flowchart which shows the operation of the information processing apparatus shown in FIG. It is a functional block diagram which shows the structure of the information processing system which concerns on 2nd Embodiment of this invention. It is the schematic which shows the structure of the neural network shown in FIG. It is a graph which shows an example of the time change of the general load excluding the load generated by the calculation of the neural network among the loads of the information processing apparatus shown in FIG. It is a figure which shows an example which converts a neural network into a block diagram. It is a figure which shows an example of the structure of a neural network. It is a figure which represented the neural network shown in FIG. 12 as an adjacency matrix. It is a block diagram which converted the neural network shown in FIG. This is another example of a block diagram converted from a neural network. It is a figure which graphed the block diagram shown in FIG. It is a figure which shows the procedure of the allocation of parallel processing based on the processing dependency shown in FIG.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. The same or corresponding parts in the drawings are designated by the same reference numerals, and the description thereof will not be repeated.

[First Embodiment]
{1. Constitution}
{1.1. Configuration of information processing system 100}
FIG. 1 is a functional block diagram showing a configuration of an information processing system 100 according to an embodiment of the present invention. With reference to FIG. 1, the information processing system 100 includes an information processing device 1 and a storage device 2.

The information processing device 1 is an electronic control unit (Electronic Control Unit) mounted on a vehicle (not shown). In the present embodiment, the information processing device 1 is an image recognition device. The information processing device 1 acquires a front image 61 of the front of the vehicle from a camera mounted on the vehicle, and detects a pedestrian from the acquired front image 61. The front image 61 is a frame included in the moving image. The information processing device 1 generates result data 62 indicating a pedestrian detection result and outputs the result data 62 to a car navigation device (not shown).

The storage device 2 is a non-volatile storage device, for example, a flash memory. The storage device 2 stores the neural network 20, the preprocessing program 41, the result output program 42, the control program 43, and the calculation amount data 52.

The neural network 20, the preprocessing program 41, and the result output program 42 are programs for image recognition processing, and are executed by the arithmetic unit 11. The neural network 20 is used in this embodiment to detect a pedestrian. The configuration of the neural network 20 will be described later. The calculation amount data 52 records the calculation amount of each of the plurality of intermediate layers included in the neural network 20.

The preprocessing program 41 executes preprocessing such as adjusting the size of the front image 61 in order to input the front image 61 to the neural network 20. The result output program 42 generates the result data 62 based on the detection result of the neural network 20. The control program 43 is used to control the entire information processing apparatus 1, and executes a process other than the process related to the image recognition process.

{1.2. Configuration of information processing device 1}
With reference to FIG. 1, the information processing apparatus 1 includes a calculation unit 11, a calculation target selection unit 12, and a calculation instruction unit 13.

The calculation unit 11 executes the calculation of the neural network 20, the calculation of the preprocessing program 41, the calculation of the result output program 42, and the calculation of the control program 43.

The calculation target selection unit 12 selects an intermediate layer to be calculated from a plurality of intermediate layers included in the neural network 20 based on the structure of the neural network 20 and the calculation amount data 52.

The calculation instruction unit 13 instructs the calculation unit 11 to perform the calculation of the intermediate layer selected by the calculation target selection unit 12.

FIG. 2 is a diagram showing a hardware configuration of the information processing device 1 shown in FIG. With reference to FIG. 2, the information processing unit 1 includes a CPU (Central Processing Unit) 101, a RAM (Random access memory) 102, a ROM (Read only memory) 103, an input unit 104, and an output unit 105. Be prepared.

The CPU 101 controls the information processing device 1 by executing a program loaded in the RAM 102. The CPU 101 functions as a calculation unit 11. Further, the CPU 101 functions as a calculation target selection unit 12 and a calculation instruction unit 13 by executing the control program 43.

The RAM 102 is the main memory of the information processing device 1. The ROM 103 stores the BIOS (Basic Input / Output System) of the information processing device 1.

The input unit 104 acquires the program and the like stored in the front image 61 and the storage device 2, and supplies the acquired data to the RAM 102. The output unit 105 supplies the calculation result of the CPU 101 to the external device connected to the information processing device 1. The calculation result is, for example, result data 62 showing a pedestrian detection result.

{1.3. Configuration of neural network 20}
FIG. 3 is a schematic view showing the configuration of the neural network 20 shown in FIG. With reference to FIG. 3, the neural network 20 includes an input layer 21, convolution layers 22, 23A, 24A and 24B, pooling layers 23B and 24C, fully connected layers 25 and 26, and an output layer 27. In FIG. 3, the display of the nodes included in the neural network 20 is omitted.

In the following description, the convolution layers 22, 23A, 24A and 24B, the pooling layers 23B and 24C, and the fully connected layers 25 and 26 may be collectively referred to as "intermediate layer 20A".

The input layer 21 receives the pixel data of the preprocessed front image 61, and supplies the received pixel data of the front image 61 to the convolution layer 22.

The convolution layer 22 performs a convolution calculation on the pixel data received from the input layer 21, and supplies the result of the convolution calculation to each of the convolution layers 23A and 24A.

The convolution layer 23A further performs a convolution calculation based on the result of the convolution calculation received from the convolution layer 22. The convolution layer 23A supplies the result of the convolution calculation to the pooling layer 23B. The pooling layer 23B statistically processes the result of the convolution operation received from the convolution layer 23A, and supplies the statistical processing result to the fully connected layer 25.

The convolution layer 24A further performs a convolution calculation based on the result of the convolution calculation received from the convolution layer 22. The convolution layer 24A supplies the result of the convolution calculation to the convolution layer 24B. The convolution layer 24B further performs a convolution calculation based on the result of the convolution calculation received from the convolution layer 24A. The convolution layer 24B supplies the result of the convolution calculation to the pooling layer 24C. The pooling layer 24C statistically processes the result of the convolution operation received from the convolution layer 24B, and supplies the statistical processing result to the fully connected layer 25.

The fully connected layer 25 receives statistical processing results from each of the pooling layers 23B and 24C, and performs an operation using the received statistical processing results. The fully connected layer 25 supplies the calculation result to the fully connected layer 26. The fully connected layer 26 further calculates the calculation result received from the fully connected layer 25, and supplies the calculation result to the output layer 27.

The output layer 27 outputs a pedestrian detection result 28 indicating whether or not a pedestrian is detected from the front image 61 based on the calculation result received from the coupling layer 27. The pedestrian detection result 28 is used to generate the result data 62.

{1.4. Dependencies in Neural Network 20}
In the neural network 20, each of the intermediate layer 20A and the output layer 27 is a separate program. That is, the calculation unit 11 can individually calculate each of the intermediate layer 20A and the output layer 27. In order to start the calculation of one intermediate layer included in the intermediate layer 20A, it is necessary that the calculation of all the intermediate layers having a dependency relationship with the one intermediate layer is completed.

For example, the convolution layer 24B is dependent on the convolution layers 22 and 24A. Therefore, the calculation of the convolution layer 24B cannot be started unless the calculation of the convolution layers 22 and 24A is completed.

The dependency relationship in the neural network 20 will be described in detail. The intermediate layer 20A is dependent on the intermediate layer located upstream. “Upstream” indicates the direction in which the input layer 21 is viewed from one intermediate layer included in the neural network 20. Downstream refers to the direction in which the output layer 27 is viewed from one intermediate layer.

For example, since the convolution layers 22 and 23A are located upstream of the pooling layer 23B, the pooling layer 23B depends on the convolution layers 22 and 23A. Since the convolution layers 24A and 24B and the pooling layer 24C are not located upstream of the pooling layer 23B, the pooling layer 23B does not depend on the convolution layers 24A and 24B and the pooling layer 23C.

The two intermediate layers in parallel positional relationship do not depend on each other. Specifically, each of the convolution layer 23A and the pooling layer 23B does not depend on the convolution layers 24A and 24B and the pooling layer 24C. On the contrary, the convolution layers 24A and 24B and the pooling layer 24C do not depend on the convolution layer 23A and the pooling layer 23B.

{1.5. Computation data 52}
FIG. 4 is a diagram showing an example of the calculation amount data 52 shown in FIG. With reference to FIG. 4, the calculation amount data 52 records the calculation amount of each of the intermediate layer 20A and the output layer 27 included in the neural network 20.

The calculation of the calculation amount recorded in the calculation amount data 52 will be described. FIG. 5 is a diagram illustrating a method of calculating the calculation amount of the convolution layer included in the neural network 20.

When the image data 66 is input to the convolution layer with reference to FIG. 5, the image data 66 is a two-dimensional matrix, and the pixel values correspond to the elements of the matrix. In this case, the horizontal size and the vertical size of the input data 66 are Ir and Ic.

The horizontal and vertical sizes of the filter 67 used in the convolution layer are Fr and Fc. There may be at least one filter 67. The number of filters 67 is Fn.

In the calculation in the convolution layer, padding data is added to the outer circumference of the input data. This is to filter the input data 66. The padding data are all 0. The upper, lower, right and left side sizes of the padding data are Pt, Pb, Pl and Pr. As a result, the calculated amount N of the convolution layer is calculated by the following equation (1).

In formula (1), Sh is the horizontal stride of the filter 67 and Sv is the vertical stride of the filter 67.

When the intermediate layer is a fully connected layer, the calculation amount of the fully connected layer is set by setting Fr, Fc, Sh and Sv of the equation (1) to 1 and setting Pt, Pb, Pl and Pr to 0. It is calculated.

The amount of calculation in the pooling layer is tightened according to the content of statistical processing executed in the pooling layer.

The calculation amount of the input layer 21 is not recorded in the calculation amount data 52. The pixel data of the preprocessed front image 61 does not substantially change in the input layer 21. This is because the input layer 21 simply passes the pixel data of the preprocessed front image 61 to the convolution layer 22, and does not perform the calculation.

{2. motion}
FIG. 6 is a diagram showing an example of time variation of the general load 55 among the loads of the CPU 101 shown in FIG. With reference to FIG. 6, the general load 55 is a numerical value obtained by subtracting the load excluding the load generated by the calculation of the neural network 20 from the load of the CPU 101. The general load 55 includes a load generated by executing each of the preprocessing program 41, the result output program 42, and the control program 43.

Hereinafter, the process in which the information processing apparatus 1 selects the intermediate layer to be calculated based on the load of the CPU 101 will be described with reference to FIG. In FIG. 6, in order to show the magnitude relationship between the margin load, the expected load, and the total expected load in an easy-to-understand manner, the margin load, the expected load, and the total expected load are shown with reference to the upper limit of the load. The margin load, the expected load, and the total expected load will be described later.

(Time t11)
The preprocessing of the front image 61 input from the camera is completed at time t11. The information processing device 1 starts the pedestrian detection process using the neural network 20 at time t11. At time t11, the calculation target selection unit 12 selects the convolution layer 22 as the calculation target from the plurality of intermediate layers 20A.

Specifically, the calculation target selection unit 12 determines that the calculationable intermediate layer 20A is the convolution layer 22 at time t11. This is because the preprocessed front image 61 is input to the convolution layer 22 via the input layer 21.

The calculation target selection unit 12 acquires the calculation amount of the convolution layer 22 that can be calculated from the calculation amount data 52. The calculation target selection unit 12 calculates the load of the CPU 101 generated by the execution of the convolution layer 22 as the expected load 221 of the convolution layer 22 based on the acquired calculation amount. The load of the CPU 101 is calculated based on the number of clocks, the number of cores, and the like of the CPU 101, and a well-known method can be used.

The calculation target selection unit 12 calculates the margin load 551 at time t11 based on the general load 55 at time t11 and the preset upper limit load. As shown in FIG. 6, the upper limit load is 90% in this embodiment.

The calculation target selection unit 12 compares the calculated expected load 221 with the margin load 5511 at time t11. When the calculated expected load 221 is less than or equal to the margin load 551 at time t11, the calculation target selection unit 12 selects the convolution layer 22 as the calculation target.

The calculation instruction unit 13 instructs the calculation unit 11 to start the calculation of the convolution layer 22 selected by the calculation target selection unit 12. The calculation unit 11 starts the calculation of the convolution layer 22 in response to the instruction of the calculation instruction unit 13.

When the expected load 221 of the convolution layer 22 is larger than the margin load 551 at time t11 at time t11, the calculation target selection unit 12 waits until a predetermined time elapses from time t11. The calculation target selection unit 12 compares the expected load 221 of the convolution layer 22 with the margin load at the time when a predetermined time has elapsed.

(Time t12)
With reference to FIG. 6, the calculation unit 11 completes the calculation of the convolution layer 22 at time t12. At time t12, the calculation target selection unit 12 newly selects the middle layer to be calculated from the intermediate layer 20A.

Since the calculation of the convolution layer 22 is completed, the calculation target selection unit 12 specifies the convolution layers 23A and 24A connected to the convolution layer 22 as operable intermediate layers. This is because the convolution layers 23A and 24A are located downstream of the convolution layer 22 for which the calculation has been completed, and are directly connected to the convolution layer 22.

The calculation target selection unit 12 calculates the expected load 231A of the convolution layer 23A based on the calculation amount of the convolution layer 23A recorded in the calculation amount data 52. The calculation target selection unit 12 calculates the expected load 241A of the convolution layer 24A based on the calculation amount of the convolution layer 24A recorded in the calculation amount data 52. The calculation target selection unit 12 calculates the total of the calculated expected loads 231A and 241A as the total expected load 562.

Although FIG. 6 shows an example in which the expected loads 231A and 241A are the same, the expected loads 231A and 241B may be different from each other.

The calculation target selection unit 12 acquires the general load 55 of the CPU 101 at the time t12, and calculates the margin load 552 at the time t12 based on the acquired general load 55.

The calculation target selection unit 12 compares the calculated total expected load 562 with the margin load 532 at time t12. In the example shown in FIG. 6, since the total expected load 562 is larger than the margin load 552 at time t12, the calculation target selection unit 12 determines that both the convolution layers 23A and 24A cannot be calculated in parallel.

In this case, the calculation target selection unit 12 compares each of the expected loads 231A and 241A with the margin load 552. Since each of the expected loads 231A and 241A has a margin load of 552 or less, the calculation target selection unit 12 selects either one of the convolution layers 23A and 24A as the calculation target.

Specifically, the calculation target selection unit 12 selects a calculation target based on the number of intermediate layers located downstream of each of the convolution layers 23A and 24A. In the example shown in FIG. 3, since the number of intermediate layers located downstream of the convolution layer 24A is larger than that of the intermediate layer located downstream of the convolution layer 23A, the calculation target selection unit 12 selects the convolution layer 24A as the calculation target. .. This is because the calculation end time of the neural network 20 can be advanced by giving priority to the calculation of the convolution layer 24A having a large number of downstream intermediate layers. The calculation instruction unit 13 instructs the calculation unit 11 to perform the calculation of the convolution layer 24A selected by the calculation target selection unit 12. As a result, the calculation of the convolution layer 24A is started from time t12. Alternatively, the calculation target selection unit 12 may select which of the convolution layers 23A and 24A has the larger calculation amount.

That is, when a plurality of intermediate layers can be calculated and the total expected load of the plurality of intermediate layers is larger than the margin load, the calculation target selection unit 12 calculates the calculation target based on the priority of the plurality of intermediate layers. The middle layer of is selected.

(Time t13)
The calculation unit 11 completes the calculation of the convolution layer 24A at time t13. At time t13, the calculation target selection unit 12 newly selects the calculation target intermediate layer from the intermediate layer 20A.

Specifically, the calculation target selection unit 12 specifies the convolution layers 23A and 24B as intermediate layers that can be calculated when the convolution layer 24A is completed. The calculation target selection unit 12 calculates the expected load 231A of the convolution layer 23A based on the calculation amount of the convolution layer 23A recorded in the calculation amount data 52. The calculation target selection unit 12 calculates the expected load 241B of the convolution layer 24B based on the calculation amount of the convolution layer 24B recorded in the calculation amount data 52. The calculation target selection unit 12 calculates the total of the calculated expected load 231A and the estimated load 241B as the total expected load 563.

Although FIG. 6 shows an example in which the expected loads 231A and 241B are the same, the expected loads 231A and 241B may be different from each other.

The calculation target selection unit 12 acquires the general load 55 of the CPU 101 at the time t13, and calculates the margin load 553 at the time t13 based on the acquired general load 55.

The calculation target selection unit 12 compares the calculated total expected load 563 with the margin load 533 at time t13. In the example shown in FIG. 6, the total expected load 563 is equal to or less than the margin load 533 at time t12. The calculation target selection unit 12 determines that both the convolution layers 23A and 24B can be calculated in parallel, and selects the convolution layers 23A and 24B as calculation targets.

The calculation instruction unit 13 instructs the calculation unit 11 to calculate the convolution layers 23A and 24A selected by the calculation target selection unit 12. As a result, the operations of the convolution layers 23A and 24B are started at time t13.

(After time t14)
At time t14, the operations of both the convolution layers 23A and 24B are completed. After that, the information processing apparatus 1 selects the pooling layer 23B as the calculation target at the time t14 and selects the pooling layer 24B at the time t15 by executing the same processing as described above. A detailed description of the selection of the calculation target at the times t14 and t15 will be omitted.

At time t15, the calculation of the pooling layer 23B is completed. However, the calculation target selection unit 12 cannot specify the fully connected layer 25 as an intermediate layer that can be calculated. This is because the operations of both the pooling layers 23B and 24C, which are dependent on the fully connected layer 25, have not been completed.

Therefore, the information processing apparatus 1 starts the calculation of the pooling layer 24C on condition that the expected load 231B of the pooling layer 23B is equal to or less than the expected load at the time t15 at the time t15.

When the information processing device 1 finishes the calculation of the fully connected layer 26, the information processing device 1 starts the calculation of the output layer 27. Even if the information processing device 1 compares the expected load of the output layer 27 with the expected load at the completion of the calculation of the fully coupled layer 26, and determines whether or not to start the calculation of the output layer 27 based on the comparison result. Good.

(flowchart)
FIG. 7 is a flowchart showing the operation of the information processing apparatus 1 shown in FIG. With reference to FIG. 7, the information processing apparatus 1 shows that when the front image 61 is input to the input layer 21, or when the calculation of the intermediate layer selected by the calculation target selection unit 12 is completed, FIG. 7 shows. The processing shown is started.

The calculation target selection unit 12 acquires the general load 55 at the time when the process shown in FIG. 7 is started (step S11). The calculation target selection unit 12 calculates the margin load based on the general load 55 acquired in step S11 and the preset upper limit load (step S12).

The calculation target selection unit 12 specifies an intermediate layer that can be calculated among the intermediate layers 20A based on the structure of the neural network 20 (step S13). Specifically, when the calculation of one intermediate layer is completed, the calculation target selection unit 12 determines that the intermediate layer connected to the downstream of the one intermediate layer can be calculated. When the front image 61 is input to the input layer 21, the calculation target selection unit 12 determines that the convolution layer 22 connected to the input layer 21 can be calculated.

The calculation target selection unit 12 calculates the expected load of the intermediate layer specified in step S13 (step S14). After step S14, the calculation target selection unit 12 determines whether or not the number of intermediate layers specified in step S13 is 2 or more (step S15).

When the number of intermediate layers specified in step S13 is 1 (No in step S15), the calculation target selection unit 12 compares the expected load calculated in step S14 with the margin load calculated in step S12 (step S20). ..

When the expected load is equal to or less than the margin load (Yes in step S20), the calculation target selection unit 12 selects the intermediate layer specified in step S13 as the calculation target (step S18). When the expected load is larger than the margin load (No in step S20), the calculation target selection unit 12 waits until a predetermined time elapses (step S21), and returns to step S11.

Returning to the description of step S15. When the number of intermediate layers specified in step S13 is 2 or more (Yes in step S15), the calculation target selection unit 12 calculates the total of the expected loads calculated in step S14 as the total expected load (step S16). The calculation target selection unit 12 compares the total expected load calculated in step S16 with the margin load calculated in step S12 (step S17).

When the total expected load is equal to or less than the margin load (Yes in step S17), the calculation target selection unit 12 selects all the intermediate layers specified in step S13 as calculation targets (step S18). When the total expected load is larger than the margin load (No in step S17), the calculation target selection unit 12 selects the calculation target intermediate layer based on the priority of each of the intermediate layers specified in step S12 (No). Step S19).

As described above, the information processing apparatus 1 according to the present embodiment is based on the structure of the neural network 20, the amount of each calculation of the intermediate layer 20A, and the load of the calculation unit 11, and is the intermediate layer to be calculated. Select. As a result, the information processing apparatus 1 can execute the processing using the neural network and the other processing in parallel without having to provide a CPU having a high arithmetic processing capacity.

Further, when the information processing apparatus 1 specifies a plurality of intermediate layers that can be calculated, the information processing apparatus 1 acquires the expected load of each of the plurality of intermediate layers, and does the sum of the acquired expected load and the general load 55 exceed the margin load? Judge whether or not. When the sum of the acquired expected load and the general load 55 does not exceed the margin load, the information processing apparatus 1 selects the specified plurality of intermediate layers as calculation targets. As a result, the information processing apparatus 1 can execute the operations of the plurality of intermediate layers in parallel with the other processes, so that the calculation completion of the neural network can be accelerated.

[Second Embodiment]
[1. Constitution]
[1.1. Configuration of information processing device 1A]
FIG. 8 is a functional block diagram showing the configuration of the information processing system 100A according to the second embodiment of the present invention. With reference to FIG. 8, the information processing system 100A includes an information processing device 1A instead of the information processing device 1. The information processing device 1A detects not only pedestrians but also traffic signs from the front image 61.

The information processing device 1A includes a calculation unit 11A and a calculation target selection unit 12A in place of the calculation unit 11 and the calculation target selection unit 12. The calculation unit 11A performs calculations on the neural networks 20 and 30 stored in the storage device 2. The calculation target selection unit 12A selects the calculation target intermediate layer from the intermediate layers of the neural networks 20 and 30.

The storage device 2 further stores the neural network 30 and the calculation amount data 53. The neural network 30 is used to detect traffic signs from the forward image 61. The calculation amount data 53 records the calculation amount of each of the intermediate layers included in the neural network 30.

Hereinafter, the present embodiment will be described with a focus on points different from the above-described embodiment. The description of the configuration and operation common to the above-described embodiment and the present embodiment will be omitted.

[1.2. Configuration of neural network 30]
FIG. 9 is a schematic view showing the configuration of the neural network 30 shown in FIG. With reference to FIG. 9, the neural network 30 includes an input layer 31, convolution layers 32 and 33, a pooling layer 34, fully coupled layers 35 and 36, and an output layer 37. In the neural network 30 shown in FIG. 9, the display of nodes is omitted.

In the following description, the convolutional layers 32 and 33, the pooling layer 34, and the fully connected layers 35 and 36 may be collectively referred to as "intermediate layer 30A".

The input layer 31 receives the pixel data of the preprocessed front image 61, and supplies the received pixel data of the front image 61 to the convolution layer 32.

The convolution layer 32 performs a convolution calculation on the pixel data received from the input layer 31, and supplies the result of the convolution calculation to the convolution layer 33. The convolution layer 33 further performs a convolution calculation based on the result of the convolution calculation received from the convolution layer 32. The convolution layer 33 supplies the result of the convolution operation to the pooling layer 34.

The pooling layer 34 statistically processes the result of the convolution operation received from the convolution layer 33, and supplies the statistical processing result to the fully connected layer 35.

The fully connected layer 35 receives a statistical processing result from the pooling layer 34, and performs an operation using the received statistical processing result. The fully connected layer 35 supplies the calculation result to the fully connected layer 36. The fully connected layer 36 further calculates the calculation result received from the fully connected layer 35, and supplies the calculation result to the output layer 37.

The output layer 37 outputs a sign detection result 38 indicating a traffic sign detected from the front image 61 based on the calculation result received from the coupling layer 36. The label detection result 38 is used to generate the result data 62.

[2. motion]
When the preprocessing of the front image 61 is completed, the information processing device 1A detects pedestrians and traffic signs from the preprocessed front image 61 by using the neural networks 20 and 30.

FIG. 10 is a graph showing an example of a time change of the general load 57 of the CPU 101 mounted on the information processing apparatus 1A shown in FIG. The general load 57 corresponds to the load of the CPU 101 excluding the load associated with the operations of the neural networks 20 and 30.

(Time t21)
With reference to FIG. 10, the preprocessing of the front image 61 is completed at time t21. The calculation target selection unit 12A specifies the convolution layers 22 and 32 as intermediate layers that can be calculated in the neural networks 20 and 30. This is because the neural networks 20 and 30 are independent of each other.

The calculation target selection unit 12A calculates the expected load 221 of the convolution layer 22 and the expected load 321 of the convolution layer 32. The expected load 321 is calculated based on the calculated amount of the convolution layer 32 recorded in the calculated amount data 53. The calculation target selection unit 12A calculates the sum of the calculated expected loads 221 and 321 as the expected total load 621.

The calculation target selection unit 12A acquires the general load 57 at time t22, and calculates the margin load 571 based on the acquired general load 57 and the load upper limit.

Since the expected total load 621 is equal to or less than the margin load 571, the calculation target selection unit 12A selects both the convolution layers 22 and 32 as calculation targets. The calculation instruction unit 13 instructs the calculation unit 11 to perform calculations on the convolution layers 22 and 32 selected by the calculation target selection unit 12A. As a result, the operations of the convolution layers 22 and 32 are started at time t22.

(Time t22)
At time t22, the operations of the convolution layers 22 and 32 are completed. The convolution layers 23A and 24A are located downstream of the convolution layer 22 and are connected to the convolution layer 22. The convolution layer 33 is located downstream of the convolution layer 32 and is connected to the convolution layer 32. Therefore, the calculation target selection unit 12A selects the convolution layers 23A, 24A, and 33 as the intermediate layers that can be calculated.

The calculation target selection unit 12A calculates the expected load 231A of the convolution layer 23A, the expected load 241A of the convolution layer 24A, and the expected load 331 of the convolution layer 33. The calculation target selection unit 12A calculates the total of the calculated expected loads 231A, 241A, and 331A as the total expected load 622.

The calculation target selection unit 12A calculates the margin load 572 at the time t22 based on the upper limit load and the general load 57 at the time t22.

Since the total expected load 622 is larger than the margin load 572, the calculation target selection unit 12A has a margin of the expected load 241A of the convolution layer 24A having the largest number of downstream intermediate layers among the three intermediate layers specified at time t22. Compare with load 572. Since the expected load 241A is smaller than the margin load 572, the calculation target selection unit 12A selects the convolution layer 24A as the calculation target. The calculation instruction unit 13 instructs the calculation unit 11A to perform the calculation of the convolution layer 24A selected by the calculation target selection unit 12A. As a result, the calculation of the convolution layer 24A starts at time t22.

Alternatively, the calculation target selection unit 12A may calculate the total of the expected effects of two of the three intermediate layers specified at time t22, and compare the calculated total with the margin load 572. When the calculated total is smaller than the margin load 572, these two intermediate layers are selected as calculation targets. When there are a plurality of combinations of two intermediate layers that can be selected as calculation targets, the calculation target selection unit 12A has the largest total calculation amount of the two intermediate layers, or is located downstream of the two intermediate layers. The combination with the largest total number of layers to be used may be selected.

(After time t22)
Even after the time t22, the calculation target selection unit 12A identifies the calculationable intermediate layer from each of the neural networks 20 and 30 each time the calculation of the intermediate layer is completed, and the expected load and the margin load of the specified intermediate layer. Based on and, the intermediate layer to be calculated is selected.

As described above, the information processing apparatus 1A identifies an intermediate layer that can be calculated from each of the neural networks 20 and 30, and acquires the expected load of each of the specified intermediate layers. The information processing apparatus 1A determines whether or not the sum of the acquired expected load and the general load 57 exceeds the margin load. When the sum of the acquired expected load and the general load 57 does not exceed the margin load, the information processing apparatus 1 selects all the specified intermediate layers as calculation targets. As a result, the information processing apparatus 1 can execute the operations of the plurality of neural networks and other processes in parallel.

[Modification example]
(Mutual conversion between neural network and block diagram)
FIG. 11 is a diagram showing an example of converting a neural network into a block diagram. Neural networks are weighted directed graphs. In a general block diagram, when it is regarded as a weight of a block value, the general block diagram can be considered as a weighted directed graph. That is, as shown in FIG. 11, the neural network can be converted into a block diagram.

In the transformation shown in FIG. 11, since no information is lost, the conversion from the neural network to the block diagram is reversible. That is, it is possible to convert the block diagram into a neural network.

FIG. 12 is a diagram showing an example of a neural network. FIG. 13 is a diagram showing the neural network shown in FIG. 12 as an adjacency matrix. With reference to FIG. 12, x is an input node, y is an intermediate node, and z is an output node. w is the weight and b is the bias. c is a constant having 1 as a value. When the neural network shown in FIG. 12 is regarded as a weighted directed graph, the neural network shown in FIG. 12 can be expressed as an adjacency matrix shown in FIG.

When creating a determinant of a weighted directed graph from the adjacency matrix shown in FIG. 13, the determinant is represented by the following equations (2) and (3).

In equations (1) and (2), x, y, z, w and b are represented by vectors such as X, Y, Z, W and B, respectively, and c = 1 is substituted. As a result, the equations (1) and (2) can be modified as the equations (3) and (4).

FIG. 14 is a block diagram obtained by converting the neural network shown in FIG. FIG. 14 corresponds to a diagram in which the equations (3) and (4) are represented by a block diagram.

The block diagram has a function of abstracting the control structure by layering. Normally, control designers are not interested in the internal structure of AI (artificial intelligence) models such as neural networks. Therefore, the neural network can be hidden by treating the AI model as a block having a complicated function. AI developers can proceed with development independently of control by focusing only on hidden neural networks.

Furthermore, by abstracting the structure of the neural network with layers such as an input layer and a convolution layer, the layers below it can be simplified into a combination of simple numerical operations. The lower layer is used by microprocessor mounting engineers to devise parallelization and memory allocation to increase speed. From this, converting a neural network into a block diagram has the effect of separating the development process by role and clarifying the guarantee range of quality and performance, not only for the implementation of an in-vehicle microcomputer. You can expect it.

(Selection of parallelization target)
It is assumed that the neural network designed and learned using the GPU (Graphics Processing Unit) server cannot structurally reduce the number of operations any more. In this case, the problem when implementing the neural network in the microcomputer can be narrowed down to how to improve the CPI (Clocks Per Instruction). Various methods such as low-latency instructions and memory access are known as methods for improving CPI. However, it is parallelization that contributes most to the improvement of CPI. Therefore, when developing a library, it is necessary to consider how much the operation dependency can be eliminated.

Which part of the operation has the dependency can be clearly understood by considering the structure of the block diagram before generating the code as a directed graph. By converting the neural network into a block diagram, parallelization can be examined graphically.

FIG. 15 is another example of a block diagram obtained by transforming a neural network. In the block diagram shown in FIG. 15, since the blocks connected by the arrows have a clear dependency relationship, the target of parallelization is selected for each branch.

In FIG. 15, the horizontal direction shows the time and the vertical direction shows the number of parallels. Therefore, the block diagram shown in FIG. 15 shows the processing with the maximum number of parallels and the shortest time at the present time. If the number of parallel targets is sufficient, the block diagram shown in FIG. 15 can be implemented as it is. If the number of parallel targets is not enough, you should consider the allocation. Therefore, the divided areas are assigned in order from the output node as a directed graph. As a result, the block diagram shown in FIG. 15 is graphed as shown in FIG.

FIG. 17 is a diagram showing a procedure of allocating parallel processing based on the processing dependency shown in FIG. When a dual-core CPU is the development target, the number of parallels is 2. For the sake of clarity, it is assumed that all the processes a to k shown in FIG. 16 have the same processing time. In this case, a box of the number of parallels × time series may be prepared and the processes may be arranged so that the time series is as short as possible. The algorithm is as follows.

Step 1: Arrange in the time series so that each node is one before the other nodes that are directly referenced.
Step 2: Find the "dependency", which is the total number of nodes that refer to each node.
Step 3: Place the empty box on the far right from the node with the highest dependency. However, the node is arranged so as to be located to the left of the position arranged in step 1.

(Other variants)
In the above embodiment, the information processing apparatus 1 may execute the neural network 20 with different process IDs (IDentification). For example, the image recognition process for detecting a pedestrian from the front image 61 generated at time t (k) and the image recognition process for detecting a pedestrian from the front image 61 generated at time t (k + 1) are mutually exclusive. being independent. The time t (k + 1) is a time after the time t (k).

In this case, the calculation target selection unit 12 assigns different process IDs to the image recognition process corresponding to the time t (k) and the image recognition process corresponding to the time t (k + 1). In this case, the calculation target selection unit 12 is derived from each of the neural network 20 used in the image recognition process corresponding to the time t (k) and the neural network 20 used in the image recognition process corresponding to the time t (k + 1). It suffices to specify the intermediate layer that can be calculated. The operation of the calculation target selection unit 12 after specifying the calculation target intermediate layer is the same as described above.

In the above embodiment, an example in which the calculation target selection unit 12 compares the expected load of the intermediate layer that can be calculated with the margin load has been described, but the present invention is not limited to this. The calculation target selection units 12 and 12A select the calculation target intermediate layer if the calculation target intermediate layer can be specified based on the structure of the neural network, the calculation amount of the intermediate layer, and the load of the calculation units 11 and 11A. The procedure to be performed is not particularly limited.

The execution order of the processing methods in the above-described embodiment is not limited to the description of the above-described embodiment, and the execution order may be changed without departing from the gist of the invention.

A computer program that causes a computer to perform the above-mentioned method and a computer-readable recording medium on which the program is recorded are included in the scope of the present invention. Here, examples of computer-readable recording media include flexible disks, hard disks, CD-ROMs, MOs, DVDs, DVD-ROMs, DVD-RAMs, large-capacity DVDs, next-generation DVDs, and semiconductor memories. ..

Although the embodiments of the present invention have been described above, the above-described embodiments are merely examples for carrying out the present invention. Therefore, the present invention is not limited to the above-described embodiment, and the above-described embodiment can be appropriately modified and implemented within a range that does not deviate from the gist thereof.

100,100A Information processing system 1,1A Information processing device 11, 11A Calculation unit 12, 12A Calculation target selection unit 13 Calculation instruction unit 21, 31 Input layer 22, 23A, 24A, 24B, 32, 33 Folding layer 23B, 24C, 34 Pooling layer 25,26,35,36 Fully connected layer 27,37 Output layer

Claims (5)

An arithmetic unit that executes an operation of a neural network including a plurality of intermediate layers and an operation of a predetermined program different from the neural network.
Calculation to select an intermediate layer to be calculated from the plurality of intermediate layers based on the structure of the neural network, the calculation amount data indicating the calculation amount in each of the plurality of intermediate layers, and the load of the calculation unit. Target selection section and
An information processing device including a calculation instruction unit that instructs the calculation unit to perform an operation on an intermediate layer selected by the calculation target selection unit. The information processing device according to claim 1.
In the calculation target selection unit, the sum of the load of the calculation unit generated by the calculation of the neural network and the load of the calculation unit generated by the calculation of the predetermined program is set in advance as the load of the calculation unit. An information processing device that selects the intermediate layer to be calculated so as not to exceed the upper limit of. The information processing device according to claim 1 or 2.
The plurality of intermediate layers
The first middle layer and
The second intermediate layer, which is independent of the calculation result of the first intermediate layer, is included.
The calculation target selection unit includes the load of the calculation unit generated by the calculation of the first intermediate layer, the load of the calculation unit generated by the calculation of the second intermediate layer, and the calculation generated by the calculation of the predetermined program. An information processing apparatus that selects the first intermediate layer and the second intermediate layer as the intermediate layer to be calculated when the total with the load of the unit does not exceed the upper limit of the load of the calculation unit. The information processing device according to any one of claims 1 to 3.
The calculation unit calculates the first neural network including the first intermediate layer and the second neural network including the second intermediate layer.
The calculation target selection unit includes the load of the calculation unit generated by the calculation of the first intermediate layer, the load of the calculation unit generated by the calculation of the second intermediate layer, and the calculation generated by the calculation of the predetermined program. An information processing apparatus that specifies the first intermediate layer and the second intermediate layer as the intermediate layer to be calculated when the total with the load of the unit does not exceed the upper limit of the load of the calculation unit. It is a control method of an information processing apparatus including an arithmetic unit that executes an operation of a neural network including a plurality of intermediate layers and an operation of a program different from the neural network.
A step of selecting an intermediate layer to be calculated from the plurality of intermediate layers based on the structure of the neural network, the calculation amount data indicating the calculation amount in each of the plurality of intermediate layers, and the load of the calculation unit. When,
A control method of an information processing apparatus including a step of instructing the calculation unit to perform a calculation of the selected intermediate layer.

Images