JP2019008421A - Processing method, program, information processing apparatus, and image processing apparatus - Google Patents

Abstract

To provide a processing method for performing convolution operation efficiently by using a processor capable of processing SIMD instructions.SOLUTION: In an information processing apparatus comprising a plurality of registers usable for SIMD and a processor for performing SIMD processing, a size of the register to be used is selected from a kernel size, a division amount for the kernel is determined based on a number of registers of the selected size and the kernel size, and is stored in the register for each kernel divided by the determined division amount for the kernel, and convolution operation is processed in parallel.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a processing method, a program, an information processing apparatus, and an image processing apparatus.

  Conventionally, in the field of image recognition for recognizing an object from acquired images, recognition accuracy has been improved. This is largely due to a multi-layered convolutional neural network (hereinafter referred to as “CNN”).

  In the CNN process, a plurality of convolution layers are included, and the convolution layer performs a convolution operation. In the convolution operation, since a plurality of product-sum operations are repeatedly performed, there is a problem that a lot of processing is required and it takes time.

  For example, in the technique disclosed in Patent Document 1, a convolution operation is performed using a graphics processor (Graphics Processing Unit: hereinafter referred to as “GPU”). When the image data to be processed is simply expanded in the GPU memory during the convolution operation, there is a problem that the required memory size increases and the cost increases. In order to solve this problem, in the technique disclosed in Patent Document 1, image data is divided into a plurality of data blocks, a plurality of data blocks divided into a local memory, and a plurality of filters are simultaneously read to perform a convolution operation. Calculate in parallel.

Japanese Patent Laying-Open No. 2006-4572

  However, the technique disclosed in Patent Document 1 is based on a GPU. On the other hand, for example, in the case of assuming image recognition processing by an embedded system mounted on a device such as a monitoring camera, if a GPU is mounted on the embedded system, power consumption is large and not realistic. As a result, a CPU with low power consumption must be used.

  The present invention has been made in view of the above circumstances, and an efficient convolution operation using an information processing apparatus including a processor with low power consumption, particularly a processor capable of processing a single instruction multiple data (SIMD) instruction. It aims at providing the processing method which performs.

  The above object of the present invention is achieved by the following means.

(1) a plurality of registers available for SIMD;
A method of controlling an information processing apparatus comprising a processor that performs SIMD processing,
(A) selecting a size of the register to be used from a kernel size;
(B) determining a kernel division amount from the number of registers of the selected size and the kernel size;
(C) storing in the register for each kernel divided by the determined kernel division amount, and performing parallel processing of convolution operations;
Including a processing method.

(2) When the number of kernel rows stored in the register at a time is n,
In the step (b), the processing method according to (1), wherein the n is set to a maximum value satisfying the following formula.
k− (x + j) ≧ (b + 1) n
Where j is j = (w + 1−y) / b,
b = w / q,
j and b are integers rounded up to the nearest decimal point;
w: kernel size k: number of available registers q: number of data stored in one register x: number of registers necessary for convolution operation y: number of pixels to be skipped.

(3) When the kernel size is 11 × 11,
In the step (b), the kernel is determined to be divided into two lines;
The processing method according to (1), wherein in step (c), the kernel is stored in the register every two lines, and then the convolution operation is executed every two lines.

(4) When the kernel size is 11 × 11,
In step (a), a 128-bit register is selected,
In the step (c), the processing method according to (3), wherein four input pixels are stored in a 128-bit register, and convolution operations for each four pixels are processed in parallel.

(5) When the kernel size is 5 × 5 or less,
In step (b), it is decided not to divide the kernel;
In the step (c), the kernel is stored in the register at a time, and a convolution operation is executed after the kernel is stored.

(6) When the kernel size is 5 × 5 or less,
In step (a), a 64-bit register is selected,
The processing method according to (5), wherein in step (c), two input pixels are stored in the 64-bit register, and convolution operations for each two pixels are processed in parallel.

  (7) Among the plurality of registers, a register for storing the kernel is fixed, and the result data stored in the intermediate buffer is updated while sliding the data stored in the register for the input pixel. To the processing method according to any one of (6) above.

(8) a plurality of registers available for SIMD;
A program for controlling an information processing device comprising a processor for SIMD processing,
(A) selecting a size of the register to be used from a kernel size;
(B) determining a kernel division amount from the number of registers of the selected size and the kernel size;
(C) storing in the register for each kernel divided by the determined kernel division amount, and performing parallel processing of convolution operations;
A program for performing the method.

(9) When n is the number of kernel rows stored in the register at a time,
In the step (b), the program according to (8), wherein the n is set to a maximum value satisfying the following formula.
k− (x + j) ≧ (b + 1) n
Where j is j = (w + 1−y) / b,
b = w / q,
j and b are integers rounded up to the nearest decimal point;
w: kernel size k: number of available registers q: number of data stored in one register x: number of registers necessary for convolution operation y: number of pixels to be skipped.

(10) When the kernel size is 11 × 11,
In the step (b), the kernel is determined to be divided into two lines;
The program according to (8), wherein in step (c), the kernel is stored in the register every two lines, and after storing, the convolution operation is executed every two lines.

(11) When the kernel size is 11 × 11,
In step (a), a 128-bit register is selected,
The program according to (10), wherein in step (c), four input pixels are stored in a 128-bit register, and convolution operations for each four pixels are processed in parallel.

(12) When the kernel size is 5 × 5 or less,
In step (b), it is decided not to divide the kernel;
The program according to (8), wherein in step (c), the kernel is stored in the register at a time, and a convolution operation is executed after the kernel is stored.

(13) When the kernel size is 5 × 5 or less,
In step (a), a 64-bit register is selected,
The program according to (12), wherein in the step (c), two input pixels are stored in the 64-bit register, and convolution operations for each two pixels are processed in parallel.

  (14) Of the plurality of registers, a register for storing the kernel is fixed, and the result data stored in the intermediate buffer is updated while sliding the data stored in the register for the input pixel, (8) To any one of (13) above.

(15) a plurality of registers available for SIMD;
An information processing apparatus including a processor for performing SIMD processing,
The size of the register to be used is selected from the kernel size, the number of registers of the selected size is determined, the division amount of the kernel is determined from the kernel size, and the register is determined for each kernel divided by the determined division amount of the kernel. An information processing device that stores data in parallel and performs convolution operations in parallel.

(16) When the number of kernel rows that are divided by the determined division amount and stored in the register at a time is n,
The information processing apparatus according to (15), wherein the n is set to a maximum value satisfying the following formula.
k− (x + j) ≧ (b + 1) n
Where j is j = (w + 1−y) / b,
b = w / q,
j and b are integers rounded up to the nearest decimal point;
w: kernel size k: number of available registers q: number of data stored in one register x: number of registers necessary for convolution operation y: number of pixels to be skipped.

  (17) Of the plurality of registers, a register for storing the kernel is fixed, and the result data stored in the intermediate buffer is updated while sliding the data stored in the register for input pixels. Or the information processing apparatus as described in said (16).

(18) an image acquisition unit that acquires an image generated by the imaging device;
The information processing according to any one of (15) to (17), wherein a feature of a person shown in the image and a peripheral feature indicating a shape, a position, or a type of a peripheral object of the person shown in the image are extracted. Equipment,
An image processing apparatus comprising:

  According to the processing method of the present invention, the size of a register to be used is selected from the kernel size, the kernel division amount is determined from the number of registers of the selected size and the kernel size, and the division is performed by the determined kernel division amount. Each kernel is stored in a register, and convolution operations are processed in parallel. By doing so, the convolution operation can be performed efficiently.

It is a block diagram which shows the information processing apparatus which concerns on embodiment of this invention. It is a schematic diagram which shows the state which stored the kernel of 5 * 5 size in the 64-bit register and the 128-bit register, respectively. It is a schematic diagram which shows the state which stored the kernel of 11 * 11 size in the 128-bit register. It is a schematic diagram explaining a process in case the division | segmentation amount of an 11x11 size kernel is 2 rows. It is a figure which shows the overlapping state of the window in the case of stride 4 in a CNN process. It is a figure which shows the flowchart of the CNN process which an information processing apparatus performs. It is a figure which shows the flowchart following FIG. 6A. It is a schematic diagram which shows the data storage state of a register. It is a schematic diagram which shows the data storage state of a register. It is a figure which shows the flowchart which concerns on a modification. 1 is a block diagram illustrating an image processing apparatus according to an embodiment. It is a figure which shows the functional block of an image processing apparatus.

  Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the description of the drawings, the same elements are denoted by the same reference numerals, and redundant description is omitted. In addition, the dimensional ratios in the drawings are exaggerated for convenience of explanation, and may be different from the actual ratios.

  FIG. 1 is a block diagram showing an information processing apparatus according to an embodiment of the present invention. As shown in the figure, the information processing apparatus 100 includes a SIMD processor 10, a general-purpose processor 20, and a memory 30, and these elements are connected to each other by a bus such as a data bus, a command bus, and an address bus. The information processing apparatus 100 is an information processing apparatus for an embedded system such as a monitoring camera.

  The SIMD processor 10 is a processor that executes a SIMD type (single instruction multiple data processing) instruction that calculates a plurality of data with a single instruction, and is, for example, a NEON that corresponds to a SIMD extension instruction of ARM Holdings. SIMD type processing (hereinafter simply referred to as “SIMD processing”) is also referred to as vector data processing.

  The SIMD processor 10 includes a SIMD register file 11, an ALU (Arithmetic Logic Unit) 12, a MUL (Multiplier Unit) 13, a shifter 14, and an LS (Load / Store Unit) 15.

  The SIMD register file 11 can be composed of a plurality of SIMD registers (hereinafter simply referred to as “registers”) having a length of 256 bits, 128 bits, or 64 bits. In this embodiment, the SIMD register file 111 can be used as 16 128-bit registers or 32 64-bit registers (hereinafter referred to as “128-bit registers” and “64-bit registers”, respectively). Say). If it is a 32-bit single precision floating point number or integer, four operands can be stored in a 128-bit register, and if it is a 64-bit register, two operands can be stored at one time. Such data is also called vector data.

  The ALU 12 is an arithmetic unit that performs addition, subtraction, and logical operations. The MUL 13 is an arithmetic unit that performs multiplication and division. The shifter 14 is an arithmetic unit that performs a shift process for shifting the bits to the left and right. The LS 15 performs processing of loading data from the memory 30 to the register and loading data from the register to the memory 30.

  The general-purpose processor 20 is a processor that executes general scalar data processing other than SIMD processing. The general-purpose processor 20 includes an ALU 22, MUL 23, and LS 24. Since these have the same functions as the ALU 12, MUL 13, and LS 15, description thereof will be omitted.

  The memory 30 includes a RAM such as a DRAM (Dynamic Random Access Memory) and a ROM such as a NAND flash memory. Further, an auxiliary storage device such as an HDD may be included. The memory 30 temporarily stores image data to be processed, a program for executing a CNN process executed by the information processing apparatus 100 described later, a control program for controlling a device in which the information processing apparatus 100 is incorporated, Store data etc.

(Kernel storage)
In this embodiment, when CNN processing is performed, due to hardware resource limitations as shown in FIG. 1, all of them are converted to the registers of the SIMD register file 111 at a time depending on the size of the kernel (also referred to as a filer). If you try to store it, you will run out of registers. Therefore, in this embodiment, a part of the kernel is divided and stored in a 128-bit register or a 64-bit register obtained by dividing the SIMD register file 111. Note that the size of the kernel to be applied varies depending on the process.

  FIG. 2 is a schematic diagram showing a state in which kernels having a size of 5 rows in height and 5 columns in width (hereinafter referred to as “5 × 5 size”) are stored in 64-bit registers and 128-bit registers, respectively. The kernel is stored in one or a plurality of registers for each row in order to perform SIMD processing. A 5 × 5 size kernel is composed of 25 data, and when each data size is 32 bits of a single-precision floating point number, one continuous 128-bit register can store four data.

  If five rows of data are stored in a 128-bit register, two 128-bit registers are required as shown in the figure, and only one data is stored in the second 128-bit register. Yes, the remaining three spaces are available. On the other hand, when stored in a 64-bit register, each 64-bit register can store data of two kernels, so three 64-bit registers are required. The third 64-bit register One space is generated.

  In comparison between the two, it can be seen that, in the case of a kernel of 5 × 5 size, selecting a 64-bit register has higher storage efficiency and can effectively use the register.

  Similarly, in the case of an 11 × 11 size kernel, even if either a 64-bit register or a 128-bit register is selected, one empty space is generated in the last register. FIG. 3 is a schematic diagram showing a state in which an 11 × 11 size kernel is stored in a 128-bit register. The third register (k02) has one space. In this case, the storage efficiency is the same in the 64-bit register and the 128-bit register. However, considering SIMD processing, the 128-bit register stores more data than the 64-bit register, so more data can be stored in one cycle (one load or store instruction) at a time. Since transfer is possible, it can be said that transfer efficiency is high.

  Here, in the present embodiment, the number of registers that can be allocated to the storage of the kernel is limited due to size restrictions in the SIMD register file. For example, in the case of a 128-bit register, the number of usable registers is 16. In order to store an 11 × 11 size kernel in a 128-bit register at a time, 33 (= 3 × 11) registers are required, and 16 registers are insufficient. Therefore, in the method according to the present embodiment, which will be described later, the kernel division amount is determined according to the condition such as the kernel size, the kernel is divided into a plurality of divisions with the determined division amount, and a convolution operation is performed for each divided kernel . This enables efficient CNN processing even with limited resources such as an embedded system (see FIG. 4).

(About common use of calculation results)
In this embodiment, the stride in the CNN process (AlexNet) is 4. FIG. 5 is a diagram showing an overlapping portion of the input pixel range before and after the stride for four pixels is executed. As shown in FIG. 5, when performing a convolution operation on an input pixel of i × i size using a kernel of 11 × 11 size, as shown in FIG. The input pixel is summed with the data at the corresponding position in the kernel to calculate the data of one output pixel. A thick frame is a window indicating a range of input pixels for product-sum operation with the kernel.

  Then, the next output pixel is calculated by multiplying the input pixel skipped by four pixels as shown in FIG. 5 (b) with the kernel. At this time, since the previous window and the current window partially overlap, the input pixels stored in the register can be used in common. This amount of overlap differs depending on the skip setting value and the kernel size. Here, in the following example, the number of pixels for striding the kernel is the same as the number of pixels shifted for the convolution operation (the number of stored data q described later). Moreover, it is preferable that the number of pixels to be stride is the same as or an integer multiple of the number of pixels shifted for the convolution operation. In the method according to the present embodiment described below, the input pixels in the common usable area are stored in a register and used for the convolution operation of a plurality of output pixels, thereby improving the processing efficiency.

(Processing method)
Hereinafter, the procedure of the CNN process performed by the information processing apparatus 100 will be described with reference to FIGS. 6A to 8. 6A and 6B are diagrams illustrating a flowchart of CNN processing executed by the information processing apparatus 100. FIG.

(S111)
First, the information processing apparatus 100 selects a register size from the kernel size. As described above, the register size is first selected as follows: (1) From the viewpoint of storage efficiency, the one with higher storage efficiency is selected. If there is no difference in storage efficiency, then (2) the larger register size is selected from the viewpoint of transfer efficiency. For example, if the kernel size is 5 × 5, a 64-bit register is selected from the viewpoint of storage efficiency. If the kernel size is 11 × 11, a 128-bit register is selected from the viewpoint of storage efficiency and transfer efficiency.

(Step S112)
Next, the division amount of the kernel is determined using the number of used registers and the kernel size of the register size selected in step S111.

The division amount of the kernel, that is, the division row number n is set to the maximum number (integer) satisfying the following formula (1).
k− (x + j) ≧ (b + 1) n (1)
Where j is j = (w + 1−y) / b,
b = w / q
j and b are integers rounded up to the nearest decimal point;
w: Kernel size k: Number of registers q: Number of data stored in one register x: Number of registers required for convolution calculation y: Number of pixels to be skipped.

(About formula (1))
Use k z-bit registers selected based on kernel size w × w.

  If the minimum number of bits is 32 bits of a single floating point type, z / 32 = q pieces of data are stored in the z-bit register.

  X registers are necessary for the accumulation and addition necessary for the convolution operation. The number of registers necessary for the integration and addition operations will be described later.

  With the kernel size w × w, it is necessary to calculate w / q = b times (b is an integer obtained by rounding up the decimal point) using a z-bit register in one line.

  Further, when convolution is performed by skipping input pixels by y pixels, (w + 1−y) pixels can be used in common.

Therefore, considering that q data is stored in the z-bit register, it is necessary to prepare (w + 1−y) / b = j intermediate buffers. (J is an integer with the decimal point rounded up)
Therefore, the number of remaining registers is k− (x + j) = m. When the kernel is divided into n rows, the number of registers necessary for the kernel is bn, and the number of registers necessary for input pixel data is n.
Therefore, the maximum integer n that satisfies the condition of m ≧ (b + 1) n is obtained.

(Specific example where n = 2 in this embodiment)
For example, when a 128-bit register is selected according to the kernel size of 11 × 11 (w = 11) in step S111, the number of registers k that can be assigned to SIMD processing is 16. In one 128-bit register, the number of data q to be stored is four 32-bit single precision floating point numbers.

  At least three (x) registers are required for integration and addition necessary for the convolution operation (described later). Four 32-bit data (= 128/32) are stored in the 128-bit register.

  When the kernel size is 11 × 11, it is necessary to perform calculation using a 128-bit register in one line 11/4 = 2.7 times and rounding up the decimal point three times (b = 3).

  In addition, when convolution is performed by skipping four input pixels, since eight pixels (w + 1−y = 11 + 1−4) can be used in common, three intermediate buffers may be secured (j = 8 / b). = 2.6, j = 3 by rounding up after the decimal point. Here, the added “1” is an adjustment value for adjusting to the number of data (b: 4) stored at one time in the register.

When the remaining registers are 16- (4 + 3) = 9 kernels divided into n rows, 16- (4 + 3) ≧ (b + 1) n = 4n
When the maximum integer n satisfying 9 ≧ 4n is obtained, n = 2.
In summary, of the 16 (k) 128 (z) bit registers, the ones that are used are determined to be 6 (3n) for kernel storage, 2 (n) for input pixels, and the intermediate buffer 3 (j) for use and 4 (x) for integration and addition, for a total of 15.

(When x registers (three) are required for accumulation and addition of convolution operations)
When calculating (a + b) × (c + d) = e, a and b are stored in the registers R0 and R1, respectively. The result x of a + b is stored in the register R0. Thereafter, c and d are stored in the registers R1 and R2, respectively. The result y of c + d is stored in the register R1. The result e of x * y is stored in the register R0. From the above, three registers R1 to R2 are necessary.

(Other example: When x (4) registers are required for accumulation and addition of convolution operations)
When calculating (a + b) × (c + d) = e, a and b are stored in the registers R0 and R1, respectively. The result x of a + b is stored in the register R2. Thereafter, c and d are stored in the registers R0 and R1, respectively. The result y of c + d is stored in the register R3. The result e of x * y is stored in the register R3. From the above, four registers R1 to R3 are necessary. As described above, the necessary number is merely an example, and the number of registers necessary for integration and addition varies depending on the number of variables and the availability of registers.

(Step S113)
The kernel for n rows determined by the above procedure is stored in the register. Note that the following processing will be described assuming that n = 2 is determined under the conditions of the above-described embodiment.

  7 and 8 are schematic diagrams showing the data storage state of the register. FIG. 7A shows a state in which the kernels for two rows are divided and stored in bn (= 6) registers in step S113. As shown in FIG. 3, each of the two rows of data constituting the kernel is stored in a register every four consecutive data. In the figure, each register is shown with symbols k00 to k12 corresponding to the stored data. Among these, the third registers k02 and k12 store the kernel data for three of the four storage areas, and have a free area for one data (see FIG. 3). Zero is put in this empty area.

(Step S114)
Next, zero padding is performed to enter a numerical value 0 at the right and bottom edges of the input pixel. The padding width is 1, for example. This padding width is appropriately set according to the required size of the output pixel.

(Step S115)
From the upper end and the left end, 2 × 4 input pixels corresponding to n (= 2) determined in step S112 and the number of data that can be stored in one register (4) are stored in n (= 2) registers. Store. FIG. 7B shows a state in which 2 × 4 input pixels are stored in two registers in step S115. Also in the same figure, each register shows codes d00 and d10 corresponding to the stored data.

(Step S116)
Multiply-and-accumulate the data (d00, d10) stored in the input pixel register and the register data (k00, k10) that stores the left 4 data kernels, and store the result S in the intermediate buffer 1 register. To do. In the product-sum operation here (the same applies below), convolution of 4 pixels is performed using SIMD processing for 4 pixels (d00, k00) stored in the input and kernel registers respectively. Parallel processing operations. FIG. 7C shows a state where the product-sum operation is performed in step S116 and the result S is stored in the register.

(Step S121)
Next, the next 2 × 4 input pixels after skipping 4 pixels (the number of stored data q) are stored in the input pixel register. Here, for example, data of d01 and d11 (see FIG. 7B) are respectively stored in two registers for input pixels.

(Step S122)
A product-sum operation is performed on the data stored in the input pixel register and the data in the register storing the left four data kernels, and the result A is stored in the intermediate buffer 2 register. FIG. 8A shows a state in which the product-sum operation is performed in step S122 and the result A is stored in the register.

(Step S123)
The sum of the data stored in the input pixel register and the data in the register storing the kernel for the central four data are added, and the result B is added to the intermediate buffer 1 register and stored. FIG. 8B shows a state where the product-sum operation is performed in step S123 and the result B is stored in the register.

(Step S131)
Next, 2 × 4 input pixels after skipping 4 pixels are stored in the input pixel register. Here, for example, data of d02 and d12 (see FIG. 7B) are stored in two registers for input pixels, respectively.

(Step S132)
A product-sum operation is performed on the data stored in the input pixel register and the data in the register storing the left four data kernels, and the result C is stored in the intermediate buffer 3 register. FIG. 8C shows a state where the product-sum operation is performed in step S132 and the result C is stored in the register.

(Step S133)
The sum of the data stored in the input pixel register and the data in the register storing the kernel for the central four data are added, and the result D is added to the intermediate buffer 2 register and stored. FIG. 8D shows a state where the product-sum operation is performed in step S133 and the result D is stored in the register.

(Step S134)
The data stored in the input pixel register and the data in the register storing the right four data kernels are summed, and the result E is added to the intermediate buffer 1 register and stored. FIG. 8E shows a state where the product-sum operation is performed in step S133 and the result E is stored in the register. At this time, the input pixel register stores four pieces of data, but the fourth data, that is, the twelfth data outside the window (see FIG. 5) is the fourth empty space in the kernel. Since the data (zero value) is multiplied and added, the result of the product-sum operation is not affected.

(Step S141)
In the processing so far, the data of the product-sum operation result for two rows of one output pixel is stored in the register for the intermediate buffer 1, so the stored data is transferred to a predetermined address position in the memory 30. The transfer process to the memory 30 is performed when the data for the output two pixels is prepared by executing the processes after step S131 via step S143 described later if the number of registers is sufficient. You may make it perform. As a result, processing efficiency can be improved.

(Step S142)
Since the role of the register for the intermediate buffer 1 is finished, the buffer data is sequentially slid and updated in the following procedure. First, the data in the intermediate buffer 1 register is updated with the data stored in the intermediate buffer 2 register. Next, the data in the intermediate buffer 2 register is updated with the data stored in the intermediate buffer 3 register. Finally, the intermediate buffer 3 register is updated with zero.

(Step S143)
If the process has not been completed up to the last column of input pixels (after zero padding) (NO), the process returns to step S131, and the subsequent processes are executed with the next input pixel in the window skipped by 4 pixels in step S131. . On the other hand, if the process has been completed up to the last column (YES), the process proceeds to step S144.

(Step S144)
If the process has not been completed up to the last line of the kernel, that is, if the process up to the 11th line has not been completed (NO), the process proceeds to step S145. On the other hand, if the processing up to the last row has been completed (YES), the calculation of output pixels for one pixel is completed (END). Thereafter, stride in the row direction and the calculation of all output pixels is repeated.

(S145)
The next two rows of kernels are stored (updated) in six kernel registers. For example, if the first and second lines are completed, the kernel data of the third and fourth lines are stored in the register. Since the data is stored in the register for every two rows, only the data for the last 11th row is stored in the register at the sixth time. In this case, the sum of products may be calculated by putting all zeros in three unused registers for one row.

(Step S146)
The 2 × 4 input pixels from the left end of the next two rows are stored in the input pixel register. Thereafter, the process proceeds to step S116, and the subsequent processing is executed.

  According to the method according to the present embodiment described above, the amount of power consumption as in an embedded system is limited by determining the kernel division amount, dividing the kernel by the determined division amount, and performing the calculation. Even under limited hardware resource conditions, the CNN process can be performed efficiently. In particular, it is possible to share the operation result by fixing the register for storing the kernel and updating the data in the intermediate buffer register while shifting (see FIG. 5). As a result, the CNN process can be performed more efficiently.

(Other examples)
The configurations of the information processing apparatus and the processing method described above are the main configurations for describing the features of the above-described embodiment, and are not limited to the above-described configurations, and various modifications can be made within the scope of the claims. be able to. Further, the processing executed by the configuration and processing method included in a general information processing apparatus is not excluded.

(Modification 1)
In the above-described embodiment, even when a 5 × 5 size kernel is used, the division amount is determined and the kernel is divided and stored in the register based on the determined division amount. Depending on the situation, it may be stored in the register at one time without being divided. FIG. 9 is a diagram illustrating a flowchart according to a modification.

(Step S211)
Here, the kernel size used for the CNN process is determined. If the kernel size is 5 × 5 or less, the process proceeds to step S212. On the other hand, if it exceeds 5 × 5, the process proceeds to step S111 in FIG. 6A.

(Step S212)
Store all kernel data in registers. For example, in the case of a 5 × 5 size kernel, all data of the kernel is stored in a total of 15 registers, 3 per line, using a 64-bit register. The data in these registers is fixedly used until the processing is completed.

(Step S213)
In the following, the data in the 5 × 5 size window of the input pixel is sequentially stored in the register according to a known procedure, and the convolution operation is performed by performing the product-sum operation with the data of the register storing the kernel. Each convolution operation is processed in parallel.

(Step S214)
As soon as the convolution calculation for one pixel is completed, it is transferred to the memory and the process is terminated. Thereafter, the calculation of all output pixels is repeated.

  In this way, in the modified example, when the kernel size is small, the number of times data is loaded into the register and the number of transfer processing of the store is smaller when the data is stored in the register at one time without being divided. CNN processing can be performed efficiently.

(Incorporation into image processing equipment)
The information processing apparatus 100 according to the present embodiment may be applied to an image processing apparatus 60 such as a monitoring camera. FIG. 10 is a block diagram showing a configuration of an action recognition system that recognizes a person's action in a captured video. As shown in the figure, the action recognition system includes an imaging device 50 and an image processing device 60 according to the present embodiment.

  The imaging device 50 is a general camera or a wide-angle camera, and generates image data by performing AD conversion on an image signal generated by an imaging element of the camera. The imaging device 50 can capture a moving image obtained by continuously generating image data in units of frames.

  The image processing device 60 includes an image acquisition unit 70 and the information processing device 100 described above. The image acquisition unit 70 acquires image data D1 of a moving image generated by the imaging device 50.

  FIG. 11 is a diagram mainly illustrating functional blocks of the information processing apparatus. The information processing apparatus 100 includes a human region detection unit 110, a human body feature extraction unit 120, a peripheral feature extraction unit 130, a peripheral feature filter unit 140, a behavior determination unit 150, and a learning unit 160.

  The human region detection unit 110 detects a partitioned human region including a person from the image of the image data D1. The method of detecting the human region by the human region detection unit 110 is arbitrary. For example, a difference image in the image is detected from the moving image, and the human region is detected from the difference image. The human region detection unit 110 may also use a learned neural network, template matching, a combination of HOG (Histograms of Oriented Gradients) features and SVM (Support Vector Machine), or a background difference method. Good.

  The human body feature extraction unit 120 acquires the image data D1 and the data D2 indicating the human region from the human region detection unit 110, performs predetermined arithmetic processing on the human region image, and the posture feature of the person shown in the image To extract. Then, the human body feature extraction unit 120 outputs the human posture feature data D3 shown in the extracted image to the peripheral feature filter unit 140 and the action determination unit 150.

  The human body feature extraction unit 120 extracts a human posture feature from an image using, for example, a learned CNN process. Note that the CNN process that constitutes the human body feature extraction unit 120 is learned by, for example, teacher data indicating the correspondence between the human body image and the coordinates (two-dimensional position or three-dimensional estimated position) of the joint position of the human body in the image. What has been processed is used (generally also referred to as R-CNN).

  The human body feature extraction unit 120 includes, for example, a preprocessing unit, a convolution processing unit, and first and second full coupling units. Based on the data D2 indicating the human area, the preprocessing unit normalizes the image by cutting out the human area image from the entire area image and converting it into a predetermined size and aspect ratio.

  The convolution processing unit is configured by hierarchically connecting a plurality of feature quantity extraction layers. The convolution processing unit performs the above-described convolution operation processing, activation processing, and pooling processing on the input data input from the previous layer in each feature amount extraction layer.

  For example, the first fully coupled portion is configured by a multilayer perceptron that fully couples a plurality of feature amounts. The first full combination unit fully combines a plurality of intermediate calculation result data obtained from the convolution processing unit, and generates data D3 indicating a human posture characteristic. Then, the first all combination unit outputs data D3 indicating the posture characteristics of the person to the second all combination unit and the peripheral feature filter unit 140.

  The peripheral feature extraction unit 130 acquires the image data D1 and the data D2 indicating the human region from the human region detection unit 110, performs predetermined arithmetic processing on the image around the human region, and displays the human image in the image. Extract peripheral features of surrounding objects. Then, the peripheral feature extraction unit 130 outputs the extracted peripheral feature data D4 to the peripheral feature filter unit 140.

  The peripheral feature extraction unit 130 extracts peripheral features from the image using CNN processing, for example, similarly to the human body feature extraction unit 120. In addition, the CNN process which comprises the surrounding feature extraction part 130 is what performed the learning process by the teacher data which shows the correspondence of the image of an object, the shape of the said object, the classification, the position of each site | part, etc., for example. Used. More preferably, an image obtained by performing learning processing using an image around a human body including a human body and teacher data indicating a correspondence relationship between the shape of the object and the coordinates of the positional relationship in the image is used.

  The peripheral feature filter unit 140 acquires posture feature data D3 from the human region detection unit 110 and peripheral feature data D4 from the peripheral feature extraction unit 130. Then, the peripheral feature filter unit 140 filters the peripheral feature based on the importance level data Da of the peripheral feature set in association with the posture feature. Then, the peripheral feature filter unit 140 outputs the filtered peripheral feature data D4a to the behavior determination unit 150.

  The behavior determination unit 150 acquires the human posture feature data D3 from the human body feature extraction unit 120 and the filtered peripheral feature data D4a from the peripheral feature filter unit 140. Then, the action determination unit 150 determines the action class of the person shown in the image based on the time series data of the human posture feature data D3 and the filtered peripheral feature data D4a. Further, the behavior determination unit 150 according to the present embodiment may perform time series analysis using a hierarchical LSTM (Long Short-Term Memory) which is a kind of recursive neural network. The hierarchical LSTM can recognize a relationship in a long time interval (for example, one minute before) in addition to a relationship in a short time interval (for example, the immediately preceding image frame).

  The learning unit 160 performs machine learning using teacher data so that the human body feature extraction unit 120, the peripheral feature extraction unit 130, the peripheral feature filter unit 140, and the behavior determination unit 150 can execute the processing described above.

  The learning unit 160 uses, for example, the convolution processing unit of the human body feature extraction unit 120, the first and the second, using teacher data in which a normalized human region image and a human posture characteristic (for example, a joint position) are associated with each other. 2. Adjust network parameters (for example, weighting factor, bias) of all joints.

  In addition, the learning unit 160 performs, for example, convolution processing of the peripheral feature extraction unit 130 using the teacher data in which the normalized image of the object around the person and the feature of the object (for example, the positional relationship with the human body) are associated. And network parameters (for example, weighting factor, bias) of all the coupling parts of the peripheral feature extraction unit 130 are adjusted.

  In addition, the learning unit 160 uses, for example, teacher data in which the posture characteristics of the person and the importance of the peripheral object are associated with each other, and the network parameters (for example, weighting factors and biases) of all the connection units of the peripheral feature filter unit 140. adjust.

  Further, the learning unit 160 uses, for example, time series data of human posture features and peripheral features and teacher data in which a correct behavior class is associated with the intermediate layer 61 and the behavior determination unit 150 of the behavior determination unit 150. The network parameters (for example, weighting factor, bias) of all the coupling parts are adjusted.

  Note that the learning unit 160 may perform these learning processes using, for example, a known error back propagation method. Then, the learning unit 160 stores the network parameter adjusted by the learning process in an external storage unit or the like.

  As described above, according to the image processing device 60 according to the present embodiment, the importance level of the peripheral feature is set in association with the posture feature of the human body, and the peripheral feature is filtered, so that it relates to the human action. Only surrounding objects can be extracted. As a result, the image processing apparatus 60 according to the present embodiment can estimate a human action class with high accuracy even in an environment in which the types, positions, and appearances of surrounding objects are different.

  In particular, the image processing apparatus 60 incorporating the information processing apparatus 100 according to the present embodiment extracts temporal changes in the positional relationship of the peripheral objects related to the posture of the human body based on the time series data of the posture features and the peripheral features. In addition, since the human behavior class is estimated, the human behavior class can be estimated with higher accuracy.

  The program for operating the information processing apparatus described above may be provided by a computer-readable recording medium such as a USB memory, a flexible disk, or a CD-ROM, or may be provided online via a network such as the Internet. . In this case, the program code recorded on the computer-readable recording medium may be written in assembly language or machine language.

100 Information processing apparatus 10 SIMD processor 11 SIMD register file 12 ALU
13 MUL
14 Shifter 15 LS
20 General-purpose processor 21 Scalar register file 22 ALU
23 MUL
24 LS
30 memory 60 image processing apparatus

Claims (18)

Multiple registers available for SIMD;
A method of controlling an information processing apparatus comprising a processor that performs SIMD processing,
(A) selecting a size of the register to be used from a kernel size;
(B) determining a kernel division amount from the number of registers of the selected size and the kernel size;
(C) storing in the register for each kernel divided by the determined kernel division amount, and performing parallel processing of convolution operations;
Including a processing method. When the number of kernel rows stored in the register at a time is n,
The processing method according to claim 1, wherein in step (b), n is set to a maximum value satisfying the following formula.
k− (x + j) ≧ (b + 1) n
Where j is j = (w + 1−y) / b,
b = w / q,
j and b are integers rounded up to the nearest decimal point;
w: kernel size k: number of available registers q: number of data stored in one register x: number of registers necessary for convolution operation y: number of pixels to be skipped. When the kernel size is 11 × 11,
In the step (b), the kernel is determined to be divided into two lines;
2. The processing method according to claim 1, wherein, in the step (c), the kernel is stored in the register every two lines, and then a convolution operation is executed every two lines. When the kernel size is 11 × 11,
In step (a), a 128-bit register is selected,
The processing method according to claim 3, wherein in step (c), four input pixels are stored in a 128-bit register, and convolution operations of four pixels are performed in parallel. When the kernel size is 5 × 5 or less,
In step (b), it is decided not to divide the kernel;
5. The processing method according to claim 1, wherein, in the step (c), the kernel is stored in the register at a time, and a convolution operation is executed after the kernel is stored. When the kernel size is 5 × 5 or less,
In step (a), a 64-bit register is selected,
6. The processing method according to claim 5, wherein in step (c), two input pixels are stored in the 64-bit register, and a convolution operation for each two pixels is processed in parallel.   The register for storing the kernel among the plurality of registers is fixed, and the result data stored in the intermediate buffer is updated while sliding the data stored in the register for the input pixel. The processing method as described in any one of. Multiple registers available for SIMD;
A program for controlling an information processing device comprising a processor for SIMD processing,
(A) selecting a size of the register to be used from a kernel size;
(B) determining a kernel division amount from the number of registers of the selected size and the kernel size;
(C) storing in the register for each kernel divided by the determined kernel division amount, and performing parallel processing of convolution operations;
A program for performing the method. When the number of kernel rows stored in the register at a time is n,
The program according to claim 8, wherein in the step (b), the n is set to a maximum value satisfying the following formula.
k− (x + j) ≧ (b + 1) n
Where j is j = (w + 1−y) / b,
b = w / q,
j and b are integers rounded up to the nearest decimal point;
w: kernel size k: number of available registers q: number of data stored in one register x: number of registers necessary for convolution operation y: number of pixels to be skipped. When the kernel size is 11 × 11,
In the step (b), the kernel is determined to be divided into two lines;
The program according to claim 8, wherein in step (c), the kernel is stored in the register every two lines, and after the storage, a convolution operation is executed every two lines. When the kernel size is 11 × 11,
In step (a), a 128-bit register is selected,
The program according to claim 10, wherein in step (c), four input pixels are stored in a 128-bit register, and convolution operations of four pixels are performed in parallel. When the kernel size is 5 × 5 or less,
In step (b), it is decided not to divide the kernel;
The program according to any one of claims 8, 10 and 11, wherein in the step (c), the kernel is stored in the register at a time, and a convolution operation is executed after the kernel is stored. When the kernel size is 5 × 5 or less,
In step (a), a 64-bit register is selected,
13. The program according to claim 12, wherein in step (c), two input pixels are stored in the 64-bit register, and a convolution operation for each two pixels is processed in parallel.   The register for storing the kernel among the plurality of registers is fixed, and the result data stored in the intermediate buffer is updated while sliding the data stored in the register for the input pixel. The program as described in any one of. Multiple registers available for SIMD;
An information processing apparatus including a processor for performing SIMD processing,
The size of the register to be used is selected from the kernel size, the number of registers of the selected size is determined, the division amount of the kernel is determined from the kernel size, and the register is determined for each kernel divided by the determined division amount of the kernel. An information processing device that stores data in parallel and performs convolution operations in parallel. When the number of kernel rows that are divided by the determined division amount and stored in the register at a time is n,
The information processing apparatus according to claim 15, wherein the n is set to a maximum value satisfying the following formula.
k− (x + j) ≧ (b + 1) n
Where j is j = (w + 1−y) / b,
b = w / q,
j and b are integers rounded up to the nearest decimal point;
w: kernel size k: number of available registers q: number of data stored in one register x: number of registers necessary for convolution operation y: number of pixels to be skipped.   The register for storing a kernel among the plurality of registers is fixed, and the result data stored in the intermediate buffer is updated while sliding the data stored in the register for the input pixel. The information processing apparatus described in 1. An image acquisition unit for acquiring an image generated by the imaging device;
The information processing apparatus according to any one of claims 15 to 17, wherein a feature of a person shown in the image and a peripheral feature indicating a shape, a position, or a type of a peripheral object of the person shown in the image are extracted. ,
An image processing apparatus comprising: