JP6800656B2 - Arithmetic circuit, its control method and program - Google Patents

Description

The present invention relates to an arithmetic circuit used for pattern recognition and the like, a control method thereof, and a program.

The neural network method is widely applied to image processing devices such as pattern recognition devices. Among neural networks, a calculation method called Convolutional Neural Networks (hereinafter abbreviated as CNN) is attracting attention as a method that enables robust pattern recognition with respect to fluctuations in the recognition target. For example, Patent Document 1 proposes an example applied to face recognition using image data.

FIG. 3 is a network configuration diagram showing an example of a simple CNN process. The input layer 301 corresponds to raster-scanned image data of a predetermined size when CNN processing is performed on the image data. The feature planes 303a to 303c indicate the feature planes of the first layer 308. The feature surface is a data surface corresponding to the processing result of a predetermined feature extraction operation (convolution operation and non-linear processing). Since the feature surface corresponds to the feature extraction result for recognizing a predetermined object in the upper hierarchy and is the processing result for the raster-scanned image data, the processing result is also represented by the surface. In CNN, data groups constituting a large number of feature planes are hierarchically related via arithmetic processing.

The feature planes 303a to 303c are generated by the convolution operation and the non-linear processing corresponding to the input layer 301. For example, the feature plane 303a is generated by a two-dimensional convolution operation schematically shown in the filter kernel 3021a and a non-linear transformation of the operation result. For example, a convolution operation in which the size of the filter kernel (filter coefficient matrix) is volumeSize × lowSize is processed by a product-sum operation as shown in the following equation.

ここで、「ｉｎｐｕｔ（ｘ，ｙ）」は座標（ｘ、ｙ）での参照画素値を示し、「ｏｕｔｐｕｔ（ｘ，ｙ）」は座標（ｘ、ｙ）での演算結果を示す。また、「ｗｅｉｇｈｔ（ｃｏｌｕｍｎ，ｒｏｗ）」は座標（ｘ＋ｃｏｌｕｍｎ、ｙ＋ｒｏｗ）での重み係数を示し、「ｃｏｌｕｍｎＳｉｚｅ」及び「ｒｏｗＳｉｚｅ」はカーネルサイズを示す。

Here, "input (x, y)" indicates a reference pixel value in coordinates (x, y), and "output (x, y)" indicates a calculation result in coordinates (x, y). Further, "weight (column, low)" indicates a weighting coefficient in coordinates (x + volume, y + low), and "columnSize" and "lowSize" indicate a kernel size.

In the CNN process, the product-sum operation is repeated while scanning a plurality of filter kernels on a pixel-by-pixel basis, and the final product-sum result is non-linearly converted to generate a feature plane. Since the feature surface 303a is calculated from one image data in the previous layer, the number of connections is 1. There is only one kernel 3021a for calculating the feature plane 303a. Further, kernel 3021b and kernel 3021c are filter kernels used when calculating feature planes 303b and 303c, respectively. Hereinafter, the filter kernel may be abbreviated as a filter or a kernel. The size of the filter kernel may be abbreviated as kernel size or filter size.

FIG. 4 is an example of calculating the characteristic surface 305a in the CNN treatment. The feature surface 305a is calculated from the three feature surfaces 303a to c of the previous layer 308 and is coupled to the feature surfaces 303a to c. When calculating the data of the feature surface 305a, first, a convolution operation using the kernel 3041a schematically shown is performed on the feature surface 303a, and the result is held in the cumulative adder 401. Similarly, the kernels 3042a and 3043a are subjected to convolution operations on the feature surface 303b and the feature surface 303c, respectively, and the results are cumulatively added to the cumulative adder 401.

After the convolution operation using the three types of kernels is completed, the non-linear conversion process 402 using the logistic function and the hyperbolic tangent function (tanh function) is performed. The feature surface 305a is generated by performing the above processing while scanning the entire image pixel by pixel. Similar to the process of FIG. 4, the feature surface 305b is calculated for each of the three feature surfaces of the previous layer 308 by using the convolution operations of kernel 3041b, kernel 3042b, and kernel 3043b. Further, the feature surface 307 is calculated for each of the feature surfaces 305a to 305b of the previous layer 309 by using the convolution calculation of the kernel 3061 and the kernel 3062.

It is assumed that the coefficients of each kernel are determined in advance by learning using general methods such as perceptron learning and backpropagation learning. For example, in pattern recognition or the like, a convolution operation may be performed using a kernel having a large size of 10 × 10 or more.

As described above, since the CNN process repeats a large number of kernel convolution operations, a huge number of product-sum operations are required.

For the purpose of speeding up the convolution operation, for example, in Patent Document 2, a weight coefficient common to a plurality of product-sum operation units is set, and the convolution operation is executed at high speed by performing the operation in parallel while shifting the input data. A device has been proposed.

Further, Patent Document 3 discloses a method of speeding up the entire processing by limiting the face candidate area for performing the face detection processing by using the skin color information.

Further, Patent Document 4 proposes a method of controlling the number of arithmetic units operating in parallel in a parallel arithmetic unit that executes error correction processing.

Japanese Patent Application Laid-Open No. 10-021406 JP 2010-134697 JP-A-2005-242582 WO00 / 079405

However, in Patent Document 1, when the convolution operation is processed in parallel, for example, the data transfer for supplying the data of the previous layer to be the reference data to the arithmetic unit may become a bottleneck depending on the size of the kernel. Further, even when a parallel computing device as disclosed in Patent Document 2 is realized in combination with a low-speed memory, performance commensurate with the number of computing devices (degree of parallelism) operating in parallel may not be exhibited. In addition, there are cases where the performance commensurate with the degree of parallelism of the arithmetic circuit cannot be exhibited, but here, for example, when data transfer becomes a bottleneck, data commensurate with the amount of data that can be processed by the simultaneous operation of the parallel arithmetic units is not transferred. Therefore, the power consumption of the arithmetic unit is wasted. This becomes a particular problem when the data transfer when performing the convolution operation becomes a bottleneck depending on the size of the kernel used for CNN processing.

Further, by combining the parallel arithmetic processing apparatus as disclosed in Patent Document 2 and the processing area limiting processing disclosed in Patent Document 3, CNN processing can be realized at high speed. However, although the size of the processing target area is changed by the method of Patent Document 3, in the parallel computing device as disclosed in Patent Document 2, the degree of parallelism is uniform regardless of the processing target area. Since the calculation is executed in, it may be wasted from the viewpoint of power consumption.

The present invention has been made in view of the above problems, and when data transfer becomes a bottleneck, the number of multipliers to be executed is appropriately controlled among a plurality of multipliers that can be executed in parallel. It is an object of the present invention to provide an arithmetic circuit for reducing power consumption. Another object of the present invention is to provide a control method and a program of the arithmetic circuit.

In order to solve the above problems, the arithmetic circuit according to the present invention has the following configuration. An arithmetic circuit connected to a holding device that holds the reference data of the filter arithmetic processing and the coefficient data of the filter used for the filter arithmetic processing, and repeatedly multiplys the reference data different from each other and the common coefficient data. A plurality of multipliers for executing the filter calculation process, a first data supply means for supplying the plurality of multipliers with different reference data held by the holding device, and the plurality of multipliers. The second data supply means for supplying the common coefficient data held in the holding device to the multiplication device and the time for the multiplication device to repeatedly execute the multiplication are determined by the filter size of the filter calculation process. increases as increases, when the filter size is less than a predetermined value, among the plurality of multipliers, to execute the multiplication for some multipliers, the multiplication with respect to the other multiplier It is characterized by having a control means for controlling the execution so as not to be executed.
Further, the arithmetic circuit of another aspect according to the present invention is
Multiple multipliers that perform filter arithmetic processing,
Based on the filter size in the filter calculation process, some multipliers among the plurality of multipliers are not allowed to execute the filter calculation process, and other multipliers are not allowed to execute the filter calculation process. Control means to control
It is characterized by having.
Further, the arithmetic circuit of another aspect according to the present invention is
Multiple multipliers that perform filter arithmetic processing,
Based on the data size of the reference data of the filter calculation process, some of the plurality of multipliers are made to execute the filter calculation process, and the other multipliers are subjected to the filter calculation process. Control means to control not to execute,
It is characterized by having.

According to the present invention, when data transfer becomes a bottleneck, power consumption is reduced by controlling the number of multipliers to be executed among a plurality of multipliers that can be executed in parallel in an arithmetic circuit. Can be done.

It is a block diagram which shows the structural example of the image processing apparatus provided with the arithmetic circuit which concerns on 1st Embodiment. It is a figure which shows the structure of the arithmetic circuit 22. It is a network block diagram which shows the example of CNN processing. This is an example of generating the characteristic surface 305a in the CNN treatment. It is a figure which shows the structure of the control part 501. It is a figure explaining the example of the information set in the register group 602. It is a figure explaining the structural example of a shift register. It is a figure explaining the structure of a multiplier. It is a figure explaining the structure of the cumulative adder. It is a figure explaining the structure of the nonlinear conversion processing part 509. It is a figure explaining the example of the convolution operation by the operation circuit 22. It is a time chart explaining the operation of the convolution calculation by the calculation circuit 22. It is a figure explaining the relationship between the degree of parallelism and the processing time when the kernel size is changed. It is a flowchart explaining operation of an image processing apparatus. It is a figure which schematically explains the processing example of the image processing apparatus of 2nd Embodiment. It is a flowchart explaining the operation of the image processing apparatus of 2nd Embodiment. (A) It is a figure which shows the conventional parallel operation. (B) It is a figure explaining the specific example of the method of determining the degree of parallelism of 2nd Embodiment. It is a figure explaining the example of the degree of parallelism determination table of 2nd Embodiment. It is a figure explaining the application example of the 3rd Embodiment. It is a figure which shows the example of the conventional parallel arithmetic circuit.

Hereinafter, embodiments of the present invention will be specifically described with reference to the accompanying drawings.

(First Embodiment)
First, the first embodiment of the present invention will be described. FIG. 1 is a configuration example of an image processing apparatus including a parallel arithmetic circuit according to the first embodiment of the present invention. The image processing device has a function of recognizing or detecting a specific object (image pattern) from the input image data. The image input module 20 includes an optical system, a photoelectric conversion device such as a CCD or CMOS sensor, a driver circuit for controlling the sensor, an AD converter, a signal processing circuit for controlling various image corrections, a frame buffer, and the like.

The RAM (Random Access Memory) 500 is used as an image buffer and a calculation work buffer of the calculation circuit 22. The RAM 500 holds a data group, filter coefficient data, and the like corresponding to the characteristic surface of the CNN. The RAM 500 has a role as an external data holding device that is connected to the arithmetic circuit 22 and transfers necessary data to the arithmetic circuit 22. In this embodiment, the arithmetic circuit 22 is a CNN processing unit that performs CNN processing in parallel. The configuration of the arithmetic circuit 22 will be described later.

The DMAC (Direct Memory Access Controller) 26 controls data transfer between each module or circuit on the image bus 23 and the CPU bus 30. The bridge 24 provides a bridge function between the image bus 23 and the CPU bus 30. The pre-processing module 25 performs various pre-processing for effectively performing the pattern recognition processing by the CNN processing. Specifically, image data conversion processing such as color conversion processing / contrast correction processing is processed by hardware.

The face candidate detection module 31 identifies a face candidate region to be processed by the arithmetic circuit 22. Specifically, the face candidate detection module 31 specifies a skin color region of a person within a predetermined color space converted by the preprocessing module 25, and sets the region as a processing target region for CNN processing in the arithmetic circuit 22. Information about the processing target area specified here is recorded in the RAM 500 and used in the arithmetic circuit 22.

The CPU 27 controls the operation of the entire image processing device. The ROM (Read Only Memory) 28 stores instructions that define the operation of the CPU 27 and parameter data necessary for various operations. The RAM 29 is a memory required for the operation of the CPU 27. The CPU 27 can also access the RAM 500 on the image bus 23 via the bridge 24.

FIG. 2 is a diagram showing the configuration of the arithmetic circuit 22. In the present embodiment, the arithmetic circuit 22 is described as a CNN processing unit that performs CNN processing, but the arithmetic processing performed by the arithmetic circuit 22 is not limited to the CNN processing, and other It is also possible to apply it to various filter calculation processes of. The arithmetic circuit 22 can be applied to various processing represented by a hierarchical connection relationship of arithmetic processing. For example, it can be applied to other hierarchical processes such as Restricted Boltzmann Machines and Recursive Neural Network.

Further, the calculation by the calculation circuit 22 can be applied in various cases from a one-stage layer to a plurality of layers as shown in FIG. Further, the data calculated by the arithmetic circuit 22 is not limited to two-dimensional data, but can be applied to one-dimensional data or three-dimensional or higher-dimensional data. The CNN process performed by the arithmetic circuit 22 shown in FIG. 2 sequentially calculates the feature planes from the lower layers according to the hierarchical connection relationship of the plurality of data groups as shown in FIG. That is, the arithmetic circuit 22 first calculates the feature surface 303a, the feature surface 303b, and the feature surface 303c in order using the input image data 301 as reference data. Next, the feature surface 305a and the feature surface 305b are calculated in order using the feature surface 303a as reference data. In this way, the arithmetic circuit 22 calculates the feature planes in order without being limited by the number of layers, and finally calculates the feature planes 307.

The operation of the arithmetic circuit 22 in this embodiment will be described below. In this embodiment, a case where convolution calculation processing is performed in parallel in the horizontal direction will be described. That is, a plurality of arithmetic units simultaneously operate and execute a convolution operation at a plurality of positions continuously in the horizontal direction with reference to the feature plane that is the result of the convolution operation. In the present embodiment, the number of arithmetic units (pairs of a multiplier and a cumulative adder) that operate simultaneously and execute a convolution operation is called a degree of parallelism. Since the parallel operation in the vertical direction can be performed in the same manner as the parallel operation in the horizontal direction, the description thereof will be omitted.

As shown in FIG. 5, the control unit 501 of FIG. 2 supplies to the sequencer 601 and the RAM 500 that control the timing of various signals based on the values of the register group 602 and the register group 602 that determine the basic operation of the arithmetic circuit 22. It includes a memory control unit 605 and the like that perform access arbitration.

The arithmetic unit control unit 517 generates a signal for controlling the operation of the multipliers 507a to 507n and the cumulative adders 508a to 508n according to the parallel degree designation information held in the internal register. The control signals 513a to 513n are signals that directly control the operation of the multipliers 507a to 507n, respectively. The arithmetic unit control unit 517 includes a mask register 511 that latches the arithmetic unit control signal, and an arithmetic unit control data generation unit 512 that controls the mask register 511 at a predetermined timing. The arithmetic unit control data generation unit 512 is also connected to the register group 602 of the control unit 501.

FIG. 5 is a diagram illustrating a configuration of the control unit 501. The sequence control unit 601 inputs and outputs various control signals 604 that control the operation of the arithmetic circuit 22 according to the information set in the register group 602. Similarly, the sequence control unit 601 generates a control signal 606 for controlling the memory control unit 605. The sequence control unit 601 is composed of a sequencer including a binary counter, a Johnson counter, and the like. The register group 602 is composed of a plurality of register sets (not shown), and holds information and the like for performing hierarchical CNN processing.

FIG. 6 is a diagram showing an example of information set in a plurality of register sets of the register group 602 shown in FIG. Each of the register contents 1101a, the register contents 1101b, and the register contents 1101c as examples of the information set in the plurality of register sets is the information necessary for calculating one characteristic surface. A predetermined value is written in advance from the CPU 27 to the register group 602 via the bridge 24 and the image bus 23.

Here, it is assumed that each register in the register set has a width of 32 bits. In the register content 1101a, the "final layer designation" is a register value that specifies whether or not the feature plane corresponding to the register set is the final layer. When the register value indicating "final layer designation" is 1, the characteristic surface to be calculated is the characteristic surface of the final layer, and the CNN process ends when all the calculation processes of the feature surface of the final layer are completed. The "number of reference data planes" is a value that specifies the number of feature planes in the previous layer connected to the feature plane to be calculated. For example, the "number of reference data planes" when calculating the feature plane 305a shown in FIG. "3" is set in the register value of "". The "non-linear conversion" is a register value for designating the presence / absence of the non-linear conversion processing, and when "1" is set in the register value, the non-linear conversion processing is executed. The "calculation result storage destination pointer" is an address indicating the start pointer on the RAM 500 for holding the calculation result of the target feature surface, and the calculation result is stored in the raster scan order with the pointer value as the start pointer.

The "horizontal size of the kernel" and the "vertical size of the kernel" are register values that specify the size of the kernel used for the convolution operation of the feature plane. The "degree of parallelism" is a register value that specifies the number of arithmetic units operating in parallel when executing the convolution operation of the characteristic surface. The degree of parallelism set here is set in advance according to the size of the kernel, the data transfer capacity from the RAM 500, and the like. The relationship between the kernel size (filter size) and the degree of parallelism will be described later.

The “weight coefficient storage destination” is a register value indicating the storage destination address of the weight coefficient of the kernel used for the calculation of the characteristic surface on the RAM 500. It is assumed that the weighting coefficient data has the same number of coefficient sets as the "number of reference data surfaces" and is stored in the order of raster scan from the address specified by the register value of the "weighting coefficient storage destination". That is, the coefficient data of the number of "horizontal size of kernel" x "vertical size of kernel" x "number of reference data planes" is stored in the RAM 500. The register value of the "vertical size of the reference data" and the register value of the "horizontal size of the reference data" are information indicating the number of horizontal pixels and the number of vertical lines of the reference image data or the reference feature plane, respectively.

It is assumed that the reference data is stored in the RAM 500 in the order of raster scan, starting from the address indicated by the register value of the "reference data storage destination pointer". That is, the number of reference data of "horizontal size of reference data" x "vertical size of reference data" x "number of reference data surfaces" is stored in the RAM 500. The plurality of register values described above are prepared for each feature plane to be calculated. When the register value of the "reference data storage pointer" of the feature surface to be processed by the operation is equal to the "calculation result storage destination pointer" of the feature surface to be combined in the previous layer, in this embodiment, the feature surface of the previous layer to be referenced Is combined with the calculated feature plane.

The sequence control unit 601 is a sequence related to the calculation operation according to the register values such as "horizontal size of kernel", "vertical size of kernel", "horizontal size of reference data", "vertical size of reference data" and "parallel degree". Take control. The memory control unit 605 arbitrates access for reading from the RAM 500 and writing to the RAM 500 of each data bus 607 to 609 according to the control signal 606 generated by the sequence control unit 601.

Specifically, the memory control unit 605 appropriately controls access to the memory via the image bus 603, reading of reference data 607, reading of weighting coefficient data 608, and writing of calculation result data 609. A description of access to the RAM 500 will be described later. It is assumed that the data width of the RAM 500 and the data width of each data bus 607 to 609 are all 32 bits.

The storage unit 502 and the storage unit 503 of FIG. 2 are composed of, for example, a plurality of registers and memories. The storage unit 502 is used to temporarily hold the kernel weighting coefficient data held in the RAM 500. In the case of data in which the weighting coefficient is represented by 8 bits, the storage unit 502 is composed of a plurality of registers having an 8-bit width.

Further, the storage unit 502 has the same number of registers as the kernel size in the same direction in which the convolution operations are processed in parallel. For example, when the convolution operation is processed in parallel in the horizontal direction, the number of registers in the storage unit 502 is "11" when the kernel size in the horizontal direction is "11". Since there are actually a plurality of kernel sizes, the storage unit 502 is configured by the number of registers having the maximum kernel size assumed. The control unit 501 loads the kernel weighting coefficient required for the product-sum operation processing of the next line from the RAM 500 into the register of the storage unit 502 during the shift operation of the shift register 504.

The storage unit 503 is used to temporarily hold the reference data stored in the RAM 500. For example, when the reference data is data represented by 8 bits, the storage unit 503 is composed of a plurality of 8 bit wide registers. The storage unit 503 is composed of a number of registers of "the number of data that can be processed in parallel" + "the kernel size in the same direction as the parallel processing direction-1" or more. Here, the "number of data that can be processed in parallel" is the maximum degree of parallelism of the arithmetic circuit 22. The number of registers here is a value for obtaining reference data necessary for calculating feature surface data at a plurality of positions (parallel calculation) at a time, and if the number of registers is equal to or greater than the number of the data. good. For example, assuming that the convolution operation is processed in parallel in the horizontal direction, when the kernel size is "11" and the degree of parallelism is "8", the storage unit 503 is composed of 18 or more 8-bit registers.

The control unit 501 loads the reference data required for the next column processing from the RAM 500 into the storage unit 503 during the shift operation of the shift register 505. That is, the product-sum operation processing of the convolution and the data loading from the RAM 500 operate in a pipeline on a line-by-line basis of the kernel. The number of reference data actually required is determined according to the contents of the "parallelism" register.

The shift register 504, the shift register 505, and the shift register 506 are shift registers with a data load function. The shift register 504 and the shift register 505 are composed of a plurality of registers having the same bit width as the storage unit 502 and the storage unit 503, respectively, and the shift register 506 is composed of a plurality of registers having the same bit width as the effective bit of the cumulative adder output. To do. Further, the mask register 511 is composed of a plurality of registers having the same number of bit widths as the number of multipliers 507a to 507n.

The shift register 504 is a data supply unit that sequentially supplies parameter data (weighting coefficient) common to each multiplier 507a to 507n by a shift operation. The shift register 505 is a data supply unit that supplies reference data at different positions in the previous layer in parallel to the multipliers 507a to 507n by a shift operation.

Since the shift register 504, the shift register 505, and the shift register 506 have the same basic configuration, FIG. 7 shows a configuration example of these shift registers. FIG. 7 describes an example when the number of registers is 4. The flip-flops 701a to 701a to d are multi-bit flip-flops, and latch data of a predetermined bit in synchronization with the CLOCK signal. The selectors 702a to c select the output signals OUTx (x: 0 to 2) of the flip-flops 701a to d when the value of the load signal which is the selection signal is 0, and when the value is 1, the flip-flops 701a to d Input signal INx (x: 1-3) of is selected. That is, one of the shift operation and the load operation of the flip-flops 701a to 701 is selected according to the value of the load signal. The Enalbe signal is an enable signal for data transition. When the value of the Enalbe signal is 1, the data is latched at the rising edge of the CLOCK signal, and when it is 0, the data latched by the previous clock is retained as it is (the state transition is). do not do).

The Load2 signal, the Load4 signal, and the Load5 signal in FIG. 2 correspond to the Load signal of FIG. 7, respectively, and the Enable1, Enable2 signal, and Enable3 signal of FIG. 2 correspond to the Enable signal of FIG. 7.

The shift register 504 collectively loads the initial weight coefficient data from the storage unit 502, then executes a clock number shift operation equal to the kernel size in the horizontal direction, and continuously supplies the weight coefficient data to the multipliers 507a to 507n. To do. The OUTn signal, which is the output signal of the shift register 504, is output from the final stage output of the shift register and is input to all the multipliers 507a to 507.

Similarly, the shift register 505 loads the initial data of the reference data from the storage unit 503, then executes the same clock number shift operation as the kernel size in the horizontal direction, and outputs a plurality of different reference data to the multipliers 507a to 507n. Supply at the same time. The shift register 504 and the shift register 505 operate in synchronization.

At this timing, the value obtained by multiplying the reference data by the weighting coefficient is sent to the cumulative adder. By shifting the clock corresponding to one line of the kernel size, the convolution operations for one line of the kernel at different feature plane positions are processed in parallel. Further, by repeating the operation for the number of lines of the kernel, a two-dimensional convolution operation of the feature plane position corresponding to the degree of parallelism is processed. Each unit operates according to the control signal output by the control unit 501. The multipliers 507a to 507n are a plurality of multipliers operating in parallel, and the cumulative adders 508a to 508n are a plurality of cumulative adders operating in parallel. The multipliers 507a to 507n and the cumulative adders 508a to 508n are connected one-to-one and are collectively referred to as an arithmetic unit.

FIG. 8 shows a configuration example of the multipliers 507a to 507n. The multiplier of this embodiment is composed of a general multiplier 1401 and a selector 1402. The selector 1402 selects the input value 0 instead of Input2 when the arithmetic control signal 1403 is not valid (active). When the input value 0 is selected, the signal value in the multiplier logic does not transition regardless of the input value Input1, and the current consumption does not increase due to the signal transition. The arithmetic control signal 1403 is any of the control signals 513a to 513n output from the mask register 511 in FIG. Here, Input2 may be input of reference data or coefficient data. Further, although the method of masking the input data to the multiplier 1401 with the output of the mask register 511 (calculation control signal 1403) has been described, a corresponding method such as setting the input data to 0 in the storage unit 503 may be used.

As shown in FIG. 9, the cumulative adders 508a to 508n are composed of the adder 901 and the register 902, and hold the cumulative sum of the input data according to the Latch Enable signal. The Latch Enable signal is a signal synchronized with a clock signal (not shown). When the multiplication result, which is the output of the multiplier, is stacked at 0 (that is, when the operation of the multiplier is stopped), the signal transition of the cumulative adder 508 does not occur, so that the same applies to the multiplier. There is no increase in current consumption. In this way, by stopping the operation of the multiplier and the cumulative adder by the control signals 513a to 513n output by the mask register 511, it is possible to suppress an increase in the current consumption of the multiplier and the cumulative adder. That is, the degree of parallelism of the arithmetic circuit of the present embodiment is controlled by the control signals 513a to 513n output from the mask register 511. The degree of parallelism of the arithmetic circuit is the number of arithmetic units (multipliers or cumulative adders) operating at the same time. When the degree of parallelism of the arithmetic circuit is low, the number of operating arithmetic units is small, so that the power consumption can be kept low.

The cumulative sum obtained here is loaded into the shift register 506 after the calculation for each kernel corresponding to the target feature surface is completed, and sent to the nonlinear conversion processing unit 509 at a predetermined timing. The shift register 506 is a shift register capable of holding the outputs of a plurality of cumulative adders 508a to 508n. The outputs of the cumulative adders 508a to 508n are connected to the shift register 506 only for predetermined effective bits.

As a method of controlling the degree of parallelism of the arithmetic circuit, a case where the input data of the multiplier is stacked at 0 together with the control of the reference data and the output data has been described, but the present invention is not limited to this. For example, instead of stacking the input data of the multiplier at 0, a method of controlling the operation of the corresponding register and the arithmetic unit may be used. For example, it can be realized by stopping the operation clock that controls the operation for each arithmetic unit. In that case, a logical product element may be inserted for each clock of the multipliers 507a to 507 and the cumulative adder 508a to n, and a control signal may be connected. That is, the degree of parallelism of arithmetic processing is controlled by controlling the clock supply for each arithmetic unit. Further, a method such as controlling the power supply supplied to the multipliers 507a to 507 and the cumulative adders 508a to n may be used.

FIG. 10 shows the configuration of the nonlinear conversion processing unit 509. The non-linear conversion processing unit 509 includes a non-linear conversion processing unit 1301 and a selector 1302 composed of a lookup table. Based on the lookup table, the non-linear conversion processor 1301 refers to the data held in the ROM or the like as the address data corresponding to the product-sum calculation result which is the output data of the cumulative adder to the input In. It is assumed that the nonlinear relationship of the output Out corresponding to the address value is recorded in advance in the ROM. The selector 1302 outputs the product-sum calculation result, which is the output data of the cumulative adder, as the output data of the selector 1302 as it is when the non-linear processing conversion is not performed.

The selection signal Select is connected to the control unit 501, and the selector 1302 is controlled according to the register value of the "nonlinear conversion" in the control unit 501. The data converted here is stored at a predetermined address of the RAM 500. The storage address here is also controlled according to the setting of the register group 602 of the control unit 501 and the operation of the sequence control unit 601. As described above, the control unit 501 processes the hierarchical convolution operation in parallel by controlling each timing signal and data transfer according to the contents of the register group 602.

FIG. 11 is a diagram illustrating an example of parallel processing of convolution operations by the operation circuit 22 of the present embodiment. FIG. 11 shows the data coordinates of the raster scan. It is assumed that each block (minimum one square shown schematically) of the reference data surface 1004 to be processed in parallel indicates the input image stored in the RAM 500 in the raster scan order or the pixel of the calculation result of the previous layer. Each pixel of the reference data surface 1004 is indicated by input (x, y) at coordinates 1001, x indicates a horizontal position, and y indicates a vertical position. It is assumed that each block of the feature surface 1003 to be calculated in parallel processing indicates the pixels of the calculation result in the raster scan order. Each pixel of the feature plane 1003 to be calculated is indicated by output (x, y) at coordinates 1002, x indicates a horizontal position, and y indicates a vertical position.

The feature surface area 1003 indicates an area of feature surface data calculated by a plurality of arithmetic units of the arithmetic circuit 22 of FIG. 2 simultaneously performing a convolution operation. Each pixel shown in the feature plane region 1003 is indicated by output (x, 6) at coordinates 1002, and x is 5 to 12. In the example shown in FIG. 11, the convolution calculation of the eight pixel positions of interest on the feature plane is processed at the same time. Further, at each of the pixel positions of interest, the multipliers 507a to 507n and the cumulative adders 508a to 508n corresponding to the arithmetic unit numbers 0 to 7 execute arithmetic processing, respectively. Each pixel shown in the reference data surface 1004 to be processed in parallel indicates an area of reference pixel data for calculating the area 1003 of the feature surface. In the example shown in FIG. 11, it is assumed that the kernel size is "11" in the horizontal direction and "13" in the vertical direction. When the degree of parallelism of the arithmetic circuit 22 is 8, the width calculated at the same time in the horizontal direction is "18". Therefore, the size of the area 1004 of the reference data surface is "18" in the horizontal direction and "13" in the vertical direction. ". The calculation circuit 22 simultaneously processes the convolution calculation of the reference data surface area 1004, and simultaneously calculates the feature surface area 1003. In this way, the two-dimensional convolution calculation is executed in parallel while scanning the reference data surface in units of 8 pixels in the horizontal direction and in units of 1 line in the vertical direction. In this embodiment, the data of a plurality of feature planes arranged in the horizontal direction is not limited to the case of calculating in parallel, and the feature plane data continuous in the vertical direction may be calculated in parallel. In this case, the weighting coefficient of one row of the kernel is loaded in the storage unit 502, and the horizontal continuous reference data of "parallel degree + vertical size of the kernel-1" is loaded in the storage unit 503. The filter size is the vertical size or the horizontal size of the kernel depending on the direction of parallel calculation. In FIG. 11, the vertical size of the kernel has been described as an example.

FIG. 12 is a time chart illustrating the operation of the convolution calculation by the parallel calculation circuit 22 of the present embodiment. FIG. 12 is a diagram illustrating a part of a convolution calculation process for calculating one characteristic surface. Further, all the signals shown in FIG. 12 are synchronously operated based on a clock signal (not shown), and one product-sum operation is processed for each clock signal. The Load1 signal indicates an enable signal for loading kernel weighting factor data into the storage unit 502. The control unit 501 reads the weighting coefficient data for one line of the kernel from the RAM 500 during the period when the signal is valid (the signal level is 1), and writes it in the storage unit 502. The size of one line of the kernel (data size of the weighting factor) is held in the register group 602 shown in FIG. Further, the control unit 501 determines the address of the data to be read based on the address pointer information of the weighting coefficient specified by the register group 602, the data size of the weighting coefficient, the number of reference data faces, and the like. Here, assuming that the data width of the RAM 500 is 32 bits and the data width of the weighting coefficient is 8 bits, when writing 11 weighting coefficients for one horizontal line to the storage unit 502, the loading process is completed in 3 clocks. To do.

After that, it is assumed that all the read / write cycles for the RAM 500 are completed in one clock. When the loading of the weighting factors is completed, the control unit 501 then activates the Load3 signal to start loading the reference data. As with the Road1 signal, it is assumed that the Road3 signal is in the enabled state when the signal level is 1.

The control unit 501 takes out reference data from the RAM 500 at the same time as enabling the Load3 signal, and sets it in the storage unit 503. The number of data to be set is determined from the size and parallelism of the kernel held in the register group 602. Further, the control unit 501 determines the address of the data to be read from the RAM 500 based on the address pointer information of the reference data specified by the register group 602, the size of the reference data, and the number of reference data surfaces. Since the effective digit of the reference data is 8 bits, for example, when writing 18 reference data to the storage unit 503, the writing sequence is completed in 5 cycles. In the case of the example shown in FIG. 11, since the kernel has a horizontal size of 11 and an operation parallel degree of 8, it is necessary to load reference data having a parallel degree of 11 + 8-1 = 18.

* The CLR signal is a signal for initializing the cumulative adder 508, and when the value of the signal is 0, the value of the register 902 of the cumulative adder is initialized to 0. The control unit 501 sets the value of the * CLR signal to 0 before starting the convolution calculation of the new feature plane position.

The Load2 signal is a signal for instructing the initialization of the shift register 504, and when the value of the signal is 1 and the Enable1 signal is valid (signal level 1), a plurality of weight coefficient data held in the storage unit 502 are stored. It is collectively loaded into the shift register 504. The Enable1 signal is a signal that controls the data transition of the shift register. As shown in FIG. 12, since the Enable1 signal is always set during operation, when the value of the Load2 signal is 1, the output of the storage unit 502 is latched according to the clock signal, and when the value of the Load2 signal is 0. , The shift process is continued according to the clock signal.

The sequence control unit 601 of the control unit 501 activates the Load2 signal when the number of clocks corresponding to the horizontal data size of the kernel is counted.

Further, at the same time as stopping the shift operation, the weighting coefficient data held in the storage unit 502 is collectively loaded into the shift register 504. That is, the weighting coefficients of one line are collectively loaded in the horizontal unit of the kernel, and the loaded weighting coefficients are shifted out according to the operation clock.

The Load4 signal is a signal for instructing the initialization of the shift register 505. When the value of the Load4 signal is 1 and the Enable2 signal is valid (signal level 1), the reference data held in the storage unit 503 is the shift register 505. It is loaded all at once.

The Enable2 signal is a signal that controls the data transition of the shift register. As shown in FIG. 12, since the value of the Enable2 signal is set to 1 during operation, when the value of the Load4 signal is 1, the output of the storage unit 503 is latched according to the clock signal, and the value of the Load4 signal is set. If is 0, the shift process is continued according to the clock signal.

When the sequence control unit 601 of the control unit 501 counts the number of clocks corresponding to the data size in the horizontal direction of the kernel, the Load4 signal is activated, the shift operation is stopped, and at the same time, the reference data held in the storage unit 503 is collectively loaded.

That is, the sequence control unit 601 collectively loads the reference data required for each line of the kernel in the convolution operation processing from the storage unit 503 into the shift register 505, and the shift register 505 shifts the loaded reference data according to the operation clock. To do. Further, the control unit 501 controls the Load4 signal at the same timing as the Load2 signal.

Note that the Enable1 signal and the Enable2 signal may not be loaded in time from the storage unit 502 and the storage unit 503 to the shift register 504 and the shift register 505 in the "horizontal calculation cycle" shown in FIG. In this case, the sequence control unit 601 stops the operation of the arithmetic unit by disabling the Enable1 signal and the Enable2 signal, and secures the data load time. At that time, the sequence control unit 601 also disables the latch signals of the cumulative adders 508a to 508n. Since the cumulative adders 508a to 508n continue the multiply-accumulate operation in synchronization with the clock, the kernel size is simultaneously applied to a plurality of positions of the feature plane calculated according to the shift operation of the shift register 504 and the shift register 505. Executes the product-sum operation process.

Specifically, during the shift operation period (horizontal calculation cycle period in FIG. 12) of the shift register 504 and the shift register 505, the product-sum calculation for one kernel line of the pixel positions of the plurality of feature planes is performed. Become. Further, in the kernel operation section shown in FIG. 12, a two-dimensional convolution operation result corresponding to the degree of parallelism can be obtained by repeating the operation for each column of the kernel in the vertical direction while exchanging the weighting coefficient and the reference data.

In this way, the control unit 501 controls each signal according to the kernel size and the degree of parallelism, so that the product-sum calculation process and the data (weight coefficient data and reference data) required for the product-sum calculation process are supplied from the RAM 500. Are processed in parallel. In the case of the example shown in FIG. 12, since the loading of the reference data from the RAM 500 to the storage unit 503 and the loading of the weighting coefficient from the RAM 500 to the storage unit 502 are completed within the horizontal calculation cycle, the time required for data loading is completed. Does not affect the calculation speed. However, depending on the kernel size of the convolution operation, the time required for loading data from the RAM 500 to the storage unit 502 or the storage unit 503 may not be within the horizontal calculation cycle.

FIG. 13 is a diagram showing the relationship between the data load cycle (time required for data load) and the horizontal calculation cycle when the kernel size and the degree of parallelism are changed. FIG. 13 is an example of calculating the data load cycle for each horizontal calculation cycle shown in FIG.

The degree of parallelism indicates the number of pairs of multipliers 507a to 507n and cumulative adders 508a to 508n that operate simultaneously. The "reference data load number" indicates the number of reference data required for the next calculation to be loaded from the RAM 500 into the storage unit 503 within the "horizontal calculation cycle" shown in FIG. The number of reference data loads is "parallelism + kernel horizontal size-1". The "weight coefficient load number" indicates the number of weight coefficient data to be loaded from the RAM 500 to the storage unit 502 in the "horizontal calculation cycle" shown in FIG.

In the case of this embodiment, the "total load cycle" corresponding to the total number of data load cycles required for the next calculation can be calculated using the following formula.

Total load cycle = INT ((reference data load number + 3) ÷ 4) + INT ((weight coefficient load number + 3) ÷ 4) (2)
Here, INT (n) is a function for finding the maximum integer that does not exceed n. The reference data and the weighting coefficient data are 8 bits (1 byte) each.

The formula (2) is a calculation formula when the width of the data bus is 32 bits (that is, 4 bytes can be transferred with one access) as described above. Further, it is assumed that one memory access is completed in one cycle.

The “calculation cycle of one kernel line” shown in FIG. 13 is the number of cycles for performing a convolution operation for one line of the kernel. The "one-line kernel operation cycle" corresponds to the horizontal size value of the kernel. "Required cycle" indicates the number of cycles required to perform the convolution operation for one kernel line. As shown in FIG. 13, when the "total load cycle" is smaller than the "calculation cycle of one kernel line", the "calculation cycle of one kernel line" corresponds to the value of the "required cycle". In this case, the arithmetic processing time determines the processing time (that is, the arithmetic bottleneck). In FIG. 13, when the kernel size is 13 × 13, the “total load cycle” is smaller than the “calculation cycle of one kernel line”, so the value of the “required cycle” is the same as the value of the “calculation cycle of one kernel line”. is there. For large kernel sizes, the required cycle is always equal to the "kernel arithmetic cycle".

On the other hand, when the "total load cycle" is larger than the "calculation cycle of one kernel line", the "total load cycle" corresponds to the value of the "required cycle". In this case, the data transfer time determines the processing time (ie, the memory access bottleneck). When the kernel size is small (when the kernel size is 3 × 3 in FIG. 13), the required cycle changes according to the degree of parallelism. When the degree of parallelism exceeds 7, the "total load cycle" exceeds the "calculation cycle of one kernel line", so data transfer becomes a bottleneck, and the "required cycle" is larger than the "calculation cycle of one kernel line". It becomes the value of "load cycle".

In such a case, the control unit 501 of the arithmetic circuit shown in FIG. 2 gives priority to the completion of access to the RAM 500 (data loading to the storage units 502 and 503 and data saving of the nonlinear conversion processing unit 509). That is, it controls the Enable1 signal, the Enable2 signal, the Enable3 signal, the Latch Enable signal of the cumulative adder, and the like. As a result, the shift operation of the shift register 504 and the shift register 505 and the operation of the cumulative adders 508a to 508n are stopped, and the start of the multiply-accumulate operation process is delayed until the completion of the data load.

The "processing time per pixel" is a value representing the processing time per pixel when the convolution operations for one kernel line are executed in parallel. "Processing time per pixel" is defined by the following formula.

Processing time per pixel = required cycle ÷ degree of parallelism (3)
When the kernel size is large (when the kernel size is 13 × 13 in FIG. 13), the “processing time per pixel” decreases with the degree of parallelism. That is, the larger the degree of parallelism, the shorter the overall processing time. On the other hand, when the kernel size is small (when the kernel size is 3 × 3 in FIG. 13), for example, when the parallel degree is 7, the “processing time per pixel” is larger than when the parallel degree is 6. That is, the processing time increases even though the degree of parallelism is increased. Even when the degree of parallelism is 8, the "processing time per pixel" is the same as when the degree of parallelism is 6. That is, the processing time does not change even though the degree of parallelism is increased. This is because the data transfer becomes a bottleneck and the time for loading data from the RAM 500 to the storage unit 502 and the storage unit 503 is longer than the arithmetic processing time of the multipliers 507a to 507n and the cumulative adders 508a to 508n. ..

When the degree of parallelism is 8, the number of simultaneous operations of the multipliers 507a to 507n and the cumulative adders 508a to 508n is 8, which is larger than that in the case of the degree of parallelism 6, so that the peak current consumption during calculation is larger. Since the dynamic power consumption of the device is proportional to the sum of squares of the current values, it is desirable that the degree of parallelism is as low as possible from the viewpoint of power consumption.

Therefore, in the present embodiment, when the control unit 501 is set to the maximum value of the degree of parallelism of the arithmetic circuit 22, the kernel size (filter size) that becomes about the same as the arithmetic processing time and the data transfer time is set to a predetermined value. And. If the kernel size (filter size) is less than a predetermined value, it becomes a memory access bottleneck. Therefore, based on the filter size, the control unit 501 sets the degree of parallelism of the arithmetic circuit 22 to be lower than the maximum value, and performs arithmetic processing time and data. Make it about the same as the transfer time. For example, when the kernel size (filter size) is 3 × 3, the control unit 501 sets the degree of parallelism of the arithmetic circuit 22 to 6. As a result, the processing speed of the convolution operation does not change, and the power consumption can be reduced. Instead of calculating the degree of parallelism of the arithmetic circuit 22 every time, a data table showing the correspondence between the kernel size (filter size) and the degree of parallelism of the arithmetic circuit 22 can be held in the RAM 500 in advance. The control unit 501 can set the degree of parallelism of the arithmetic circuit 22 by referring to the data table held in the RAM 500.

In the present embodiment, the degree of parallelism is set according to the kernel size from such a viewpoint. As described above, the degree of parallelism is set according to the contents of the registers shown in FIG. In FIG. 2, the control unit 501 controls the operation of the parallel computing circuit according to the degree of parallelism recorded in the register. For example, even if the number of arithmetic units (multipliers 507a to 507n and cumulative adders 508a to 508n) capable of parallel processing is 8, when the kernel size is 3 × 3, the degree of parallelism is set to 6 for calculation. Operate the vessel.

In this case, the control unit 501 loads a number of reference data corresponding to the degree of parallelism from the RAM 500 into the storage unit 503, and starts the arithmetic processing. The arithmetic unit control data generation unit 512 generates the arithmetic unit control signal 514 according to the "parallel degree" information recorded in the register group 602 prior to the arithmetic processing. For example, when the degree of parallelism is 6, the arithmetic unit control data generation unit 512 generates the arithmetic unit parallel signal “11111100”. Each bit indicates a control signal, 1 corresponds to an operation calculator, and 0 corresponds to a stop calculator.

The arithmetic unit control unit 517 activates the Enable4 signal at the timing of starting the convolution calculation, and latches the control signal in the mask register. Each latched signal (signals 513a to 513n that directly control the arithmetic unit) controls the operation of the multipliers 507a to 507n during the convolution calculation. Specifically, only the multiplier 0 to the multiplier 5 and the cumulative adder 0 to the cumulative adder 5 perform the calculation operation.

The Load5 signal is a signal for loading the result of the cumulative adder in parallel to the shift register 506, and the control unit 501 is the value of the Load5 signal and the Enable3 signal when the product-sum calculation of the parallel processing unit of the target feature surface is completed. Set to 1. FIG. 12 shows an example in which the feature plane of the coupling destination is one (that is, the feature plane is calculated by only one set of convolution operations).

When the value of the Load5 signal is 1 and the value of the Enable3 signal is 1, the shift register 506 collectively loads the output of the cumulative adder 508. At this timing, the calculated convolution calculation result is latched in the shift register 506. When the data loading to the storage unit 502 and the storage unit 503 is completed during the shift operation of the shift register 504 and the shift register 505, the control unit 501 activates the signal of Enable 3 and holds the multiplication result in the shift register 506. Shift out. The shifted-out calculation result is converted by the nonlinear conversion processing unit 509, and then stored by the control unit 501 at a predetermined address of the RAM 500 according to the size of the calculation result storage destination pointer and the reference data described in the register group 602. ..

The control unit 501 activates the Enable 3 signal in the gap of data loading from the RAM 500 to the storage unit 502 and the storage unit 503, and shifts out the calculation result from the shift register 506. Again, the control unit 501 controls the number of data to shift out according to the "parallelism" information recorded in the register group 602. When the degree of parallelism is 6, the result of 6 operations is shifted out.

As described above, the control unit 501 determines the degree of parallelism according to the "degree of parallelism" information written in the register group 602, and controls each unit.

The control unit 501 arbitrates the access to the RAM 500 of the three processing units of the storage unit 502, the storage unit 503, and the nonlinear conversion processing unit 509, and pipelines the multiply-accumulate processing and the access to the RAM 500 of the three processing units. To do.

The nonlinear conversion processing unit 509 operates in the lowest priority because the access frequency to the RAM 500 is lower than that of the storage unit 502 and the storage unit 503. That is, the non-linear conversion processing unit 509 is accessed in the time slot that is the access gap between the storage unit 502 and the storage unit 503.

FIG. 14 is a flowchart illustrating the operation of the image processing apparatus including the arithmetic circuit 22 of the present embodiment. Hereinafter, it is assumed that the flowchart is realized by the CPU 27 executing the control program. In the present embodiment, an image processing device that performs pattern recognition will be described as an example, but the image processing device of the present embodiment can be applied not only to pattern recognition processing but also to processing such as object detection.

In step S101, the CPU 27 executes various initialization processes prior to the start of the pattern recognition process of the image processing device. The CPU 27 transfers the weighting coefficient required for the operation of the arithmetic circuit 22 which is the CNN processing unit of the image processing apparatus from the ROM 28 to the RAM 500, and at the same time, the operation of the arithmetic circuit 22, that is, various registers for defining the parameters of the CNN processing. Make settings. Specifically, the CPU 27 sets a predetermined value in a plurality of register groups 602 existing in the control unit 501 of the arithmetic circuit 22.

Further, step S102 sets the degree of parallelism based on the size of the kernel and the data transfer time. As described above, the degree of parallelism is determined in advance from the kernel size, the memory access cycle, and the like. When the process of step S102 is completed, the process proceeds to step S103, each hardware module is started, and a series of pattern recognition operations is started.

First, in step S104, the image input module 20 converts the signal output by the image sensor into digital data and stores it in a frame buffer (built in the image input module 20), which is not shown in frame units. Further, when the storage in the frame buffer is completed, the preprocessing module 25 starts the image conversion process based on the predetermined start signal. The preprocessing module 25 extracts the luminance data from the image data on the frame buffer and performs contrast correction processing. Luminance data is generated by linearly converting RGB image data to YIQ (National Television Standard Committee) image data by a linear conversion process. In the contrast correction method, a generally known contrast correction process is applied to emphasize the contrast of the luminance data (Y data).

The preprocessing module 25 stores the luminance data after the contrast correction processing in the RAM 500 as a processing image. Further, in step S104, the face candidate detection module 31 operates. The face candidate detection module 31 identifies the skin color area from the YIQ color image data stored in the frame buffer, stores the specific result as the processing target area information in the RAM 500, and completes the acquisition of the image of the processing target area. When the processing for one image data is completed, the face candidate detection module 31 enables a completion signal (interrupt signal) (not shown). The CPU 27 that has received the completion signal then proceeds to step S105, activates the arithmetic circuit 22, and executes high-precision pattern recognition processing by CNN processing. The arithmetic circuit 22 uses the corrected luminance image data which is the processing result of the preprocessing module 25 and the information about the processing target area which is the processing result of the face candidate detection module 31 only for the image data of the processing target area. Performs convolution operations in parallel.

In step S105, the arithmetic circuit 22 selects the kernel to be used in the convolution arithmetic according to the set value of the register group 602. Further, in step S106, an arithmetic unit that operates in parallel is specified according to the set value of the register group 602. Next, in step S107, the arithmetic circuit 22 uses the kernel for convolution arithmetic (for example, 3041a to 3041b shown in FIG. 3) selected in step S105, and convolves with a plurality of arithmetic units specified in step S106. Process operations in parallel. In step S108, the arithmetic circuit 22 determines whether or not the processing for all the feature planes has been completed. When the processing for all the characteristic surfaces is completed, the process proceeds to step S109. In the example of FIG. 3, when the processing for all the feature planes is finished, the calculation of the feature plane 307 is finished. In step S109, the arithmetic circuit 22 generates an interrupt signal for notifying the CPU 27 of the end of processing for all characteristic surfaces. In step S110, the processes from step S104 to step S109 are executed for all the images.

Upon receiving the end notification interrupt from the control unit 501, the CPU 27 activates the DMAC 26 and transfers the final feature plane data on the RAM 500 to the RAM 29 on the CPU bus 30. The CPU 27 acquires information such as the position and size of a predetermined object to be detected from the final layer detection result placed in the RAM 500. Specifically, the final detection result is binarized and the object position and size are extracted by processing such as labeling.

In a general convolution parallel computing circuit, an arithmetic unit is often assigned to the kernel. FIG. 20 shows an example in which four coefficients having a kernel size of 2 × 2 are processed in parallel by four parallel multipliers 2100. In this case, since the kernel and the parallel computing unit have a dependency relationship, it is not easy to change the number of computing units (degree of parallelism) that operate simultaneously according to the reference data supply capacity and the like. On the other hand, in the configuration of the present embodiment, since the parallel processing is performed for each pixel position of the characteristic surface to be generated, there is no dependency between the kernel and the arithmetic unit in the first place, and the degree of parallelism of the arithmetic is changed according to the operating conditions with simple control. can do.

According to the present embodiment, in a parallel computing circuit that performs high-speed processing in parallel in parallel, the degree of parallelism of the computing circuit 22 that performs parallel computing based on the size of the kernel and the supply capacity of reference data is determined. As a result, it is possible to suppress unnecessary circuit operation and reduce power consumption without reducing the arithmetic processing speed of the arithmetic circuit 22. In the present embodiment, the case where the kernel size is small has been described as an example of the case where data transfer becomes a bottleneck, but the present case is not limited to this, and this case also applies to other cases where data transfer becomes a bottleneck. Applicable.

(Second Embodiment)
Since the configuration of the image processing apparatus of the present embodiment is the same as that of the first embodiment, the processing in which the present embodiment is different from that of the first embodiment will be described below. FIG. 15 is a diagram schematically illustrating a processing example of the image processing apparatus according to the second embodiment of the present invention. In the process of FIG. 15, the image processing device changes the degree of parallelism of the arithmetic circuit according to the data size of the face candidate region in the input image.

Here, the process of recognizing the face image area in the input image will be described as an example. The input image data 1501 is image data input from the image input module 20. The input image data 1501 undergoes the face candidate detection process 1502 by the face candidate detection module 31, and the area-limited information 1503 (coordinate information) regarding the area for which the high-precision recognition process is performed by the convolution calculation 1506 is generated. The area limitation information 1503 is information regarding the processing target area in the input image data 1501 of the convolution calculation 1506. The data in the area (white area) shown in the coordinate data 1504a and the coordinate data 1504b becomes the processing target data of the input image data 1501. In the face candidate detection process 1502, the face candidate area in the input image is acquired by a simple process with a low processing load. As a method for acquiring the face candidate area, various methods such as a method of extracting a rectangular area or an elliptical area containing specific color information and a method of using the extraction result of the previous frame (when applied to a moving image) are applied. It is possible. The pixel data of the face candidate region is the processing target data to be calculated by the convolution calculation 1506.

In the present embodiment, the arithmetic circuit 22 controls the execution of each arithmetic unit 1507 based on the size of the face candidate region. Here, the convolution calculation 1506 is executed by the calculation circuit shown in FIG. Specifically, each pixel 1507 of the feature surface calculated in parallel is calculated by the multipliers 507a to 507n and the cumulative adders 508a to 508n in FIG. 2, and the control of each arithmetic unit 1507 is controlled by the arithmetic unit control unit 517 in FIG. Is executed by. Further, FIG. 15 shows an example when the degree of parallelism of the parallel arithmetic circuit is 8. That is, each arithmetic unit 1507 that processes in parallel simultaneously executes convolution arithmetic processing of eight pixel positions of interest that are continuous in the horizontal direction. In FIG. 15, the convolution calculation is performed with reference to the input image data of the pixel region centered on the position of each arithmetic unit 1507.

Each arithmetic unit 1507 repeats the arithmetic processing in units of 8 pixels and scans the image data to complete the convolution arithmetic on one image data. When the CNN process is executed, the convolution operation is repeatedly executed using a plurality of filter coefficients (kernel coefficients) for calculating a plurality of feature planes.

The arithmetic circuit of the present embodiment can be applied not only to the face recognition process but also to the recognition process of other objects. In the recognition process of other objects, the detection process of the face candidate region can be replaced with the detection process of the candidate region of the recognition target object, and the same processing can be performed.

In the present embodiment, the arithmetic unit control signal generation unit 517 controls each arithmetic unit to be processed in parallel from the face candidate region (coordinate data indicating the face candidate region) obtained in the face candidate region detection process 1502. Generate a signal. The arithmetic unit control signal generation unit 517 outputs a control signal in synchronization with the operation of each arithmetic unit 1507 and at a timing corresponding to the operation type (convolution kernel size, etc.).

Hereinafter, the processing of the image processing apparatus of the present embodiment will be described with reference to the flowchart shown in FIG. The configuration of the image processing device and the configuration of the arithmetic circuit 22 are the same as those of the first embodiment, respectively.

In step S1601, the CPU 27 executes various initialization processes prior to the start of processing by the image processing device. The CPU 27 transfers the weighting coefficient required for the operation of the arithmetic circuit 22 which is the CNN processing unit of the image processing apparatus from the ROM 28 to the RAM 500, and sets various registers for defining the operation of the arithmetic circuit 22, that is, the parameters of the CNN processing. I do. Specifically, the CPU 27 sets a predetermined value in a plurality of register groups 602 existing in the control unit 501 of the arithmetic circuit 22.

When the various initializations are completed, the process proceeds to step S1602, and the CPU 27 starts each hardware module to start a series of face recognition processes. First, in step S1603, the image input module 20 converts the signal output by the image sensor into digital data and stores it in a frame buffer (built in the image input module 20), which is not shown in frame units. Further, when the storage in the frame buffer is completed, the preprocessing module 25 starts the image conversion process based on the predetermined start signal. The preprocessing module 25 extracts luminance data from the image data on the frame buffer and performs contrast correction processing. The preprocessing module 25 stores the luminance data after the contrast correction processing in the RAM 500 as a detection image.

Further, in step S1603, the face candidate detection module 31 operates. The face candidate detection module 31 identifies and determines the skin color region from the YIQ color image data stored in the frame buffer, and acquires a rectangular region including the skin color region in the image data as the face candidate region. In addition, the processing target area including the face candidate area for the convolution calculation is acquired. When the area acquisition process for one image data is completed, the face candidate detection module 31 enables a completion signal (interrupt signal) (not shown). The CPU 27 that has received the completion signal then proceeds to step S1604, activates the arithmetic circuit 22, and executes high-precision face recognition processing. The arithmetic circuit 22 uses the corrected luminance image data obtained by the preprocessing module 25 and the face candidate area (processing target area) acquired by the face candidate detection module 31 to obtain the image data of the processing target area. Only performs convolution arithmetic processing in parallel. In the present embodiment, the case where the face candidate detection module 31 acquires the skin color region in the input image data as the face candidate region has been described, but various methods utilizing other features such as the shape information of the recognition target are applied. It is possible to acquire the candidate area to be recognized. Further, the candidate area to be recognized may be acquired by the area-limited information designated from the outside. For example, the CPU 27 can set the area limitation information in advance according to a predetermined condition, cut out a predetermined small area, and acquire it as a candidate area to be recognized. Further, the candidate area is not limited to the rectangular area, and can be expressed by another shape such as an ellipse.

In the present embodiment, first, in step S1604, the face candidate detection module 31 specifies the size of the face candidate area from the face candidate area acquired in step S1603. The size of the face candidate region specified here is the size of the face candidate region in the same direction as the direction of the parallel calculation (the number of pixels in the width direction or the number of pixels in the height direction of the face candidate region). Next, in step S1605, the degree of parallelism at the time of parallel calculation is determined according to the acquired size.

FIG. 17 is a diagram illustrating a method of determining the degree of parallelism in the present embodiment. When determining the degree of parallelism, the same result is obtained regardless of whether the input image data as the reference data or the feature plane data which is the calculation result is used, but for the sake of simplicity, the calculation result is shown in FIG. This will be described using feature plane data. FIG. 17A is a diagram showing an operation result of a conventional parallel operation. The image data 1701 indicates an image data surface (feature surface) of the calculation result. FIG. 17 (a) and. The white area (mask) in FIG. 17B shows the area 1702 of the feature surface corresponding to the face candidate area in the input data image on the image data surface 1701 of the calculation result. This white area is a characteristic surface calculated by the arithmetic circuit 22 by convolution calculation processing with reference to the data of the face candidate area, and its size corresponds to the size of the face candidate area in the input image data. It is a thing. Hereinafter, this white area is referred to as a calculation target area.

FIG. 17A shows a case where the number of arithmetic units (multipliers 507a to 507n and cumulative adders 508a to 508n) that can be calculated in parallel is eight. When the operations on the calculation target area 1702 are processed in parallel, one row of pixels in the calculation target area 1702 is calculated by two repetitions of processing by the eight arithmetic units shown in each of the block 1703a and the block 1703b. However, when the eight arithmetic units shown in the block 1703b process, the necessary operations are the two arithmetic units of the arithmetic unit numbers 0 and 1, and the arithmetic units shown by the arithmetic unit numbers 2 to 7 are unnecessary arithmetic processing. To execute.

In the present embodiment, as shown in FIG. 17B, the degree of parallelism of the arithmetic circuit 22 is determined according to the size of the calculation target region 1702. In the case of FIG. 17B, the degree of parallelism is set to 5. In this case, in both the five arithmetic units shown in block 1704a and the five arithmetic units shown in block 1704b, the arithmetic units (multipliers 507a to 507n and cumulative adders 508a to 508n) perform only necessary operations. .. In this case, as in the case where the degree of parallelism is 8, it is possible to calculate one line of pixels in the calculation target area by repeating the process twice. That is, the processing speed is the same even when the degree of parallelism is lowered.

When the size of the calculation target area is not an integral multiple of the number of arithmetic units that can operate in parallel (maximum degree of parallelism), the optimum degree of parallelism can be calculated by, for example, the following formula.

Optimal degree of parallelism = n-INT ((n-m) / (INT (w / n) +1)) ... (4)
Here, W indicates the size of the calculation target area, n indicates the number of arithmetic units that can operate in parallel (maximum degree of parallelism), m indicates the remainder of w / n, and INT (x) indicates the numerical value x. Indicates a function that returns the largest integer that does not exceed.

According to the equation (4), it is possible to perform arithmetic processing with the lowest degree of parallelism without reducing the processing speed. That is, the power consumption can be reduced without reducing the processing speed.

Further, instead of calculating the degree of parallelism each time using the above equation (4), the arithmetic unit control data generation unit 512 included in the arithmetic unit control unit 517 refers to the data table holding the pre-calculated values. It is also possible to determine the degree of parallelism with. For example, it is assumed that the RAM 500 holds a data table that holds a correspondence between the width or height of the calculation target area and the degree of parallelism of the arithmetic circuit 22. The arithmetic unit control data generation unit 512 can control the degree of parallelism of the arithmetic circuit 22 by accessing the table held in the RAM 500. FIG. 18 is an example of a data table, and when the width of the calculation target area is between 9 and 12, the degree of parallelism is controlled to be low. Since the data size of the calculation target area on the feature surface has a correspondence relationship with the data size of the face candidate area in the input image data, a table that holds the correspondence relationship between the data size of the face candidate area and the degree of parallelism can be used. .. In this case, the degree of parallelism of the arithmetic circuit 22 can be controlled directly from the data size of the face candidate region acquired by the face candidate detection module 31, so that the process is simple.

As described above, when the arithmetic unit control unit 517 uses the table, the control processing of the arithmetic circuit 22 according to the present embodiment can be realized on a smaller circuit scale.

The control of the degree of parallelism is realized by controlling the operation of the multipliers 507a to 507n by the arithmetic unit control unit 517 as in the first embodiment. Further, the control unit 501 controls the operation of the entire arithmetic processing according to the degree of parallelism determined by the arithmetic unit control unit 517.

In step S1606, the arithmetic circuit 22 executes the convolution arithmetic at the determined degree of parallelism. When the arithmetic processing for all the feature planes is completed in step S1607, the process proceeds to step S1608, and the arithmetic circuit 22 generates an interrupt signal to the CPU 27. It should be noted that the calculation processing for all the feature planes is completed when the calculation of the feature plane 307 shown in FIG. 3 is completed.

In step S1609, the processes from step S1603 to step S1608 are executed for all the images.

As described above, according to the present embodiment, the degree of parallelism is determined according to the size of the calculation target area (the size of the calculation target area in the same direction as the parallel processing direction), thereby reducing the power consumption while maintaining the processing speed. It becomes possible to make it.

In the present embodiment, the face candidate detection module 31 identifies a plurality of skin color regions with respect to one input image data input from the image input module 20, and a plurality of face candidate regions are generated from one input image data. It may be acquired and the processing target area may be acquired. In this case, the CNN processing can be performed on all the processing target areas by sequentially performing the processing shown in FIG. 16 on each of the plurality of processing target areas. The control of the degree of parallelism of the arithmetic circuit in this case is as described above.

Further, by collectively performing the processing shown in FIG. 16 for a plurality of processing target areas, CNN processing can be performed on all the processing target areas. In this case, the arithmetic unit control unit 517 controls the degree of parallelism according to the number of processing target areas.

For example, when the number of a plurality of processing target areas is small, the processing time of the arithmetic circuit 22 may be shorter than the specified time. In this case, the power consumption can be reduced by lowering the degree of parallelism of the arithmetic circuit 22. That is, the minimum degree of parallelism that can be processed within the processing time specified by the image processing apparatus is set.

The degree of parallelism can be easily determined, for example, by having the arithmetic unit control data generation unit 512 have table data. That is, it is possible to hold the table data that holds the correspondence between the number of processing target areas and the corresponding degree of parallelism in the RAM 500. Alternatively, a method of having a plurality of registers for setting a predetermined determination threshold value and determining by comparison with the register value may be used. The control unit 501 controls the operation of the entire arithmetic processing according to the degree of parallelism determined by the arithmetic unit control unit 517.

As described above, even when there are a plurality of processing target areas in one image, it is possible to reduce the power consumption by determining the degree of parallelism according to the number of processing target areas.

(Third Embodiment)
Since the configuration of the image processing apparatus of the present embodiment is the same as that of the first embodiment, the processing in which the present embodiment is different from that of the first embodiment will be described below. In the hierarchical CNN process of the present embodiment, the filter calculation process is performed hierarchically a plurality of times, but a processing unit (not shown) performs a subsampling process for each layer to hold the subsampling processed feature surface in the RAM 500. .. In the hierarchical CNN process, the subsampling process may be the case where the convolution operation result of the previous layer is thinned out to be the reference data of the next layer, or the case where the convolution operation result is pooled to be the reference data of the next layer. is there. In the pooling process, the feature surface generated in the previous layer is reduced by using an average value filter or a maximum value filter. The processing unit of the present embodiment performs either or both of the subsampling processing and the pooling processing.

Generally, in the subsample processing, the feature plane is reduced to 1/2 times or 1/4 times in the horizontal and vertical directions. In the CNN process, since the processing unit performs processing such as subsampling processing, the size of the feature surface in the subsequent stage is often smaller than that of the input image. FIG. 19 is a diagram showing an example of a characteristic surface when the processing unit performs subsampling processing. The feature surface 2001, the feature surface 2002, and the feature surface 2003 are the feature surfaces to be processed in the first layer, the second layer, and the third layer, respectively, and the feature surface 2002 is subsampled 1/2 times both horizontally and vertically. On the feature surface 2003, the subsampling is further performed 1/2 times. In such a case, as shown in the parallel processing area 2004, in the horizontal direction 8 parallel arithmetic unit, when processing the feature surface 2002 and the feature 2003, a useless arithmetic operation occurs. The same applies when the processing unit performs the pooling process.

In the present embodiment, the arithmetic unit control data generation unit 512 controls the number of arithmetic units (degree of parallelism) operating in parallel based on the subsampling situation for each layer. That is, when the arithmetic circuit 22 processes the feature surface 2002, the operations of the arithmetic unit 6 and the arithmetic unit 7 are stopped, and the processing is performed with a degree of parallelism of 6. Further, when the arithmetic circuit 22 processes the feature surface 2003, the operations of the arithmetic units 3 to 7 are stopped and the processing is performed with a parallel degree of 3.

As described above, since the parallel computing circuit of the present embodiment can control the degree of parallelism in units of the number of computing units operating in parallel, the computing units are effectively utilized and consumed according to the size of the calculation target data (reference data). It is possible to reduce power consumption. In other words, by controlling the number of arithmetic units operating in parallel (degree of parallelism) based on the size of the feature surface that is the processing target area (reference data), unnecessary arithmetic operations are eliminated and power consumption is reduced. can do.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

500 RAM
501 Control unit 502 Storage unit 503 Storage unit 504 Shift register 505 Shift register 506 Shift register 507a to 507n Multiplier 508a to 508n Cumulative adder 509 Non-linear conversion unit

Claims (29)

An arithmetic circuit connected to a holding device that holds reference data for filter arithmetic processing and coefficient data of a filter used for the filter arithmetic processing.
A plurality of multipliers that execute the filter calculation process by repeatedly multiplying the reference data that is different from each other and the coefficient data that is common to each other.
A first data supply means for supplying the plurality of multipliers with different reference data held in the holding device.
A second data supply means for supplying the common coefficient data held in the holding device to the plurality of multipliers, and
The time for the multiplier to repeatedly execute the multiplication increases as the filter size of the filter calculation process increases, and when the filter size is equal to or less than a predetermined value, a power of a part of the plurality of multipliers is used. An arithmetic circuit having a control means for controlling an arithmetic unit to execute the multiplication and not causing another multiplier to execute the multiplication. A first storage held in the holding device for temporarily storing the reference data to be supplied to the multiplier for performing the multiplication, and the stored reference data being loaded into the first data supply means. Means and
A second storage means that temporarily stores the coefficient data held in the holding device for supplying to the filter size and loads the stored coefficient data into the second data supply means.
The arithmetic circuit according to claim 1, further comprising. The holding device holds a table showing the correspondence between the filter size and the number of multipliers that perform the multiplication.
The arithmetic circuit according to claim 2, wherein the control means refers to the table and determines the number of the multipliers that perform the multiplication based on the filter size. A claim, wherein the number of reference data loaded from the holding device into the first data supply means is one less than the sum of the filter size and the number of multipliers performing the multiplication. The arithmetic circuit according to any one of 1 to 3. The arithmetic circuit according to any one of claims 1 to 4, wherein each of the first data supply means and the second data supply means includes a shift register. 1. The control means is characterized in that, by outputting an arithmetic control signal to each of the plurality of multipliers, the multiplier is controlled to execute the multiplication or not to execute the multiplication. The arithmetic circuit according to any one of items 5 to 5. The control means has a mask register that connects to each of the plurality of multipliers and outputs the arithmetic control signal, and when controlling the multiplier so as not to execute the multiplication, the mask register The arithmetic circuit according to claim 6, wherein the unit masks the supply of the reference data or the coefficient data to the multiplier and controls the signal of the multiplier so as not to transition. The arithmetic circuit according to any one of claims 1 to 7, further comprising a plurality of cumulative adders that cumulatively add the multiplication results of the plurality of multipliers. A conversion means that performs non-linear conversion processing on the output data of each of the plurality of cumulative adders, and
A selection means for selecting and outputting either the output data of each of the plurality of cumulative adders and the output data of the conversion means, and
The arithmetic circuit according to claim 8, further comprising. An arithmetic circuit connected to a holding device that holds reference data for filter arithmetic processing and coefficient data of a filter used for the filter arithmetic processing.
A plurality of multipliers that execute the filter calculation process by repeatedly multiplying the reference data that is different from each other and the coefficient data that is common to each other.
A first data supply means for supplying the plurality of multipliers with different reference data held in the holding device.
A second data supply means for supplying the common coefficient data held in the holding device to the plurality of multipliers, and
An input means for inputting image data from the outside as the reference data, and
A detection means that detects a processing target area from the input image data and acquires pixel data of the processing target area as the reference data.
Based on the data size of the processing target area, among the plurality of multipliers, to execute the multiplication for some multipliers, controls not to execute the multiplication to the other multiplier Control means and
An arithmetic circuit characterized by having. The holding device further holds a table showing the correspondence between the number of the multipliers performing the multiplication and the data size of the processing target area.
The arithmetic circuit according to claim 10, wherein the control means refers to the table held in the holding device to determine the number of multipliers that perform the multiplication. It is an arithmetic circuit connected to a holding device that holds reference data of a plurality of times of filter arithmetic processing and coefficient data of a filter used for the filter arithmetic processing.
A plurality of multipliers that execute the filter calculation process by repeatedly multiplying the reference data that is different from each other and the coefficient data that is common to each other.
A first data supply means for supplying the plurality of multipliers with different reference data held in the holding device.
A second data supply means for supplying the common coefficient data held in the holding device to the plurality of multipliers, and
With respect to the plurality of data output as a result of the first filter calculation process, the data obtained by the pooling process or the subsampling process is acquired as reference data of the second filter calculation process and held in the holding device. Processing means to make
Based on the data size of the reference data of the second filtering operation, among the plurality of multipliers, to execute the multiplication for some multipliers, the multiplication with respect to the other multiplier Control means to control so that it is not executed,
An arithmetic circuit characterized by having. The holding device further holds a table showing the correspondence between the number of the multipliers performing the multiplication and the data size of the reference data of the second filter calculation process.
12. The arithmetic circuit according to claim 12, wherein the control means refers to the table held in the holding device to determine the number of multipliers that perform the multiplication. An image processing apparatus having the arithmetic circuit according to any one of claims 1 to 11 and processing image data as the reference data. It is a control method of an arithmetic circuit connected to a holding device that holds the reference data of the filter arithmetic processing and the coefficient data of the filter used for the filter arithmetic processing.
A multiplication step in which the filter calculation process is executed by a plurality of multipliers by repeatedly multiplying the reference data that is different from each other and the coefficient data that is common to each other.
A first supply step of supplying the plurality of multipliers with different reference data held in the holding device by the first data supply means.
A second supply step of supplying the common coefficient data held by the holding device to the plurality of multipliers by the second data supply means, and
The time for the multiplier to repeatedly execute the multiplication increases as the filter size of the filter calculation process increases, and when the filter size is equal to or less than a predetermined value, a power of a part of the plurality of multipliers is used. A control step in which a control means controls so that a multiplier is made to perform the multiplication and another multiplier is not made to perform the multiplication.
A method characterized by having. It is a control method of an arithmetic circuit connected to a holding device that holds the reference data of the filter arithmetic processing and the coefficient data of the filter used for the filter arithmetic processing.
A multiplication step in which the filter calculation process is executed by a plurality of multipliers by repeatedly multiplying the reference data that is different from each other and the coefficient data that is common to each other.
A first supply step of supplying the plurality of multipliers with different reference data held in the holding device by the first data supply means.
A second supply step of supplying the common coefficient data held by the holding device to the plurality of multipliers by the second data supply means, and
An input process in which image data is input from the outside as the reference data by an input means,
An acquisition step of detecting a processing target area from the input image data and acquiring pixel data of the processing target area as the reference data by a detection means.
Based on the data size of the processing target area, among the plurality of multipliers, to execute the multiplication for some multipliers, other control means so as not to execute the multiplication with respect to the multiplier Control process controlled by
A method characterized by having. It is a control method of an arithmetic circuit connected to a holding device that holds reference data of a plurality of filter arithmetic processes and coefficient data of a filter used for the multiple filter arithmetic processes.
A multiplication step in which the filter calculation process is executed by a plurality of multipliers by repeatedly multiplying the reference data that is different from each other and the coefficient data that is common to each other.
A first supply step of supplying the plurality of multipliers with different reference data held in the holding device by the first data supply means.
A second supply step of supplying the common coefficient data held by the holding device to the plurality of multipliers by the second data supply means, and
With respect to the data output as a result of the first filter calculation process, the data obtained by the pooling process or the subsampling process is acquired by the processing means as the reference data of the second filter calculation process, and is stored in the holding device. The acquisition process to hold and
Based on the data size of the reference data of the second filtering operation, among the plurality of multipliers, to execute the multiplication for some multipliers, the multiplication with respect to the other multiplier A control process that is controlled by control means so that it is not executed,
A method characterized by having. A control program of an arithmetic circuit connected to a holding device that holds reference data for filter arithmetic processing and coefficient data of a filter used for the filter arithmetic processing.
A multiplication step in which a plurality of multipliers execute the filter calculation process by repeatedly multiplying the reference data that is different from each other and the coefficient data that is common to each other.
A first supply step of causing the plurality of multipliers to supply the first data supply with the different reference data held in the holding device .
A second supply step of causing the second data supply means to supply the common coefficient data held by the holding device to the plurality of multipliers.
The time for the multiplier to repeatedly execute the multiplication increases as the filter size of the filter calculation process increases, and when the filter size is equal to or less than a predetermined value, a power of a part of the plurality of multipliers is used. A control step that causes an arithmetic unit to execute the multiplication and a control means to control other multipliers so that the multiplication is not executed.
A program characterized by having a computer execute. A control program of an arithmetic circuit connected to a holding device that holds reference data for filter arithmetic processing and coefficient data of a filter used for the filter arithmetic processing.
A multiplication step in which a plurality of multipliers execute the filter calculation process by repeatedly multiplying the reference data that is different from each other and the coefficient data that is common to each other.
A first supply step of causing the first data supply means to supply the plurality of multipliers with different reference data held in the holding device.
A second supply step of causing the second data supply means to supply the common coefficient data held by the holding device to the plurality of multipliers.
An input step in which image data is input to the input means from the outside as the reference data, and
An acquisition step of causing a detection means to detect a processing target area from the forced image data and acquiring pixel data of the processing target area as the reference data.
Based on the data size of the processing target area, among the plurality of multipliers, to execute the multiplication for some multipliers, other control means so as not to execute the multiplication with respect to the multiplier Control steps to be controlled by
A program characterized by having a computer execute. It is a control program of an arithmetic circuit connected to a holding device that holds reference data of a plurality of filter arithmetic processes and coefficient data of a filter used for the multiple filter arithmetic processes.
A multiplication step in which a plurality of multipliers execute the filter calculation process by repeatedly multiplying the reference data that is different from each other and the coefficient data that is common to each other.
A first supply step of causing the first data supply means to supply the plurality of multipliers with different reference data held in the holding device.
A second supply step of causing the second data supply means to supply the common coefficient data held by the holding device to the plurality of multipliers.
With respect to the data output as a result of the first filter calculation process, the processing means is made to acquire the data obtained by the pooling process or the subsampling process as the reference data of the second filter calculation process, and the holding device is used. The acquisition process to hold and
Based on the data size of the reference data of the second filtering operation, among the plurality of multipliers, to execute the multiplication for some multipliers, the multiplication with respect to the other multiplier A control step that causes the control means to control it so that it is not executed,
A program characterized by having a computer execute. Multiple multipliers that perform filter arithmetic processing,
Based on the filter size in the filter calculation process, some multipliers among the plurality of multipliers are not allowed to execute the filter calculation process, and other multipliers are not allowed to execute the filter calculation process. Control means to control
An arithmetic circuit characterized by having. Multiple multipliers that perform filter arithmetic processing,
Based on the data size of the reference data of the filter calculation process, some of the plurality of multipliers are made to execute the filter calculation process, and the other multipliers are subjected to the filter calculation process. Control means to control not to execute,
An arithmetic circuit characterized by having. The reference data is a processing target area of image data, and is
Based on the data size of the processing target area, the control means causes some of the plurality of multipliers to execute the filter calculation process, and causes the other multipliers to perform the filter calculation process. The arithmetic circuit according to claim 22, wherein the processing is controlled so as not to be executed. The arithmetic circuit according to any one of claims 1 to 14 and 21 to 23, wherein the filter arithmetic processing is a convolution arithmetic processing for feature extraction. The arithmetic circuit according to any one of claims 1 to 14 and 21 to 24, wherein the arithmetic circuit performs an arithmetic for Convolutional Neural Networks. The arithmetic circuit according to any one of claims 1 to 14 and 21 to 25, wherein the coefficient data of the filter used in the filter arithmetic processing is determined by machine learning. The control means has claims 1 to 14 and 21 to 26 that control the number of the multipliers to perform the processing among the plurality of multipliers based on the transfer time of the data used for the processing of the plurality of multipliers. The arithmetic circuit according to any one of the items. A control method for an arithmetic circuit having a plurality of multipliers for executing filter arithmetic processing.
Based on the filter size in the filter calculation process, some multipliers among the plurality of multipliers are not allowed to execute the filter calculation process, and other multipliers are not allowed to execute the filter calculation process. A method of controlling an arithmetic circuit, which is characterized in that it is controlled to. A control method for an arithmetic circuit having a plurality of multipliers for executing filter arithmetic processing.
Based on the data size of the reference data of the filter calculation process, some of the plurality of multipliers are made to execute the filter calculation process, and the other multipliers are subjected to the filter calculation process. A method of controlling an arithmetic circuit, which is characterized in that it is controlled so as not to be executed .

Images