WO2020155044A1 - 卷积计算的装置、方法、处理器和可移动设备 - Google Patents

卷积计算的装置、方法、处理器和可移动设备 Download PDF

Info

Publication number
WO2020155044A1
WO2020155044A1 PCT/CN2019/074249 CN2019074249W WO2020155044A1 WO 2020155044 A1 WO2020155044 A1 WO 2020155044A1 CN 2019074249 W CN2019074249 W CN 2019074249W WO 2020155044 A1 WO2020155044 A1 WO 2020155044A1
Authority
WO
WIPO (PCT)
Prior art keywords
unit
result
multiplying
specific
output
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2019/074249
Other languages
English (en)
French (fr)
Inventor
韩峰
杨康
谷骞
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
SZ DJI Technology Co Ltd
Original Assignee
SZ DJI Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by SZ DJI Technology Co Ltd filed Critical SZ DJI Technology Co Ltd
Priority to PCT/CN2019/074249 priority Critical patent/WO2020155044A1/zh
Priority to CN201980005258.1A priority patent/CN111279364A/zh
Publication of WO2020155044A1 publication Critical patent/WO2020155044A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
    • G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means

Definitions

  • This application relates to the field of information technology, and more specifically, to a device, method, processor, and movable device for convolution calculation.
  • CNN Convolutional Neural Network
  • the embodiments of the present application provide a device, method, processor, and movable device for convolution calculation, which can improve the efficiency of convolution calculation.
  • a device for convolution calculation including: a multiplying and adding unit array, including M rows and N columns of multiplying and adding units, and a specific multiplying and adding unit of the M rows and N columns of multiplying and adding units is used for The feature value input to the specific multiplication and addition unit is multiplied by the weight value corresponding to the specific multiplication and addition unit, and the multiplied product is added to the previous multiplication and addition output result, and the added sum is Output as the output result of the specific multiply-add unit, where the specific multiply-add unit is any one of the multiply-add units of the M rows and N columns, and the previous multiply-add output result is the The output result of the previous multiplication and addition unit of the specific multiplication and addition unit in the column where the specific multiplication and addition unit is located or zero, M and N are both positive integers; the accumulation unit array includes 1 row of N accumulation units, the N accumulations The units respectively correspond to the N columns of the multiplying and adding unit array, and a specific accumulating unit of the N
  • a method of convolution calculation including: inputting a weight value to a multiplying and adding unit array, wherein the multiplying and adding unit array includes M rows and N columns of multiplying and adding units; Input feature value; multiply the feature value input to the specific multiply-add unit and the weight value corresponding to the specific multiply-add unit by the specific multiply-add unit in the multiply-add unit of the M rows and N columns, and The multiplied product is added to the previous multiplication and addition output result, and the added sum is output as the output result of the specific multiplication and addition unit, wherein the specific multiplication and addition unit is the M rows and N columns Any multiply-add unit in the multiply-add unit, the previous multiply-add output result is the output result of the previous multiply-add unit of the specific multiply-add unit in the column where the specific multiply-add unit or zero, M and N All are positive integers; the output result of the last multiplication unit of the column corresponding to the specific accumulation unit is added to the previous accumulation output result through the specific accumulation unit in the accumulation unit
  • a processor including the convolution calculation device of the first aspect.
  • a movable device including the convolution calculation apparatus of the first aspect; or, the processor of the third aspect.
  • a computer storage medium stores program code, and the program code can be used to instruct to execute the method of the second aspect.
  • the technical solution of the embodiment of the present application can use one multiplying and adding unit to complete all calculations of the weight value corresponding to the multiplying and adding unit, which reduces data movement in the calculation process, reduces the bandwidth of input and output data, and can improve convolution. Calculation efficiency.
  • Fig. 1 is a schematic diagram of a convolution operation process of a convolutional neural network according to an embodiment of the present application.
  • Fig. 2 is a structural diagram of a technical solution applying an embodiment of the present application.
  • FIG. 3 is a schematic structural diagram of a mobile device according to an embodiment of the present application.
  • Fig. 4 is a schematic diagram of a convolution calculation device according to an embodiment of the present application.
  • Fig. 5 is a schematic diagram of a convolution calculation device according to another embodiment of the present application.
  • Fig. 6 is a schematic diagram of a convolution kernel mapping according to an embodiment of the present application.
  • Fig. 7 is a schematic diagram of a multiply-add unit of an embodiment of the present application.
  • Fig. 8 is a schematic diagram of an accumulation unit of an embodiment of the present application.
  • FIG. 9 is a schematic flowchart of a method of convolution calculation according to an embodiment of the present application.
  • the size of the sequence number of each process does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, rather than corresponding to the embodiments of the present application.
  • the implementation process constitutes any limitation.
  • Fig. 1 shows a schematic diagram of the convolution operation process of the convolutional neural network.
  • the convolution operation of the convolutional neural network will calculate a set of input weight values and a set of input feature maps (Input Feature Map, IFM) and output a set of output feature maps (Output Feature Map, OFM) ).
  • the input weight value is called a filter or convolution kernel.
  • the input feature map is the output feature map of the previous layer.
  • the output feature map is the feature map obtained after the input feature map is processed by the current layer.
  • the convolution kernel and the input and output feature maps can be represented as a multi-dimensional matrix.
  • a convolution operation of the convolutional layer of the convolutional neural network is at least part of the eigenvalues (data units) of the input feature matrix and the convolution kernel matrix.
  • the weight value performs an inner product operation.
  • the convolution operation of the convolutional layer can adopt a sliding window method, starting from the upper left corner of the input eigenvalue matrix, taking the size of the convolution kernel as the window, and sliding the window to the lower right corner of the input eigenvalue matrix in turn to produce a complete two-dimensional Output feature matrix.
  • the convolution calculation device extracts an input eigenvalue of the window size from the input eigenvalue matrix, and performs an inner product operation with the convolution kernel to generate an output eigenvalue.
  • the three-dimensional output feature matrix of the convolutional layer can be obtained.
  • Fig. 2 is a structural diagram of a technical solution applying an embodiment of the present application.
  • the system 200 may include a convolution calculation device 210 and a memory 220.
  • the memory 220 is used to store data to be processed, for example, input feature maps and weight values, and store processed data, such as output feature maps.
  • the memory 220 may be a static random access memory (Static Random Access Memory, SRAM).
  • the convolution calculation device 210 includes a Multiply Accumulate Unit (MAU) 211, an IFM input module 212, a weight value input module 213, and an OFM storage module 214.
  • the weight value input module 213 is responsible for reading the weight value from the memory 220 and sending it to the MAU 211 in a specific format.
  • the IFM input module 212 is responsible for reading the input feature map data from the memory 220 and sending it to the MAU 211 for convolution operation.
  • the MAU211 can include a systolic array and a buffer for storing intermediate calculation results. When performing the convolution operation, MAU211 first loads the weight value sent by the weight value input module 213 into the systolic array.
  • the input feature map data is sent from the IFM input module 212 to the systolic array, it is combined with the weight value loaded in advance Multiply and accumulate. If the intermediate result is buffered in the buffer in MAU 211, the systolic array output result will continue to be multiplied and accumulated with the intermediate result in the buffer again. If the result of the multiplication and accumulation is still the intermediate result of the convolution operation, it is stored in the buffer of the MAU, otherwise it is output to the lower-level module OFM storage module 214 for subsequent processing.
  • the OFM storage module 214 assembles the convolution calculation result output by the MAU 211 into the data format stored in the memory 220, and then writes it into the memory 220.
  • the weight value sharing feature of the convolutional neural network is used to reduce data movement in the calculation process, thereby reducing the bandwidth of input and output data, thereby improving the efficiency of convolution calculation.
  • the technical solutions of the embodiments of the present application can be applied to portable devices.
  • the movable device may be a drone, an unmanned boat, an autonomous vehicle or a robot, etc., which is not limited in the embodiment of the present application.
  • FIG. 3 is a schematic structural diagram of a movable device 300 according to an embodiment of the present application.
  • the movable device 300 may include a power system 310, a control system 320, a sensing system 330, and a processing system 340.
  • the power system 310 is used to provide power for the movable device 300.
  • the power system of the UAV may include an electronic governor (referred to as an ESC), a propeller, and a motor corresponding to the propeller.
  • the motor is connected between the electronic governor and the propeller, and the motor and the propeller are arranged on the corresponding arm; the electronic governor is used to receive the drive signal generated by the control system, and provide the drive current to the motor according to the drive signal to control the motor Rotating speed.
  • the motor is used to drive the propeller to rotate, thereby providing power for the flight of the drone.
  • the sensing system 330 can be used to measure the posture information of the mobile device 300, that is, the position information and state information of the mobile device 300 in space, such as three-dimensional position, three-dimensional angle, three-dimensional velocity, three-dimensional acceleration, and three-dimensional angular velocity.
  • the sensing system 330 may include, for example, at least one of sensors such as a gyroscope, an electronic compass, an inertial measurement unit (IMU), a vision sensor, a global positioning system (GPS), a barometer, and an airspeed meter.
  • sensors such as a gyroscope, an electronic compass, an inertial measurement unit (IMU), a vision sensor, a global positioning system (GPS), a barometer, and an airspeed meter.
  • the sensor system 330 can also be used to collect images, that is, the sensor system 330 includes a sensor for collecting images, such as a camera.
  • the control system 320 is used to control the movement of the movable device 300.
  • the control system 320 can control the movable device 300 according to a preset program instruction.
  • the control system 320 may control the movement of the movable device 300 according to the posture information of the movable device 300 measured by the sensor system 330.
  • the control system 320 may also control the movable device 300 according to the control signal from the remote controller.
  • the control system 320 may be a flight control system (flight control), or a control circuit in the flight control.
  • the processing system 340 can process the images collected by the sensing system 330.
  • the processing system 340 may be an image signal processing (Image Signal Processing, ISP) chip.
  • ISP Image Signal Processing
  • the processing system 340 may be the system 200 in FIG. 2, or the processing system 340 may include the system 200 in FIG. 2.
  • the movable device 300 may further include other components not shown in FIG. 3, which is not limited in the embodiment of the present application.
  • FIG. 4 shows a schematic diagram of a device 400 for convolution calculation according to an embodiment of the present application.
  • the device 400 may be the MAU 211 in FIG. 2.
  • the device 400 may include a multiplying and adding unit array 410 and an accumulating unit array 420.
  • the multiplying and adding unit array 410 includes M rows and N columns of multiplying and adding units (Multiply AccumulateCell, MC), where M and N are both positive integers.
  • M and N are both positive integers.
  • the specific multiplying and adding unit in the multiplying and adding unit of the M rows and N columns is used to multiply the feature value input to the specific multiplying and adding unit and the weight value corresponding to the specific multiplying and adding unit, and the multiplied The product of is added to the previous multiplication and addition output result, and the added sum is output as the output result of the specific multiplication and addition unit, where the specific multiplication and addition unit is the multiplication and addition unit of the M rows and N columns
  • the previous multiply-add output result is the output result of the previous multiply-add unit of the specific multiply-add unit in the column where the specific multiply-add unit or zero.
  • the previous multiply-add output result is zero, and the multiply-add unit multiplies the corresponding feature value and the weight value and outputs it downward, that is, outputs to the next multiply-add unit in the column.
  • the previous multiply-add output result is the output result of the previous multiply-add unit in the column where the multiply-add unit multiplies the corresponding feature value and the weight value, and then the previous multiply-add output result Add, and output the added sum downward.
  • the multiplying and adding unit of the last row output to the accumulating unit of the column, and for other multiplying and adding units, output to the next multiplying and adding unit of the column.
  • the accumulating cell array includes 1 row of N accumulating cells (AccumulateCell, ACC), the N accumulating cells respectively corresponding to the N columns of the multiplying cell array, and a specific accumulating cell of the N accumulating cells is used to
  • the output result of the last multiplication and addition unit of the column corresponding to the specific accumulation unit is added to the previous accumulation output result, and the added sum is output as the output result of the specific accumulation unit, wherein the previous accumulation
  • the output result is the output result of the previous accumulation unit of the specific accumulation unit or zero.
  • the previous accumulation output result is zero, and the accumulation unit only needs to output the output result of the last multiplication and accumulation unit in the column to the next accumulation unit.
  • the previous accumulation output result is the output result of the previous accumulation unit of the accumulation unit, and the accumulation unit will output the output of the last multiplication unit in the column. The result is added to the previous cumulative output result, and the added sum is output.
  • the various units or modules in the embodiments of the present application may be specifically implemented by circuits.
  • the multiplying and adding unit may be a multiplying and adding circuit, but the embodiments of the present application are not limited thereto, and they may also be implemented in other ways.
  • the apparatus 400 may further include: a weight value injection module 430 and a feature value injection module 440.
  • the weight value injection module 430 is configured to input a weight value to the multiplying and adding unit array 410;
  • the feature value injection module 440 is configured to input feature values to the multiplying and adding unit array 410.
  • the weight value injection module 440 is connected to the multiply-add unit of the first row of the multiply-add unit array 410, wherein for each column of the multiply-add unit, the weight value is transferred from the multiply-add unit of the first row to the corresponding multiply-add unit . After the weight value of each column multiplication and addition unit is transferred to the corresponding multiplication and addition unit, the weight value is latched at the same time.
  • the weight value injection module 430 and each column multiplication and addition unit of the multiplication and addition unit array 410 has only one interface (that is, the interface between the weight value injection module 430 and the first row of the multiplication and addition unit). Only one weight value can be transmitted in a period.
  • the weight input can be divided into two stages: shift and load.
  • shift phase the weight value injection module 430 sequentially sends the weight values required by the same column of multiplication and addition units to the multiplication and addition unit array 410 through the same interface.
  • the received weight values are sequentially passed down from the multiplying and adding unit at the interface.
  • the multiplying and adding units of the same column in the systolic array simultaneously load the weight values into their respective buffers.
  • the weight value injection module 430 inputs weight values for two adjacent column multiplying and adding units, there will be a delay of one clock cycle.
  • the feature value injection module 440 is connected to the multiplying and adding unit of the first column of the multiplying and adding unit array 410, wherein for each row of the multiplying and adding unit, the feature value is sequentially from the multiplying and adding unit of the first column to the next column of multiplying and adding unit transfer.
  • the feature value injection module 440 inputs feature values for two adjacent rows of multiplying and adding units, there is a delay of one clock cycle.
  • the multiplying and adding unit array 410 may correspond to the weight value of the convolution kernel in the following manner:
  • Kd ⁇ Kh ⁇ Kw convolution kernel For the Kd ⁇ Kh ⁇ Kw convolution kernel, the Kd Kh ⁇ Kw two-dimensional images expanded in the depth direction are mapped to the corresponding multiplying and adding unit along the column direction of the multiplying and adding unit array, where one weight The value corresponds to a multiplying and adding unit, Kd, Kh, and Kw are all positive integers, representing the depth, height and width of the convolution kernel respectively;
  • the multiple convolution kernels are mapped along the row direction of the multiplying and adding unit array.
  • one convolution kernel is mapped to the multiplying and adding unit array multiple times.
  • FIG. 6 shows a schematic diagram of convolution kernel mapping.
  • Kh and Kw in Figure 6 are 3 respectively.
  • Convolution kernel 0 and convolution kernel 1 are respectively expanded into Kd 3 ⁇ 3 two-dimensional graphs, and then the two-dimensional graphs obtained by convolution kernel 0 expansion are sequentially mapped along the 0-2 columns of the multiplying and adding unit array, The two-dimensional image obtained by the expansion of the convolution kernel 1 is sequentially mapped along the 3 to 5 column multiplying and adding units of the systolic array.
  • the first two-dimensional image of the convolution kernel 0 is mapped to the multiplying and adding unit of the 0th to 2nd rows of the systolic array, the Kdth two-dimensional The graph is mapped to the last three rows of multiply and add units in columns 0 to 2.
  • the first two-dimensional image of the convolution kernel 1 is mapped to the multiply-add unit of rows 0-2 of the 3rd to 5th columns of the systolic array, and the Kd-th two-dimensional image is mapped to the last three rows of the 3rd to 5th columns. unit.
  • the mapping method of the accumulating unit array is similar to that of the above multiplying unit array. As shown in FIG. 6, the convolution kernel 0 is mapped to the accumulation unit in the 0th to 2nd columns, and the convolution kernel 1 is mapped to the accumulation unit in the 3rd to 5th columns. The calculation result of convolution kernel 0 is output from the accumulation unit in the second column, and the calculation result of convolution kernel 1 is output from the accumulation unit in the fifth column.
  • Fig. 7 shows a schematic diagram of a multiplying and adding unit according to an embodiment of the present application.
  • the specific multiply-add unit may include:
  • the weight value shift register 701 is used for buffering and transferring the weight value along the column where the specific multiplying and adding unit is located, and latching the weight value corresponding to the specific multiplying and adding unit to the weight value register 702;
  • the weight value register 702 is used to buffer the weight value corresponding to the specific multiplying and adding unit
  • the characteristic value shift register 703 is used for buffering and transferring characteristic values along the row where the specific multiplying and adding unit is located, and latching the characteristic values to the characteristic value register 704;
  • the characteristic value register 704 is used for buffering characteristic values
  • the multiplication circuit 705 is configured to multiply the weight value in the weight value register 702 and the feature value in the feature value register 704, and output the multiplied product to the product register 706;
  • the product register 706 is used to buffer the product multiplied by the multiplication circuit 705;
  • the addition circuit 707 is configured to add the product in the product register 706 to the previous output result of multiplication and addition, and output the added sum down the column where the specific multiplication and addition unit is located.
  • the weight value shift register 701 is responsible for buffering the weight value sent from the weight value injection module 430 or the previous multiplication and addition unit. In the shift phase of the weight value input, the weight value buffered by the weight value shift register 701 will be passed down to the next multiplication and addition unit. During the loading phase of the weight input, the weight value buffered by the weight value shift register 701 will be latched into the weight value register 702.
  • the feature value shift register 703 is responsible for buffering the feature value data sent from the feature value injection module 440 or the multiplication and addition unit on the left. The feature value data buffered by the feature value shift register 703 will be latched into the feature value register 704, and at the same time will be sent to the right multiplying unit.
  • the multiplication circuit 705 is responsible for multiplying the weight value and the characteristic value buffered in the weight value register 702 and the characteristic value register 704, and the operation result is sent to the product register 706.
  • the addition circuit 707 is responsible for accumulating the data in the product register 706 and the previous multiplication and addition output result sent above and then transferring it downward.
  • FIG. 8 shows a schematic diagram of an accumulation unit of an embodiment of the present application.
  • the specific accumulation unit may include:
  • the filter circuit 801 is configured to filter the output result of the last multiplication and addition unit of the column corresponding to the specific accumulation unit according to the step value calculated by the convolution, and output the filtered result to the multiplication and addition unit result register 802;
  • the multiply-add unit result register 802 is used to buffer the result filtered by the filter circuit 801;
  • the delay circuit 803 is configured to delay the previous accumulation output result according to the expansion value calculated by the convolution, and output the delayed result to the accumulation unit result register 804;
  • the accumulation unit result register 804 is used to buffer the delayed result of the delay circuit 803;
  • the first-stage addition circuit 805 is configured to add the result in the multiplication and addition unit result register 802 with the result in the accumulation unit result register 804, and output the added sum to the sum register 806;
  • the sum register 806 is used to buffer the sum added by the first-stage addition circuit 805.
  • the specific accumulation unit may further include: a second-stage addition circuit 807.
  • the sum register 806 is used to output the sum in the sum register 806 to the next accumulation unit when the column corresponding to the specific accumulation unit is not the last column corresponding to the specific convolution kernel ;
  • the column corresponding to the specific accumulation unit is the last column corresponding to the specific convolution kernel, output the sum in the sum register 806 to the second-stage addition circuit 807;
  • the second-stage addition circuit 807 is configured to combine the sum in the sum register 806 with the sum in the intermediate result buffer module when the column corresponding to the specific accumulation unit is the last column corresponding to the specific convolution kernel.
  • the intermediate results of the specific convolution kernel are added, and the added sum is output to the result processing module.
  • the filtering circuit 801 can filter out the redundant result output by the multiplying and adding unit array 410 according to the parameter stride value input during the convolution calculation, and at the same time, send the filtered result to the multiplying and adding unit result register 802.
  • the effective result output by the multiplying and adding unit of each column needs to be accumulated with the effective result output by the corresponding other column.
  • the two columns output valid results may be separated by some clock cycles. Therefore, the delay circuit 803 needs to delay the result output by the left accumulation unit by a specified clock period and send it to the accumulation unit result register 804. The number of delayed clock cycles is calculated from the input parameter Dilation value during the convolution calculation.
  • the first-stage adding circuit 805 is responsible for accumulating the data buffered in the multiplying-adding unit result register 802 and the accumulating unit result register 804 and then sending it to the sum register 806.
  • the first accumulating unit does not need to receive the result output by the left accumulating unit.
  • the last accumulating unit does not output the result buffered by the sum register 806 to the right multiplying unit.
  • the two-stage addition circuit 807 accumulates the result buffered by the sum register 806 and the intermediate result read back from the intermediate result buffer module and then outputs the result to the result processing module.
  • the apparatus 400 may further include: a result processing module 450.
  • the result processing module 450 is configured to process the result output by the accumulation unit array 410.
  • the apparatus 400 may further include: an intermediate result cache module 460.
  • the result processing module 450 outputs the result output by the accumulation unit array 410 when the result output by the accumulation unit array 410 is the final result of the specific convolution kernel; in the accumulation unit When the result output by the array 410 is the intermediate result of the specific convolution kernel, the result output by the accumulation unit array 410 is buffered in the intermediate result buffer module 460.
  • a convolution kernel is mapped to the multiplying and adding unit array 410 multiple times.
  • the calculation result of the convolution kernel is composed of calculation results obtained by multiple mappings. That is to say, the calculation results obtained from the first few mappings in the multiple mappings are intermediate results, which need to be added to the calculation results obtained from the next mapping to finally obtain the final result of the convolution kernel.
  • the result processing module 450 buffers the output result to the intermediate result buffer module 460 when the result output by the accumulation unit array 410 is the intermediate result of the specific convolution kernel, so as to be added to the next output result; in the accumulation unit When the result output by the array 410 is the final result of the specific convolution kernel, the result output by the accumulation unit array 410 is output.
  • the intermediate result cache module 460 is used to cache the intermediate result of each convolution kernel.
  • the intermediate result buffer module may include: N first-in-first-out queues (First Input First Output, FIFO).
  • each Kw FIFO of the N FIFOs forms a group for buffering an intermediate result of a convolution kernel.
  • Kw FIFOs form a group to buffer the intermediate results of a convolution kernel, which can make full use of the resources of Kw FIFOs, thereby improving the utilization of FIFO.
  • the device 400 may further include a control module 470.
  • the control module 470 can be used to control the input of the weight value and the characteristic value to the multiplying and adding unit array 410 and controlling the calculation of the multiplying and adding unit array 410 and the accumulating unit array 420.
  • control module 470 can be used to control the processing of each module in the device 400 to obtain the calculation result.
  • the control module 470 may first control the weight value injection module 430 to load the weight value sent by the weight value input module into the multiplying unit array 410, and then control the feature value injection module 440 to multiply and add the feature value input sent by the IFM input module
  • the unit array 410 and the multiplying and adding unit array 410 and the accumulating unit array 420 are controlled to perform convolution operations. After all the feature map data is sent to the multiply-add unit array 410 and the convolution operation is completed, the above process is repeated in turn until all the convolution operations are completed.
  • the technical solution of the embodiment of the present application can use one multiplying and adding unit to complete all calculations of the weight value corresponding to the multiplying and adding unit, which reduces data movement in the calculation process, reduces the bandwidth of input and output data, and can improve convolution. Calculation efficiency.
  • the convolution calculation device of the embodiment of the present application is described above, and the convolution calculation method of the embodiment of the present application is described below.
  • the method of convolution calculation in the embodiment of the application is the method of the convolution calculation device of the embodiment of the application or the device including the device of the convolution calculation embodiment of the application implements the technical solution of the embodiment of the application, related description
  • the foregoing embodiments can be referred to, and for the sake of brevity, the details are not repeated here.
  • FIG. 9 shows a schematic flowchart of a method 900 for convolution calculation according to an embodiment of the present application.
  • the method 900 includes:
  • the output result output of the accumulation unit array includes 1 row of N accumulation units, the N accumulation units respectively correspond to the N columns of the multiply-add unit array, and the previous accumulation output result is the specific The output result of the previous accumulation unit of the accumulation unit or zero.
  • the inputting the weight value to the multiplying and adding unit array includes: inputting the weighting value through the multiplying and adding unit of the first row of the multiplying and adding unit array, wherein, for each column Unit, the weight value is transferred from the multiply-add unit in the first row to the corresponding multiply-add unit.
  • each column of the multiplication and addition unit simultaneously latches the weight value.
  • the inputting the feature value to the multiplying and adding unit array includes: inputting the feature value through the multiplying and adding unit of the first column of the multiplying and adding unit array, wherein for each row Adding unit, the characteristic value is transferred from the multiplying and adding unit of the first column to the multiplying and adding unit of the next column in turn.
  • the multiplying and adding unit array corresponds to the weight value of the convolution kernel in the following manner:
  • Kd ⁇ Kh ⁇ Kw convolution kernel For the Kd ⁇ Kh ⁇ Kw convolution kernel, the Kd Kh ⁇ Kw two-dimensional images expanded in the depth direction are mapped to the corresponding multiplying and adding unit along the column direction of the multiplying and adding unit array, where one weight The value corresponds to a multiplying and adding unit, Kd, Kh, and Kw are all positive integers, representing the depth, height and width of the convolution kernel respectively;
  • the multiple convolution kernels are mapped along the row direction of the multiplying and adding unit array.
  • one convolution kernel is mapped to the multiply-add unit array multiple times.
  • the method further includes: processing a result output by the accumulation unit array.
  • the processing the result output by the accumulation unit array includes: when the result output by the accumulation unit array is the final result of a specific convolution kernel, the accumulation unit The result output of the array output; when the result output by the accumulation unit array is an intermediate result of the specific convolution kernel, the result output by the accumulation unit array is buffered to the intermediate result cache module.
  • the intermediate result buffer module includes: N first-in-first-out queues FIFOs, where, for a convolution kernel with a size of Kd ⁇ Kh ⁇ Kw, each of the N FIFOs Kw FIFOs form a group for buffering the intermediate results of a convolution kernel.
  • the specific multiplying and adding unit includes: a weight value shift register, a weight value register, a characteristic value shift register, a characteristic value register, a multiplying circuit, a product register, and an adding circuit
  • the weight value is buffered by the weight value shift register and the weight value is transferred along the column where the specific multiplying and adding unit is located, and the weight value corresponding to the specific multiplying and adding unit is latched to the weight value register
  • the characteristic value shift register is buffered and the characteristic value is transferred along the row of the specific multiplying and adding unit, and the characteristic value is latched to the characteristic value register; the weight value and the total value in the weight value register are processed by the multiplication circuit.
  • the feature value in the feature value register is multiplied, and the multiplied product is output to the product register; the product in the product register is combined with the previous output result of multiplication and addition through the addition circuit Add, and output the added sum down the column where the specific multiplying and adding unit is located.
  • the specific accumulation unit includes: a filter circuit, a multiply-add unit result register, a delay circuit, an accumulation unit result register, a first-stage addition circuit, and a sum register;
  • the filtering circuit filters the output result of the last multiplication and addition unit of the column corresponding to the specific accumulation unit according to the step value calculated by the convolution, and outputs the filtered result to the multiplication and addition unit result register;
  • the delay circuit delays the previous accumulation output result according to the expansion value calculated by the convolution, and outputs the delayed result to the accumulation unit result register; the multiplication and addition are performed by the first-stage addition circuit
  • the result in the unit result register is added to the result in the accumulation unit result register, and the added sum is output to the sum register.
  • the specific accumulation unit further includes: a second-stage addition circuit; wherein, when the column corresponding to the specific accumulation unit is not the last column corresponding to the specific convolution kernel, Output the sum in the sum register to the next accumulation unit; when the column corresponding to the specific accumulation unit is the last column corresponding to the specific convolution kernel, output the sum in the sum register to the first Two-stage addition circuit; through the second-stage addition circuit, when the column corresponding to the specific accumulation unit is the last column corresponding to the specific convolution kernel, the sum in the sum register and the intermediate result are cached The intermediate results of the specific convolution kernel in the module are added.
  • a second-stage addition circuit wherein, when the column corresponding to the specific accumulation unit is not the last column corresponding to the specific convolution kernel, Output the sum in the sum register to the next accumulation unit; when the column corresponding to the specific accumulation unit is the last column corresponding to the specific convolution kernel, output the sum in the sum register to the first Two-stage addition circuit; through the second-stage addition circuit, when the column
  • the method further includes: controlling the input of the weight value and the characteristic value to the multiplying and adding unit array, and controlling the calculation of the multiplying and adding unit array and the accumulating unit array .
  • An embodiment of the present application also provides a processor, which includes the aforementioned convolution calculation apparatus of the embodiment of the present application.
  • the processor may be the convolution calculation device 210 in FIG. 2, where the MAU 211 may be the convolution calculation device in the embodiment of the application.
  • An embodiment of the present application further provides a movable device, which may include the convolution calculation apparatus of the foregoing embodiment of the present application; or, include the foregoing processor of the foregoing embodiment of the present application.
  • An embodiment of the present application also provides a computer storage medium, and the computer storage medium stores program code, and the program code may be used to instruct to perform the convolution calculation method of the foregoing embodiment of the present application.
  • the disclosed system, device, and method may be implemented in other ways.
  • the device embodiments described above are merely illustrative.
  • the division of the units is only a logical function division, and there may be other divisions in actual implementation, for example, multiple units or components may be combined or It can be integrated into another system, or some features can be ignored or not implemented.
  • the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may also be electrical, mechanical or other forms of connection.
  • the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments of the present application.
  • the functional units in the various embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit.
  • the above-mentioned integrated unit can be implemented in the form of hardware or software functional unit.
  • the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium.
  • the technical solution of this application is essentially or the part that contributes to the existing technology, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium It includes several instructions to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application.
  • the aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program code .

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • General Health & Medical Sciences (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Computational Linguistics (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Artificial Intelligence (AREA)
  • Neurology (AREA)
  • Complex Calculations (AREA)

Abstract

一种卷积计算的装置（400）、方法、处理器和可移动设备。该装置（400）包括：乘加单元阵列（410），包括M行N列的乘加单元，所述M行N列的乘加单元中的特定乘加单元用于对输入到所述特定乘加单元的特征值和所述特定乘加单元对应的权重值进行相乘，并将相乘后的积与前一乘加输出结果相加，并将相加后的和作为所述特定乘加单元的输出结果输出；累加单元阵列（420），包括1行N个累加单元，所述N个累加单元分别与所述乘加单元阵列（410）的N列对应，所述N个累加单元中的特定累加单元用于将所述特定累加单元对应列的最后一个乘加单元的输出结果与前一累加输出结果相加，并将相加后的和作为所述特定累加单元的输出结果输出。通过上述技术方案，能够提高卷积计算的效率。

Description

卷积计算的装置、方法、处理器和可移动设备
版权申明
本专利文件披露的内容包含受版权保护的材料。该版权为版权所有人所有。版权所有人不反对任何人复制专利与商标局的官方记录和档案中所存在的该专利文件或者该专利披露。
技术领域
本申请涉及信息技术领域，并且更具体地，涉及一种卷积计算的装置、方法、处理器和可移动设备。
背景技术
卷积神经网络(Convolutional Neural Network，CNN)是一种机器学习算法，它被广泛应用于目标识别、目标检测以及图像的语义分割等计算机视觉任务。
目前卷积神经网络的卷积计算过程中，或者数据处理的并行度较低，或者需要较多的数据搬移操作，导致效率较低。因此，如何提高卷积计算的效率，成为卷积神经网络设计中一个亟待解决的技术问题。
发明内容
本申请实施例提供了一种卷积计算的装置、方法、处理器和可移动设备，能够提高卷积计算的效率。
第一方面，提供了一种卷积计算的装置，包括：乘加单元阵列，包括M行N列的乘加单元，所述M行N列的乘加单元中的特定乘加单元用于对输入到所述特定乘加单元的特征值和所述特定乘加单元对应的权重值进行相乘，并将相乘后的积与前一乘加输出结果相加，并将相加后的和作为所述特定乘加单元的输出结果输出，其中，所述特定乘加单元为所述M行N列的乘加单元中的任一乘加单元，所述前一乘加输出结果为所述特定乘加单元所在列中所述特定乘加单元的前一个乘加单元的输出结果或零，M和N均为正整数；累加单元阵列，包括1行N个累加单元，所述N个累加单元分别与所述乘加单元阵列的N列对应，所述N个累加单元中的特定累加单元 用于将所述特定累加单元对应列的最后一个乘加单元的输出结果与前一累加输出结果相加，并将相加后的和作为所述特定累加单元的输出结果输出，其中，所述前一累加输出结果为所述特定累加单元的前一个累加单元的输出结果或零。
第二方面，提供了一种卷积计算的方法，包括：向乘加单元阵列输入权重值，其中，所述乘加单元阵列包括M行N列的乘加单元；向所述乘加单元阵列输入特征值；通过所述M行N列的乘加单元中的特定乘加单元对输入到所述特定乘加单元的特征值和所述特定乘加单元对应的权重值进行相乘，并将相乘后的积与前一乘加输出结果相加，并将相加后的和作为所述特定乘加单元的输出结果输出，其中，所述特定乘加单元为所述M行N列的乘加单元中的任一乘加单元，所述前一乘加输出结果为所述特定乘加单元所在列中所述特定乘加单元的前一个乘加单元的输出结果或零，M和N均为正整数；通过累加单元阵列中的特定累加单元将所述特定累加单元对应列的最后一个乘加单元的输出结果与前一累加输出结果相加，并将相加后的和作为所述特定累加单元的输出结果输出，其中，所述累加单元阵列包括1行N个累加单元，所述N个累加单元分别与所述乘加单元阵列的N列对应，所述前一累加输出结果为所述特定累加单元的前一个累加单元的输出结果或零。
第三方面，提供了一种处理器，包括第一方面的卷积计算的装置。
第四方面，提供了一种可移动设备，包括第一方面的卷积计算的装置；或者，第三方面的处理器。
第五方面，提供了一种计算机存储介质，该计算机存储介质中存储有程序代码，该程序代码可以用于指示执行上述第二方面的方法。
本申请实施例的技术方案，可以利用一个乘加单元完成该乘加单元对应的权重值的所有计算，减少了计算过程中的数据搬移，降低了输入和输出数据的带宽，从而能够提高卷积计算的效率。
附图说明
图1是本申请实施例的卷积神经网络的卷积操作过程的示意图。
图2是应用本申请实施例的技术方案的架构图。
图3是本申请实施例的可移动设备的示意性架构图。
图4是本申请一个实施例的卷积计算的装置的示意图。
图5是本申请另一个实施例的卷积计算的装置的示意图。
图6是本申请实施例的卷积核映射的示意图。
图7是本申请实施例的乘加单元的示意图。
图8是本申请实施例的累加单元的示意图。
图9是本申请实施例的卷积计算的方法的示意性流程图。
具体实施方式
下面将结合附图，对本申请实施例中的技术方案进行描述。
应理解，本文中的具体的例子只是为了帮助本领域技术人员更好地理解本申请实施例，而非限制本申请实施例的范围。
还应理解，在本申请的各种实施例中，各过程的序号的大小并不意味着执行顺序的先后，各过程的执行顺序应以其功能和内在逻辑确定，而不应对本申请实施例的实施过程构成任何限定。
还应理解，本说明书中描述的各种实施方式，既可以单独实施，也可以组合实施，本申请实施例对此并不限定。
本申请实施例的技术方案可以应用于各种深度学习算法，例如卷积神经网络，但本申请实施例对此并不限定。
图1示出了卷积神经网络的卷积操作过程的示意图。
如图1所示，卷积神经网络的卷积操作会对输入的一组权重值和一组输入特征图(Input Feature Map，IFM)进行运算后输出一组输出特征图(Output Feature Map，OFM)。输入的权重值被称为滤波器(Filter)或卷积核。输入特征图为上一层的输出特征图。输出特征图为输入特征图经过当前层运算后得到的特征图。卷积核和输入、输出特征图都可以被表示为一个多维矩阵，卷积神经网络的卷积层的一次卷积运算为输入特征矩阵的至少部分特征值(数据单元)与卷积核矩阵的权重值进行内积操作。
卷积层的卷积运算可以采用滑动窗口方式，以输入特征值矩阵的左上角为起点，以卷积核的大小为窗口，依次滑动窗口到输入特征矩阵的右下角，产生一个完整的二维输出特征矩阵。每次滑动窗口后，卷积计算装置都会从输入特征值矩阵中提取一个窗口大小的输入特征值，将其与卷积核进行内积操作，产生一个输出特征值。依照上述方式，依次产生所有的二维输出特征 矩阵后，便可得到该卷积层的三维输出特征矩阵。
图2是应用本申请实施例的技术方案的架构图。
如图2所示，系统200可以包括卷积计算装置210和存储器220。
存储器220用于存储待处理的数据，例如，输入特征图和权重值，以及存储处理后的数据，例如输出特征图。存储器220可以为静态随机存取存储器(Static Random Access Memory，SRAM)。
卷积计算装置210包括乘累加单元(Multiply Accumulate Unit，MAU)211、IFM输入模块212、权重值输入模块213和OFM存储模块214。权重值输入模块213负责从存储器220中读出权重值，并按特定格式送到MAU211。IFM输入模块212负责将输入特征图数据从存储器220中读出，并将其送到MAU 211中进行卷积运算。MAU211可以包括脉动阵列和用来存储中间计算结果的缓存。进行卷积运算时，MAU211首先将权重值输入模块213送入的权重值装载到脉动阵列，之后，当输入特征图数据从IFM输入模块212送入脉动阵列后，将其与提前装载的权重值进行乘累加。如果MAU 211中的缓存中缓存了中间结果，则脉动阵列输出结果还会继续与该缓存中的中间结果再进行一次乘累加。乘累加的结果如果仍为卷积运算的中间结果，则将其存储到MAU的缓存中，否则输出到下级模块OFM存储模块214中进行后续处理。OFM存储模块214将MAU 211输出的卷积计算结果组装为存储器220中存储的数据格式，然后将其写入存储器220。
在本申请实施例中，利用卷积神经网络权重值共享的特性，减少计算过程中的数据搬移，进而降低输入和输出数据的带宽，从而提高卷积计算的效率。
在一些实施例中，本申请实施例的技术方案可以应用于可移动设备中。该可移动设备可以是无人机、无人驾驶船、自动驾驶车辆或机器人等，本申请实施例对此并不限定。
图3是本申请实施例的可移动设备300的示意性架构图。
如图3所示，可移动设备300可以包括动力系统310、控制系统320、传感系统330和处理系统340。
动力系统310用于为该可移动设备300提供动力。
以无人机为例，无人机的动力系统可以包括电子调速器(简称为电调)、螺旋桨以及与螺旋桨相对应的电机。电机连接在电子调速器与螺旋桨之间， 电机和螺旋桨设置在对应的机臂上；电子调速器用于接收控制系统产生的驱动信号，并根据驱动信号提供驱动电流给电机，以控制电机的转速。电机用于驱动螺旋桨旋转，从而为无人机的飞行提供动力。
传感系统330可以用于测量可移动设备300的姿态信息，即可移动设备300在空间的位置信息和状态信息，例如，三维位置、三维角度、三维速度、三维加速度和三维角速度等。传感系统330例如可以包括陀螺仪、电子罗盘、惯性测量单元(Inertial Measurement Unit，IMU)、视觉传感器、全球定位系统(Global Positioning System，GPS)、气压计、空速计等传感器中的至少一种。
传感系统330还可用于采集图像，即传感系统330包括用于采集图像的传感器，例如相机等。
控制系统320用于控制可移动设备300的移动。控制系统320可以按照预先设置的程序指令对可移动设备300进行控制。例如，控制系统320可以根据传感系统330测量的可移动设备300的姿态信息控制可移动设备300的移动。控制系统320也可以根据来自遥控器的控制信号对可移动设备300进行控制。例如，对于无人机，控制系统320可以为飞行控制系统(飞控)，或者为飞控中的控制电路。
处理系统340可以处理传感系统330采集的图像。例如，处理系统340可以为图像信号处理(Image Signal Processing,ISP)类芯片。
处理系统340可以为图2中的系统200，或者，处理系统340可以包括图2中的系统200。
应理解，上述对于可移动设备300的各组成部件的划分和命名仅仅是示例性的，并不应理解为对本申请实施例的限制。
还应理解，可移动设备300还可以包括图3中未示出的其他部件，本申请实施例对此并不限定。
图4示出了本申请一个实施例的卷积计算的装置400的示意图。该装置400可以为图2中的MAU 211。
如图4所示，该装置400可以包括乘加单元阵列410和累加单元阵列420。
乘加单元阵列410，包括M行N列的乘加单元(Multiply AccumulateCell，MC)，M和N均为正整数。
所述M行N列的乘加单元中的特定乘加单元用于对输入到所述特定乘加单元的特征值和所述特定乘加单元对应的权重值进行相乘，并将相乘后的积与前一乘加输出结果相加，并将相加后的和作为所述特定乘加单元的输出结果输出，其中，所述特定乘加单元为所述M行N列的乘加单元中的任一乘加单元，所述前一乘加输出结果为所述特定乘加单元所在列中所述特定乘加单元的前一个乘加单元的输出结果或零。
例如，对于第一行的乘加单元，前一乘加输出结果为零，该乘加单元将相应特征值和权重值相乘后向下输出，即输出到所在列的下一个乘加单元。对于其他行的乘加单元，前一乘加输出结果为所在列的前一个乘加单元的输出结果，该乘加单元将相应特征值和权重值相乘后，再与前一乘加输出结果相加，将相加后的和向下输出。其中，对于最后一行的乘加单元，输出给所在列的累加单元，对于其他乘加单元，输出给所在列的下一个乘加单元。
累加单元阵列，包括1行N个累加单元(AccumulateCell，ACC)，所述N个累加单元分别与所述乘加单元阵列的N列对应，所述N个累加单元中的特定累加单元用于将所述特定累加单元对应列的最后一个乘加单元的输出结果与前一累加输出结果相加，并将相加后的和作为所述特定累加单元的输出结果输出，其中，所述前一累加输出结果为所述特定累加单元的前一个累加单元的输出结果或零。
例如，若某一累加单元为卷积核对应的第一个累加单元，则前一累加输出结果为零，该累加单元只需将所在列的最后一个乘加单元的输出结果输出给下一个累加单元。若某一累加单元不是卷积核对应的第一个累加单元，则前一累加输出结果为该累加单元的前一个累加单元的输出结果，该累加单元将所在列的最后一个乘加单元的输出结果与前一累加输出结果相加，并将相加后的和输出。
应理解，本申请实施例中的各种单元或模块具体可以由电路实现，例如，乘加单元可以为乘加电路，但本申请实施例对此并不限定，它们也可以由其他方式实现。
可选地，在本申请一个实施例中，如图5所示，该装置400还可以包括：权重值注入模块430和特征值注入模块440。
权重值注入模块430，用于向所述乘加单元阵列410输入权重值；
特征值注入模块440，用于向所述乘加单元阵列410输入特征值。
所述权重值注入模块440与所述乘加单元阵列410的第一行乘加单元连接，其中，对于每一列乘加单元，权重值从第一行的乘加单元传递到对应的乘加单元。每一列乘加单元在权重值传递到对应的乘加单元后，同时将权重值锁存。
具体而言，权重值注入模块430与乘加单元阵列410的每一列乘加单元仅有一个接口(即权重值注入模块430与第一行乘加单元之间的接口)，该接口每个时钟周期仅能传输一个权重值。权重输入可分为移位和装载两个阶段。在移位阶段，权重值注入模块430依次将同一列乘加单元需要的权重值通过同一个接口依次送入乘加单元阵列410。在乘加单元阵列410中，接收到的权重值从接口处的乘加单元依次向下传递。在装载阶段，脉动阵列中同一列的乘加单元同时将权重值装载到各自的缓存中。权重值注入模块430为相邻两列乘加单元输入权重值时会有一个时钟周期的延迟。
所述特征值注入模块440与所述乘加单元阵列410的第一列乘加单元连接，其中对于每一行乘加单元，特征值从第一列的乘加单元依次向下一列的乘加单元传递。
具体而言，特征值注入模块440与乘加单元阵列410的每一行乘加单元仅有一个接口(即特征值注入模块440与第一列乘加单元之间的接口)，该接口每个时钟周期仅能传输一个特征值。在乘加单元阵列410中，接收到的特征值从接口处的乘加单元依次向右传递直至最后一个乘加单元。特征值注入模块440为相邻两行乘加单元输入特征值时会有一个时钟周期的延迟。
可选地，在本申请一个实施例中，所述乘加单元阵列410可以按照如下方式与卷积核的权重值对应：
对于Kd×Kh×Kw大小的卷积核，在深度方向展开的Kd个Kh×Kw的二维图，沿所述乘加单元阵列的列方向映射到对应的乘加单元上，其中，一个权重值对应一个乘加单元，Kd、Kh和Kw均为正整数，分别表示卷积核的深度、高度和宽度；
多个卷积核沿所述乘加单元阵列的行方向进行映射。
可选地，在Kd×Kh>M的情况下，将一个卷积核分多次映射到所述乘加单元阵列上。
例如，图6示出了卷积核映射的示意图。图6中Kh和Kw分别为3。卷积核0和卷积核1分别展开为Kd个3×3的二维图，然后卷积核0展开得 到的二维图沿着乘加单元阵列的0～2列乘加单元依次映射，卷积核1展开得到的二维图沿着脉动阵列的3～5列乘加单元依次映射。假如乘加单元阵列的高度M等于Kd×3，则卷积核0的第一个二维图映射在脉动阵列第0～2列的第0～2行的乘加单元，第Kd个二维图映射到第0～2列的最后三行乘加单元。卷积核1的第一个二维图映射在脉动阵列第3～5列的第0～2行的乘加单元，第Kd个二维图映射到第3～5列的最后三行乘加单元。
累加单元阵列的映射方式与上方的乘加单元阵列的映射方式类似。如图6所示，卷积核0映射到第0～2列的累加单元，卷积核1映射到第3～5列的累加单元。其中卷积核0的计算结果从第2列的累加单元输出，卷积核1的计算结果从第5列的累加单元输出。
图7示出了本申请一个实施例的乘加单元的示意图。
如图7所示，特定乘加单元可以包括：
权重值移位寄存器701，用于缓存以及沿所述特定乘加单元所在列传递权重值，并将所述特定乘加单元对应的权重值锁存到权重值寄存器702；
权重值寄存器702，用于缓存所述特定乘加单元对应的权重值；
特征值移位寄存器703，用于缓存以及沿所述特定乘加单元所在行传递特征值，并将特征值锁存到特征值寄存器704；
特征值寄存器704，用于缓存特征值；
相乘电路705，用于对所述权重值寄存器702中权重值和所述特征值寄存器704中的特征值进行相乘，并将相乘后的积输出到积寄存器706；
积寄存器706，用于缓存所述相乘电路705相乘后的积；
相加电路707，用于将所述积寄存器706中的积与所述前一乘加输出结果相加，并将相加后的和沿所述特定乘加单元所在列向下输出。
具体而言，权重值移位寄存器701负责缓存从权重值注入模块430或上一个乘加单元送来的权重值。在权重值输入的移位阶段，权重值移位寄存器701缓存的权重值会向下传递到下一个乘加单元。在权重输入的装载阶段，权重值移位寄存器701缓存的权重值会被锁存到权重值寄存器702。特征值移位寄存器703负责缓存从特征值注入模块440或左边的乘加单元送来的特征值数据。特征值移位寄存器703缓存的特征值数据会被锁存到特征值寄存器704，同时还会被送到右边的乘加单元。相乘电路705负责将权重值寄存器702和特征值寄存器704缓存的权重值和特征值进行乘法操作，操作结果 被送到积寄存器706中。相加电路707负责将积寄存器706中的数据和上面送入的前一乘加输出结果再次累加后向下传递。
图8示出了本申请一个实施例的累加单元的示意图。
如图8所示，特定累加单元可以包括：
过滤电路801，用于根据卷积计算的步长值，过滤所述特定累加单元对应列的最后一个乘加单元的输出结果，并将过滤后的结果输出到乘加单元结果寄存器802；
乘加单元结果寄存器802，用于缓存所述过滤电路801过滤后的结果；
延迟电路803，用于根据卷积计算的膨胀值对所述前一累加输出结果进行延迟，并将延迟后的结果输出到累加单元结果寄存器804；
累加单元结果寄存器804，用于缓存所述延迟电路803延迟后的结果；
第一阶段相加电路805，用于将所述乘加单元结果寄存器802中的结果与所述累加单元结果寄存器804中的结果相加，并将相加后的和输出到和寄存器806；
和寄存器806，用于缓存所述第一阶段相加电路805相加后的和。
可选地，所述特定累加单元还可以包括：第二阶段相加电路807。
在这种情况下，所述和寄存器806，用于在所述特定累加单元对应的列不为特定卷积核对应的最后一列时，将所述和寄存器806中的和输出到下一个累加单元；在所述特定累加单元对应的列为所述特定卷积核对应的最后一列时，将所述和寄存器806中的和输出到所述第二阶段相加电路807；
所述第二阶段相加电路807，用于在所述特定累加单元对应的列为所述特定卷积核对应的最后一列时，将所述和寄存器806中的和与中间结果缓存模块中的所述特定卷积核的中间结果相加，并将相加后的和输出到结果处理模块。
具体而言，乘加单元计算的结果中会存在冗余的结果(无效的结果)。以图6中第1列乘加单元为例，该列乘加单元中的权重值与特征值矩阵中的第1列的特征值的乘积为有效的结果，而与特征值矩阵中的第0列的特征值的乘积为冗余的结果，需要过滤掉。因此，过滤电路801可以根据卷积计算时输入的参数步长(Stride)值过滤掉乘加单元阵列410输出的冗余的结果，同时，将过滤后的结果送入乘加单元结果寄存器802。
另一方面，每列乘加单元输出的有效的结果需要与相应的另一列输出 的有效的结果累加。而这两列输出有效的结果的时间可能间隔一些时钟周期。因此，延迟电路803需要将左边累加单元输出的结果延迟指定时钟周期后送入累加单元结果寄存器804。延迟的时钟周期数由卷积计算时输入的参数膨胀(Dilation)值计算得到。
第一阶段相加电路805负责将乘加单元结果寄存器802和累加单元结果寄存器804中缓存的数据进行累加后送入和寄存器806。
在将卷积运算的卷积核映射时，连续Kw个累加单元会映射到同一个卷积核，Kw的大小和卷积核的宽度相同。Kw个累加单元中，第一个累加单元不需要接收左边累加单元输出的结果，同时，最后一个累加单元也不会将和寄存器806缓存的结果输出到右边的乘加单元，它只会在第二阶段相加电路807中将和寄存器806缓存的结果与从中间结果缓存模块中读回的中间结果累加后输出到结果处理模块。
可选地，在本申请一个实施例中，如图5所示，该装置400还可以包括：结果处理模块450。
结果处理模块450，用于处理所述累加单元阵列410输出的结果。
可选地，该装置400还可以包括：中间结果缓存模块460。
在这种情况下，所述结果处理模块450，在所述累加单元阵列410输出的结果为特定卷积核的最终结果时，将所述累加单元阵列410输出的结果输出；在所述累加单元阵列410输出的结果为所述特定卷积核的中间结果时，将所述累加单元阵列410输出的结果缓存到所述中间结果缓存模块460。
具体而言，在Kd×Kh>M的情况下，一个卷积核分多次映射到所述乘加单元阵列410上。在这种情况下，该卷积核的计算结果由多次映射得到的计算结果组成。也就是说，多次映射中的前几次映射得到的计算结果为中间结果，要与下一次映射得到的计算结果相加，最终得到该卷积核的最终结果。因此，结果处理模块450在累加单元阵列410输出的结果为特定卷积核的中间结果时，将输出的结果缓存到中间结果缓存模块460，以便于与下一次输出的结果相加；在累加单元阵列410输出的结果为特定卷积核的最终结果时，将累加单元阵列410输出的结果输出。
中间结果缓存模块460用于缓存每个卷积核的中间结果。可选地，中间结果缓存模块可以包括：N个先入先出队列(First Input First Output，FIFO)。
对于Kd×Kh×Kw大小的卷积核，所述N个FIFO中的每Kw个FIFO 组成一组，用于缓存一个卷积核的中间结果。
Kw个FIFO组成一组缓存一个卷积核的中间结果，可以充分利用Kw个FIFO的资源，从而可以提高FIFO的利用率。
可选地，在本申请一个实施例中，如图5所示，该装置400还可以包括：控制模块470。
控制模块470可用于控制权重值和特征值到所述乘加单元阵列410的输入，以及控制所述乘加单元阵列410和所述累加单元阵列420的计算。
具体而言，控制模块470可用于控制装置400中各模块的处理，以得到计算结果。例如，控制模块470可以首先控制权重值注入模块430将权重值输入模块送入的权重值装载到乘加单元阵列410，然后控制特征值注入模块440将IFM输入模块送入的特征值输入乘加单元阵列410，并控制乘加单元阵列410和累加单元阵列420进行卷积运算。待所有的特征图数据送入乘加单元阵列410并完成卷积运算后，再依次重复上述过程，直至完成所有的卷积运算。
本申请实施例的技术方案，可以利用一个乘加单元完成该乘加单元对应的权重值的所有计算，减少了计算过程中的数据搬移，降低了输入和输出数据的带宽，从而能够提高卷积计算的效率。
以上描述了本申请实施例的卷积计算的装置，下面描述本申请实施例的卷积计算的方法。本申请实施例的卷积计算的方法为前述本申请实施例的卷积计算的装置或包括本申请实施例的卷积计算的装置的设备实施本申请实施例的技术方案时的方法，相关描述可以参考前述实施例，以下为了简洁，在此不再赘述。
图9示出了本申请实施例的卷积计算的方法900的示意性流程图。
如图9所示，所述方法900包括：
910，向乘加单元阵列输入权重值，其中，所述乘加单元阵列包括M行N列的乘加单元；
920，向所述乘加单元阵列输入特征值；
930，通过所述M行N列的乘加单元中的特定乘加单元对输入到所述特定乘加单元的特征值和所述特定乘加单元对应的权重值进行相乘，并将相乘后的积与前一乘加输出结果相加，并将相加后的和作为所述特定乘加 单元的输出结果输出，其中，所述特定乘加单元为所述M行N列的乘加单元中的任一乘加单元，所述前一乘加输出结果为所述特定乘加单元所在列中所述特定乘加单元的前一个乘加单元的输出结果或零，M和N均为正整数；
940，通过累加单元阵列中的特定累加单元将所述特定累加单元对应列的最后一个乘加单元的输出结果与前一累加输出结果相加，并将相加后的和作为所述特定累加单元的输出结果输出，其中，所述累加单元阵列包括1行N个累加单元，所述N个累加单元分别与所述乘加单元阵列的N列对应，所述前一累加输出结果为所述特定累加单元的前一个累加单元的输出结果或零。
可选地，在本申请一个实施例中，所述向乘加单元阵列输入权重值，包括：通过所述乘加单元阵列的第一行乘加单元输入权重值，其中，对于每一列乘加单元，权重值从第一行的乘加单元传递到对应的乘加单元。
可选地，在本申请一个实施例中，在权重值传递到对应的乘加单元后，每一列乘加单元同时将权重值锁存。
可选地，在本申请一个实施例中，所述向所述乘加单元阵列输入特征值，包括：通过所述乘加单元阵列的第一列乘加单元输入特征值，其中对于每一行乘加单元，特征值从第一列的乘加单元依次向下一列的乘加单元传递。
可选地，在本申请一个实施例中，所述乘加单元阵列按照如下方式与卷积核的权重值对应：
对于Kd×Kh×Kw大小的卷积核，在深度方向展开的Kd个Kh×Kw的二维图，沿所述乘加单元阵列的列方向映射到对应的乘加单元上，其中，一个权重值对应一个乘加单元，Kd、Kh和Kw均为正整数，分别表示卷积核的深度、高度和宽度；
多个卷积核沿所述乘加单元阵列的行方向进行映射。
可选地，在本申请一个实施例中，在Kd×Kh>M的情况下，将一个卷积核分多次映射到所述乘加单元阵列上。
可选地，在本申请一个实施例中，所述方法还包括：处理所述累加单元阵列输出的结果。
可选地，在本申请一个实施例中，所述处理所述累加单元阵列输出的结果，包括：在所述累加单元阵列输出的结果为特定卷积核的最终结果 时，将所述累加单元阵列输出的结果输出；在所述累加单元阵列输出的结果为所述特定卷积核的中间结果时，将所述累加单元阵列输出的结果缓存到中间结果缓存模块。
可选地，在本申请一个实施例中，所述中间结果缓存模块包括：N个先入先出队列FIFO，其中，对于Kd×Kh×Kw大小的卷积核，所述N个FIFO中的每Kw个FIFO组成一组，用于缓存一个卷积核的中间结果。
可选地，在本申请一个实施例中，所述特定乘加单元包括：权重值移位寄存器、权重值寄存器、特征值移位寄存器、特征值寄存器、相乘电路、积寄存器和相加电路；其中，通过所述权重值移位寄存器缓存以及沿所述特定乘加单元所在列传递权重值，并将所述特定乘加单元对应的权重值锁存到所述权重值寄存器；通过所述特征值移位寄存器缓存以及沿所述特定乘加单元所在行传递特征值，并将特征值锁存到所述特征值寄存器；通过所述相乘电路对所述权重值寄存器中权重值和所述特征值寄存器中的特征值进行相乘，并将相乘后的积输出到所述积寄存器；通过所述相加电路将所述积寄存器中的积与所述前一乘加输出结果相加，并将相加后的和沿所述特定乘加单元所在列向下输出。
可选地，在本申请一个实施例中，所述特定累加单元包括：过滤电路、乘加单元结果寄存器、延迟电路、累加单元结果寄存器、第一阶段相加电路和和寄存器；其中，通过所述过滤电路，根据卷积计算的步长值，过滤所述特定累加单元对应列的最后一个乘加单元的输出结果，并将过滤后的结果输出到所述乘加单元结果寄存器；通过所述延迟电路，根据卷积计算的膨胀值对所述前一累加输出结果进行延迟，并将延迟后的结果输出到所述累加单元结果寄存器；通过所述第一阶段相加电路将所述乘加单元结果寄存器中的结果与所述累加单元结果寄存器中的结果相加，并将相加后的和输出到所述和寄存器。
可选地，在本申请一个实施例中，所述特定累加单元还包括：第二阶段相加电路；其中，在所述特定累加单元对应的列不为特定卷积核对应的最后一列时，将所述和寄存器中的和输出到下一个累加单元；在所述特定累加单元对应的列为所述特定卷积核对应的最后一列时，将所述和寄存器中的和输出到所述第二阶段相加电路；通过所述第二阶段相加电路，在所述特定累加单元对应的列为所述特定卷积核对应的最后一列时，将所述和寄存器 中的和与中间结果缓存模块中的所述特定卷积核的中间结果相加。
可选地，在本申请一个实施例中，所述方法还包括：控制权重值和特征值到所述乘加单元阵列的输入，以及控制所述乘加单元阵列和所述累加单元阵列的计算。
本申请实施例还提供了一种处理器，该处理器包括前述本申请实施例的卷积计算的装置。
例如，该处理器可以为图2中的卷积计算装置210，其中，MAU 211可以为本申请实施例的卷积计算的装置。
本申请实施例还提供了一种可移动设备，该可移动设备可以包括上述本申请实施例的卷积计算的装置；或者，包括上述本申请实施例的处理器。
本申请实施例还提供了一种计算机存储介质，该计算机存储介质中存储有程序代码，该程序代码可以用于指示执行上述本申请实施例的卷积计算的方法。
本领域普通技术人员可以意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，能够以电子硬件、计算机软件或者二者的结合来实现，为了清楚地说明硬件和软件的可互换性，在上述说明中已经按照功能一般性地描述了各示例的组成及步骤。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
在本申请所提供的几个实施例中，应该理解到，所揭露的系统、装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另外，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口、装置或单元的间接耦合或通信连接，也可以是电的，机械的或其它的形式连接。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的 部分或者全部单元来实现本申请实施例方案的目的。
另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以是两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。
所述集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分，或者该技术方案的全部或部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(ROM，Read-Only Memory)、随机存取存储器(RAM，Random Access Memory)、磁碟或者光盘等各种可以存储程序代码的介质。
以上所述，仅为本申请的具体实施方式，但本申请的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本申请揭露的技术范围内，可轻易想到各种等效的修改或替换，这些修改或替换都应涵盖在本申请的保护范围之内。因此，本申请的保护范围应以权利要求的保护范围为准。

Claims (29)

  1. 一种卷积计算的装置，其特征在于，包括：
    乘加单元阵列，包括M行N列的乘加单元，所述M行N列的乘加单元中的特定乘加单元用于对输入到所述特定乘加单元的特征值和所述特定乘加单元对应的权重值进行相乘，并将相乘后的积与前一乘加输出结果相加，并将相加后的和作为所述特定乘加单元的输出结果输出，其中，所述特定乘加单元为所述M行N列的乘加单元中的任一乘加单元，所述前一乘加输出结果为所述特定乘加单元所在列中所述特定乘加单元的前一个乘加单元的输出结果或零，M和N均为正整数；
    累加单元阵列，包括1行N个累加单元，所述N个累加单元分别与所述乘加单元阵列的N列对应，所述N个累加单元中的特定累加单元用于将所述特定累加单元对应列的最后一个乘加单元的输出结果与前一累加输出结果相加，并将相加后的和作为所述特定累加单元的输出结果输出，其中，所述前一累加输出结果为所述特定累加单元的前一个累加单元的输出结果或零。
  2. 根据权利要求1所述的装置，其特征在于，所述装置还包括：
    权重值注入模块，用于向所述乘加单元阵列输入权重值；
    特征值注入模块，用于向所述乘加单元阵列输入特征值。
  3. 根据权利要求2所述的装置，其特征在于，所述权重值注入模块与所述乘加单元阵列的第一行乘加单元连接，其中，对于每一列乘加单元，权重值从第一行的乘加单元传递到对应的乘加单元。
  4. 根据权利要求3所述的装置，其特征在于，每一列乘加单元用于在权重值传递到对应的乘加单元后，同时将权重值锁存。
  5. 根据权利要求2至4中任一项所述的装置，其特征在于，所述特征值注入模块与所述乘加单元阵列的第一列乘加单元连接，其中对于每一行乘加单元，特征值从第一列的乘加单元依次向下一列的乘加单元传递。
  6. 根据权利要求1至5中任一项所述的装置，其特征在于，所述乘加单元阵列用于按照如下方式与卷积核的权重值对应：
    对于Kd×Kh×Kw大小的卷积核，在深度方向展开的Kd个Kh×Kw的二维图，沿所述乘加单元阵列的列方向映射到对应的乘加单元上，其中，一个权重值对应一个乘加单元，Kd、Kh和Kw均为正整数，分别表示卷积核 的深度、高度和宽度；
    多个卷积核沿所述乘加单元阵列的行方向进行映射。
  7. 根据权利要求6所述的装置，其特征在于，在Kd×Kh>M的情况下，将一个卷积核分多次映射到所述乘加单元阵列上。
  8. 根据权利要求1至7中任一项所述的装置，其特征在于，所述装置还包括：
    结果处理模块，用于处理所述累加单元阵列输出的结果。
  9. 根据权利要求8所述的装置，其特征在于，所述装置还包括：
    中间结果缓存模块；
    其中，所述结果处理模块用于，在所述累加单元阵列输出的结果为特定卷积核的最终结果时，将所述累加单元阵列输出的结果输出；在所述累加单元阵列输出的结果为所述特定卷积核的中间结果时，将所述累加单元阵列输出的结果缓存到所述中间结果缓存模块。
  10. 根据权利要求9所述的装置，其特征在于，所述中间结果缓存模块包括：
    N个先入先出队列FIFO，其中，对于Kd×Kh×Kw大小的卷积核，所述N个FIFO中的每Kw个FIFO组成一组，用于缓存一个卷积核的中间结果。
  11. 根据权利要求1至10中任一项所述的装置，其特征在于，所述特定乘加单元包括：
    权重值移位寄存器，用于缓存以及沿所述特定乘加单元所在列传递权重值，并将所述特定乘加单元对应的权重值锁存到权重值寄存器；
    权重值寄存器，用于缓存所述特定乘加单元对应的权重值；
    特征值移位寄存器，用于缓存以及沿所述特定乘加单元所在行传递特征值，并将特征值锁存到特征值寄存器；
    特征值寄存器，用于缓存特征值；
    相乘电路，用于对所述权重值寄存器中权重值和所述特征值寄存器中的特征值进行相乘，并将相乘后的积输出到积寄存器；
    积寄存器，用于缓存所述相乘电路相乘后的积；
    相加电路，用于将所述积寄存器中的积与所述前一乘加输出结果相加，并将相加后的和沿所述特定乘加单元所在列向下输出。
  12. 根据权利要求1至11中任一项所述的装置，其特征在于，所述特定累加单元包括：
    过滤电路，用于根据卷积计算的步长值，过滤所述特定累加单元对应列的最后一个乘加单元的输出结果，并将过滤后的结果输出到乘加单元结果寄存器；
    乘加单元结果寄存器，用于缓存所述过滤电路过滤后的结果；
    延迟电路，用于根据卷积计算的膨胀值对所述前一累加输出结果进行延迟，并将延迟后的结果输出到累加单元结果寄存器；
    累加单元结果寄存器，用于缓存所述延迟电路延迟后的结果；
    第一阶段相加电路，用于将所述乘加单元结果寄存器中的结果与所述累加单元结果寄存器中的结果相加，并将相加后的和输出到和寄存器；
    和寄存器，用于缓存所述第一阶段相加电路相加后的和。
  13. 根据权利要求12所述的装置，其特征在于，所述特定累加单元还包括：
    第二阶段相加电路；
    其中，所述和寄存器，用于在所述特定累加单元对应的列不为特定卷积核对应的最后一列时，将所述和寄存器中的和输出到下一个累加单元；在所述特定累加单元对应的列为所述特定卷积核对应的最后一列时，将所述和寄存器中的和输出到所述第二阶段相加电路；
    所述第二阶段相加电路，用于在所述特定累加单元对应的列为所述特定卷积核对应的最后一列时，将所述和寄存器中的和与中间结果缓存模块中的所述特定卷积核的中间结果相加，并将相加后的和输出到结果处理模块。
  14. 根据权利要求1至13中任一项所述的装置，其特征在于，所述装置还包括：
    控制模块，用于控制权重值和特征值到所述乘加单元阵列的输入，以及控制所述乘加单元阵列和所述累加单元阵列的计算。
  15. 一种卷积计算的方法，其特征在于，包括：
    向乘加单元阵列输入权重值，其中，所述乘加单元阵列包括M行N列的乘加单元；
    向所述乘加单元阵列输入特征值；
    通过所述M行N列的乘加单元中的特定乘加单元对输入到所述特定乘 加单元的特征值和所述特定乘加单元对应的权重值进行相乘，并将相乘后的积与前一乘加输出结果相加，并将相加后的和作为所述特定乘加单元的输出结果输出，其中，所述特定乘加单元为所述M行N列的乘加单元中的任一乘加单元，所述前一乘加输出结果为所述特定乘加单元所在列中所述特定乘加单元的前一个乘加单元的输出结果或零，M和N均为正整数；
    通过累加单元阵列中的特定累加单元将所述特定累加单元对应列的最后一个乘加单元的输出结果与前一累加输出结果相加，并将相加后的和作为所述特定累加单元的输出结果输出，其中，所述累加单元阵列包括1行N个累加单元，所述N个累加单元分别与所述乘加单元阵列的N列对应，所述前一累加输出结果为所述特定累加单元的前一个累加单元的输出结果或零。
  16. 根据权利要求15所述的方法，其特征在于，所述向乘加单元阵列输入权重值，包括：
    通过所述乘加单元阵列的第一行乘加单元输入权重值，其中，对于每一列乘加单元，权重值从第一行的乘加单元传递到对应的乘加单元。
  17. 根据权利要求16所述的方法，其特征在于，在权重值传递到对应的乘加单元后，每一列乘加单元同时将权重值锁存。
  18. 根据权利要求15至17中任一项所述的方法，其特征在于，所述向所述乘加单元阵列输入特征值，包括：
    通过所述乘加单元阵列的第一列乘加单元输入特征值，其中对于每一行乘加单元，特征值从第一列的乘加单元依次向下一列的乘加单元传递。
  19. 根据权利要求15至18中任一项所述的方法，其特征在于，所述乘加单元阵列按照如下方式与卷积核的权重值对应：
    对于Kd×Kh×Kw大小的卷积核，在深度方向展开的Kd个Kh×Kw的二维图，沿所述乘加单元阵列的列方向映射到对应的乘加单元上，其中，一个权重值对应一个乘加单元，Kd、Kh和Kw均为正整数，分别表示卷积核的深度、高度和宽度；
    多个卷积核沿所述乘加单元阵列的行方向进行映射。
  20. 根据权利要求19所述的方法，其特征在于，在Kd×Kh>M的情况下，将一个卷积核分多次映射到所述乘加单元阵列上。
  21. 根据权利要求15至20中任一项所述的方法，其特征在于，所述方法还包括：
    处理所述累加单元阵列输出的结果。
  22. 根据权利要求21所述的方法，其特征在于，所述处理所述累加单元阵列输出的结果，包括：
    在所述累加单元阵列输出的结果为特定卷积核的最终结果时，将所述累加单元阵列输出的结果输出；在所述累加单元阵列输出的结果为所述特定卷积核的中间结果时，将所述累加单元阵列输出的结果缓存到中间结果缓存模块。
  23. 根据权利要求22所述的方法，其特征在于，所述中间结果缓存模块包括：
    N个先入先出队列FIFO，其中，对于Kd×Kh×Kw大小的卷积核，所述N个FIFO中的每Kw个FIFO组成一组，用于缓存一个卷积核的中间结果。
  24. 根据权利要求15至23中任一项所述的方法，其特征在于，所述特定乘加单元包括：权重值移位寄存器、权重值寄存器、特征值移位寄存器、特征值寄存器、相乘电路、积寄存器和相加电路；
    其中，通过所述权重值移位寄存器缓存以及沿所述特定乘加单元所在列传递权重值，并将所述特定乘加单元对应的权重值锁存到所述权重值寄存器；
    通过所述特征值移位寄存器缓存以及沿所述特定乘加单元所在行传递特征值，并将特征值锁存到所述特征值寄存器；
    通过所述相乘电路对所述权重值寄存器中权重值和所述特征值寄存器中的特征值进行相乘，并将相乘后的积输出到所述积寄存器；
    通过所述相加电路将所述积寄存器中的积与所述前一乘加输出结果相加，并将相加后的和沿所述特定乘加单元所在列向下输出。
  25. 根据权利要求15至24中任一项所述的方法，其特征在于，所述特定累加单元包括：过滤电路、乘加单元结果寄存器、延迟电路、累加单元结果寄存器、第一阶段相加电路和和寄存器；
    其中，通过所述过滤电路，根据卷积计算的步长值，过滤所述特定累加单元对应列的最后一个乘加单元的输出结果，并将过滤后的结果输出到所述乘加单元结果寄存器；
    通过所述延迟电路，根据卷积计算的膨胀值对所述前一累加输出结果进行延迟，并将延迟后的结果输出到所述累加单元结果寄存器；
    通过所述第一阶段相加电路将所述乘加单元结果寄存器中的结果与所述累加单元结果寄存器中的结果相加，并将相加后的和输出到所述和寄存器。
  26. 根据权利要求25所述的方法，其特征在于，所述特定累加单元还包括：第二阶段相加电路；
    其中，在所述特定累加单元对应的列不为特定卷积核对应的最后一列时，将所述和寄存器中的和输出到下一个累加单元；在所述特定累加单元对应的列为所述特定卷积核对应的最后一列时，将所述和寄存器中的和输出到所述第二阶段相加电路；
    通过所述第二阶段相加电路，在所述特定累加单元对应的列为所述特定卷积核对应的最后一列时，将所述和寄存器中的和与中间结果缓存模块中的所述特定卷积核的中间结果相加。
  27. 根据权利要求15至26中任一项所述的方法，其特征在于，所述方法还包括：
    控制权重值和特征值到所述乘加单元阵列的输入，以及控制所述乘加单元阵列和所述累加单元阵列的计算。
  28. 一种处理器，其特征在于，包括根据权利要求1至14中任一项所述的卷积计算的装置。
  29. 一种可移动设备，其特征在于，包括：
    根据权利要求1至14中任一项所述的卷积计算的装置；或者，
    根据权利要求28所述的处理器。
PCT/CN2019/074249 2019-01-31 2019-01-31 卷积计算的装置、方法、处理器和可移动设备 Ceased WO2020155044A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
PCT/CN2019/074249 WO2020155044A1 (zh) 2019-01-31 2019-01-31 卷积计算的装置、方法、处理器和可移动设备
CN201980005258.1A CN111279364A (zh) 2019-01-31 2019-01-31 卷积计算的装置、方法、处理器和可移动设备

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2019/074249 WO2020155044A1 (zh) 2019-01-31 2019-01-31 卷积计算的装置、方法、处理器和可移动设备

Publications (1)

Publication Number Publication Date
WO2020155044A1 true WO2020155044A1 (zh) 2020-08-06

Family

ID=71002822

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/074249 Ceased WO2020155044A1 (zh) 2019-01-31 2019-01-31 卷积计算的装置、方法、处理器和可移动设备

Country Status (2)

Country Link
CN (1) CN111279364A (zh)
WO (1) WO2020155044A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112836803A (zh) * 2021-02-04 2021-05-25 珠海亿智电子科技有限公司 一种提高卷积运算效率的数据摆放方法
CN114120082A (zh) * 2021-11-23 2022-03-01 西南交通大学 一种图像加速卷积计算方法、系统、设备及可读存储介质
CN116384451A (zh) * 2022-12-27 2023-07-04 中科南京智能技术研究院 一种适用于全连接层的脉动阵列单元及其脉动阵列结构

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116009813B (zh) * 2021-10-20 2026-04-03 Oppo广东移动通信有限公司 一种数据处理方法及装置、存储介质
CN116415629A (zh) * 2021-12-31 2023-07-11 北京思丰可科技有限公司 一种卷积神经网络处理方法及装置

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160328838A1 (en) * 2015-05-01 2016-11-10 Applied Research LLC. Automatic target recognition system with online machine learning capability
CN107918794A (zh) * 2017-11-15 2018-04-17 中国科学院计算技术研究所 基于计算阵列的神经网络处理器
CN108520297A (zh) * 2018-04-02 2018-09-11 周军 可编程深度神经网络处理器
CN108537330A (zh) * 2018-03-09 2018-09-14 中国科学院自动化研究所 应用于神经网络的卷积计算装置及方法
CN109284821A (zh) * 2017-07-19 2019-01-29 华为技术有限公司 一种神经网络运算装置

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109997132B (zh) * 2017-05-17 2023-05-23 谷歌有限责任公司 低时延矩阵乘法部件

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160328838A1 (en) * 2015-05-01 2016-11-10 Applied Research LLC. Automatic target recognition system with online machine learning capability
CN109284821A (zh) * 2017-07-19 2019-01-29 华为技术有限公司 一种神经网络运算装置
CN107918794A (zh) * 2017-11-15 2018-04-17 中国科学院计算技术研究所 基于计算阵列的神经网络处理器
CN108537330A (zh) * 2018-03-09 2018-09-14 中国科学院自动化研究所 应用于神经网络的卷积计算装置及方法
CN108520297A (zh) * 2018-04-02 2018-09-11 周军 可编程深度神经网络处理器

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112836803A (zh) * 2021-02-04 2021-05-25 珠海亿智电子科技有限公司 一种提高卷积运算效率的数据摆放方法
CN114120082A (zh) * 2021-11-23 2022-03-01 西南交通大学 一种图像加速卷积计算方法、系统、设备及可读存储介质
CN116384451A (zh) * 2022-12-27 2023-07-04 中科南京智能技术研究院 一种适用于全连接层的脉动阵列单元及其脉动阵列结构
CN116384451B (zh) * 2022-12-27 2025-08-26 中科南京智能技术研究院 一种适用于全连接层的脉动阵列单元及其脉动阵列结构

Also Published As

Publication number Publication date
CN111279364A (zh) 2020-06-12

Similar Documents

Publication Publication Date Title
TWI834729B (zh) 神經網路處理器及其卷積操作方法
CN111915001B (zh) 卷积计算引擎、人工智能芯片以及数据处理方法
EP4379608B1 (en) Semiconductor memory device employing processing in memory (pim) and method of operating the semiconductor memory device
CN111279364A (zh) 卷积计算的装置、方法、处理器和可移动设备
CN110020723B (zh) 神经网络处理单元及包括该神经网络处理单元的片上系统
WO2020019174A1 (zh) 数据存取的方法、处理器、计算机系统和可移动设备
KR102499396B1 (ko) 뉴럴 네트워크 장치 및 뉴럴 네트워크 장치의 동작 방법
US12198053B2 (en) Integrated circuit that extracts data, neural network processor including the integrated circuit, and neural network device
US11024005B2 (en) Optical flow tracking device and method
US20160267349A1 (en) Methods and systems for generating enhanced images using multi-frame processing
CN108629406B (zh) 用于卷积神经网络的运算装置
CN112005251B (zh) 运算处理装置
CN108496188A (zh) 神经网络训练的方法、装置、计算机系统和可移动设备
WO2021102946A1 (zh) 计算装置、方法、处理器和可移动设备
CN108717571B (zh) 一种用于人工智能的加速方法和装置
WO2021232422A1 (zh) 神经网络的运算装置及其控制方法
US20210311703A1 (en) Neural network device for neural network operation, operating method of the neural network device, and application processor including the same
CN111932616A (zh) 一种利用并行计算加速的双目视觉惯性里程计方法
CN115294280A (zh) 三维重建方法、装置、设备、存储介质和程序产品
CN118982453B (zh) 一种适用于资源受限环境的slam硬件加速架构
WO2024230476A9 (zh) 视觉惯性融合定位方法及设备
WO2019019013A1 (zh) 处理图像的方法、芯片、处理器、系统和可移动设备
JP2017010255A (ja) 画像認識装置および画像認識方法
Bevara et al. VLSI implementation of high throughput parallel pipeline median finder for IoT applications
WO2022017129A1 (zh) 目标对象检测方法、装置、电子设备及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19912995

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19912995

Country of ref document: EP

Kind code of ref document: A1