US20220129741A1 - Implementation of a neural network in multicore hardware - Google Patents

Implementation of a neural network in multicore hardware Download PDF

Info

Publication number
US20220129741A1
US20220129741A1 US17/500,415 US202117500415A US2022129741A1 US 20220129741 A1 US20220129741 A1 US 20220129741A1 US 202117500415 A US202117500415 A US 202117500415A US 2022129741 A1 US2022129741 A1 US 2022129741A1
Authority
US
United States
Prior art keywords
layer
layer group
pass
data
core
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
US17/500,415
Other languages
English (en)
Inventor
Xiran Huang
Fernando Escobar
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Imagination Technologies Ltd
Original Assignee
Imagination Technologies Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from GB2016226.9A external-priority patent/GB2599910B/en
Priority claimed from GB2016225.1A external-priority patent/GB2599909B/en
Priority claimed from GB2016227.7A external-priority patent/GB2599911B/en
Application filed by Imagination Technologies Ltd filed Critical Imagination Technologies Ltd
Assigned to IMAGINATION TECHNOLOGIES LIMITED reassignment IMAGINATION TECHNOLOGIES LIMITED ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: Huang, Xiran, ESCOBAR, Fernando
Publication of US20220129741A1 publication Critical patent/US20220129741A1/en
Assigned to FORTRESS INVESTMENT GROUP (UK) LTD reassignment FORTRESS INVESTMENT GROUP (UK) LTD SECURITY INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: IMAGINATION TECHNOLOGIES LIMITED
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
    • G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/10—Interfaces, programming languages or software development kits, e.g. for simulating neural networks
    • G—PHYSICS
    • G11—INFORMATION STORAGE
    • G11C—STATIC STORES
    • G11C11/00—Digital stores characterised by the use of particular electric or magnetic storage elements; Storage elements therefor
    • G11C11/54—Digital stores characterised by the use of particular electric or magnetic storage elements; Storage elements therefor using elements simulating biological cells, e.g. neuron

Definitions

  • a Deep Neural Network is a type of artificial neural network that can be used for machine learning applications.
  • a DNN can be used in signal processing applications, including image processing and computer vision applications.
  • DNNs have been implemented in applications where power resources are not a significant factor. Despite this, DNNs have application in a number of different technical fields in which the resources of the hardware used to implement the DNNs is such that power consumption, processing capabilities, or silicon area are limited. There is therefore a need to implement hardware that is configured to implement a DNN (or at least part thereof) in an efficient manner—for example, in a manner that requires less silicon area or less processing power when operating. Moreover, DNNs can be configured in a number of different ways for a variety of different applications. There is therefore also a need for hardware for implementing a DNN to be flexible to be able to support a variety of DNN configurations. As DNNs become more complex in structure and are required to process larger volumes of data in shorter periods of time (for example, for inference in real-time applications) there is a growing need for hardware that has greater processing capabilities, while maintaining or increasing area- and power-efficiency.
  • a multicore hardware implementation of a deep neural network is disclosed.
  • a plurality of layers of the network is arranged in plurality of layer groups.
  • the input data to the network comprises a multidimensional tensor including one or more traversed dimensions, being dimensions that are traversed by strides in at least one layer of a first layer group, and one or more non-traversed dimensions.
  • the hardware implementation is configured to split the evaluation of the first layer group into at least a first pass and a second pass, along one of the traversed dimensions or one of the non-traversed dimensions.
  • a first core is configured to evaluate the first layer group for the first pass, to generate a first portion of output data.
  • a second core is configured to evaluate the first layer group for the second pass, to generate a second portion of output data.
  • the hardware implementation is configured to combine the first portion of output data and the second portion of output data to produce the output data of the first layer group.
  • a method for implementing a deep neural network in multicore hardware according to claim 1 .
  • the layer groups of the neural network are evaluated by splitting the calculations into passes whose evaluation is distributed across multiple cores. Splitting the evaluation into passes along a traversed dimension or a non-traversed dimension can enable layer groups to be processed layer group by layer group, in parallel.
  • Evaluating a layer group means evaluating each layer of the layer group.
  • the layer groups may be defined such that all layers of a single layer group can be evaluated in a single hardware pass. For example, all layers of a single layer group can be evaluated in a hardware pipeline of a given core without needing to write any data to the one or more second memory devices or the one or more first memory devices.
  • a layer group may consist of a single layer of the neural network. In other cases, a layer group may consist of multiple layers.
  • a layer group may comprise any one or any two or more of: a single convolutional layer, a single pooling layer, a single activation layer, a single normalisation layer, and a single layer of element-wise operations.
  • a layer group may include at most one convolutional layer.
  • a cost of accessing the one or more second memory devices may be less than a cost of accessing the one or more first memory devices.
  • the cost may be in terms of time or power consumption.
  • the one or more first memory devices may be provided as off-chip memory; the one or more second memory devices may be provided as on-chip memory.
  • the traversed dimensions may include an X dimension (columns) and a Y dimension (rows), and optionally a batch dimension, B.
  • the input data comprises one or more additional, non-traversed dimensions, such as a channel dimension, C.
  • the input data may be processed by a number of filters, to produce a respective number of output channels F.
  • the tensor (output data of the first layer group) is reassembled in the one or more second memory devices.
  • the second and any subsequent layer groups may be split differently from the first layer group, or might not be split at all.
  • the splitting may be planned before a training phase or inference phase of the neural network begins.
  • the splitting may be planned in advance based on the parameters of the layers in the various layer groups, and the tensor size at the input and output of each layer.
  • the deep neural network may be a convolutional neural network.
  • the method may further comprise: splitting the evaluation of the second layer group into at least a first pass and a second pass, along one of the traversed dimensions or one of the non-traversed dimensions; allocating the first pass for the second layer group to the first core and allocating the second pass for the second layer group to the second core; using the first core, evaluating the second layer group for the first pass, to generate a first portion of output data; and using the second core, evaluating the second layer group for the second pass, to generate a second portion of output data, each of said first portion of output data and said second portion of output data comprising a plurality of data elements.
  • the evaluation of the first layer group may be split along a first dimension and the evaluation of the second layer group may be split along a second dimension.
  • the first dimension and second dimension may be different dimensions.
  • the method is flexible, in that the different layer groups need not be split along the same dimension.
  • the split can be chosen according to whatever is more efficient for the layer group in question, given the capabilities of the hardware.
  • the first layer group may be split according to input channel, whereby the first pass comprises evaluating the first layer group for a first input channel and the second pass comprises evaluating the first layer group for a second input channel.
  • the input channel dimension is an example of a non-traversed dimension.
  • the first pass comprises evaluating the first layer group for a first set of input channels and the second pass comprises evaluating the first layer group for a second set of input channels.
  • the first layer group may be split according to output channel, whereby the first pass comprises evaluating the first layer group for a first output channel and the second pass comprises evaluating the first layer group for a second output channel.
  • the output channel dimension is another example of a non-traversed dimension.
  • the first pass comprises evaluating the first layer group for a first set of output channels and the second pass comprises evaluating the first layer group for a second set of output channels.
  • the method may further comprise storing output data of a final one of the plurality of layer groups in the one or more first memory devices.
  • intermediate data generated when evaluating the layer groups is stored in the one or more second memory devices, and the output data generated by evaluating the final layer group is stored in the one or more first memory devices.
  • the method may further comprise: analysing at least one property of the input data for the first layer group; and determining based on the analysing that it should be split into passes along said one of the dimensions, wherein the splitting of the input data for the first layer group is performed in response to said determining.
  • the at least one property may comprise the size of the input data in at least one traversed dimension, and determining that the input data should be split into passes may comprise determining that the size in the at least one traversed dimension is not larger than a threshold size.
  • the evaluation (and input data) of the layer group may be split along the at least one traversed dimension.
  • the evaluation of the layer group may be split up along one of the other dimensions.
  • the processing of larger tensors may be divided between the cores along one or more traversed dimensions
  • smaller tensors may be divided between the cores along other dimensions, which may be non-traversed dimensions such as batch, input channel or output channel dimensions.
  • the tensor If the tensor is large, it can be split into tiles along at least one traversed dimension and processed depth-first in these tiles. This may reduce the amount of data that needs to be stored in the one or more second memory devices, because it might not be necessary to store an entire intermediate tensor in them at the same time. On the other hand, if the sensor is small, it can be split into passes. In this case, even though it is necessary to store an entire intermediate tensor in the one or more second memory devices, this might not be a limiting factor because the tensor was smaller to begin with. In other words, the present method can allow parallel hardware to be utilised efficiently whether an input tensor is small or large.
  • Splitting the evaluation of the first layer group may comprise splitting it into the first pass, the second pass and a third pass.
  • the method may further comprise: allocating the third pass to the first core; and after evaluating, using the first core, the first layer group for the first pass, evaluating, using the first core, the first layer group for the third pass. This may be done when the layer group is split into a number of passes that is greater than the number of cores in the multicore hardware.
  • the input data for the first layer group may comprise a first traversed dimension and a second traversed dimension, wherein the input data is stored in the one or more first memory devices ordered according to the first dimension first, and wherein the evaluation of the first layer group is split along the first dimension.
  • the evaluation may be split in this way, in particular if the input data is not large enough in the second dimension to enable splitting in that dimension.
  • the input data may be stored row-first—in other words, ordered according to the X dimension.
  • the evaluation of the first layer group may be split into passes in the X dimension.
  • the input data may be stored column-first (ordered according to the Y dimension first), and the evaluation of the first layer group may be split into passes along the Y dimension.
  • the one or more second memory devices may comprise: a local memory for the first core that is accessible only to the first core; a local memory for the second core that is accessible only to the second core; and a shared memory that is accessible to the first core and the second core, the method optionally comprising storing the first portion of output data and the second portion of output data in the shared memory.
  • the first layer group may comprise a convolutional layer, and the method may further comprise: storing weight values and/or bias values for the convolutional layer in the shared memory, if the evaluation was split along one of the traversed dimensions; or splitting up the weight values and/or bias values and storing them in the local memories, if the evaluation was split along one of the non-traversed dimensions.
  • the same weights/biases need to be applied to every part of the data, and therefore every core needs access to all of the same weights/biases.
  • the evaluation is split according to input channels or output channels of the layer group, it may be possible for each core to process one or more input channels or output channels, while other cores process one or more other input channels or output channels.
  • the weights/biases associated with a given input channel or output channel may be stored locally to the core that needs to use them.
  • the evaluation may be split according to input channel, wherein the first layer group comprises no convolutional layer, and wherein the method comprises splitting up parameters of the first layer group and storing them in the local memories.
  • the parameters may include parameters controlling pooling, normalisation, or element-wise operations.
  • the parameters may include stride and/or window size, for the respective input channel, for example.
  • Each core may comprise a plurality of processing elements configured to process input data to evaluate a layer of the neural network.
  • the plurality of cores may be configured to store output data of a final one of the plurality of layer groups in the one or more first memory devices.
  • the splitter unit may be configured to analyse at least one property of the input data for the first layer group; and determine based on the analysing that it should be split into passes along said one of the dimensions, wherein the splitter unit is configured to, in response, split the input data for the first layer group along said one of the dimensions.
  • the at least one property may comprise the size of the input data in at least one traversed dimension, wherein the splitter unit is configured to determine that the input data should be split into passes by determining that the size in the at least one traversed dimension is not larger than a threshold size.
  • the one or more second memory devices may comprise: a local memory for the first core that is accessible only to the first core; a local memory for the second core that is accessible only to the second core; and a shared memory that is accessible to the first core and the second core.
  • the plurality of cores may be configured to store the first portion of output data and the second portion of output data in the shared memory.
  • the first layer group may comprise a convolution layer.
  • the splitter unit may be configured to store weight values and/or bias values for the convolutional layer in the shared memory if the evaluation was split along one of the traversed dimensions; and split up the weight values and/or bias values and store them in the local memories if the evaluation was split along one of the non-traversed dimensions.
  • the splitter unit can be configured to split up parameters of the first layer group and store them in the local memories.
  • an artificial intelligence accelerator system comprising a hardware implementation as claimed in any of claims 13 to 16 , or as summarized above. Also provided is an artificial intelligence accelerator system configured to perform a method as summarized above or a method according to any of claims 1 to 12 . Such artificial intelligence accelerator systems may be embodied in hardware on an integrated circuit.
  • an integrated circuit definition dataset that, when processed in an integrated circuit manufacturing system, configures the integrated circuit manufacturing system to manufacture an artificial intelligence accelerator system as claimed in claim 17 .
  • Non-transitory computer readable storage medium having stored thereon a computer readable description of an artificial intelligence accelerator system as claimed in claim 17 that, when processed in an integrated circuit manufacturing system, causes the integrated circuit manufacturing system to manufacture an integrated circuit embodying the artificial intelligence accelerator system.
  • a non-transitory computer readable storage medium having stored thereon a computer readable description of an artificial intelligence accelerator system as claimed in claim 17 which, when processed in an integrated circuit manufacturing system, causes the integrated circuit manufacturing system to: process, using a layout processing system, the computer readable description of the artificial intelligence accelerator system so as to generate a circuit layout description of an integrated circuit embodying the artificial intelligence accelerator system; and manufacture, using an integrated circuit generation system, the artificial intelligence accelerator system according to the circuit layout description.
  • an integrated circuit manufacturing system configured to manufacture an artificial intelligence accelerator system as claimed in claim 17 .
  • an integrated circuit manufacturing system comprising: a non-transitory computer readable storage medium having stored thereon a computer readable description of an artificial intelligence accelerator system as claimed in claim 17 ; a layout processing system configured to process the computer readable description so as to generate a circuit layout description of an integrated circuit embodying the artificial intelligence accelerator system; and an integrated circuit generation system configured to manufacture the artificial intelligence accelerator system according to the circuit layout description.
  • the layout processing system may be configured to determine positional information for logical components of a circuit derived from the integrated circuit description so as to generate a circuit layout description of an integrated circuit embodying the artificial intelligence accelerator system.
  • FIG. 1 is a schematic diagram of an example deep neural network (DNN);
  • DNN deep neural network
  • FIG. 2 is a schematic diagram of example data in a DNN
  • FIG. 3 is a block diagram of an example hardware implementation of a convolution layer of a DNN
  • FIG. 4 is a block diagram of an example convolution engine of FIG. 3 ;
  • FIG. 5 is a schematic diagram illustrating a flow of data according to an example hardware implementation
  • FIG. 6 illustrates the data dependency within a plurality of layers of a DNN, for a first block of data
  • FIG. 7 illustrates the data dependency within the plurality of layers of FIG. 6 , for a second block of data
  • FIG. 8 is a conceptual illustration of the concept of overlap-data
  • FIG. 9 schematically illustrates an example allocation of memory associated with a hardware implementation of a DNN
  • FIG. 10 illustrates a DNN with four layers and no branching
  • FIG. 11A illustrates an example grouping of layers for a DNN with four layers, having branching
  • FIG. 11B illustrates an alternative example grouping, for the DNN of FIG. 11A ;
  • FIG. 12 is a block diagram of a multicore hardware implementation, according to an example.
  • FIG. 13 illustrates an example of execution order in the multicore hardware of FIG. 12 , when splitting input data into tiles and processing the tiles depth-first, according to a first aspect
  • FIGS. 14A-14B illustrate synchronisation between cores, in another example in which input data is split into tiles and processed depth-first;
  • FIG. 15 is a flowchart showing a method for implementing a convolutional neural network in multicore hardware, when splitting input data into tiles;
  • FIG. 16 illustrates an example of execution order, when splitting the evaluation of layers into passes, according to a second aspect
  • FIG. 17 is a flowchart showing a method for implementing a convolutional neural network in multicore hardware, when splitting the evaluation of the layers into passes;
  • FIG. 18 shows a computer system in which a graphics processing system is implemented
  • FIG. 19 shows an integrated circuit manufacturing system for generating an integrated circuit embodying a graphics processing system.
  • a Deep Neural Network is a form of artificial neural network comprising a plurality of interconnected layers that can enable the DNN to perform signal processing tasks, including, but not limited to, computer vision tasks.
  • FIG. 1 illustrates an example DNN 100 that comprises a plurality of layers 102 - 1 , 102 - 2 , 102 - 3 . Each layer 102 - 1 , 102 - 2 , 102 - 3 receives input data, and processes the input data in accordance with the layer to produce output data. The output data is either provided to another layer as input data to that layer, or is output as the final output data of the DNN.
  • the first layer 102 - 1 receives the original input data 104 to the DNN 100 and processes the input data in accordance with the first layer 102 - 1 to produce output data.
  • the output data of the first layer 102 - 1 becomes the input data to the second layer 102 - 2 , which processes the input data in accordance with the second layer 102 - 2 to produce output data.
  • the output data of the second layer 102 - 2 becomes the input data to the third layer 102 - 3 , which processes the input data in accordance with the third layer 102 - 3 to produce output data.
  • the output data of the third layer 102 - 3 is output as output data 106 of the DNN.
  • each layer of a DNN may be one of a plurality of different types.
  • Example DNN layer types include, but are not limited to: a convolution layer, an activation layer, a normalisation layer, a pooling layer and a fully connected layer. It will be evident to a person of skill in the art that these are example DNN layer types and that this is not an exhaustive list and there may be other DNN layer types.
  • a convolution layer is configured to convolve the input data using weights associated with that layer.
  • each convolution layer is associated with a plurality of weights w 1 . . . w i , which may also be referred to as filter weights or coefficients.
  • the weights may be grouped to form or define one or more filters or kernels.
  • FIG. 2 illustrates an example of the format of data 200 utilised in a DNN.
  • the data 200 used in the DNN may be arranged as p planes of data, where each plane has a dimension x ⁇ y.
  • a DNN may comprise one or more convolution layers, each of which is associated with a plurality of filters, each of which comprises a plurality of weights.
  • Each filter has a dimension m ⁇ n ⁇ p (that is, each filter comprises a set of m ⁇ n ⁇ p weights w) and is applied to the input data according to a convolution operation across several steps (known as strides) s and t, as illustrated in FIG. 2 .
  • a convolutional neural network which is a specific type of DNN that is effective for image recognition and classification, generally comprises a plurality of convolution layers.
  • a (single-core) hardware implementation of a convolution layer may comprise a hardware module or block (which may be referred to herein as a “processing element”) that is configured to calculate the sum of the products between the weights forming all or a portion of a filter and input data values forming all or a portion of a window (which may be referred to as a filter-window calculation). Since typically a high number of such filter-window calculations is performed, some hardware implementations may comprise a plurality of such processing elements so that more than one filter-window calculation can be performed in parallel.
  • Preparing each processing element to perform a filter-window calculation involves reading the appropriate input data and weights for each filter-window calculation from one or more first memory devices and providing it to one of the processing elements. Typically, it is necessary to transfer a large amount of data from the memory to the processing elements. If this is not done efficiently, it can result in a high memory bandwidth requirement, and high-power consumption, for providing the input data and weights to the processing elements. This is particularly the case when the memory is “off-chip”—that is, implemented in a different integrated circuit or semiconductor die from the processing elements.
  • blocks (also referred to as “tiles”) of data may be processed in a depth-wise manner, through a plurality of layers of the neural network. Intermediate outputs of the layers within the plurality of layers can be stored in one or more second memory devices.
  • the one or more second memory devices may be smaller in size than the one or more first memory devices that are used for bulk storage of data, but may be quicker and/or more energy efficient to access.
  • at least one of the one or more second memory devices is provided on-chip (that is, in the same integrated circuit or semiconductor die as the processing elements).
  • FIG. 3 illustrates an example single-core hardware implementation 300 of a convolution layer of a DNN.
  • the hardware implementation 300 comprises a plurality of convolution engines 302 , a plurality of accumulators 304 , an accumulation buffer 306 , a coefficient buffer 308 , and an input buffer 310 .
  • Each convolution engine 302 together with its respective accumulator 304 and its share of the resources of the accumulation buffer 306 , represents a processing element 318 .
  • the hardware implementation further comprises a double data rate (DDR) SDRAM 312 , which is an off-chip memory; an on-chip memory (OCM) 314 , and an output buffer 316 .
  • DDR double data rate
  • OCM on-chip memory
  • Each convolution engine 302 comprises hardware logic configured to receive a set of weights, e.g. ⁇ w 1 . . . w 8 ⁇ , that represent all or a portion of a filter, and a set of input data values, e.g. ⁇ d 1 . . . d 8 ⁇ , that represent all or a portion of a window, and perform a multiply-accumulate calculation on the received weights and input data values as depicted in FIG. 4 .
  • each convolution engine 302 may comprise a plurality of multipliers 402 , each of which is configured to multiply a weight (w i ) and a corresponding input data value (d i ) to produce a multiplication output value.
  • the multipliers 402 are followed by a plurality of adders 404 that form an adder tree to calculate the sum of the multiplication outputs.
  • the convolution engine 302 comprises eight multipliers 402 , but in other examples there may be more or fewer multipliers. For example, in some cases there may be 128 multipliers.
  • the adder tree comprises Z ⁇ 1 adders 404 .
  • the multiply-accumulate calculation is pipelined.
  • the multipliers 402 and adders 404 may be divided into a number of pipeline stages with a register stage (not shown) before the first pipeline stage and between each pair of pipeline stages.
  • the multipliers may form a first pipeline stage and the adders may be divided into layers wherein the subsequent pipeline stages comprise one or more layers of adders.
  • the example hardware implementation 300 of FIG. 3 comprises four convolution engines 302 ; however, it will be evident to a person of skill in the art that the methods and principles described herein are applicable to hardware implementations with any number of convolution engines.
  • the hardware implementation may comprise a plurality of accumulators 304 .
  • Each accumulator 304 receives the output of one convolution engine 302 and adds the output to the previous convolution engine output that relates to the same filter. Since the convolution engine may not generate or produce outputs that relate to the same filter in consecutive cycles the partial results of one or more filters may be stored in an accumulation buffer 306 and then the appropriate partial result may be provided to the accumulator each cycle by the accumulation buffer 306 .
  • the accumulation buffer 306 may be able to store partial results related to 128 different filters.
  • the coefficient buffer 308 comprises memory (not shown) to store a plurality of weights related to a convolution layer (or fully connected layer) of the DNN, and hardware logic (not shown) to provide the weights to the convolution engines 302 for processing in a predetermined order over a plurality of cycles.
  • the plurality of weights may comprise all of the weights related to the convolution layer or only a portion of the weights related to a convolution layer.
  • the coefficient buffer 308 is shown as a single module, the coefficient buffer 308 may be implemented by a plurality of coefficient buffers that each form a bank, for example.
  • the input buffer 310 comprises memory (not shown) to store a plurality of input data values related to a convolution layer (or fully connected layer) of the DNN and hardware logic (not shown) to provide the input data values to the convolution engines 302 for processing in a predetermined order over a plurality of cycles.
  • the plurality of input data values may comprise all of the input data values related to the convolution layer or only a portion of the input data values related to a convolution layer.
  • the input buffer 310 is shown in FIG. 3 as a single module, the input buffer 310 may be implemented by a plurality of input buffers that each form a bank, for example.
  • the DDR 312 is coupled to the on-chip memory 314 , for providing weight data to the on-chip memory 314 .
  • the DDR 312 is also coupled to the input buffer 310 , for providing blocks of input data to the input buffer 310 .
  • the on-chip memory 314 is coupled to the coefficient buffer 308 , for providing weight data to the coefficient buffer 308 .
  • the on-chip memory 314 is also coupled to the input buffer 310 , for providing intermediate output data (which comprises input data to subsequent layers) to the input buffer 310 .
  • the on-chip memory 314 is also coupled to the DDR 312 , for providing blocks of output data of the end layer of the plurality of layers to the DDR 312 .
  • the accumulation buffer 306 is coupled to the output buffer 316 , to allow the output buffer to receive intermediate output data of the layers within the plurality of layers, as well as the output data of the end layer.
  • the output buffer 316 is coupled to the on-chip memory 314 , for providing the intermediate output data and output data of the end layer to the on-chip memory 314 .
  • the various connections are shown separately in the example of FIG. 3 ; but, in some embodiments, some or all of them may be provided by one or more shared bus connections. It should also be understood that other connections may be provided, as an alternative to or in addition to those illustrated in FIG. 3 .
  • the output buffer 316 may be coupled to the DDR 312 , for providing output data directly to the DDR 312 .
  • the DDR 312 may be coupled to the coefficient buffer 308 , for providing weight data directly to the coefficient buffer 308 .
  • not all of the connections illustrated in FIG. 3 are necessary.
  • the DDR 312 need not always be coupled to the input buffer 310 —the input buffer could alternatively obtain input data from the DDR 312 via the on-chip memory 314 .
  • FIG. 5 is a simplified schematic diagram, to better illustrate the flow of data according to one example of a hardware implementation like that of FIG. 3 .
  • FIG. 5 shows a plurality of layers (consisting of two layers 502 - 1 , 502 - 2 ) of a neural network. These may be any two successive layers of the network. Note also that the plurality of layers is not limited to two layers.
  • the diagram shows the input data 504 to the first layer 502 - 1 of the plurality of layers.
  • the intermediate output data 506 of the first layer 502 - 1 forms the input data to the second layer 502 - 2 (which is the end layer in this two-layer example).
  • the diagram also shows the output data 508 of the second layer 502 - 2 (end layer).
  • the DDR 312 is a first memory device, with a relatively high cost of reading/writing.
  • the on-chip memory 314 is a second memory device, with a relatively lower cost of reading/writing. For example, it may take more time and energy to read/write a given amount of data from/to the DDR than from/to the on-chip memory 314 .
  • DDR memory is just one example of a bulk memory storage device, and other memory/storage technologies can also be used, such as flash memory.
  • the DDR 312 stores input data for the first layer 502 - 1 of the plurality of layers 502 - 1 , 502 - 2 .
  • the DDR 312 also stores weight data representing weights for the plurality of layers 502 - 1 , 502 - 2 .
  • the weight data is transferred from the DDR 312 to the on-chip memory 314 .
  • the coefficient buffer 308 reads the weight data from the on-chip memory 314 and provides the weight data to the processing elements 318 .
  • the input buffer 310 reads, from the DDR 312 , a first subset of the input data, representing a first block of the input data 504 .
  • This first subset of the input data 504 (also referred to herein as a “leading tile”) is shown shaded in grey in FIG. 5 and may comprise one or more planes of data.
  • the input buffer 310 provides the first subset of input data 504 to the processing elements 318 .
  • the processing elements 318 process the weight data and the first subset of the input data 504 to evaluate each layer of the plurality of layers and thereby calculate a first block of output data 508 for the end layer.
  • This first block of output data 508 is shown shaded in grey in FIG. 5 and may comprise one or more planes of data.
  • the output buffer 316 may write the first block of output data 508 to the on-chip memory 314 . From there, the first block of output data 508 may be transferred to the DDR 312 . Alternatively, the output buffer may write the output data directly into the DDR memory 312 (as indicated by the dashed arrow in FIGS. 3 and 5 ).
  • the data input to and output from each layer is arranged in one or more planes (also referred to herein as input and output data sets).
  • the number of planes depends on the weight data. In particular, the weights making up each filter take a predetermined number of input planes as input data, and each filter outputs a plane of output data. The number of planes of data produced by a given layer is thus equal to the number of filters applied by that layer.
  • Each plane of data may represent a feature map.
  • the three planes of input data 504 might represent three feature maps generated by an earlier layer of the neural network (if the first layer 502 - 1 in the drawing is not the first layer of the neural network).
  • the three planes of input data 504 may represent three planes of input data.
  • Input data having multiple planes arises frequently in image-processing tasks, for example.
  • Such input planes may include but are not limited to: Red, Green, and Blue (RGB) planes; or Hue, Saturation, and Value (HSV) planes.
  • the input planes could also include depth data as one of the input planes.
  • the processing elements 318 In order to calculate the first block of output data, the processing elements 318 must evaluate each layer of the plurality of layers (as mentioned above).
  • the coefficient buffer 308 reads the weight data for the first layer 502 - 1 from the on-chip memory 314 .
  • the processing elements 318 process the weight data for the first layer 502 - 1 and the first subset of the input data 504 to evaluate the first layer 502 - 1 of the plurality of layers, and thereby calculate output data 506 of the first layer.
  • This is a block of output data of the first layer 502 - 1 corresponding to the first block of input data 504 . In other words, it is a subset of the complete set of output data of the neural network for the first layer 502 - 1 .
  • the output buffer 316 writes the block of output data of the first layer 502 - 1 to the on-chip memory 314 . It represents intermediate output data that will be needed as input data to the next layer (second layer 502 - 2 ).
  • the input buffer 310 reads, from the on-chip memory 314 , the block of the output data 506 of the first layer 502 - 1 that was stored there by the output buffer 316 .
  • the coefficient buffer 308 reads the weight data for the second layer 502 - 2 from the on-chip memory 314 .
  • the processing elements 318 process the weight data for the second layer 502 - 2 and the block of the output data 506 of the preceding (first) layer, to evaluate the second layer 502 - 2 and thereby calculate output data of the second layer 502 - 2 .
  • This procedure continues for any subsequent internal layers of the plurality of layers—saving the intermediate output data in the on-chip memory 314 each time, and reading this intermediate output data to provide the input data to the next layer. This continues until the end layer is reached.
  • the second layer 502 - 2 is the end layer.
  • the output of the second layer 502 - 2 is therefore the first block of the output data 508 .
  • the output buffer 316 may write the first block of output data 508 to the on-chip memory 314 . From there, the first block of output data 508 may be transferred to the DDR 312 . Alternatively, the output buffer 316 may write the output data directly to the DDR 312 (dashed arrow).
  • Processing a subset (tile) of input data through multiple layers of the neural network in this way, can efficiently utilise the processing resources of the processing elements 318 .
  • Writing the intermediate output data to the on-chip memory 314 and reading it from the on-chip memory 314 to process the next layer reduces the need to read and write data from the off-chip DDR 312 , which can help to reduce power consumption.
  • the on-chip memory 314 can be significantly smaller than the off-chip DDR 312 , because the on-chip memory 314 only needs to store data associated with the current block of input data, the current block of output data, and the associated blocks of intermediate output data for the internal layers, during each pass.
  • the size of the block to be processed in each pass can be chosen based on the available storage capacity of the on-chip memory 314 , the number of layers in the plurality of layers, and the structure of the neural network.
  • An example approach for memory management in the on-chip memory 314 will be described below.
  • FIG. 6 shows the input data to three layers (not shown explicitly) of a neural network; the output data of the first layer (here labelled as layer 0); the output data of the second layer (labelled layer 1); and the output data of the end layer.
  • the output data of the end layer is a single feature map created by a single filter.
  • the other layers may be thought of as convolutional layers, with a single filter, creating a single output feature map, as shown in simplified form in the drawings.
  • the scope of the present disclosure is not limited in this way—these are simplifications purely for ease of understanding.
  • a single filter could be applied to multiple input channels (feature maps), or multiple filters could be applied to one input channel or multiple input channels, thereby processing and/or generating multiple planes of data.
  • Methods and hardware implementations according to the present disclosure can be used equally in such scenarios.
  • a block of output data (Tile 1 ) in the end layer it is possible to trace backwards through the network to find the intermediate data and input data upon which this output block depends.
  • Tile 1 a block of data in the layer 1 output that corresponds to Tile 1 in the end layer, in the sense that Tile 1 in layer 1 is needed to calculate Tile 1 in the end layer.
  • the filter has a kernel size greater than 1. Because of this kernel size, Tile 1 in the layer 1 output is larger than Tile 1 in the end layer output. In other words, Tile 1 in the end layer output depends on some additional data in the layer 1 output. For example, for a 3 ⁇ 3 kernel and a stride of 1, one extra row of output data is needed in the layer 1 output. This expansion in the amount of data required is indicated by the sloped dashed line in FIG. 6 .
  • Each Tile is delimited by solid lines, in this drawing.
  • Tile 1 in the input data, corresponding to Tile 1 in layer 0.
  • the stride in layer 0 is 1; therefore, the input data is approximately the same size as the layer 0 output (subject to padding). Again, additional data is needed at the boundary, due to the kernel size being greater than 1.
  • Tile 1 in the input data is larger again than Tile 1 in the layer 0 output.
  • Tile 1 in the input data makes up the first subset (block 1) of input data.
  • Tile 1 in the output data of the end layer makes up the output block.
  • the total amount of data needed for the first tile grows successively from the end block back to the input block. It depends on the parameters (kernel size, stride, dilation) of all the intervening layers. Also indicated in FIG. 6 is the “Overlap”. This is data that can be reused in a subsequent pass—that is, reused when calculating a subsequent Tile.
  • the Overlap data is indicated by the dashed horizontal lines in each layer of data.
  • Overlap 1 in the layer 1 output is the portion of Tile 1 that can be (re)used in calculating a subsequent block (Tile 2 ) of the end layer output.
  • Overlap 1 in the layer 0 output is the portion of Tile 1 that can be (re)used in calculating Tile 2 of the layer 1 output.
  • FIG. 7 illustrates the data dependency for a second block.
  • the second block (Tile 2 ) of output data of the end layer is adjacent to the first block (Tile 1 ).
  • Tile 2 depends on amounts of intermediate data (and ultimately input data) that increase as we progress backwards through the layers. However, some of this data is common with the dependency of Tile 1 in the output data of the end layer.
  • Overlap 1 in the output of layer 1 has already been calculated when evaluating layer 1 for Block 1 (Tile 1 ).
  • the only additional data that needs to be calculated is the block Tile 2 in layer 1.
  • moving backwards to the layer 0 output the only additional data that needs to be calculated is the block Tile 2 in layer 0.
  • Block 2 Block 2
  • each Tile in FIG. 7 is delimited by solid lines.
  • the Overlap data is delimited by dashed lines.
  • the dotted lines indicate the uppermost extremity of the data dependency (that is, the first row of data needed to calculate the Tile in the next layer).
  • the sloped dashed lines indicate the lowermost extremity of the data dependency (that is the last row of data needed to calculate the Tile in the next layer).
  • Tile 2 in the layer 1 output depends on Overlap 1 and Tile 2 in the layer 0 output.
  • Tile 2 incorporates Overlap 2 , which will be reused in the next pass to calculate Tile 3 of the subsequent layer, and so on.
  • the Overlap data constitutes a non-disposable portion of the data.
  • the remainder of each Tile constitutes a disposable portion of the data.
  • the Overlap data is needed not only for calculating output data for the end layer for the current block but will also be needed later, for calculating output data for the end layer for a subsequent block (subsequent Tile).
  • the remainder of the data in each Tile is needed for calculating output data for the end layer for a current block, but it can be deleted/overwritten after that.
  • FIG. 8 illustrates individual data elements in the input data and the output data of each of three layers of a neural network (layer 0, layer 1, and an end layer).
  • Each circle represents a data element (which could be, for example, a pixel in an image or feature map).
  • Each element depends on a certain sub-group of elements in the preceding layer.
  • the connecting lines in FIG. 8 indicate the elements of the preceding layer upon which a given element depends.
  • the 3rd element in the output of the end layer depends on the 2 nd , 3 rd and 4 th elements in the output of layer 1.
  • the fourth element in the output of the end layer depends on the 3 rd , 4 th and 5 th elements in the layer 1 output.
  • the end layer has a kernel size of 3 and a stride of 1.
  • Layer 1 has a kernel size of 3 and a stride of 2.
  • Layer 0 has a kernel size of 3 and a stride of 1. Because layer 0 and the end layer each have a stride of 1, the output of layer 0 is approximately the same size as the input data, and the output of the end layer is approximately the same size as the output of layer 1 (subject to the need for padding). Meanwhile, because layer 1 has a stride of 2, the output of layer 1 is approximately half the size of the output of layer 0.
  • the dashed lines represent the boundaries between blocks (Tiles). That is, the 3 rd data element shown in the end layer is the final data element of the first block (Tile 1 ). Taking the 4th element in the end layer output as an example, it can be seen that each element in the end layer output depends on: 3 elements in the output of layer 1; 7 elements in the output of layer 0; and 9 elements in the input data. Elements in Tile 1 , other than the Overlap 1 elements, are shown with diagonal hatching. Three such elements (1-3) are shown, by way of example, in the end layer output.
  • Some elements can be discarded after the end layer output for the current block (Tile 1 ) has been calculated. These are referred to as “disposable” elements. Some other elements will be used also for calculating the end layer output for the next block (Tile 2 ). These are referred to as “non-disposable” elements. In particular, the Overlap 1 elements (coloured solid black in FIG. 8 ) are non-disposable elements. Elements 7 and 8 in the input data are used to calculate elements 6 and 7 in the output of layer 0, during the first pass through the plurality of layers, to calculate the end layer output data for the current block (Tile 1 ). However, they are also needed to calculate elements 8 and 9 in the layer 0 output.
  • Element 7 in the layer 0 output and elements 3 and 4 in the layer 1 output are also Overlap 1 elements, since they will also be needed for calculations in the second pass (element 7 in the layer 0 output will be needed to calculate element 5 in the layer 1 output; and elements 3 and 4 in the layer 1 output will be needed to calculate elements 4 and 5 in the end layer output).
  • Element 6 of the input data is used in the calculation of elements 5, 6, and 7 of the layer 0 output. However, each of these three elements must be calculated in the first pass, for calculating the first output block (Tile 1 ). Element 6 of the input data is not needed for any additional calculations in the second pass; therefore, it is “disposable” data. Similarly, element 6 in the layer 0 output is used to calculate element 4 in the layer 1 output, but is not needed further in the second pass (provided that element 4 in the layer 1 output is retained). Elements of Tile 2 (other than the Overlap 2 elements) are shown as empty circles. Overlap 2 elements are shown shaded in grey. These will be reused in the calculation of Tile 3 and are therefore non-disposable data. A few Tile 3 elements are shown at the bottom of the drawing, with horizontal hatching.
  • the present inventors have recognised that the disposable data, is only required for calculating the next layer, in the present pass through the plurality of layers. Once the next layer in the present pass has been calculated, the disposable data from the preceding layer can be deleted/overwritten.
  • the non-disposable data (Overlap) must be retained, for all layers of the present pass, in order that it can be used in the second pass, for calculation of the second block. This recognition can enable a reduction in unnecessary read/write operations from/to the DDR 314 , and also a reduction in unnecessary re-calculation of intermediate output data.
  • FIG. 9 illustrates one example memory allocation for the on-chip memory 314 that exploits these characteristics.
  • the memory management strategy divides the on-chip memory 314 into three sections.
  • the first section 802 (which may be referred to as “swap” memory) is used for storing the disposable data.
  • the second section 804 (which may be referred to as “heap” memory) is used for storing the non-disposable data.
  • the third section 806 (coefficient memory) is used for storing the weight data for the various layers.
  • a plurality of slots A, B, C are defined.
  • Within a given pass (to calculate a given block of output data) the method proceeds as follows. The disposable output of layer 0 is written to slot A.
  • the non-disposable output of layer 0 is written to the start of the second section 804 .
  • the data is read from these two locations.
  • the disposable output of layer 1 is written to slot B and the non-disposable output of layer 1 is written to the second section 804 —for example, appended to the non-disposable output of layer 0.
  • the disposable data from layer 0 is no longer needed. Consequently, during the evaluation of layer 2, the contents of slot A can be overwritten with the disposable output of layer 2.
  • the non-disposable output of layer 2 is once again appended to the data in the second section 804 .
  • neither slot B nor slot A should be overwritten.
  • the contents of slot B (disposable output of layer 1) are needed again for the evaluation of layer 5, because of the branching.
  • the disposable output of layer 3 is written to slot C.
  • the disposable output of layer 4 is written to slot A, once again overwriting the contents of that slot, which are no longer needed for further calculations.
  • the disposable output data from layer 1 is read from slot B, and the disposable output data from layer 4 is read from slot A. (The non-disposable output data from layers 1 and 4 is read from the second section 804 .)
  • each slot can be chosen based on the maximum size of disposable data to be stored in that slot during a pass.
  • the second section 804 is double buffered, in this example.
  • the non-disposable output data (Overlap) for all layers of the current block is stored in one set of memory locations.
  • the Overlap is retrieved from this set of memory locations, as needed.
  • the Overlap for the next block is stored in a second set of memory locations.
  • the hardware implementation alternates between using the first set and the second set of memory locations—reading from one set while writing to the other set.
  • the disposable and non-disposable portions of the data can be identified by analysing the structure of each layer of the plurality of layers.
  • the allocation of the disposable data of each layer to a particular slot in the first section 802 of the on-chip memory 314 can be done by analysing the overall layer structure of the plurality of layers.
  • This analysis can be conducted by software, either dynamically at runtime or when mapping a given neural network to a given hardware implementation. Alternatively, it may be defined manually by a designer of the hardware implementation. However, the use of software facilitates the mapping of a variety of different neural networks to a given hardware implementation.
  • the plurality of layers of the neural network that are evaluated together in each pass could include all layers of the neural network, but this is not essential.
  • the plurality of layers may consist of a subset of the layers of the overall neural network. Depending on the structure of the neural network, different groupings of layers may be found advantageous.
  • FIG. 10 shows a simple neural network consisting of four layers, with no branches and each layer simply depending on the output of the preceding layer. This leads to a large degree of freedom in the choice of how to group the layers together.
  • the layers could be grouped into one or more pluralities of layers according to any of the following grouping strategies:
  • FIG. 11A shows a slightly more complex network structure. Again, there are four layers, with each layer depending on the preceding layer. However, there is an additional branch from layer 2 to layer 4.
  • the network is split into two pluralities of two layers each (layers 1-2 and 3-4), indicated by the dashed boxes in the drawing. Note, however, that this might not be optimal in terms of memory access requirements.
  • the output of layers 1-2 will be written to the DDR 312 . This will be read from the DDR 312 when evaluating layer 3 (the first layer of layers 3-4); however, the output of layer 2 is also needed for the evaluation of layer 4. Therefore, the same data will have to be read a second time from the DDR 312 in order to evaluate layer 4.
  • FIG. 11A shows a slightly more complex network structure. Again, there are four layers, with each layer depending on the preceding layer. However, there is an additional branch from layer 2 to layer 4.
  • the network is split into two pluralities of two layers each (layers 1-2 and 3-4), indicated by the dashed boxes
  • 11B shows an alternative grouping of the layers, which may be more efficient in terms of memory access requirements.
  • layer 1 is processed by itself, and layers 2-4 are grouped into a plurality of layers. By doing this, both of the branches are captured within the plurality of layers.
  • the output data of layer 1 will be read once from the DDR 312 , as input data for the first layer (layer 2) of the plurality of layers.
  • the output data of layer 2 can be handled within the plurality of layers, using the memory management strategy described above with reference to FIG. 9 .
  • the example was given of a convolutional layer of a neural network, operating on two-dimensional (2D) image data in a plurality of p planes or feature maps.
  • 2D two-dimensional
  • the blocks (tiles) of input and output data can be constructed in a variety of ways.
  • Each block can be defined in one, two, or more dimensions.
  • a block defined in one dimension may form a 2D stripe (in a horizontal or vertical direction).
  • the overlap data may be defined above and below the stripe (for a horizontal stripe) or to the left and right of the stripe (for a vertical stripe).
  • the diagrams of FIGS. 5, 6, and 7 are drawn with this in mind. (They are also applicable to 1D blocks defined in 1D data.)
  • FIGS. 5-8 indicate data being split into blocks in the Y (height) dimension, this is non-limiting. As explained above, the same methodology can be applied in the X (width) dimension. Nevertheless, there may be a preference for splitting in one dimension versus another dimension, in some cases. For example, if the data is stored row-wise in memory, and the individual data elements do not align neatly with a burst size used to read/write the memory, then it may be preferable to split the tiles in the Y (height) dimension.
  • a block may be defined in two dimensions, forming a 2D rectangle of data.
  • the overlap data may then be defined above, below, to the left, and to the right of this rectangle.
  • the overlap data from a current block may be useful for calculating multiple subsequent blocks of output data—for example, a block to the right of, a block below, and a block below-right of the current block (assuming blocks are processed in raster-scan order).
  • the same principles can likewise be extended to blocks in a higher number of dimensions, with overlap data in multiple directions in the multidimensional space.
  • the depth-first processing of the neural network was discussed in terms of individual layers of the network. This is appropriate for the implementation 300 of FIG. 3 , which is designed primarily to process a single convolutional layer. However, it is also possible to provide more elaborate hardware implementations, which add additional processing modules to the implementation 300 , and thereby enable multiple layers to be processed in a single pass through the hardware implementation. For instance, a pooling module, element-wise operations module, normalisation module, and activation module can be provided (see GB 2566702 A, for example). This could allow—for example—a convolutional layer, followed by an activation function, followed by a pooling operation, to be carried out in a single pass through the hardware implementation.
  • a layer group is a group of layers that can be processed together by the hardware implementation, in a single pass through the hardware.
  • a layer group may consist of a single layer (as in the examples described above with reference to FIG. 3 ); however, where the hardware has the necessary capabilities, a layer group may consist of more than one layer.
  • FIG. 12 shows a block diagram of the hardware implementation 600 . It comprises three cores 350 - 0 , 350 - 1 , and 350 - 2 . Each core has a respective local on chip memory (OCM) 314 a , which is accessible only to that core.
  • the hardware implementation 600 also comprises a shared on-chip memory 314 b . All of the cores are able to access (that is, read from and write to) the shared OCM 314 b via a high bandwidth interconnect 610 .
  • a splitter unit 620 is provided, which is configured to manage the division of work between the cores.
  • FIG. 13 illustrates schematically one example of processing tiles in a depth-first fashion, using the different cores 350 of the hardware implementation 600 .
  • the individual tiles are like those described previously above.
  • the tiles are divided up along the Y-dimension of the input data tensor. This is convenient when the data is stored in memory row-first (that is, X-dimension first).
  • the neural network includes five layer groups (labelled LG0 to LG4). Each layer group is indicated as comprising three layers, denoted by the circles in the drawing. (It will be understood that this is not limiting—different layer groups can have different numbers of layers.)
  • the layer groups are to be evaluated by the hardware implementation in two “tile groups” (labelled Tile Group 0 and Tile Group 1).
  • a tile group consists of the set of layer groups that are to be evaluated depth-first together, after splitting the data into tiles.
  • the input data for the first layer group in a tile group is read from the DDR 312 , and the output data from the final layer group in the tile group is written to the DDR 312 .
  • intermediate data (in tiles) is written to the OCM 314 by each layer group, and read from the OCM 314 by the next layer group.
  • the intermediate data may be stored in part in the local OCMs 314 a , and in part in the shared OCM 314 b . This will be explained in greater detail below.
  • the input data stored in the DDR 312 is split into six tiles (labelled Tile 0 to Tile 5) for the evaluation of Tile Group 0.
  • Tile 0 to Tile 5 There are only three cores 350 ; therefore, not all of the tiles can be processed in parallel, in this tile group. Instead, the first three tiles are allocated to the three cores, respectively, for a first multicore tile pass (Multicore Tile Pass 0).
  • a multicore tile pass means the evaluation of all of the layer groups of a tile group, for N successive tiles, where N is the number of cores.
  • Tile Group 0 consists of the first three layer groups LG0-LG2.
  • Multicore Tile Pass 0 Core 0 evaluates the layer groups LG0-LG2 based on Tile 0; Core 1 evaluates the layer groups based on Tile 1; and Core 2 evaluates the layer groups based on Tile 2. This is indicated by the solid-lined arrows in FIG. 13 .
  • the hardware implementation 600 proceeds to Multicore Tile Pass 1, in which Core 0 evaluates the layer groups LG0-LG2 based on Tile 3; Core 1 evaluates the layer groups based on Tile 4; and Core 2 evaluates the layer groups based on Tile 5. This completes the evaluation of the first tile group.
  • FIG. 13 is primarily intended to illustrate the execution order of the tiles by the cores. It does not offer a detailed picture of which portions of data are stored in which memory. Further details will be provided below, with reference to FIG. 14 , to explain how the data is arranged in the memories 312 and 314 .
  • the output tiles of LG0 are written to the OCM 314 , and read from the OCM 314 as input tiles for LG1.
  • the output tiles of LG1 are written to the OCM 314 , and read from the OCM as input tiles for LG2.
  • the output data of the first tile group namely, the output tiles of LG2—are written to the DDR 312 , before the hardware implementation 600 proceeds to evaluate Tile Group 1.
  • Tile Group 1 consists of two layer groups LG3-LG4. As shown, the data is split into just three tiles for this tile group.
  • the number of tiles can be chosen based on the size of the input data to the tile group, dependent on the size on the OCM 314 .
  • the number of tiles may also be influenced by the number and configuration of the layers within the Tile Group, because these will affect the amount of data that needs to be stored in the OCM 314 during the evaluation of the tile group for each tile.
  • Tile Group 1 can be evaluated in a single multicore tile pass, because the number of tiles is not greater than the number of cores.
  • the output tiles of LG3 are written to the OCM 314 , and read from the OCM 314 as input tiles for LG4.
  • the output tiles of LG4 are written to the DDR 312 .
  • FIGS. 14A-14B illustrate (for a different example), how tiles may be allocated to and processed by multiple cores.
  • FIG. 14A shows output data of three layer groups LG0-LG2, split into tiles.
  • LGxTn means “Layer Group x, Tile n”.
  • the sizes of the tiles in each layer are defined by the data-dependency in the neural network layers. Starting from the final layer group, the size of the leading tile expands from the leading output tile LG2T0 in the final layer group to the leading output tile LG1T0 in the middle layer group, and expands again to the leading output tile LG0T0 in the initial layer group LG0.
  • each layer group subsequent tiles after the leading tile LGxT0 have uniform size, within that layer group—with the exception of the trailing tile.
  • the trailing tile can shrink as one moves backwards through the layer groups from the final layer group to the initial layer group. This can be seen comparing the trailing tile LG2T7 of the final layer group with the trailing tile LG1T7 of the middle layer group. Moreover, the trailing tile LG1T7 “disappears” completely in the initial layer group, in the sense that there are only seven tiles, instead of eight, in this layer group.
  • the tiles are processed in a multicore architecture with four cores (C0-C3).
  • the tiles are shaded/hatched according which core processes them.
  • FIG. 14B shows how the layer groups are processed by the cores, over time, with arrows indicating data-dependencies between tiles.
  • the first multicore tile pass the first four tiles are processed by the four cores C0-C3, respectively.
  • the leading tile LG0T0 is larger than the other tiles, and therefore core C0 takes longer to evaluate the layer group based on this tile than the other cores C1-C3 take to evaluate it based on the other tiles.
  • each core waits until all of the cores have finished evaluating the layer group.
  • cores C1-C3 wait for C0 to finish the leading tile LG0T0.
  • This provides a simple but effective way to achieve synchronisation between the cores.
  • all four cores continue with the evaluation of the second layer group, followed by the evaluation of the third layer group.
  • the evaluation of the leading tile LG2T0 is quicker than the other tiles. This is because the leading tile LG2T0 has been selected to be smaller than subsequent tiles LG2T1-LG2T7. This is done in order to avoid that the leading tile grows too large in the initial layer.
  • the aim is to try to balance the division of labour between the cores, and to try to avoid that a core finishes its work significantly earlier or later than any of the others.
  • the evaluation of a layer group for a given tile depends on the overlap data of the preceding output tile in the preceding layer group.
  • the evaluation of LG1T1 depends on both LG0T1 and LG0T0.
  • the evaluation of LG1T2 depends on LG0T2 and LG0T1, and so on.
  • This dependency between tiles within a multicore tile pass is shown by the solid arrows in FIG. 14B .
  • the dependency means that each core needs access to overlap data that was produced by another of the cores.
  • the overlap (non-disposable) data of each tile is stored in the shared OCM 314 b .
  • the remainder of the data (that is, the disposable data) can be stored in the local OCM 314 a of each core, because no other core needs to access it. This can avoid the need for each core to recalculate the overlap data, while also avoiding the costly overhead that may be involved in writing the overlap data to the DDR 312 . Meanwhile, the local OCM 314 a may be less costly to access than the shared OCM 314 b , providing improved efficiency compared with an implementation that stores all of the data in a shared OCM.
  • LG1T2 could start as soon as LG0T1 and LG0T2 are finished.
  • evaluation of LG1T0 could start at any time (that is, an indeterminate amount of time) after the evaluation of LG0T0 completes.
  • This kind of “free-running” approach can indeed be adopted in some other implementations.
  • the present implementation has the benefit of simplicity, as it avoids the added complexity necessary to keep track of the free-running.
  • the end of the first multicore tile pass is indicated by the thick black vertical line in FIG. 14B .
  • the cores proceed with a second multicore tile pass, to evaluate the layer groups based on the remaining tiles.
  • LG1-LG2 it will be understood that the fifth tile depends on overlap data that was produced when evaluating the fourth tile in the previous multicore tile pass.
  • the overlap data from LG0T0, LG0T1 and LG0T2 is used by the cores “immediately”, in the evaluation of LG1T1, LG1T2 and LG1T3, respectively, the overlap data from LG0T3 needs to be retained in the shared OCM 314 b until the second multicore tile pass, when it is used by core C0 in the evaluation of LG1T4.
  • the overlap data from LG1T3 is retained in the shared OCM 314 b until it is used by core C0 in the evaluation of LG2T4.
  • the splitter unit 620 defines the plurality of layer groups and arranges them in tile groups. The arrangement in tile groups can be done as described above with reference to FIGS. 10-11B .
  • the input data for the first layer group is provided in the DDR 312 .
  • the splitter unit 620 analyses the input data for the first layer group. In particular, it checks whether the size of the input data in the Y-dimension is above a threshold. If so, it determines (step 716 ) that the input data can be split along the Y-dimension into tiles. In other words, the input data will be split into tiles if it is of a sufficiently large size in the Y-dimension.
  • the splitter unit splits the data into tiles (step 718 ) and allocates each tile to one of the available cores (step 719 ). If there are more tiles than cores, then the splitter unit 620 will control the cores to carry out several multicore tile passes. In general, the splitter unit tries to split the input data into tiles in such a way that all cores are occupied. Therefore it may try (where possible) to split the input data into a number of tiles that is an integer multiple of the number of cores. The number of tiles (and number of multicore tile passes) will be influenced by the capacity of the OCM 314 .
  • step 720 at least two cores evaluate the first layer group based at least two respective tiles. That is, each core evaluates the first layer group based on one tile.
  • the splitter unit is configured to identify a disposable portion and a non-disposable (overlap) portion of each output tile. It instructs the cores to store these different portions in different places.
  • the first core 350 - 0 writes the disposable portion of a first output tile to its local OCM 314 a (step 722 ) and writes the overlap portion of the first output tile to the shared OCM 314 b (step 724 ).
  • the second core 350 - 1 writes the disposable portion of a second output tile to its local OCM 314 a (step 722 ), and writes the overlap portion of the second output tile to the shared OCM 314 b (step 724 ).
  • steps 720 - 724 Only two instances of steps 720 - 724 are shown in the flowchart, implying the presence of two cores; however, it will be understood that there are as many parallel instances of these steps as there are cores occupied with the evaluation. Note that step 722 and 724 may also be performed in the opposite order or in parallel.
  • any core that has finished its evaluation ahead of the others waits for all of the cores to finish their evaluation of the first layer group before proceeding.
  • the method proceeds to step 730 . If there are further layer groups in the current tile group, each core moves on to evaluate the next layer group based on its tile. When evaluating the second and subsequent layer groups, each core reads the necessary disposable data from the preceding layer group from its own local OCM 314 a , and reads the necessary overlap data from the preceding layer group from the shared OCM 314 b.
  • step 730 the method proceeds to step 732 , checking whether there are further tiles to be processed. This represents the end of one multicore tile pass. If there are further tiles to be processed, the cores proceed with a second multicore tile pass, in order to evaluate all of the layer groups of the current tile group for the next set of tiles. The method continues looping over multicore tile passes until all of the tiles have been processed. Then, in step 732 , when there are no more tiles to be processed, the evaluation of the tile group is complete and the method ends. The method may be repeated for subsequent tile groups, as necessary.
  • the splitter unit 620 determined that the input data could be split into tiles based on the size of the input data (in particular, in the Y-dimension).
  • the splitting into tiles depends on the tensor size being large enough.
  • a method is provided for doing this. An example of the method will now be described with reference to FIGS. 12, 16, and 17 .
  • the evaluation can be split across cores along one of the other dimensions. This may include spitting the evaluation of a layer group along the X-dimension, which is one of the traversed dimensions.
  • it may include splitting the evaluation of a layer group across one of the non-traversed dimensions—in particular, splitting across input channels, C, or splitting across output channels, F.
  • the input data for the layer group is not split up between the cores—it is stored in the shared OCM 314 b and accessed from there by each of the cores 350 .
  • This is a distinction from the use of tiles in the first aspect (as exemplified in FIGS. 13-15 ), in which both the data and the evaluation are split up among the cores.
  • each core evaluates the layer group for a respective channel (or set of channels) of the input data.
  • each core When splitting along the F-dimension, each core evaluates the layer group for a respective filter (or set of filters), producing a respective output channel, or set of output channels.
  • each core evaluates the layer group for a respective column (or set of columns) of the input data.
  • FIG. 16 illustrates schematically an example of splitting the evaluation of a plurality of layer groups into multiple passes, split across multiple cores, evaluating one layer group at a time.
  • Layer group LG2 comprises two layers; the other layer groups comprise three layers each.
  • the input data for the first layer group is stored in the DDR 312 .
  • the first layer group LG0 is split into three passes, to be evaluated by different cores. The division is along the X-dimension.
  • Each of the three cores C0-C2 evaluates a respective one of the three passes. In each pass, the respective core processes a portion of the input data to produce a portion of the output data.
  • the different cores may access overlapping portions of the input data, but they generate distinct portions of the output data. Unlike the case for FIG. 13 , the data is not split into separate tiles in this example. (It would be difficult to do this in the X-dimension, in the current example, because the data is arranged row-first and the data elements in the X-dimension do not line up with bursts of data in the memory.) Instead each core accesses the portion of data it needs from the shared OCM 314 b . As noted already, each core generates a respective portion of output data. When all the cores have evaluated the first layer group, the portions of output data are combined in the OCM 314 .
  • the next two layer groups LG1-LG2 are not split; therefore, each is evaluated in a single pass by one core.
  • the following layer group LG3 is split into six passes across output channels (the F-dimension). As there are more passes here than cores, the cores C0-C2 firstly evaluate the layer group for the first three passes in the F-dimension (labelled 0, 1, and 2 in the diagram). Then, the cores C0-C2 evaluate the layer group for the second set of three passes (labelled 3, 4, and 5, in the diagram), to complete the layer group.
  • the final layer group LG4 is again processed in a single pass by core C0.
  • the output of the first four layer groups LG0-LG3 is combined in the OCM 314 ; the output of the final layer group LG4 is written to the DDR 312 .
  • the splitting is flexible, in that different layer groups can be split up along different dimensions (compare LG0 with LG3, for example). This is achieved by combing the complete output of each layer group in the OCM 314 , which makes the splitting of one layer group independent of the splitting of the other layer groups.
  • This is another distinction from the tiled-based splitting of FIG. 13 , in which each core operates on a specific tile and the correspondence between tiles is preserved as the layer groups of a tile group are evaluated depth-first.
  • the splitter unit 620 defines the plurality of layer groups.
  • the input data for the first layer group is provided in the DDR 312 .
  • the splitter unit 620 analyses the input data, to check whether it can be split into tiles. In particular, the splitter unit checks whether the size of the input data in the Y dimension is above a threshold. In step 746 , the splitter unit 620 determines that the size is not above the threshold—therefore, the data is not suitable for tiling.
  • the splitter unit 620 splits the evaluation of the layer group into passes (step 748 ), and allocates each pass to a core (step 749 ).
  • the cores evaluate the layer group for their respective passes, to generate respective portions of output data, in step 750 . If it is determined, in step 752 , that there are further passes to complete in the current layer group, the cores evaluate the layer group for the necessary further passes (as in the example of layer group LG3 in FIG. 16 ). When it is determined, in step 752 that there are no further passes to evaluate for the present layer group, the method proceeds to step 754 , and the portions of output data from each of the passes are combined, to produce the output data of the layer group.
  • step 758 This output data is written to the OCM 314 in step 756 .
  • step 758 it is checked whether there is a further layer group to evaluate. If so, the method proceeds to step 748 again, to split the evaluation of the next layer group into passes. Note that, although only two instances of step 750 are shown in the flowchart, implying the presence of two cores, it will be understood that there are as many parallel instances of this step as there are cores occupied with the evaluation.
  • the output data of each layer group is written to the shared OCM 314 b , so that it is accessible to all cores for the evaluation of the following layer group.
  • the manner in which the evaluation is split, in step 748 , and the manner in which the portions of output data are combined, in step 754 will depend on the characteristics of the layers in the layer group, and the dimension along which the evaluation is split.
  • the evaluation may be split by input channel C—whereby each core evaluates the layer group for a respective channel (or set of channels) of the input data. For instance, if the layer group includes a convolutional layer, with one filter, and each core processes a different input channel, this would mean that each core applies one channel of the filter kernel to the respective channel of the input data.
  • the respective output portions produced by the different cores will then need to be combined by element-wise summation, to produce the output of the convolution. This can be done by an element-wise operations module of one of the cores 350 , if available, or by another processing module, separate from the cores 350 .
  • the weights (and biases) for the different channels of the kernel may be stored in the local OCM 314 a of each respective core, since each core only needs access to the weights for the channel (or channels) that it is responsible for evaluating.
  • the respective output portions produced by the different cores can be combined by concatenating them in the input channel dimension (P-dimension).
  • P-dimension the input channel dimension
  • the step of combining the respective output portions consists of concatenating them, this can be achieved by the individual cores simply writing the output portions to the appropriate locations in the shared OCM 314 b .
  • the “combining” of step 754 and the “writing” of step 756 may be carried out together, and may be carried out by each core immediately following the “evaluation” step 750 .
  • the parameters governing the element-wise operations for each input channel can be split up and stored in the respective different local OCMs 314 a .
  • the parameters governing the processing of each channel can be split up into the respective different local OCMs 314 a.
  • each core will be responsible for evaluating a given filter or set of filters.
  • the weights (and biases) for the relevant filters can be stored in the local OCM 314 a of the respective cores.
  • the layer group includes a convolutional layer, and the evaluation is split across the cores along one of the traversed dimensions (for example, the X-dimension), then the weights and biases are stored in the shared OCM 314 b , where they are accessible to all of the cores.
  • Some layers may be difficult, inefficient or impossible to split across multiple cores.
  • the layers LG1, LG2, and LG4 in FIG. 16 are not split across cores at all—they are executed in their entirety by a single core (potentially leaving the other cores idle, if there is no other work to occupy them). This approach can be used, for example, to process a fully connected layer comprising one filter.
  • the method of FIG. 15 can be used to process large input data tensors (that is, if the size is above the threshold, in the step 714 of analysing the size of the input data tensor).
  • the method of FIG. 17 can be used to process small input data tensors (if the size is below the threshold, in step 714 ′).
  • the threshold used in step 714 and the threshold used in step 714 ′ may be the same, such that either the method of FIG. 15 is followed, or the method of FIG. 17 is followed, by the hardware implementation 600 .
  • a processing system may comprise a hardware implementation as described above, or may be configured to perform a method as described above.
  • the processing system may be an artificial intelligence accelerator system—such as a neural network accelerator (NNA)—or a graphics processing system/graphics processing unit (GPU).
  • NNA neural network accelerator
  • GPU graphics processing system/graphics processing unit
  • FIG. 18 shows a computer system in which a processing system according to the present disclosure may be implemented.
  • the computer system comprises a CPU 902 , an NNA 904 , a memory 906 and other devices 914 , such as a display 916 , speakers 918 and a camera 919 .
  • a processing block 910 (corresponding to hardware implementation 600 ) is implemented on the NNA 904 . In other examples, the processing block 910 may be implemented on the CPU 902 .
  • the components of the computer system can communicate with each other via a communications bus 920 .
  • a store 912 (corresponding to DDR 312 ) is implemented as part of the memory 906 .
  • FIG. 18 illustrates one implementation of an artificial intelligence accelerator system
  • a similar block diagram could be drawn for a graphics processing system—for example, by replacing either the CPU 902 or the NNA 904 with a GPU, or by adding the GPU as an additional unit.
  • the processing block 910 can be implemented in the GPU.
  • the hardware implementation 600 of FIG. 12 is shown as comprising a number of functional blocks. This is schematic only and is not intended to define a strict division between different logic elements of such entities. Each functional block may be provided in any suitable manner. It is to be understood that intermediate values described herein as being formed by a hardware implementation need not be physically generated by the hardware implementation at any point and may merely represent logical values which conveniently describe the processing performed by the hardware implementation between its input and output.
  • the hardware implementations described herein may be embodied in hardware on an integrated circuit.
  • the hardware implementations described herein may be configured to perform any of the methods described herein.
  • any of the functions, methods, techniques or components described above can be implemented in software, firmware, hardware (e.g., fixed logic circuitry), or any combination thereof.
  • the terms “module,” “functionality,” “component”, “element”, “unit”, “block” and “logic” may be used herein to generally represent software, firmware, hardware, or any combination thereof.
  • the module, functionality, component, element, unit, block or logic represents program code that performs the specified tasks when executed on a processor.
  • a computer-readable storage medium examples include a random-access memory (RAM), read-only memory (ROM), an optical disc, flash memory, hard disk memory, and other memory devices that may use magnetic, optical, and other techniques to store instructions or other data and that can be accessed by a machine.
  • RAM random-access memory
  • ROM read-only memory
  • optical disc optical disc
  • flash memory hard disk memory
  • hard disk memory and other memory devices that may use magnetic, optical, and other techniques to store instructions or other data and that can be accessed by a machine.
  • Computer program code and computer readable instructions refer to any kind of executable code for processors, including code expressed in a machine language, an interpreted language or a scripting language.
  • Executable code includes binary code, machine code, bytecode, code defining an integrated circuit (such as a hardware description language or netlist), and code expressed in a programming language code such as C, Java® or OpenCL.
  • Executable code may be, for example, any kind of software, firmware, script, module or library which, when suitably executed, processed, interpreted, compiled, executed at a virtual machine or other software environment, cause a processor of the computer system at which the executable code is supported to perform the tasks specified by the code.
  • a processor, computer, or computer system may be any kind of device, machine or dedicated circuit, or collection or portion thereof, with processing capability such that it can execute instructions.
  • a processor may be any kind of general purpose or dedicated processor, such as a CPU, GPU, NNA, System-on-chip, state machine, media processor, an application-specific integrated circuit (ASIC), a programmable logic array, a field-programmable gate array (FPGA), or the like.
  • a computer or computer system may comprise one or more processors.
  • HDL hardware description language
  • An integrated circuit definition dataset may be, for example, an integrated circuit description.
  • a method of manufacturing at an integrated circuit manufacturing system, a hardware implementation as described herein. Furthermore, there may be provided an integrated circuit definition dataset that, when processed in an integrated circuit manufacturing system, causes the method of manufacturing a hardware implementation to be performed.
  • An integrated circuit definition dataset may be in the form of computer code, for example as a netlist, code for configuring a programmable chip, as a hardware description language defining hardware suitable for manufacture in an integrated circuit at any level, including as register transfer level (RTL) code, as high-level circuit representations such as Verilog or VHDL, and as low-level circuit representations such as OASIS (RTM) and GDSII.
  • RTL register transfer level
  • RTM high-level circuit representations
  • GDSII GDSI
  • one or more intermediate user steps may be required in order for a computer system configured for generating a manufacturing definition of an integrated circuit to execute code defining an integrated circuit so as to generate the manufacturing definition of that integrated circuit.
  • FIG. 19 shows an example of an integrated circuit (IC) manufacturing system 1002 which is configured to manufacture a hardware implementation as described in any of the examples herein.
  • the IC manufacturing system 1002 comprises a layout processing system 1004 and an integrated circuit generation system 1006 .
  • the IC manufacturing system 1002 is configured to receive an IC definition dataset (e.g. defining a hardware implementation as described in any of the examples herein), process the IC definition dataset, and generate an IC according to the IC definition dataset (e.g. which embodies a hardware implementation as described in any of the examples herein).
  • the processing of the IC definition dataset configures the IC manufacturing system 1002 to manufacture an integrated circuit embodying a hardware implementation as described in any of the examples herein.
  • the layout processing system 1004 is configured to receive and process the IC definition dataset to determine a circuit layout.
  • Methods of determining a circuit layout from an IC definition dataset are known in the art, and for example may involve synthesising RTL code to determine a gate level representation of a circuit to be generated, e.g. in terms of logical components (e.g. NAND, NOR, AND, OR, MUX and FLIP-FLOP components).
  • a circuit layout can be determined from the gate level representation of the circuit by determining positional information for the logical components. This may be done automatically or with user involvement in order to optimise the circuit layout.
  • the layout processing system 1004 may output a circuit layout definition to the IC generation system 1006 .
  • a circuit layout definition may be, for example, a circuit layout description.
  • the IC generation system 1006 generates an IC according to the circuit layout definition, as is known in the art.
  • the IC generation system 1006 may implement a semiconductor device fabrication process to generate the IC, which may involve a multiple-step sequence of photo lithographic and chemical processing steps during which electronic circuits are gradually created on a wafer made of semiconducting material.
  • the circuit layout definition may be in the form of a mask which can be used in a lithographic process for generating an IC according to the circuit definition.
  • the circuit layout definition provided to the IC generation system 1006 may be in the form of computer-readable code which the IC generation system 1006 can use to form a suitable mask for use in generating an IC.
  • the different processes performed by the IC manufacturing system 1002 may be implemented all in one location, e.g. by one party.
  • the IC manufacturing system 1002 may be a distributed system such that some of the processes may be performed at different locations, and may be performed by different parties.
  • some of the stages of: (i) synthesising RTL code representing the IC definition dataset to form a gate level representation of a circuit to be generated, (ii) generating a circuit layout based on the gate level representation, (iii) forming a mask in accordance with the circuit layout, and (iv) fabricating an integrated circuit using the mask may be performed in different locations and/or by different parties.
  • processing of the integrated circuit definition dataset at an integrated circuit manufacturing system may configure the system to manufacture a hardware implementation without the IC definition dataset being processed so as to determine a circuit layout.
  • an integrated circuit definition dataset may define the configuration of a reconfigurable processor, such as an FPGA, and the processing of that dataset may configure an IC manufacturing system to generate a reconfigurable processor having that defined configuration (e.g. by loading configuration data to the FPGA).
  • an integrated circuit manufacturing definition dataset when processed in an integrated circuit manufacturing system, may cause an integrated circuit manufacturing system to generate a device as described herein.
  • the configuration of an integrated circuit manufacturing system in the manner described above with respect to FIG. 19 by an integrated circuit manufacturing definition dataset may cause a device as described herein to be manufactured.
  • an integrated circuit definition dataset could include software which runs on hardware defined at the dataset or in combination with hardware defined at the dataset.
  • the IC generation system may further be configured by an integrated circuit definition dataset to, on manufacturing an integrated circuit, load firmware onto that integrated circuit in accordance with program code defined at the integrated circuit definition dataset or otherwise provide program code with the integrated circuit for use with the integrated circuit.
  • performance improvements may include one or more of increased computational performance, reduced latency, increased throughput, and/or reduced power consumption.
  • performance improvements can be traded-off against the physical implementation, thereby improving the method of manufacture. For example, a performance improvement may be traded against layout area, thereby matching the performance of a known implementation but using less silicon. This may be done, for example, by reusing functional blocks in a serialised fashion or sharing functional blocks between elements of the devices, apparatus, modules and/or systems.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Software Systems (AREA)
  • Computing Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Neurology (AREA)
  • Computer Hardware Design (AREA)
  • Image Analysis (AREA)
  • Semiconductor Memories (AREA)
  • Design And Manufacture Of Integrated Circuits (AREA)
US17/500,415 2020-10-13 2021-10-13 Implementation of a neural network in multicore hardware Pending US20220129741A1 (en)

Applications Claiming Priority (6)

Application Number Priority Date Filing Date Title
GB2016226.9 2020-10-13
GB2016226.9A GB2599910B (en) 2020-10-13 2020-10-13 Implementation of a neural network in multicore hardware
GB2016225.1A GB2599909B (en) 2020-10-13 2020-10-13 Implementation of a neural network in multicore hardware
GB2016225.1 2020-10-13
GB2016227.7A GB2599911B (en) 2020-10-13 2020-10-13 Implementation of a neural network in multicore hardware
GB2016227.7 2020-10-13

Publications (1)

Publication Number Publication Date
US20220129741A1 true US20220129741A1 (en) 2022-04-28

Family

ID=78000644

Family Applications (3)

Application Number Title Priority Date Filing Date
US17/500,840 Active 2042-03-30 US11875248B2 (en) 2020-10-13 2021-10-13 Implementation of a neural network in multicore hardware
US17/500,415 Pending US20220129741A1 (en) 2020-10-13 2021-10-13 Implementation of a neural network in multicore hardware
US17/500,206 Active 2042-05-21 US11853866B2 (en) 2020-10-13 2021-10-13 Implementation of a neural network in multicore hardware

Family Applications Before (1)

Application Number Title Priority Date Filing Date
US17/500,840 Active 2042-03-30 US11875248B2 (en) 2020-10-13 2021-10-13 Implementation of a neural network in multicore hardware

Family Applications After (1)

Application Number Title Priority Date Filing Date
US17/500,206 Active 2042-05-21 US11853866B2 (en) 2020-10-13 2021-10-13 Implementation of a neural network in multicore hardware

Country Status (3)

Country Link
US (3) US11875248B2 (fr)
EP (3) EP3985573A1 (fr)
CN (3) CN114358237A (fr)

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US12093696B1 (en) 2018-04-20 2024-09-17 Perceive Corporation Bus for transporting output values of a neural network layer to cores specified by configuration data
US11995533B1 (en) 2018-12-05 2024-05-28 Perceive Corporation Executing replicated neural network layers on inference circuit
US11615322B1 (en) * 2019-05-21 2023-03-28 Perceive Corporation Compiler for implementing memory shutdown for neural network implementation configuration
KR20220073306A (ko) * 2020-11-26 2022-06-03 에스케이하이닉스 주식회사 스토리지 장치 및 그 동작 방법
GB2625739A (en) * 2022-12-22 2024-07-03 Imagination Tech Ltd Mapping neural networks to hardware
WO2024167491A1 (fr) * 2023-02-08 2024-08-15 Google Llc Exécution efficace de calculs d'un réseau entièrement convolutif à sorties multiples
CN118312327A (zh) * 2024-06-06 2024-07-09 北京壁仞科技开发有限公司 硬件资源分配方法、电子设备与存储介质

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9679192B2 (en) * 2015-04-24 2017-06-13 Adobe Systems Incorporated 3-dimensional portrait reconstruction from a single photo
US10380413B2 (en) * 2017-07-13 2019-08-13 Robert Bosch Gmbh System and method for pose-invariant face alignment
GB2566702B (en) 2017-09-20 2021-11-03 Imagination Tech Ltd Hardware implementation of a deep neural network with variable output data format
US10380788B2 (en) * 2017-10-12 2019-08-13 Ohio State Innovation Foundation Fast and precise object alignment and 3D shape reconstruction from a single 2D image
US10832133B2 (en) * 2018-05-31 2020-11-10 Neuralmagic Inc. System and method of executing neural networks
CN111222561B (zh) * 2019-12-31 2023-06-09 深圳云天励飞技术股份有限公司 图像识别神经网络处理方法、装置与系统
US11544191B2 (en) * 2020-03-26 2023-01-03 Intel Corporation Efficient hardware architecture for accelerating grouped convolutions

Also Published As

Publication number Publication date
CN114358237A (zh) 2022-04-15
CN114358239A (zh) 2022-04-15
US11875248B2 (en) 2024-01-16
CN114358238A (zh) 2022-04-15
US20220147832A1 (en) 2022-05-12
EP3985573A1 (fr) 2022-04-20
EP3985572A1 (fr) 2022-04-20
US11853866B2 (en) 2023-12-26
US20220121914A1 (en) 2022-04-21
EP3985570A1 (fr) 2022-04-20

Similar Documents

Publication Publication Date Title
US20220129741A1 (en) Implementation of a neural network in multicore hardware
US12050986B2 (en) Neural network architecture using convolution engines
US20210174180A1 (en) Hardware Implementation of a Neural Network
GB2599909A (en) Implementation of a neural network in multicore hardware
CN114722994A (zh) 在硬件中实现全连接神经网络层
GB2599910A (en) Implementation of a neural network in multicore hardware
US12061972B2 (en) Hardware implementation of a neural network
US20240143986A1 (en) Methods and systems for executing a neural network on a neural network accelerator
GB2599911A (en) Implementation of a neural network in multicore hardware
GB2621217A (en) Implementation of a neural network in multicore hardware
GB2621218A (en) Implementation of a neural network in multicore hardware
GB2611658A (en) Hardware implementation of a neural network
GB2625215A (en) Implementation of a neural network
US20240232596A1 (en) Implementing neural networks in hardware
US20240232597A1 (en) Mapping neural networks to hardware
Aiello Design of optimized architectures for non-linear convolutional neural networks
GB2602494A (en) Implementing fully-connected neural-network layers in hardware

Legal Events

Date Code Title Description
AS Assignment

Owner name: IMAGINATION TECHNOLOGIES LIMITED, UNITED KINGDOM

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:HUANG, XIRAN;ESCOBAR, FERNANDO;SIGNING DATES FROM 20211014 TO 20211015;REEL/FRAME:058630/0238

STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

AS Assignment

Owner name: FORTRESS INVESTMENT GROUP (UK) LTD, NEW YORK

Free format text: SECURITY INTEREST;ASSIGNOR:IMAGINATION TECHNOLOGIES LIMITED;REEL/FRAME:068221/0001

Effective date: 20240730