WO2020072619A1 - Addressing bottlenecks for deep neural network execution of a graphics processor unit - Google Patents

Abstract

A method includes receiving a non-optimized deep neural network (DNN), identifying sets of contributing and/or non-contributing synapse vectors, and generating an optimized DNN based on the non-optimized DNN. A method includes loading two strings into a first register, loading contents of the first register into an on-chip register, loading a first set of bits of the on-chip register into a second register, loading a second set of bits of the on-chip register into a third register, computing on the second register and third register, and writing the contents of the second register and third register to off-chip memory. A method includes extending a parallel thread instruction set architecture of a processor. A processor includes a plurality of floating point units including a data fission unit and an instruction unit. A runtime system includes an off-chip memory, registers, and on-chip memory. The runtime system includes a synapse vector elimination kernel.

Description

ADDRESSING BOTTLENECKS FOR DEEP NEURAL NETWORK EXECUTION OF A GRAPHICS PROCESSOR UNIT

STATEMENT OF GOVERNMENTAL INTEREST

[0001] This invention was made with government support under CCF-1438996,

IIS153901 1 , and CCF1628991 awarded by the National Science Foundation. The government has certain rights in the invention.

CROSS-REFERENCE TO RELATED APPLICATION

[0002] The present application claims priority to United States patent application no. 62/740,267, filed October 2, 2018. The priority application is hereby incorporated by reference.

FIELD OF THE DISCLOSURE

[0003] The present disclosure generally relates to a system and method for addressing bottlenecks for deep neural network execution in a graphics processor unit (GPU). More particularly, methods and systems for facilitating synapse vector elimination and near compute data fission.

BACKGROUND

[0004] The background description provided herein is for the purpose of generally presenting the context of the disclosure. Work of the presently named inventors, to the extent it is described in this background section, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against the present disclosure.

[0005] As user demand for state-of-the-art technologies in the domains of computer vision, speech recognition, and natural language processing (NLP) continues to increase, system designers are tasked with supporting increasingly sophisticated machine learning (ML) capabilities. An important trend that impacts the design of current and future intelligent systems is the convergence of industry toward deep learning as the computational engine providing these services. Large technology companies are using deep neural networks (DNNs) as the primary technique underpinning machine learning for vision, speech, and NLP tasks. With an increasing number of queries requiring DNN computation on the critical path, a significant challenge emerges vis-a-vis the large gap between the amount of computation required to process a DNN-based query relative to a traditional browser centric query such as web search. Researchers have recently been investigating the role of accelerators such as application-specific integrated circuits (ASICs) and field-programmable gate arrays (FPGAs) to help bridge this gap. However, these specialized hardware solutions require substantial adjustments across the hardware-software stack as well as re-designing and redeploying of servers which is an obstacle for wide-scale adoption. To avoid this burden on existing infrastructure, deep learning frameworks have embraced commodity accelerators like GPUs. However, significant improvement beyond current GPU performance is needed to bridge the scalability gap for DNN computation.

[0006] Like biological neural networks, DNNs are intrinsically resilient to both minor numerical adjustments and eliding spurious neurons and synapses. This characteristic can be leveraged to achieve performance improvement. However, reduction of computation and data movement may not directly translate to performance improvement. Optimization techniques from prior work may create a mismatch between the algorithm and underlying architecture, and/or may not be designed to address the real hardware bottlenecks, leaving two open challenges in the way of realizing performance benefits. Specifically, that of irregular computation and lack of bottleneck optimization.

[0007] Regarding irregular computation, prior work has focused on network pruning, a machine learning technique that reduces the DNN topology by reducing the memory footprint. However, past work has failed to realize performance benefits on graphics processor units (GPUs). Although network pruning may significantly reduce the amount of raw computation (/.e., floating-point operations), the hardware-inefficient irregular DNN topology may outweigh the benefits and results in substantial slowdown due to increased branch divergence and uncoalesced memory access on GPUs. Removed computation may occur sporadically throughout the DNN topology, limiting benefits on commodity

architectures. GPU hardware, which requires contiguous data structures for efficient computation, presents a significant challenge when omitting arbitrary neurons/ synapses.

[0008] Branch divergence occurs wherein some threads partitioned into a group by hardware need to execute instructions different from the other threads into the group. The hardware may be designed such that all of the threads in a group must execute instructions in lockstep. This requires that divergent sections of code are executed sequentially, causing omitted computation due to noncontiguous data structures/ memory to result in idle hardware, rather than efficient execution. Uncoalesced memory access occurs in the memory subsystem when multiple threads in a thread group are stored in noncontiguous addresses, such that when the threads issue memory instructions, to nonconsecutive addresses, the memory bus is underutilized. The slowdown caused by branch divergence and uncoalesced memory access may be large ( e.g ., 61 x).

[0009] Regarding lack of bottleneck optimization, prior work has focused on improving off- chip memory using compression, removing non-contributing bits to increase the effective bandwidth. This technique, however, fails to provide significant speedups for DNNs. For example, off-chip data packing may result in a speedup of less than four percent. On-chip compression has not been effectively utilized in past approaches, because frequently reformatting data is difficult to achieve without introducing significant overhead.

[0010] Therefore, techniques for addressing irregular computation and lack of bottleneck optimization are needed, to achieve performance benefits by reducing computation while aligning the reduced computation with underlying hardware.

BRIEF SUMMARY

[0011] In one aspect, a computer-implemented method of optimizing a deep neural network includes receiving a non-optimized deep neural network having a set of synapse vectors, identifying one or both of a contributing subset of the set of synapse vectors in the deep neural network, and a non-contributing subset of the set of synapse vectors in the deep neural network; and generating, based on the non-optimized deep neural network, an optimized deep neural network, the optimized deep neural network having less inferential accuracy than the non-optimized deep neural network.

[0012] In another aspect, a computer-implemented method of improving the effective bandwidth of a deep neural network includes loading two strings of n-significant bits of two respective floating-point values into a first register, wherein the two strings of n-significant bits are contiguous with respect to the first register, and wherein n is an even integer, loading the contents of the first register into an on-chip register, loading the first n/2 bits of the on-chip register into a second register, and loading the second n/2 bits of the on-chip register into a third register. The method may also include modifying, by computation, the contents of the second register and the third register and writing the contents of the second register and third register to off-chip memory.

[0013] In another aspect, a method of improving the effective bandwidth of a DNN executing in a graphics processor unit may include extending the parallel thread execution (PTX) instruction set architecture (ISA) of the graphics processor unit with a first instruction configured to invoke a 16-bit data fission unit (DFU) and a second instruction configured to invoke an 8-bit DFU, wherein the 16-bit DFU is parameterized with a 32-bit source register and two contiguous 32-bit destination registers, and wherein the 8-bit DFU is parameterized with a 32-bit source register and four contiguous 32-bit destination registers.

[0014] In another aspect, a processor may include a plurality of floating-point units (FPUs), wherein each FPU includes a data fission unit (DFU) to apply a data fusion instruction, the instruction specifying a source register and a plurality of contiguous destination registers; and an execution unit to execute the instruction to generate a fused result comprising the result of converting an n-bit floating-point representation to an IEEE single-precision value.

[0015] In another aspect, a DNN runtime operation system may include an off-chip memory, a processor including a set of registers, and a synapse vector elimination kernel application comprising a set of computer-executable instructions stored in the off-chip memory, and a data fission application comprising a set of computer-executable instructions stored in the off-chip memory. When the instructions of the synapse vector elimination kernel are executed by the computer processor, the instructions may cause the DNN runtime operation system to receive a non-optimized trained DNN, wherein the non-optimized trained DNN includes a set of synapse vectors. The synapse vector elimination kernel application may include further instructions that, when executed, cause the DNN operating system to identify a non-contributing subset of the set of synapse vectors, prune the non-contributing subset from the set of synapse vectors to produce an optimized DNN, wherein the optimized DNN loses some inferential accuracy. The synapse vector elimination kernel application may include further instructions that, when executed, cause the DNN operating system to tune the optimized DNN to recover at least some of the lost inferential accuracy, receive input formatted for the non-optimized trained DNN, reorganize the input formatted for the non-optimized trained DNN, and apply the reorganized input to the optimized DNN. When the instructions stored in the data fission application are executed, the instructions may cause the DNN operation runtime to load two strings of n-significant bits of two respective floating-point values into a first register, wherein the two strings of n-significant bits are contiguous with respect to the first register, and wherein n is an even integer, load the contents of the first register into an on-chip register, load the first n/2 bits of the on-chip register into a second register, load the second n/2 bits of the on-chip register into a third register, modify, by computation, the contents of the second register and the third register, and write the contents of the second register and third register to off-chip memory.

[0016] In another aspect, a method of operating a deep neural network operation runtime may include receiving a non-optimized trained deep neural network having a set of synapse vectors, pruning a non-contributing subset of synapse vectors from the trained deep neural network to produce an optimized deep neural network, wherein the optimized deep neural network loses some inferential accuracy, tuning the optimized deep neural network to recover at least some of the lost inferential accuracy, receiving input formatted for the non- optimized trained deep neural network, reorganizing the input formatted for the non- optimized trained deep neural network, and applying the reorganized input to the optimized deep neural network, where applying the reorganized input to the optimized deep neural network comprises one or both of (i) software-based data fission and (ii) a hardware-based data fission unit.

BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 depicts an example deep neural network (DNN) operation runtime, according to an embodiment,

[0018] Figure 2A depicts a GPU architecture flow diagram, according to an embodiment,

[0019] Figure 2B depicts an architecture-efficient flow diagram for performing of synapse vector elimination, according to an embodiment,

[0020] Figure 3 depicts a flow diagram for synapse reordering step and matrix truncation step, according to an embodiment,

[0021] Figure 4 depicts an example computational flow diagram, according to one embodiment and scenario,

[0022] Figure 5 depicts an example data fission unit, according to an embodiment,

[0023] Figure 6 depicts an example method for optimizing a DNN, according to an embodiment,

[0024] Figure 7 depicts an example method for improving the effective bandwidth of a DNN, according to an embodiment,

[0025] Figure 8 depicts an example method improving the effective bandwidth of a DNN, according to an embodiment;

[0026] Figure 9 depicts an example method of operating a DNN operation runtime, according to an embodiment; and

[0027] Figure 10 depicts a block diagram of a computing system including a GPU for implementing a DNN operation runtime, according to one embodiment and scenario.

DETAILED DESCRIPTION

[0028] Although the following text sets forth a detailed description of numerous different embodiments, it should be understood that the legal scope of the description is defined by the words of the claims set forth at the end of this text. The detailed description is to be construed as exemplary only and does not describe every possible embodiment since describing every possible embodiment would be impractical, if not impossible. Numerous alternative embodiments could be implemented, using either current technology or technology developed after the filing date of this patent, which would still fall within the scope of the claims. [0029] It should also be understood that, unless a term is expressly defined in this patent using the sentence "As used herein, the term " " is hereby defined to mean . . . " or a similar sentence, there is no intent to limit the meaning of that term, either expressly or by implication, beyond its plain or ordinary meaning, and such term should not be interpreted to be limited in scope based on any statement made in any section of this patent (other than the language of the claims). To the extent that any term recited in the claims at the end of this patent is referred to in this patent in a manner consistent with a single meaning, that is done for sake of clarity only so as to not confuse the reader, and it is not intended that such claim term be limited, by implication or otherwise, to that single meaning. Finally, unless a claim element is defined by reciting the word "means" and a function without the recital of any structure, it is not intended that the scope of any claim element be interpreted based on the application of 35 U.S.C. § 1 12(f).

[0030] In general, this work describes methods and systems for optimizing DNNs for execution on GPUs. A first technique, synapse vector elimination, reduces the total problem size by automatically locating and discarding non-contributing synapses in a DNN to improve performance. Generally, non-contributing synapses are those synapses having negligible or no impact on the output results. To address the limitation of irregular computation, it may be necessary to preserve existing architectural optimizations in original GPU-efficient applications. Utilizing this insight, synapse vector elimination transforms the DNN data layout, producing computations that efficiently leverage GPU hardware.

[0031] A second technique, near-compute data fission, mitigates the GPU on-chip memory bandwidth bottleneck by optimizing the utilization of integer units during DNN execution. On-chip memory is closer to the functional units than off-chip memory. To address the prior work’s limitation of providing only off-chip bandwidth optimization, the present techniques may support low-overhead very fine-grained data conversion by focusing on low decompression overhead, rather than high compression ratio. The present techniques also include a Data Fission Unit (DFU) that can be added to existing GPU hardware to obviate data fission overhead to realize additional benefits on future generations of GPU hardware ( e.g ., resulting in a 1 Ax speedup over software-only near-compute data fission).

EXAMPLE DEEP NEURAL NETWORK OPERATION RUNTIME

[0032] Figure 1 depicts various aspects of an example DNN operation runtime 100, including a GPU DNN execution framework for optimizing DNN inference by tailoring the DNN to the underlying architecture. The operation runtime 100 may include a baseline infrastructure, a synapse vector elimination layer, and a data fission layer. The baseline infrastructure may include an offline training phase for optimizing the DNN topology. The two optimizations, synapse vector elimination and near-compute data fission, may work in concert to apply optimizations automatically and transparently to unmodified DNN

applications. As discussed herein, near-compute data fission may include hardware acceleration using a GPU hardware extension to mitigate overhead. Synapse vector elimination may add minimal overhead, and as such, implementing synapse vector elimination in hardware may not be worth the additional costs.

[0033] The operation runtime 100 may include a set of training input 102 and a DNN configuration 104 that specifies the topology of the DNN. Using the training inputs 102, a set of DNN parameters are adjusted iteratively in a training module 106 until the classification loss function converges. The training module 106 produces a trained DNN model 108.

[0034] After the training module 106 produces the baseline trained DNN model 108, a synapse search module 1 10 may determine a set of non-contributing synapse vectors, which are groups of synapses that are architecturally efficient to eliminate on the GPU. As discussed below, the synapse search module 110 may locate and remove non-contributing synapse vectors from the trained DNN 108, wherein the non-contributing synapse vectors may be defined as any vector highly correlated with another vector. As illustrated in Figure 1 , the synapse search module 110 may produce a reduced DNN model 112 that has some set of synapse vectors eliminated from the computation.

[0035] The operation runtime 100 may include a fine tuning module 114, which may produce a fine-tuned DNN model 116. Although the retained synapse vectors in the reduced DNN module 112 are chosen to be representative of those that were eliminated by the synapse search module 1 10, the nuanced impact of the missing, eliminated synapse vectors may result in accuracy degradation if used directly. To remedy this, the fine tuning module 1 14 may refine the DNN parameters by applying a small number of DNN training iterations. This process may allow the DNN model to fully recover accuracy that is lost from minor perturbations of the weights or topology. By using fine tuning after applying synapse vector elimination, the operation runtime 100 may produce a DNN model having negligible loss in inference accuracy.

[0036] Beyond the training mechanism used by the operation runtime 100 to produce an efficient DNN model, the operation runtime 100 may service a DNN application using by allowing inputs formatted for an unoptimized DNN model to be applied to a DNN model including synapse vector elimination optimizations. Specifically, a synapse vector elimination kernel 120 may reorganize input activation values prior to inference so that the values can be applied to an optimized DNN model. A detailed description of the architecture-efficient synapse vector elimination kernel is provided below. Although reorganization may consume some kernel execution time, this is an overhead dwarfed by the reduction in computation facilitated by synapse vector elimination.

[0037] In addition to reducing the size of the DNN using synapse vector elimination, the operation runtime 100 may optimize the key GPU bottleneck and on-chip memory

bandwidth, within the inference kernel using near-compute data fission. Near-compute data fission may pack DNN weights and activations into on-chip memory by removing non contributing bits from the numerical representation. Because this technique resides in the low-level computational DNN kernels at runtime, no further changes are required to the baseline infrastructure to utilize this optimization. Near-compute data fission is discussed further below.

EXAMPLE SYNAPSE VECTOR ELIMINATION

[0038] Synapse vector elimination removes non-contributing synapses from DNNs, thereby reducing the total computation required for the DNN to process its inputs. Synapse vector elimination may include discovering and removing performance-exploitable

noncontributing synapses. Previous network pruning techniques produce an inefficient mapping of operations to hardware. As noted, these techniques modify the computational kernel to be irregular, limiting performance benefits due to branch divergence and

uncoalesced memory access. Instead, synapse vector elimination retains a hardware- efficient design by transforming DNN inputs for similarly-structured but smaller DNN computations. Many DNNs have a large number of synapses that can potentially be eliminated. Without considering the underlying architecture, the selection of non-contributing synapses is fairly straightforward: network pruning techniques simply select the synapses with the lowest weights. In general, the granularity of synapses that should be removed is constrained by the architecture, thus the selection of synapses becomes a multi-dimensional optimization problem. The present techniques include a search technique to solve the multi dimensional optimization problem based on the correlation matrix formed by the architectural groups of synapse weights.

[0039] Figure 2A depicts a GPU architecture data flow diagram 200. Matrix operations {e.g., matrix multiplication) are performed on a weight matrix of size KxM and an input matrix of size KxN to generate an output matrix of size MxN. Figure 2B depicts a GPU

architecture-efficient data flow diagram 220, corresponding to the data flow diagram 200, for performing synapse vector elimination, that avoids the performance pitfalls associated with network pruning by applying a preprocessing step to efficiently rearrange computation.

Specifically, the data flow diagram 200 includes an original neural network computation step (a) is carried out by multiplying an M c K weight matrix 202 by a K c N input (output) matrix 204 of the previous layer. The corresponding synapse vector elimination step (b) of Figure 2B depicts preprocessing an input matrix 210 and weight matrix 212 to reduce the total problem size. The weight matrix 210 may be preprocessed offline, allowing the weight matrix 210 to be reused many times, while the input matrix 212 may be preprocessed during runtime to allow seamless switching between the original computation step (a) and the synapse vector elimination optimized computation step (b). Smaller matrices 214 resulting from the step (b) may be provided to a matrix multiplication algorithm 216. Performance benefits may result from applying a computationally inexpensive transformation that reduces the size of the inner dimension (/.e., K in the figure) of the matrix multiplication algorithm 216.

[0040] Figure 3 is a flow diagram 300 depicting a synapse reordering step (A) and a matrix truncation step (B), which taken together, may correspond to the synapse vector elimination step (b) of Figure 2B. First, the synapse reordering step (A) may efficiently reposition rows and columns of a neural network’s matrices so that the matrices are easier to manipulate. Next, matrix truncation may be used to reduce the amount of computation required for matrix multiplication while preserving the uniform data structure of the matrix. Finally, a correction factor may be applied to the matrices to retain the scale of the output values.

[0041] In the reordering step (A), synapses may be reordered to simplify the task of discarding unwanted synapses. The central goal of reordering is to preserve a matrix data structure’s uniformity without diminishing the gains of skipping synapses. An efficient method of grouping the retained and discarded synapses may include grouping the synapses ( e.g ., rows in a weight matrix 304 and columns in an input matrix 302) together based on whether the synapses will be discarded or retained. The input matrix 302 and the weight matrix 304 may correspond, respectively, to the input matrix 212 and the weight matrix 212 of Figure 2B. In some embodiments, an equivalent reordering may be applied to the transpose of the input matrix 302.

[0042] The number of discarded synapse weights, D, is known before the reordering synapses step (A). A matrix passed to the synapse reordering step (A) may be partitioned at column K - D so that the K - D columns on the left represent the retained synapse group and the D columns on the right represent the discarded one. After defining this partition point, some of the synapses that are to be retained may already be contained in the retained synapse partition. There may be an equal number of synapses to be retained as discarded that are in the incorrect partition. Therefore, a pairing between misplaced retained synapses and misplaced discarded synapses may be created. The passed matrix may then be reordered by swapping the two columns for each of these pairs. After swapping all of the misplaced columns, the retained and discarded synapses are strictly separated at column K - D.

[0043] The synapse reordering step (A) is more efficient than naively copying all retained synapses into a separate buffer. Experimentally, more than 50% of synapses are needed to retain accuracy, and using this discarding rate, synapse reordering is at least 1 Ax faster than copying retained synapses. As noted, it is impractical to design hardware for synapse reordering, since there is little overhead involved in synapse vector elimination.

[0044] The matrix truncation step (B) includes reducing the dimensions of the input matrix 302 and the weight matrix 304 to reduce the required amount of computation. The following equation describes the formula for computing the value of a neuron (/.e., a cell of the output matrix), where Out is the output matrix, W is the weight matrix 304, In is the previous layer matrix, i is the input index ( e.g ., the convolution kernel index or the fully connected input vector index), and j represents the input neuron index:

[0045] After the synapse reordering step (A), the output of the matrix multiplication is the same as it would be without reordering. Only the order of the weighted sum is changed. Therefore, the output is equivalent to the following equation, wherein the K - D synapses in the ordered matrices, W’ and In’ are summed first, and then the discarded synapses D are summed:

[0046] To remove the computation for the discarded synapses, the summation is stopped at the partitioned K - D instead of at K. In other words, the last D columns are sliced from the input matrix 302, and the last D rows are sliced from the weight matrix 304.

[0047] To compensate for the discarded synapses in each summation, the magnitude of the retained synapses is increased, so that the expected value of the original and optimized results match. Assuming that the synapses are all drawn from a similar distribution, the expected value is equal to the expected value of any single synapse, multiplied by the number of synapses:

[0048] The expected value of this sum, after removing the discarded synapse, may be represented similarly:

[0049] To match the expected value from synapse vector elimination to the original expected value, the weighted sum may be scaled by the ratio between the unadjusted expected value from the synapse vector elimination and the original expected value. This produces a final expression for the synapse discarded summation:

[0050] The synapse search module 110 of Figure 1 may use the above method of efficiently discarding synapses to find the synapses that are non-contributing to the final output. Trying all combinations of synapses is intractable, because a DNN may include many ( e.g ., thousands or more) synapses, and each synapses may be either retained or discarded, leading to 2^#synapses possibilities.

[0051] Prior methods of discarding synapses from the DNN at a per-synapse granularity are trivial, discarding those synapses with near-zero weights. Such a pruning strategy may be effective for pruning sporadic synapses, GPU-efficient optimizations must discard synapses in groups to exploit wide-vector unit hardware. A synapses vector pruning search mechanism may choose to retain or discard each architectural group of synapses, referred to herein as synapse vectors, rather than single synapses. Such vector-wise synapse pruning does not allow the use of near-zero weight pruning, so a new method is required. Specifically, instead of discarding synapses with weights nearest to zero, a subset of the synapse vectors that are representative of the entire set of synapses are retained.

[0052] Representativeness may be determined by computing a correlation matrix, p, for the synapse vectors, where S_x is the synapse vector for the group of synapses at index x\

[0053] For each synapse vector S*, a set of synapse vectors is generated that S* can represent. S_t maybe defined to be representative of S_j when the correlation between the two synapse vectors, denoted r_{ί 7}·, is above a representative correlation threshold a:

[0054] The operation runtime 100 may select the synapse vector that represents the most synapse vectors, , to be retained in the output DNN from synapse vector elimination, while the non-contributing synapse vectors represented by the retained one are removed. The operation runtime 100 may repeat the process of selecting the representative vectors until all synapse vectors are either retained or discarded.

[0055] In some embodiments, the representative correlation threshold a may be adjusted. In addition to selecting non-contributing synapse vectors, synapse vector elimination can be parameterized to discard marginally-contributing synapse vectors by adjusting the representative correlation threshold. As the correlation threshold is lowered, the number of synapse vectors that can be represented by a single synapse increases. This capability can be used to enact approximate computing, essentially shedding small amounts of accuracy to realize improved performance. In some embodiments, a user may be provided with a facility for adjusting the accuracy, such as by adjusting a numeric accuracy scale or by otherwise providing an input upon which the accuracy of the system may be based.

[0056] The number of such readily available performance-accuracy trade-off

configurations is limited due to large DNN memory footprints, when each configuration is stored in memory separately. To greatly increase the flexibility of synapse vector elimination, applied to marginally-contributing synapses, the operation runtime 100 may dynamically build DNNs using combinations of layers that were trained with varying correlation thresholds. Each <layer, correlation threshold> pair may be fine-tuned independently of the others, allowing arbitrary combinations of these pairs to be composed during runtime without requiring a new DNN model for each combination. Given a performance or accuracy constraint, the operation runtime 100 may quickly select an appropriate set of correlation thresholds for each of the layers. To do this, the operation runtime 100 may be configured to build a Pareto frontier of configurations during training and to select the configuration that is nearest to the user-specified goal during runtime. As discussed below, the configurations and trained networks may be stored in an electronic database.

EXAMPLE NEAR-COMPUTE DATA FISSION

[0057] As noted above, some embodiments of the present techniques may include a technique to achieve speedup by alleviating the DNN processing bottleneck on GPUs by effectively exchanging one hardware resource for another. There are three main hardware resources on a GPU that are susceptible to becoming a bottleneck: the functional units, the off-chip memory bandwidth, and the on-chip memory bandwidth. Empirical data demonstrates that GPU-based systems are greatly limited by on-chip memory bandwidth due to optimized matrix multiplication, the main underlying GPU kernel for DNN inference, which makes use of loop tiling. Loop tiling optimization allows on-chip memory storage and registers to be traded for off-chip memory bandwidth and on-chip memory bandwidth, respectively. While there may be sufficient on-chip memory storage to sufficiently reduce off-chip memory bandwidth, the on-chip memory bandwidth remains a bottleneck due to the limited number of registers available for loop tiling. As an example, the state-of-the-art Titan X (Pascal) GPU provides 1 1 single-precision TFLOPS (i.e. 44 TB/s), but its on-chip memory bandwidth is limited to 3.6 TB/s (frequency c # shared memory banks c bus width = 1 GHz c 28 banks c 128 bytes). While loop tiling at the register level mitigates this throughput gap, on-chip memory bandwidth is still the limiting resource due to the limited number of registers available for tiling.

[0058] To alleviate the on-chip memory bandwidth bottleneck, unused functional unit cycles can be leveraged to compress on-chip memory. Unfortunately, existing GPU memory compression techniques may only leverage off-chip memory by compressing the data in off- chip memory, while storing the decompressed data in on-chip memory. Although this can reduce off-chip memory bandwidth, it provides no benefit for DNNs because as noted, on- chip memory bandwidth is the performance bottleneck. Moving existing memory

compression techniques closer to the functional units is more complex than simply applying an existing compression technique at a different place in the memory hierarchy. The central challenge when moving the compressed data closer to the compute is that the

decompression overhead can outweigh the gains of reduced memory bandwidth and storage. The bandwidth for on-chip memory, however, is much greater than that of off-chip memory, making the size of the compressed data format less critical. The differences in proximity to functional units and available bandwidth cause a fundamental shift in the compression design space. While off-chip data packing focuses on larger reductions in memory bandwidth, a solution to this problem for DNNs must focus on minimizing decompression overhead. Therefore, a near-compute data fission technique is hereby disclosed which mitigates the GPU bottleneck in the system by targeting on-chip memory bandwidth. The present techniques achieve speedup by treating fission overhead as the paramount characteristic of the design.

[0059] In particular, in an embodiment, a near-compute data fission technique fuses multiple values into a single value of lesser size in on-chip memory to improve effective bandwidth. Near-compute data fission directly improves performance, because DNN computation is bottlenecked by on-chip memory bandwidth. Although on-chip memory bandwidth is a key limitation of DNN performance, fission at the on-chip level of the memory hierarchy requires very frequent data reformatting, causing excessive overhead, unless the data format is carefully chosen. In some embodiments, a standard CUDA-supported half precision format may be used; however, such a format may be insufficient for near-compute data fission. A new format is disclosed herein that results in far better performance due to reduced reformatting overheads. To exploit the non-contributing bits further, by reducing the reformatting overhead, the present techniques include a hardware device allowing conversion to narrower numerical representations.

[0060] Figure 4 depicts an example computational flow diagram. An original computation (a) including no fission may include loading data from off-chip memory into registers. To improve performance, the values in registers may be stored into an on-chip memory scratchpad for future reuse. An application may read from and compute on the data stored in scratchpad memory many times. Finally, the result may be written to off-chip memory.

[0061] Computation (b) may include the addition of near-compute data fission. As in the original computation (a), data is loaded from the off-chip memory into the register file.

Instead of writing directly to the scratchpad memory, multiple values are fused into a single element by data packing. Similarly, each time the application reads from the scratchpad memory, data fission unpacking is applied to the value before it is computed on. This process removes the non-contributing bits from the numerical representation in the on-chip memory. The fused data may not be stored into the off-chip memory because as noted above, the off-chip memory utilization is already very low.

[0062] Figure 4 depicts three near-compute data fission reduced precision floating-point data formats. The three formats include a first data format IEEE half format (c), a Deft-16 data format (d), and a Deft-16Q data format (e). Each of the three formats include a sign (S), a mantissa (M) that specifies the precision, and an exponent (E) that denotes dynamic range. Empirical testing has shown that the IEEE half format (c) ( e.g ., a IEEE 754 half precision data format) results in excessive reformatting overhead, resulting in slowdown, due to the complex conversion taking several cycles. The Deft-16 data format (d) takes advantage of a special floating-point format defined as the 16 most significant bits of the IEEE single-precision floating-point format, including a data type with 8 exponent bits and 7 mantissa bits, which provides sufficient precision and dynamic range for DNN workloads. Inexpensive shift and bitwise operations may be the only operations necessary to apply fission to values formatted in the Deft-16 data format (d).

[0063] The Deft-16 data format (d) may be further optimized by removing one instruction from the fission process, by allowing the most significant bits of one value to spill into the least significant bits of the other value. Such a data format is the Deft-16Q data format (e). Despite only reducing the fission process by a single logical AND instruction, the Deft-16 quick format ( i.e ., Deft-16Q data format (e)) performs substantially better than the Deft-16 data format (d) and the I EE Half data format (c). When applying near-compute data fission, the fission operation is on the critical path. Nevertheless, while the optimized fission process using Deft-16Q data format (e) may be specific to 16-bit data and may introduce some overhead. To address both of these limitations, a small additional hardware unit may be used to perform the fission operation, as discussed in the next section.

EXAMPLE DATA FISSION HARDWARE

[0064] In general, a Data Fission Unit (DFU) may be used to accelerate fission in conjunction with the techniques discussed with respect to Figure 4. The DFU may be a lightweight GPU hardware extension that is replicated for each floating-point unit to maintain high throughput. A central design goal of the DFU may be the minimization of area overhead. For this reason, the DFU may be specialized for the data representations that are most likely to be beneficial in a GPU context. In particular, the DFU may be specifically targeted to accelerate the fission of custom 8-bit floating-point and Deft-16Q representations, such as Deft-16Q data format (e) discussed with respect to Figure 4.

[0065] In an embodiment, DFU fission operations may be accessed via a parallel thread execution (PTX) ISA extension. For example, two instructions may be added to PTX, dfu_cvt_16 and dfu_cvt_8, which may provide the ability to invoke the 16-bit and 8-bit DFUs, respectively. The 16-bit DFU operation may be parameterized with a source .b32 {i.e., a 32- bit conversion-only data type in PTX) register and two contiguous .f32 {i.e., a 32-bit floating point data type in PTX) destination registers. The 8-bit DFU operation may be similar, except the 8-bit DFU may be parameterized by four destination registers and an immediate floating-point exponent bitwidth. The dfu_cvt instructions may be executed by the DFU, which is integrated into the microarchitecture of a microprocessor as an extension of the arithmetic-logic unit (ALU) of the microprocessor. This extension may add the DFU to each floating-point unit, so the conversion throughput is sufficiently high to provide enough data for all of the floating-point units. The flexibility of a variable-width exponent may allow low- precision 8-bit values to be more versatile, outweighing any negligible area cost. In particular, empirical data demonstrates that the area-efficient design of the DFU requires only 0.22% area overhead when replicated for each floating-point unit. In addition to allocating the DFU for each floating-point unit, the throughput of the DFU may be increased by specifying, in some embodiments, that the 32-bit floating-point destination registers are contiguous. Using contiguous registers may allow the DFU to use 64-bit and 128-bit register write operations when writing two and four 32-bit values, as produced by 16-bit and 8-bit data fission, respectively. [0066] Turning to Figure 5, one embodiment of a DFU 500 is depicted, wherein the DFU 500 applies 16-bit fission using the Deft-16Q data format (e) discussed with respect to Figure 4. The DFU 500 may only require a single zero-padded bitwise shift to prepare two values for computation. Figure 5 depicts an 8-bit floating-point representation 502 and a 32- bit floating-point representation 504 ( e.g ., a 32-bit representation), wherein the respective sign bits are denoted by "S", the respective exponent bits are denoted by Έ", and the respective mantissa bits are denoted by "M". Figure 5 also depicts an adder 506.

[0067] The size of the exponent of the 8-bit floating-point representation 502 may be adjusted from 7 bits to 1 bit, denoted by N in Figure 5, depending on the exponent length encoded into a DFU instruction {e.g., dfu_cvt_8). The 8-bit floating-point representation 502 may include a fixed exponent encoding, the fixed exponent encoding having a fixed offset (/.e., a bias) based on the bit width of the 8-bit floating-point representation 502. The respective biases of the 8-bit floating-point representation 502 and the 32-bit floating-point representation 504 may differ. Thus, the DFU 500 may determine the difference between the two biases using the adder 506, and then add this difference to the exponent bits of the 32-bit floating-point representation 504. Because the GPU architecture executes threads in each thread group in lockstep, the bias difference may be reused when applying fission to all of the fused values in a given thread group. The mantissa bits of the 32-bit floating-point representation 504, which may also be of variable length, may be shifted to the left, so that the most significant bit of the mantissa is aligned with the most significant bit of the 32-bit floating-point representation 504. After alignment, the shifted mantissa may be zero-padded to 23 bits and used as the mantissa of the 32-bit floating-point representation 504. The sign bit of the 8-bit floating-point representation 502 may be directly transferred from the 8-bit floating-point representation 502 to the 32-bit floating point representation 504. Leveraging the DFU 500, which provides single-cycle fission operations, the cost of performing near compute data fission is significantly reduced.

[0068] Specifically, empirical information produced by evaluating the end-to-end real- system GPU performance characteristics of when applying both synapse vector elimination and near-compute data fission indicates that applying each of the two optimization techniques in isolation provides significant speedup of 1.5x and 1.6x geometric means across the applications for synapse vector elimination and near-compute data fission, respectively. When both techniques are applied, an average speedup of 2.1 x is achieved, showing the substantial performance benefit of deploying the present techniques. By leveraging the DFU for accelerated data fission in the end-to-end system, the speedup for all applications of the end-to-end system improves performance substantially when leveraging the DFU to facilitate efficient near-compute data fission. Empirical testing indicates the end- to-end speedup averages 2.1 c with Deft-16Q and that it increases to 2.5x with Deft-16Q and 2.6c with Deft-8H. As Deft-16Q and Deft-16H may have the same data movement characteristics, the difference between the two represents the removal of most of the overhead of performing data fission in software. The additional speedup achieved by Deft- 8H is due to the substantial reduction in the amount of data moved compared to Deft-16Q and Deft-16H.

EXAMPLE METHODS

[0069] Figure 6 depicts an example method 600 for optimizing a DNN. The method 600 may include receiving a non-optimized DNN having a set of synapse vectors (block 602).

For example, as discussed with respect to Figure 1 , the set of synapse vectors may comprise and/or compose a trained DNN, such as trained DNN 108. In some embodiments, a sub-part of the trained DNN 108 ( e.g ., a subset of nodes of the trained DNN) may be received. The method 600 may include identifying one or both of (i) a contributing subset of the set of synapse vectors in the DNN, and (ii) a non-contributing subset of the set of synapse vectors in the DNN (block 604). The synapse search module 1 10 may perform the identifying step of the method 600, in some embodiments. The identifying may include identifying a contributing and/or non-contributing subset of the set of synapse vectors, wherein representativeness is determined as discussed with respect to Figure 3.

Specifically, the technique of synapse vector elimination may be used. The method 600 may include computing a correlation matrix as discussed with respect to Figure 3 to determine the representativeness of a first synapse vector of a second synapse vector. The method 600 may select a synapse vector that is most representative of a set/ subset of synapse vectors. In some embodiments, a Pareto frontier may be used to approximate a

performance goal and/or an accuracy goal of the DNN to determine a correlation threshold.

[0070] The method 600 may include generating, based on the non-optimized DNN, an optimized DNN, the optimized DNN having less inferential accuracy than the non-optimized DNN (block 606). The optimized DNN may correspond to the reduced DNN 112 of Figure 1. The method 600 may include a fine-tuning module such as the fine-tuning module 1 14 producing a fine-tuned DNN, corresponding to the fine-tuned DNN 116 of Figure 1. In some embodiments, the optimized DNN may include only the contributing subset of the original DNN. Generating the optimized DNN may also include pruning the non-optimized DNN to remove the non-contributing subset (/.e., modifying the original DNN in place). The pruning may include synapse reordering, matrix truncation, and/or scale adjustment as discussed above, as well as tuning the DNN to improve inferential accuracy. Tuning the DNN to improve inferential accuracy may include refining the optimized DNN by applying training iterations. [0071] Figure 7 depicts a method 700 for improving the effective bandwidth of a DNN.

The method 700 may include loading two strings of n-significant bits of two respective floating-point values into a first register, wherein the two strings of n-significant bits are contiguous with respect to the first register, and wherein n is an even integer (block 702). The method 700 may also include loading the contents of the first register into an on-chip register (block 704), loading the first n/2 bits of the on-chip register into a second register (block 706), loading the second n/2 bits of the on-chip register into a third register (block 708), and modifying, by computation, the contents of the second register and the third register (block 710). The method 700 may also include writing the contents of the second register and third register to off-chip memory (block 712). The registers in method 700 may correspond to, for example, the registers described with respect to the Deft-16Q data format (e) of Figure 4.

[0072] Figure 8 depicts a method 800 for improving the effective bandwidth of a DNN executing in a GPU. The method 800 may include extending the parallel thread execution (PTX) instruction set architecture (ISA) of the GPU with a first instruction configured to invoke a 16-bit data fission unit (DFU) and a second instruction configured to invoke an 8-bit DFU, wherein the 16-bit DFU is parameterized with a 32-bit source register and two contiguous 32-bit destination registers, and wherein the 8-bit DFU is parameterized with a 32-bit source register and four contiguous 32-bit destination registers (block 802). For example, the 16-bit DFU and/or the 8-bit DFU may correspond, respectively, to the DFU 500 of Figure 5.

[0073] Figure 9 depicts an example method 900 of operating a DNN operation runtime. The method 900 may include receiving a non-optimized trained DNN having a set of synapse vectors (block 902). The set of synapse vectors may correspond to the set of synapse vectors in the trained DNN 108. The method 900 may include pruning a non contributing subset of synapse vectors from the trained DNN to produce an optimized DNN, wherein the optimized DNN loses some inferential accuracy (block 904). The pruning may include a synapse search module such as the synapse search module 100 generating an optimized DNN, such as reduced DNN 112. The method 900 may include a fine tuning module, such as fine tuning module 114, tuning the optimized DNN to recover some lost inferential accuracy (block 906). The method 900 may further include receiving input formatted for the non-optimized trained DNN (block 908). The input may correspond to the input 122 of Figure 1. The method 900 may include reorganizing the input formatted for the non-optimized trained DNN to generate reorganized input (block 910). For example, a set of activation values corresponding to the input 122 may be reorganized, so that the activation values can be applied to the optimized DNN model using software-based data fission and/or a hardware DFU, as discussed above.

EXAMPLE COMPUTING DEVICE

[0074] Figure 10 depicts an example block diagram 1000 including a computing device 1002, in which the DNN operation runtime may be implemented. The device

1002 may be communicatively coupled to a network 1004 and a database 1006. The device may also be associated with, or include, an input device 1008 and an output device 1010. The device 1002 may be a laptop computer, desktop computer, tablet, smartphone device, or any other suitable type of personal and/or mobile computing device. The device 1002 may include a GPU 1020, a CPU 1022, a memory 1024, and an input/output (I/O) controller 1026.

[0075] While referred to in the singular, the GPU 1020 may include any suitable number of processors one or more types ( e.g ., one or more GPUs, cores, etc.). The GPU 1020 may be any suitable GPU (e.g., a Titan X (Pascal) GPU). As depicted in the diagram 1000, the GPU 1020 may include both off-chip and on-chip memory

regions. The off-chip memory regions may include a constant memory 1030 and a generic memory 1032, which may include a texture memory 1034. The on-chip

memory region may include one or more multiprocessor each including an instruction cache 1040, a shared memory 1042, a functional unit 1044, and a set of registers

1046. More than one functional unit 1044 of different types may be included,

depending on the GPU design. The GPU may share data between the on-chip and off-chip memories, and the on-chip and off-chip regions may be linked by additional caching layers/ controllers (not depicted).

[0076] In general, the GPU 1020 access to off-chip memory is slower than the

GPU 1020 access than on-chip memory. Off-chip memory is typically composed of dynamic random-access memory (DRAM). In manufacturing, DRAM is often

optimized for cost rather than speed. Off-chip memory is also located further away from the GPU 1020 than on-chip memory. On-chip memory is typically composed of static dynamic-access memory (SRAM). SRAM is more expensive to produce, and is faster. On a circuit board containing the GPU 1020 processor, global memory

(including local memory, constant memory, texture memory, etc.) is typically located in DRAM hardware chips that surround the GPU 1020 hardware chip. Shared

memory, registers, and caches are typically located in the GPU 1020 hardware chip itself, and are typically SRAM-based. However, the GPU 1020 may have a different memory layout, depending on the embodiment, [0077] As noted above, on-chip memory bandwidth is a key bottleneck for DNN execution on GPUs. Prior work focuses on improving compression in off-chip memory, which fails to provide significant speedups for DNNs. Synapse vector elimination and near-compute data fission mitigate GPU on-chip memory bandwidth bottlenecks by optimizing the utilization of integer units during DNN execution.

Unlike the prior work’s techniques, the present techniques support low-overhead very fine-grained data conversion. Rather than focusing on high compression ratio, the present techniques focus on low decompression overhead. The DFU obviates data fission overhead and will continue to realize additional benefits on future generations of GPU hardware. Near-compute data fission may improve performance by efficiently packing on-chip memory, as described.

[0078] The off-chip memory may receive a non-optimized DNN corresponding to, for example, the trained DNN 108 of Figure 1. The non-optimized DNN may include a set of synapse vectors. The synapse vector elimination kernel application may create an optimized DNN from the non-optimized DNN as discussed above. Once the optimized DNN is created, the DNN runtime operation may load the optimized DNN and operate it in conjunction with data fission software and/or hardware components.

[0079] For example, a data fission application comprising a set of computer executable instructions may be stored in the off-chip region. When executed by a computer processor ( e.g ., the GPU 1020 or the CPU 1022), the instructions may cause the DNN operation system to perform data fission operations as described with respect to Figure 4 and Figure 5. The off-chip region of Figure 10 may correspond to the off-chip memory of Figure 4. An application corresponding to a synapse vector elimination kernel application may load data from the off-chip region into the registers 1046. The data may be stored in an on-chip memory scratchpad (not depicted). The application may read from and write to the on-chip memory scratchpad many times, before writing the data back to off-chip memory. In an embodiment, the data may be fused and the fission process used as described above.

[0080] For example, a DFU may be included in the off-chip region and/or the on- chip region of the GPU 1020, and the DFU may read data from, and write data to, the on-chip registers 1046, as described above. In general, near-compute data fission packs DNN weights and activations into on-chip memory by removing non contributing bits from the numerical representation. This technique may reside in low-level DNN computational DNN kernels at runtime. Therefore, no further changes may be necessary to the baseline DNN infrastructure. As described with respect to Figure 4 and Figure 5, multiple values may be fused into a single value of lesser size in on-chip memory to improve effective bandwidth. This technique results in a direct improvement, because DNN computations are typically bottlenecked by on-chip memory bandwidth.

[0081] While referred to in the singular, the CPU 1022 may include any suitable number of processors of one or more types ( e.g ., one or more CPUs, graphics processing units (GPUs), cores, etc.). The memory 1024 may include one or more persistent memories {e.g., a hard drive and/or solid state memory) and may store data used by and/or output by one or more applications, including a DNN operation runtime. Generally, the CPU 1022 may be configured to execute software instructions stored in the memory 1024, which may be loaded into a random-access memory unit (RAM) of the memory 1024. The software instructions may launch, initialize, and/or execute the DNN operation runtime. In some cases, the software instructions may load and/or store DNN models {e.g., the trained DNN, reduced DNN, and/or fine-tuned DNN depicted in Figure 1 ) from the memory 1024, or from the database 1006. The software instructions may also retrieve DNN configurations and/or a DNN training data set from the memory 1024 or the database 1006 which correspond to, respectively, the training set 102 and/ the DNN configuration 104 of Figure 1 . The software instructions may correspond to the synapse search module 1 10 of Figure 1 and/or the fine-tuning module 1 14 of Figure 1.

[0082] One or more DNN model and/or DNN model parameters may be transmitted by the computing device 1002. For example, the CPU 1022 may receive/ retrieve a DNN model from the network 1004 or the database 1006. The CPU 1022 may initialize the DNN model using the parameters {e.g., weights). The CPU 1022 may execute instructions causing the DNN model to be loaded in the GPU 1020.

The GPU 1020 may perform operations {e.g., training, operation, and/or optimization) of the DNN model, producing an output. The CPU 1022 may receive the output and process the output further, such as by displaying the output in the output 1010, storing the output in the database 1006, etc. In some embodiments the CPU 1022 may perform more or less work.

[0083] The CPU 1022 may execute some or all of the software instructions. For example, the CPU 1022 may include code that, when executed, initializes a DNN runtime operation runtime, as depicted in Figure 1 . The GPU 1020 may execute some or all of the software instructions, in some embodiments. For example, the DNN operation runtime 100 may be initialized by the CPU 1022, and then certain functions ( e.g ., DNN execution) handed off to the GPU 1020. The CPU 1022 may process results of the GPU 1020 computations, such as further processing,

transmission via the network 1004, display in the output device 1010, etc.

[0084] The network 1004 may be a single communication network, or may include multiple communication networks of one or more types {e.g., one or more wired

and/or wireless local area networks (LANs), and/or one or more wired and/or wireless wide area networks (WANs) such as the Internet). The database may be any

suitable database {e.g., a Structured Query Language (SQL) database, a NoSQL database, a flat file database, etc.). The input 1008 may be a computer peripheral device, such as a keyboard, mouse, or other device. The output 1010 may be any suitable output device such as a speaker, display device, etc. In an embodiment, the input 1008 and output 1010 may be joined in a single device, such as in a touch

screen.

ADDITIONAL CONSIDERATIONS

[0085] The following considerations also apply to the foregoing discussion. Throughout this specification, plural instances may implement operations or structures described as a single instance. Although individual operations of one or more methods are illustrated and described as separate operations, one or more of the individual operations may be performed concurrently, and nothing requires that the operations be performed in the order illustrated. These and other variations, modifications, additions, and improvements fall within the scope of the subject matter herein.

[0086] Unless specifically stated otherwise, discussions herein using words such as “processing,”“computing,”“calculating,”“determining,”“presenting,”“displaying,” or the like may refer to actions or processes of a machine {e.g., a computer) that manipulates or transforms data represented as physical {e.g., electronic, magnetic, or optical) quantities within one or more memories {e.g., volatile memory, non-volatile memory, or a combination thereof), registers, or other machine components that receive, store, transmit, or display information.

[0087] As used herein any reference to“one embodiment” or“an embodiment” means that a particular element, feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. The appearances of the phrase“in one embodiment” in various places in the specification are not necessarily all referring to the same embodiment.

[0088] As used herein, the terms“comprises,”“comprising,”“includes,”“including,”“has,” “having” or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, article, or apparatus that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Further, unless expressly stated to the contrary,“or” refers to an inclusive or and not to an exclusive or. For example, a condition A or B is satisfied by any one of the following: A is true (or present) and B is false (or not present), A is false (or not present) and B is true (or present), and both A and B are true (or present).

[0089] In addition, use of “a” or“an” is employed to describe elements and components of the embodiments herein. This is done merely for convenience and to give a general sense of the invention. This description should be read to include one or at least one and the singular also includes the plural unless it is obvious that it is meant otherwise.

[0090] Upon reading this disclosure, those of skill in the art will appreciate still additional alternative structural and functional designs for implementing the concepts disclosed herein, through the principles disclosed herein. Thus, while particular embodiments and

applications have been illustrated and described, it is to be understood that the disclosed embodiments are not limited to the precise construction and components disclosed herein. Various modifications, changes and variations, which will be apparent to those skilled in the art, may be made in the arrangement, operation and details of the method and apparatus disclosed herein without departing from the spirit and scope defined in the appended claims.

Claims

WHAT IS CLAIMED:

1. A computer-implemented method of optimizing a deep neural network (DNN), the method comprising:

receiving a non-optimized DNN having a set of synapse vectors,

identifying one or both of (i) a contributing subset of the set of synapse vectors in the DNN, and (ii) a non-contributing subset of the set of synapse vectors in the DNN; and

generating, based on the non-optimized DNN, an optimized DNN, the optimized DNN having less inferential accuracy than the non-optimized DNN.

2. The method of claim 1 ,

wherein identifying one or both of (i) the contributing subset and (ii) the non contributing subset includes identifying a representative subset of the set of synapse vectors.

3. The method of claim 2, wherein identifying the representative subset of the set of synapse vectors includes:

computing a correlation matrix including the correlation of each of the set of synapse vectors to the set of synapse vectors, wherein a first synapse vector in the set of synapse vectors is representative of a second synapse vector in the set of synapse vector when the correlation between the first synapse and the second synapse is greater than one or more representative correlation thresholds; and

determining, for each of the set of synapse vectors, the synapse vector that is most representative of the set of synapse vectors.

4. The method of claim 3, wherein determining the synapse vector that is most representative of the set of synapse vectors includes counting the number of instances wherein the synapse vector is representative as compared to the set of synapse vectors.

5. The method of claim 3, wherein computing the correlation matrix of the set of synapse vectors is performed by calculating the covariance of a first synapse vector and a second synapse vector, divided by the square root of the product of the variance of the first synapse vector and the second synapse vector.

6. The method of claim 3, wherein the DNN is comprised of multiple layers individually trained with respect to the one or more representative correlation thresholds.

7. The method of claim 6, wherein the one or more representative correlation thresholds are chosen based on a Pareto frontier to approximate one or both of (i) a user- specified performance goal, and (ii) a user-specified accuracy goal.

8. The method of claim 1 ,

wherein generating the optimized DNN includes generating an optimized output DNN including only the contributing subset.

9. The method of claim 1 ,

wherein generating the optimized DNN includes pruning, from the non- optimized DNN, the non-contributing subset.

10. The method of claim 9, wherein pruning the non-contributing subset includes one or more of (i) synapse reordering, (ii) matrix truncation, or (iii) scale adjustment.

1 1. The method of claim 1 , further comprising:

tuning the DNN to improve inferential accuracy.

12. The method of claim 11 , wherein tuning the DNN to improve inferential accuracy includes refining the optimized DNN by applying a small number of training iterations.

13. A method of improving the effective bandwidth of a DNN, the method comprising:

loading two strings of /7-significant bits of two respective floating-point values into a first register, wherein the two strings of /7-significant bits are contiguous with respect to the first register, and wherein n is an even integer,

loading the contents of the first register into an on-chip register,

loading the first n/2 b\ts of the on-chip register into a second register,

loading the second n/2 b\ts of the on-chip register into a third register,

modifying, by computation, the contents of the second register and the third register; and

writing the contents of the second register and third register to off-chip memory.

14. A method of improving the effective bandwidth of a DNN executing in a graphics processor unit (GPU), the method including: extending the parallel thread execution (PTX) instruction set architecture (ISA) of the GPU with a first instruction configured to invoke a 16-bit data fission unit (DFU) and a second instruction configured to invoke an 8-bit DFU,

wherein the 16-bit DFU is parameterized with a 32-bit source register and two contiguous 32-bit destination registers, and

wherein the 8-bit DFU is parameterized with a 32-bit source register and four contiguous 32-bit destination registers.

15. A processor comprising:

a plurality of floating-point units (FPUs), wherein each FPU includes a data fission unit (DFU) to apply a data fusion instruction, the instruction specifying a source register and a plurality of contiguous destination registers; and

an execution unit to execute the instruction to generate a fused result comprising the result of converting an n- bit floating-point representation to an IEEE single-precision value.

16. A DNN runtime operation system comprising:

an off-chip memory,

a computer processor including:

a first register, a second register, and a third register; and

an on-chip memory;

a synapse vector elimination kernel application comprising a set of computer- executable instructions stored in the off-chip memory that, when executed by the computer processor, causes the DNN runtime operation system to:

receive, in the off-chip memory, a non-optimized trained DNN, wherein the non-optimized trained DNN includes a set of synapse vectors,

identify a non-contributing subset of the set of synapse vectors, prune, from the set of synapse vectors, the non-contributing subset to produce an optimized DNN, wherein the optimized DNN loses some inferential accuracy, tune the optimized DNN to recover at least some of the lost inferential accuracy,

receive input formatted for the non-optimized trained DNN,

reorganize the input formatted for the non-optimized trained DNN, and apply the reorganized input to the optimized DNN;

and

a data fission application comprising a set of computer-executable instructions stored in the off-chip memory that, when executed by the computer processor, causes the DNN operation system to: load two strings of /7-significant bits of two respective floating-point values into a first register, wherein the two strings of /7-significant bits are contiguous with respect to the first register, and wherein n is an even integer,

load the contents of the first register into an on-chip register,

load the first n/2 bits of the on-chip register into a second register,

load the second n/2 bits of the on-chip register into a third register, modify, by computation, the contents of the second register and the third register, and

write the contents of the second register and third register to off-chip memory.

17. A method of operating a DNN operation runtime, the method comprising: receiving a non-optimized trained DNN having a set of synapse vectors, pruning a non-contributing subset of synapse vectors from the trained DNN to produce an optimized DNN, wherein the optimized DNN loses some inferential accuracy, tuning the optimized DNN to recover at least some of the lost inferential accuracy,

receiving input formatted for the non-optimized trained DNN, reorganizing the input formatted for the non-optimized trained DNN to generate reorganized input, and

applying the reorganized input to the optimized DNN, where applying the reorganized input to the optimized DNN includes one or both of (i) software-based data fission and (ii) a hardware-based data fission unit.’

18. A DNN operation system, comprising:

one or more GPUs,

a memory storing computer-executable instructions that, when executed, cause the DNN operation system to:

receive an input,

optimize a non-optimized trained DNN using a synapse vector elimination kernel, analyze the input using a near-compute data fission technique; and

generate an output.

19. The DNN operation system of claim 18, further comprising:

a DFU,

wherein the memory includes further computer-executable instructions that, when executed, cause the DNN operation system to:

accelerate the near-compute data fission technique using the DFU.

20. The DNN operation system of claim 18, the memory including further computer-executable instructions that, when executed, cause the DNN operation system to: receive a correlation parameter, and

modify, based on the correlation parameter, the synapse vector elimination kernel.