JP2023162348A - Data sparsity monitoring during neural network training - Google Patents

Abstract

To provide a device monitoring data sparsity during neutral network training.SOLUTION: An electronic device includes: a processor which executes training iterations during a training process for a neural network; and a sparsity monitor. The sparsity monitor acquires, during a monitoring interval in each of one or more monitoring periods, intermediate data outputted by at least some intermediate nodes of the neural network during training iterations that occur during each monitoring interval; generates on the basis of at least in part the intermediate data, one or more values representing sparsity characteristics for the intermediate data; and sends, to the processor, the one or more values representing the sparsity characteristics. The processor performs subsequent training iterations based at least in part on the values representing the sparsity characteristics.SELECTED DRAWING: Figure 9

Description

Some electronic devices implement the operations of artificial neural networks, or more simply "neural networks." In general, a neural network is a computational structure that includes internal elements that have similarities to biological neural networks, such as the neural networks of biological brains. A neural network can be trained to perform a specific task by using known instances of training data to configure the neural network's internal elements, thereby making the neural network , specific tasks can be performed on unknown instances of input data. For example, one particular task performed by a neural network is to identify whether an image contains image elements such as faces or vehicles. When training a neural network to perform image identification, images that are known to contain (or not contain) an image element are processed through the neural network to determine whether the image element is present in an unknown image. Internal elements are configured to produce appropriate output when later processing the unknown image to identify whether the unknown image is

A neural network includes within its internal elements a set of artificial neurons, or "nodes," that are interconnected with each other in a configuration similar to the way neurons are interconnected via synapses in the brain of an organism. A basic neural network can be visualized as a weighted graph structure whose nodes include input nodes, intermediate nodes, and output nodes. Within a neural network, each node other than the output node is connected to one or more downstream nodes via directed edges with associated weights. In operation, the input nodes of the first layer of the neural network receive input from external sources and process the input to generate input values. The input nodes forward the input values to intermediate nodes in the next layer of the neural network. The receiving intermediate node weights the received input based on the weight of the corresponding directed edge, ie, adjusts the received input, such as by multiplying by a weighting value. Each intermediate node sums its corresponding weighted received inputs, and possibly a bias value, to generate an internal value, and uses that internal value to evaluate the intermediate node's activation function to generate a resulting value. generate. The intermediate node then forwards the resulting value as an input value to the intermediate node of the next layer of the neural network. The input values are used to generate internal values and evaluate activation functions as described above. In this way, the value progresses through the intermediate nodes within the layers of the neural network until the final layer of intermediate nodes transfers the resulting value to the output node for the neural network, thereby producing the output for the neural network. Ru. Continuing with the example above, the output produced by the output node, and therefore the output from the neural network, indicates whether the image is likely to contain (or not contain) the specified image element, e.g. 0 and 1.

As mentioned above, values transferred along directed edges between nodes in a neural network are weighted according to the weight associated with each directed edge. Train the neural network to produce a desired output, such as the identification of image elements within an image as described above, by setting weights associated with directed edges during the training process such that the desired output is produced by the neural network. can be trained. When training a neural network, multiple instances of training data with expected or desired outputs are processed within the neural network and actual outputs are generated from the output nodes. Continuing with the neural network example above, an instance of training data is known to contain (or not contain) a particular image element, and therefore the image element is present (or not present) in the image. Contains a digital image in which the neural network is expected to produce an output indicating the possibility. After each instance of training data is processed within the neural network to produce an actual output, the error value or "loss" between the actual output and the corresponding expected output is determined by the mean squared error, log loss, or another Calculated using an algorithm. The loss is then processed backwards through the neural network, i.e., "backpropagated" through the neural network to reduce the error in the instance of the training data. Weights associated with directed edges are adjusted. Thereby, the response of the neural network to that particular instance of training data and all subsequent instances of input data is adjusted. For example, one backpropagation technique involves calculating the slope of the loss with respect to the weight of each directed edge in the neural network. Each gradient is then multiplied by a training factor or "learning rate" to calculate a weight adjustment value. The weight adjustment value is then used in calculating an updated value for the corresponding weight, eg, added to the existing value for the corresponding weight.

For some neural networks, especially those with a large number of intermediate nodes, large amounts of intermediate data can be generated as output from the intermediate nodes. Therefore, the device used to train a neural network by carrying out the above-mentioned calculations during the training process (i.e. when an instance of training data is processed through the neural network and the loss is back-propagated) This places a significant computational burden on the processor and other functional blocks within the system. Additionally, storing and retrieving intermediate data during the training process requires large amounts of memory and adds delay to the training process. Accordingly, designers have proposed multiple techniques for using the properties of intermediate data to improve the efficiency of computations that require intermediate data and the storage of intermediate data. For example, some system designers may detect zeros in intermediate data from multiple intermediate nodes, i.e., the "sparsity" of the intermediate data from these intermediate nodes, and proposed using more efficient computations (e.g., sparse matrix operations) where the use of computations is acceptable. As another example, designers have proposed using compressed storage formats to store intermediate data in memory based on the sparsity of the intermediate data. Unfortunately, detecting sparsity in intermediate data during the training process is tedious and data-intensive. This has acted to limit the widespread adoption of techniques that exploit the sparsity of intermediate data during training iterations.

1 is a block diagram illustrating a neural network, according to some embodiments. FIG. 1 is a block diagram illustrating an electronic device, according to some embodiments. FIG. 1 is a block diagram illustrating a sparsity monitor, according to some embodiments. FIG. FIG. 3 is a timeline diagram illustrating monitoring cycles and periods, according to some embodiments. FIG. 2 is a block diagram illustrating the flow of data and information to and from a sparsity monitor, according to some embodiments. 2 is a state diagram illustrating transitions between various modes of operation in a sparsity monitor, according to some embodiments. FIG. FIG. 3 illustrates an example of pseudocode that dynamically adjusts the monitoring period of a sparsity monitor in active mode, according to some embodiments. FIG. 3 illustrates an example pseudocode for transitioning from active mode to hibernation mode and vice versa, according to some embodiments. 2 is a flowchart illustrating a process for monitoring sparsity in intermediate data during training iterations of a training process for a neural network, according to some embodiments. FIG. 3 illustrates a portion of a flowchart illustrating a process for using active mode and hibernation mode in a sparsity monitoring device, according to some embodiments. FIG. 3 illustrates a portion of a flowchart illustrating a process for using active mode and hibernation mode in a sparsity monitoring device, according to some embodiments. 2 is a flowchart illustrating a process for fast termination of a monitoring period by a sparsity monitor, according to some embodiments.

Like reference numerals refer to like diagrammatic elements throughout the drawings and description.

The following description is presented to enable any person skilled in the art to make and use the described embodiments, and is provided in the light of a particular application and its requirements. Various modifications to the described embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications. The described embodiments are therefore not limited to those shown, but are to be accorded the widest scope consistent with the principles and features disclosed herein.

(term)
In the following description, various terminology is used to describe the embodiments. Below is a brief and general explanation of some of these terms. The term may have important additional aspects that are not described herein for the sake of clarity and brevity. It is therefore noted that the description is not intended to limit the term.

Functional Block: A functional block refers to a group, collection, and/or set of one or more interrelated circuit elements, such as integrated circuit elements or discrete circuit elements. Circuit elements are "interrelated" in that the circuit elements share at least one characteristic. For example, interrelated circuit elements may be included in, fabricated on, or otherwise coupled to a particular integrated circuit chip or portion thereof; may be involved in the performance of functions (computation or processing functions, memory functions, etc.) or may be controlled by a common control element. A functional block may include any number of circuit elements from a single circuit element (e.g., a single integrated circuit logic gate) to millions or billions of circuit elements (e.g., integrated circuit memory). can.

Sparsity: The number and/or arrangement of zeros (and/or other designated values) found in intermediate data output by intermediate (or "hidden") nodes within a neural network. For example, a zero may be output by an intermediate node due to an activation function (eg, a regularized linear unit (ReLU) activation function, etc.) that produces a zero result. One simple calculation of sparsity is a ratio. In this case, if sparsity is determined based on zeros in the intermediate data, then assuming 100 intermediate nodes, 25% sparsity means that 25 intermediate nodes output zeros and the remaining 75 Exists when an intermediate node outputs another value.

Training iteration: A training iteration includes some or all of the operations for processing a single instance of training data through the neural network during the neural network training process. The operations of a training iteration include processing a given instance of training data through the neural network to generate an output from an output node of the neural network, in addition to intermediate data from intermediate nodes of the neural network. The operations of training iterations may also include backpropagation of loss values associated with a given instance of training data to set weights within the neural network, and the like.

(neural network)
As mentioned above, a neural network is a computational structure that includes internal elements (i.e., nodes and directed edges) that are trained to perform specific tasks. FIG. 1 is a block diagram illustrating a neural network 100 including an input node 102, intermediate nodes 104 in layers 110, 112, output nodes 106, and directed edges 108, according to some embodiments. For clarity, only the two directed edges and layers are labeled).

Depending on the nature of the internal elements of neural network 100, neural network 100 can be a "discriminative" network or a "generative" network. A discriminable network is configured to process instances of input data and output a result indicating whether a particular pattern is likely to exist in an instance of input data; A neural network configured to classify. For example, a discriminable network is a result that indicates whether an image element such as a face or a road sign is likely to be present in a digital image, or whether a particular sound or word is likely to be present in digital audio, etc. may be configured to output. A generative network is a neural network configured to generate instances of output data that include patterns similar to a particular pattern. For example, a generative network may be configured to generate digital images that include patterns that resemble faces or road signs, audio that includes patterns that resemble particular sounds or words, and the like.

The described embodiment implements a training process for a neural network. During the training process, weights associated with directed edges and other values are set so that the neural network can later be used to perform specific tasks. In some embodiments, when training a neural network, during each of a plurality of training iterations, a separate instance of training data having an expected or desired output is processed through the neural network, and from an output node Generate actual output. In these embodiments, after each instance of training data is processed through the neural network to produce an actual output, an error value or "loss" between the actual and expected outputs for that instance of training data is determined. is calculated using root mean square error, log loss, or another algorithm. The loss is then processed backwards through the neural network, i.e. by "backpropagating" through the neural network, the directed edges within the neural network are used to reduce the error for instances of the training data. The weight associated with is adjusted. Thereby, the response of the neural network to instances of training data and subsequent instances of input data is adjusted. For example, in some embodiments, backpropagation includes computing the slope of the loss with respect to the weights for each directed edge in the neural network. Each gradient is then multiplied by a training factor or "learning rate" to calculate a weight adjustment value. The weight adjustment value is then used in calculating an updated value for the corresponding weight, eg, added to the existing value for the corresponding weight.

Although an example neural network is shown in FIG. 1, in some embodiments different arrangements of nodes and/or layers are present in the neural network. For example, a neural network can include a large number (sometimes hundreds of thousands or millions) of intermediate nodes arranged in numerous layers, with each layer of intermediate nodes receiving input values and generating input values. Transfer the resulting value to an intermediate node or output node in the next layer. As another example, in some embodiments different topologies or connectivity of nodes are used and/or convolutional networks, radial-based networks, recurrent neural networks, autoencoders, Markov chains, trust networks, residual Different types of nodes are used, such as the arrangement and types of nodes used in neural networks, including networks. In general, the described embodiments can be implemented with any configuration of neural networks that can detect and use sparsity in intermediate data as described herein.

(overview)
In the described embodiment, a sparsity monitor within the electronic device monitors sparsity in intermediate data output by intermediate nodes within the neural network during training iterations of the neural network training process. For monitoring, the sparsity monitor obtains intermediate data output from at least some of the intermediate nodes in the neural network. The sparsity monitor then analyzes the obtained intermediate data to determine sparsity, ie, intermediate data equal to zero (or another specific value). As part of the analysis, the sparsity monitor generates a value representative of the sparsity characteristics of the intermediate data. For example, a sparsity monitor represents the current sparsity, average or median sparsity of intermediate data from one or more intermediate nodes for a certain number of training iterations, etc. as a value representing the sparsity property. can generate a value. The sparsity monitor then sends a value representative of the sparsity characteristic to a processor within the electronic device. The processor controls one or more aspects of performing subsequent training iterations based at least in part on the value representing the sparsity property. For example, the processor determines, based on the sparsity characteristics, the type of computations performed for subsequent training iterations, the type of compression used to compress and store intermediate data in memory for subsequent training iterations, etc. The type, etc. can be determined and configured.

In some embodiments, intermediate data is obtained from intermediate nodes as described during a "monitoring interval" within a "monitoring period." For example, in some embodiments, the monitoring interval is Nms (eg, 50ms, 100ms, etc.) out of a monitoring period of Mms (eg, 1s, 2s, etc.), and Mms>Nms. As another example, in some embodiments, the monitoring period and/or monitoring interval includes a certain number of training repetitions, e.g., the monitoring interval includes K training repetitions (e.g., 25, 50, etc.) , the monitoring period is Z training iterations (eg, 500, 1000, etc.), and Z>K. In these embodiments, the monitoring interval and monitoring period may not be calculated in terms of time (in fact, they may be variable times depending on how long the training iterations take). In some of these embodiments, the monitoring period and corresponding monitoring interval are repeated and may be repeated as long as the training process continues. For example, a monitoring interval of Nms (e.g., 60ms) may occur every Mms (e.g., 250ms), and a monitoring interval of Z training repetitions (e.g., 20 training repetitions) may occur every (For example, it may occur every 2 seconds).

In these embodiments, the sparsity monitoring device operates in one of a hibernation mode and an active mode. In general, a sparsity monitor starts in active mode and/or if the sparsity between monitoring intervals is "unstable", i.e. changing by more than a certain amount between monitoring intervals. transition from hibernation mode to active mode. A sparsity monitor transitions from active mode to hibernation mode when sparsity between monitoring intervals is "stable", ie, varying less than a certain amount between monitoring intervals.

When the sparsity monitor starts operating in active mode (e.g., at startup, when initially monitoring sparsity during the neural network training process, when transitioning from hibernation mode, etc.), the monitoring period An active mode initial value is set for the intermediate node, where the active mode initial value is the shortest monitoring period used by the sparsity monitor. The sparsity monitor then periodically determines whether the sparsity of the intermediate data from the corresponding training interval is stable, and if so, increases the length of the monitoring period (e.g. double or otherwise increase the length of the monitoring period). In some embodiments, the sparsity monitor monitors sparsity in the intermediate data for individual intermediate nodes and/or different groups of intermediate nodes (e.g., intermediate nodes within layers of a neural network, regions of a neural network, etc.). may be monitored and the monitoring period may be adjusted individually for individual intermediate nodes and/or groups of intermediate nodes. The sparsity monitor continues to periodically determine whether the sparsity is stable and in this way determines whether the sparsity is stable and in this way increases the sparsity to the maximum length, i.e., the hibernation mode monitoring period. Increase the length of the monitoring period (if sparsity is found to be unstable, the monitoring period may be gradually shortened to return to the same shortness as the initial active mode value). In active mode, the sparsity monitor transitions to hibernation mode if the monitoring period is increased to the hibernation mode monitoring period for each monitored intermediate node and the sparsity is still stable.

When the sparsity monitoring device operates in hibernation mode, the monitoring period is fixed to the hibernation mode monitoring period for all intermediate nodes, so the monitoring period is the maximum length for each intermediate node. If it is determined that the sparsity of a certain number of intermediate nodes is unstable in hibernation mode, the sparsity monitoring device transitions from hibernation mode to active mode. When transitioning to active mode, the monitoring period is reduced to the active mode initial value and the sparse monitoring device operates as described above.

In some embodiments, the sparsity monitor supports "fast" termination of sparsity monitoring during intermediate node monitoring intervals. In these embodiments, intermediate data from one or more training iterations is found to have less than a certain sparsity (e.g., 5%, 10%, etc.) and thus the data is relatively full/sparse. If so, the sparsity monitor stops acquiring and processing intermediate data for the remaining training iterations within the monitoring interval. In these embodiments, values representative of sparsity properties may be generated based on intermediate data from fewer than all training iterations within a normal monitoring interval.

In some embodiments, the aforementioned values representative of sparsity characteristics of the intermediate data are trends or patterns of sparsity in the intermediate data for one or more intermediate nodes for two or more training iterations, or including. For example, a trend can include a trend (eg, increase, decrease, percentage of increase or decrease, etc.) in sparsity values for one or more intermediate nodes over a particular number of training iterations. As another example, the pattern may include a grouping of intermediate nodes that output zero (or another value), a layer of intermediate nodes that has more than a predetermined number of intermediate nodes that output zero (or another value), It may include patterns such as a block ratio (eg, a block ratio of zero if the sparsity is based on intermediate data with a value of zero).

In some embodiments, the values representing the sparsity characteristic are organized into multiple "stages" similar to the bins of a histogram, with each stage indicating a respective value or range of values for the sparsity characteristic. For example, in some embodiments, the stage has a sparsity between 10% and 0% (so that zero or another specified value is applied to the intermediate data for one or more intermediate nodes for one or more training iterations). The first stage is associated with sparsity of 20% to 11%, the third stage is associated with sparsity of 30% to 21%, etc. . In these embodiments, the sparsity monitor returns a stage value (e.g., a value of 1 for the first stage, a value of 2 for the second stage, etc.) to the processor as a value representing the sparsity characteristic. obtain.

By tracking sparsity in intermediate data output by intermediate nodes using a sparsity monitor, the described embodiments enable a processor to track sparsity in the data output of intermediate nodes in a neural network. allows controlling the processing of intermediate data for subsequent training iterations based on current real-time information about This can help the processor process instances of training data through the neural network more efficiently, which can save power, speed up processing operations, etc. By processing individual instances of training data more efficiently, the training process can be faster and require less power. This improves the performance of the electronic device, resulting in greater user satisfaction with the electronic device. Additionally, by using hibernation and active modes and their associated dynamically adjustable monitoring periods, sparse monitoring equipment can unnecessarily burden computational resources, communication interfaces, etc. within an electronic device. It is possible to monitor sparsity without applying .

(electronic device)
In the described embodiments, the electronic device performs operations to, among other things, determine sparsity in intermediate data during training iterations of a training process for a neural network. FIG. 2 is a block diagram illustrating an electronic device 200, according to some embodiments. As can be seen in FIG. 2, electronic device 200 includes a processor 202, a memory 204, and a sparsity monitor 206. Generally, processor 202, memory 204, and sparsity monitor 206 are implemented in hardware, ie, using various circuitry and devices. For example, processor 202, memory 204, and sparsity monitor 206 can be fabricated entirely on one or more semiconductor chips, such as one or more discrete semiconductor chips, and may include discrete circuit elements. It can be manufactured from semiconductor chips in combination with semiconductor chips, it can be manufactured from only discrete circuit elements, and so on. As described herein, some or all of processor 202, memory 204, and sparsity monitor 206 perform operations related to determining sparsity of intermediate data.

Processor 202 is a functional block that performs computational operations in electronic device 200. For example, processor 202 may be one or more central processing unit (CPU) cores, graphics processing unit (GPU) cores, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), etc. These may also be included. In some embodiments, processor 202 includes circuit elements or functional blocks, such as pipelines, execution units, computational units, etc., that execute program code that cause the circuit elements/functional blocks to perform associated operations. In some embodiments, processor 202 is specialized to perform specific processing operations, and in some cases includes hardware circuitry for such specific purposes.

Memory 204 is a functional block within electronic device 200 that implements memory (eg, “main” memory) operations for electronic device 200. Memory 204 stores data and instructions for use by fourth generation double data rate synchronous dynamic random access memory (DDR4 SDRAM), static random access memory (SRAM), and/or functional blocks within electronic device 200 and control circuitry for processing access to data and instructions stored in the memory circuitry and for performing other control or configuration operations. In some embodiments, intermediate data for training iterations of the neural network training process is stored in and read from memory 204.

Sparsity monitor 206 is a functional block that performs operations to track, analyze, and report sparsity of intermediate data output from intermediate nodes during training iterations of a training process for a neural network. In general, tracking sparsity includes obtaining intermediate data for at least some intermediate nodes of the neural network and possibly generating/storing records of sparsity. Analyzing the sparsity may include determining, based on the obtained intermediate data, a value representing the sparsity of the intermediate data for one or more intermediate nodes for a particular training iteration and/or for a combination of two or more training iterations. including determining a value representative of a sparsity characteristic of the sparsity, such as a value representative of sparsity (eg, an average, trend, or pattern of sparsity) for one or more intermediate nodes. Reporting sparsity includes transmitting, via a file, message, etc., a value representative of the sparsity characteristic of the sparsity to one or more other functional blocks (eg, processor 202, etc.).

FIG. 3 is a block diagram illustrating a sparsity monitor 206, according to some embodiments. As can be seen in FIG. 2, the sparsity monitor 206 includes a sparsity information (INFO) list 300, a sparsity processor 302, a sparsity stage calculator (CALC) 304, and a scheduler 306. Sparsity information list 300 is calculated based on intermediate data obtained from one or more intermediate nodes in the neural network for one or more training iterations and/or intermediate data for one or more training iterations. This is a functional block that stores the value. For example, in some embodiments, sparsity information list 300 includes one or more intermediate nodes obtained (e.g., captured or read) from intermediate nodes during each of one or more training iterations. Contains/stores actual values of intermediate data for the node (and possibly each of a plurality of intermediate nodes). As another example, in some embodiments, the sparsity information list 300 is calculated or derived from actual intermediate data, such as sparsity averages, sums, pattern identifiers, etc. of or within the intermediate data. Contains value. The sparsity information list 300 also includes the identifier of the training iteration(s), the monitoring period and/or monitoring interval when the intermediate data was obtained, the identifier of the intermediate node from which the intermediate data was obtained, and the intermediate data and Metadata may be stored internally, such as/or other information to identify sources of sparsity.

Sparsity processor 302 performs operations to analyze intermediate data obtained during one or more training iterations, monitoring intervals, and/or monitoring periods to determine values representing sparsity characteristics of the intermediate data. It is a functional block. In other words, sparsity processor 302 determines whether individual intermediate nodes and/or groups thereof (e.g., intermediate nodes within one or more layers or regions of a neural network) for one or more training iterations, monitoring periods, and/or monitoring intervals. compute or determine a value representing a sparsity property for a node (such as a node); For example, sparsity processor 302 may determine the actual sparsity for individual intermediate nodes and/or groups thereof (ie, intermediate nodes that output zero and/or another value). As another example, sparsity processor 302 may calculate a mean, median, or standard value of intermediate data sparsity for individual intermediate nodes and/or groups thereof for one or more training iterations. As discussed above, sparsity processor 302 calculates the number of training iterations, monitoring periods, and/or periods used to calculate mean, median, or standard values, and other values representative of the sparsity characteristics of the sparsity. Alternatively, historical or past intermediate data for monitoring intervals and/or values calculated from these intermediate data may be maintained in the sparsity information list 300. As yet another example, sparsity processor 302 calculates or determines a pattern of sparsity, such as a group or area of nodes that output zeros in the intermediate data during one or more training iterations, monitoring periods, or monitoring intervals. obtain. As yet another example, sparsity processor 302 may calculate or determine sparsity trends for one or more intermediate nodes during two or more training iterations, monitoring periods, or monitoring intervals. As yet another example, sparsity processor 302 may calculate or determine a zero block ratio in intermediate data for a group of intermediate nodes for one or more training iterations, monitoring periods, or monitoring intervals.

In addition to analyzing intermediate data, in some embodiments sparsity processor 302 produces output that includes values representing sparsity characteristics. For example, sparsity processor 302 may generate one or more electronic files, messages, etc. that include values representing sparsity characteristics and/or information based thereon. In some embodiments, sparsity processor 302 communicates output to one or more other functional blocks within electronic device 200 (eg, processor 202).

Sparsity stage calculator 304 is a functional block that performs operations to calculate a "stage" of sparsity for one or more intermediate nodes for one or more training iterations, monitoring periods, and/or monitoring intervals. be. In some embodiments, each stage indicates a value or range of values for each of the sparsity properties from a set of possible sparsity values. For example, in some embodiments, the stage has a sparsity between 10% and 0% (so that zero or another specific value is intermediate for one or more intermediate nodes for one or more training iterations). the first stage is associated with sparsity of 20% to 11%, the third stage is associated with sparsity of 30% to 21%, etc. include. In these embodiments, given a set of intermediate data from one or more intermediate nodes for one or more training iterations, monitoring periods, or monitoring intervals, the sparsity stage calculator 304 calculates a value representing the sparsity characteristic. A stage value (eg, a value of 1 for the first stage, a value of 2 for the second stage, etc.) can be calculated. Therefore, in some embodiments, representing sparsity values for training iterations, monitoring periods, or monitoring intervals as simplified values (e.g., with fewer bits) makes communicating values representing sparsity more efficient. It can be done. In some embodiments, the number of stages used is determined by the number of stages used by processor 202 in processing subsequent instances of training data, such as sparse computation mechanisms/kernels, data compressed versions, etc., available to processor 202. Based on multiple changes that can be made.

Scheduler 306 is a functional block that performs operations to control when/how sparsity monitor 206 obtains and/or analyzes intermediate data. In some embodiments, scheduler 306 determines the mode of operation in which sparsity monitor 206 operates. In these embodiments, scheduler 306 may select active mode if sparsity is unstable, ie, varies by more than a predetermined amount between monitoring periods or intervals. During this active mode, the length of the monitoring period is dynamically adjusted based on the continued stability (or instability) of the sparsity. In contrast, if the sparsity is stable, ie, varies less than a predetermined amount between monitoring periods or intervals, scheduler 306 may select hibernation mode. During this hibernation mode, the length of the monitoring period is maintained at a maximum value. Active mode and hibernation mode are discussed in more detail below.

Although sparsity monitor 206 is illustrated with a particular configuration of functional blocks in FIG. 3, in some embodiments sparsity monitor 206 includes a different number and/or configuration of functional blocks. . For example, in some embodiments, sparsity processor 302 includes elements such as sparsity stage calculator 304 and scheduler 306. Typically, sparsity monitor 206 includes sufficient functional blocks to perform the operations described herein.

Although processor 202 and sparsity monitor 206 are shown as separate in FIG. 2, in some embodiments processor 202 includes sparsity monitor 206. For example, in some embodiments, processor 202 is a neural network accelerator, ie, a processor that is designed to efficiently perform neural network operations and includes sparsity monitor 206. In general, any configuration of processor 202 and/or sparsity monitor 206 that performs the operations described herein may be used in the described embodiments.

Although the hardware entity sparsity monitor 206 is described as performing certain operations, in some embodiments a different entity performs the operations. For example, in some embodiments, processor 202 performs some or all of the operations attributable to sparsity monitor 206 (and the functional blocks included therein). In these embodiments, processor 202 may execute program code that causes processor 202 to perform operations.

Although electronic device 200 is shown as including certain functional blocks and circuit elements, electronic device 200 is simplified for convenience of explanation. In some embodiments, electronic device 200 includes additional or different functional blocks, subsystems, and/or circuit elements, or is included in a device that has additional or different functional blocks, subsystems, and/or circuit elements. It will be done. For example, electronic device 200 or devices may include a display subsystem, a power subsystem, an input/output (I/O) subsystem, and the like. Generally, electronic device 200 includes sufficient functional blocks, etc. to perform the operations described herein.

Electronic device 200 can be or be included in any device that performs computer operations. For example, electronic device 200 may include a server computer, a desktop computer, a laptop computer, a wearable computing device, a tablet computer, a virtual or augmented reality device, a smartphone, an artificial intelligence (AI) or machine learning device, a network appliance, a toy, an audiovisual It can be or be included in equipment, home appliances, vehicles, etc., and/or a combination thereof.

(Monitoring period and monitoring interval)
In the described embodiment, a sparsity monitor (e.g., sparsity monitor 206) uses a "monitor period" and a "monitor interval" to determine sparsity in intermediate data for training iterations for a neural network. Monitor sex. FIG. 4 is a timeline diagram illustrating a monitoring period 400 and a monitoring interval 402, according to some embodiments. Scheduler 306 may use monitoring periods and monitoring intervals similar to those shown in FIG. 4 for individual intermediate nodes or groups thereof during active mode. In that case, the monitoring period is dynamically adjusted based on the sparsity stability of the intermediate data. In contrast, during hibernation mode, the monitoring periods are all the same length for all intermediate nodes.

For each of the multiple training iterations that occur, during each monitoring interval 402 (indicated by cross-hatching in FIG. 4) that occurs during the "active" portion of the corresponding monitoring period 400, the scheduler 306 A sparsity processor 302 obtains intermediate data from one or more intermediate nodes for final analysis, as described in this paper. For example, if the monitoring interval is Ams, sparsity processor 302 may obtain intermediate data from one or more intermediate nodes for B training iterations that occur within Ams (although other This may be terminated early with fast termination, as described in section 1. For the remainder of the monitoring period 400 (shown in white in FIG. 4), i.e., during the "idle" portion of the monitoring period, the scheduler 306 causes the sparsity processor 302 to skip retrieving intermediate data from intermediate nodes. . In some embodiments, scheduler 306 directs sparsity processor 302 and/or other functional blocks within sparsity monitor 206 to conserve power, avoid heat generation, etc. during idle portions of the monitor period. to a low-power operating state.

In FIG. 4, the monitoring periods 400 1 to N-1 have the same length of time and/or the same number of training repetitions, such as 1 s or 250 training repetitions, and at the beginning of each monitoring period 400 The monitoring interval 402 that occurs is a particular length of time and/or a particular number of repetitions, such as 100 ms or 50 training repetitions. During monitoring periods 1 through N-1, which may follow other/past monitoring periods (not shown), sparsity processor 302 determines that sparsity is stable and therefore the monitoring period can be increased. For example, sparsity processor 302 determines whether sparsity is determined based on intermediate data from one or more intermediate nodes for training iterations occurring during monitoring periods 1 through N-1 (and has changed by less than a predetermined amount (e.g., 5%, 10%, or another value) during the possibly earlier monitoring period (not shown) and thus the sparsity can be determined to be stable. can. Accordingly, scheduler 306 may increase the length of monitoring period 400 for one or more intermediate nodes, as indicated by monitoring period N. In FIG. 4, the monitoring period N is approximately twice as long as the monitoring period 1 to N-1, but arbitrarily increased lengths may be used up to the maximum monitoring period value.

Although a particular monitoring period 400 and monitoring interval 402 are shown in FIG. 4, different configurations of monitoring periods and/or monitoring intervals are used in some embodiments. For example, monitoring interval 402 may occur at a time other than at the beginning of monitoring period 400, more than one monitoring interval 402 may occur during each monitoring period 400, and so on. Generally, the described embodiments use a monitoring period 400 and a monitoring interval 402 that are sufficient to perform the operations described herein.

(Flow of data and information)
In the described embodiment, a sparsity monitor (eg, sparsity monitor 206) receives and generates various data and information for monitoring sparsity in intermediate data. FIG. 5 is a block diagram illustrating the flow of data and information to and from a sparsity monitor, according to some embodiments. Note that FIG. 5 is shown as an example of some embodiments; in other embodiments, different functional blocks receive and generate some or all of the data and information. Additionally, although neural network 502 is illustrated in FIG. 5 using a simplified representation similar to that of neural network 100, neural network 502 has nodes of corresponding type and connectivity. It may include different configurations and/or types of nodes, such as convolutional neural networks, radial-based networks, recurrent neural networks, etc.

As can be seen in FIG. 5, processor 202 performs operations to process instances of training data 500 through neural network 502 during a training process for neural network 502. For example, assuming neural network 502 classifies an image as containing (or not containing) a particular image element, processor 202 processes the image (e.g., a digital file containing image data) through the neural network. may generate an output indicating whether the image element is present within a given image. By processing an instance of training data 500 through neural network 502, processor 202 receives information from one or more intermediate nodes (and possibly other nodes (e.g., input nodes)) in the neural network Intermediate data 504 related thereto is generated. For example, for an intermediate node with an activation function, such as a Regularized Linear Unit (ReLU), the processor activates the activation function based on the corresponding weighted input to the intermediate node to generate an output from the intermediate node. The function may be evaluated (ie, the result of the activation function is computed). Outputs from intermediate nodes serve as intermediate data from or associated with those nodes.

Intermediate data 504 may be communicated directly from processor 202 to sparsity monitor 206 , or by being stored in memory during computation by processor 202 and then read from memory 204 by sparsity monitor 206 . Obtained by sparsity monitor 206. Additionally, sparsity monitor 206 obtains neural network information (INFO) 508 from processor 202 (or another entity within electronic device 200). This information is information about neural networks that can be used in sparsity calculations. Neural network information 508 includes information regarding the type of neural network 502, the number, configuration/connectivity and/or characteristics of some or all of the nodes within neural network 502, the specific tasks performed by neural network 502, the implementation It includes information such as the number of training iterations performed, which instances of training data are processed, etc.

Sparsity monitor 206 uses intermediate data 504 for one or more training iterations, monitoring periods or intervals, and neural network information 508 to determine a value representing a sparsity characteristic 510 for intermediate data 504. . To determine a value representing a sparsity characteristic, sparsity monitor 206 determines the current sparsity (e.g., zero sparsity) for one or more intermediate nodes for one or more training iterations, monitoring periods, or monitoring intervals. or another value), average, median or standard sparsity for one or more intermediate nodes, trend sparsity for one or more intermediate nodes, sparsity for one or more intermediate nodes patterns, zero block ratio, etc. values can be calculated. In some embodiments, sparsity monitor 206 determines a value representing sparsity for a particular set of one or more intermediate nodes, such as individual intermediate nodes, intermediate nodes within a layer of a neural network, etc.

The particular value representing the sparsity characteristic determined by the sparsity monitor 206 depends on the intermediate nodes and/or training iterations, monitoring periods, or monitoring intervals at which sparsity is determined. For example, in some embodiments, the value representing sparsity characteristic 510 is a number (percentage, average, etc.), pattern identifier, trend indicator, etc. As another example, in some embodiments, the value representing sparsity property 510 is a string, such as a 0/1 string, indicating a sequence of training iterations. In this case, each of the one or more intermediate nodes outputs zero or other values. As another example, in some embodiments the value representing sparsity property 510 is a stage. Through this stage, a corresponding range of sparsity is represented by each digit or digits.

The sparsity monitor 206 may store (e.g., the sparsity information list 300) in, for example, the memory 204 and/or local memory within the sparsity monitor 206 and/or store information representing sparsity characteristics. may be output to one or more other functional blocks within electronic device 200. For example, in some embodiments, sparsity monitor 206 generates a file containing values representative of sparsity characteristics and sends the file to processor 202 (such as by storing it in a shared memory location within memory 204). provide. Processor 202 then controls one or more aspects of performing subsequent training iterations based at least in part on the one or more values representative of the sparsity property. For example, processor 202 may determine the particular type of computation to be performed for subsequent training iterations (e.g., which sparsity computation mechanisms/kernels and/or shorthand computations are used, if any). The compression used when storing intermediate data in memory 204 for subsequent training iterations may be selected, and so on.

In some embodiments, processor 202 adjusts the performance level (e.g., , voltage and/or control clock) are disabled, powered down or otherwise reduced. For example, if the sparsity is higher and therefore more sparse matrix computations are used for subsequent training iterations, the processor 202 may perform the computation units, pipelines, memory interface elements, controllers and/or computations. Other functional blocks that are not used for this purpose may be disabled.

(Active mode and hibernation mode)
In some embodiments, a sparsity monitor (e.g., sparsity monitor 206) operates in one of active and hibernation modes when monitoring sparsity of intermediate data during training iterations of a neural network training process. On the other hand it works. FIG. 6 is a state diagram illustrating transitions between various modes of operation, according to some embodiments. As can be seen in FIG. 6, the states in which the sparsity monitor can operate include off 600, active mode 602, and hibernation mode 604. Off 600, the sparsity monitor is activated when the electronic device 200 is not processing an instance of training data, is shut down, and/or when the sparsity monitor is in a low power mode (provided that the sparsity monitor is do not monitor sparsity in intermediate data, such as in cases where the data is at least capable of preserving at least some functionality. The sparsity monitor receives a request from a processor (e.g., processor 202) to start/restart sparsity monitoring when it receives an "on" signal, such as when power is applied to the sparsity monitor. transition from off 600 to active mode 602 (or, alternatively, in some embodiments to hibernation mode 604), such as upon receiving an instance of training data/processing of training iterations by the processor. . The sparsity monitor receives a request from a processor (e.g., processor 202) to stop sparsity monitoring if it receives an "off" signal, such as when power to the sparsity monitor is shut off. If so, a transition is made from active mode 602 or hibernation mode 604 to off 600, such as when the processing of an instance of training data/training iteration is not detected by the processor.

In active mode 602, the sparsity monitor determines if the intermediate data for different training iterations has or may have different levels of sparsity (i.e., different numbers or patterns of intermediate nodes producing zero or another value). etc., to monitor relatively unstable/changing sparsity. A fixed monitoring period is not well suited for monitoring sparsity because sparsity is unstable (may be decreasing). Thus, in active mode 602, the sparsity monitor determines whether an intermediate node and/or its group (e.g., a layer of a neural network) based on the current sparsity and history of sparsity for the intermediate node and/or group. Dynamically adjust the monitoring period for intermediate nodes within a neural network, intermediate nodes within an area of a neural network, intermediate nodes of a particular type, etc.).

FIG. 7 is a diagram illustrating an example of pseudocode that dynamically adjusts the monitoring period of a sparsity monitor in active mode 602, according to some embodiments. As can be seen in Figure 7, the sparsity monitor has a sufficient "history" of sparsity, i.e., for one or more training iterations or monitoring intervals (and more specifically, for a certain number of training First generate sparsity records for one or more intermediate nodes (for an iteration or a certain number of monitoring intervals). Next, the sparsity monitor establishes the current sparsity, ie, the current value of sparsity. The sparsity monitor then compares the current sparsity with a particular value of historical/past sparsity (shown in Figure 7 as the first position in the historical sparsity record). to determine whether the current sparsity is less than a threshold away from the historical sparsity. That is, this is a case where the sparsity changes by a small amount between the history sparsity and the current sparsity. In such a case, the sparsity is stable and increases the monitoring period by a predetermined amount (this is shown as a factor of 2 in FIG. 7, but various increases may be used). Increases in the monitoring period are limited to the maximum hibernation monitoring period. This is typically the longest monitoring period used in either active mode 602 or hibernation mode 604. In other words, as long as the sparsity is relatively stable in active mode, the sparsity monitor periodically increases the monitoring period until it reaches a maximum length equal to the hibernation monitoring period. In some embodiments, although not shown in FIG. 7, if it is determined that the monitored sparsity is unstable after increasing the monitoring period, the monitoring period may be decreased/shortened (e.g. , halved, reduced by a certain amount, etc.). However, such a reduction only occurs until the monitoring period is equal to the active mode initial value (minimum value of the monitoring period).

As mentioned above, in some embodiments, in active mode, intermediate data output by individual intermediate nodes may be individually monitored by a sparsity monitor. In some of these embodiments, the monitoring period can be set to the resolution of individual intermediate nodes. In other words, different intermediate nodes may be monitored using different monitoring periods. For example, the sparsity monitoring device may use a first monitoring period to monitor intermediate data of a first intermediate node or a first group of intermediate nodes, and use a second monitoring period to monitor intermediate data of a first intermediate node or a first group of intermediate nodes. A second intermediate node or a second group of intermediate nodes may be monitored, and so on. This provides the sparsity monitor with fine-grained resolution for detecting sparsity (or sparsity patterns, averages, trends, etc.) within intermediate nodes.

When the sparsity monitor operates in active mode 602, upon determining that sparsity is relatively stable, the sparsity monitor transitions to hibernation mode 604. In hibernation mode 604, the monitoring period is not dynamically adjusted and the same monitoring period (ie, hibernation mode monitoring period) is used for all intermediate nodes. If the sparsity is relatively unstable, ie, changing by more than a certain amount, the sparsity monitor transitions from hibernation mode 604 to active mode 602.

FIG. 8 is a diagram illustrating an example of pseudocode for transitioning from active mode to hibernation mode and vice versa, according to some embodiments. As can be seen from Figure 8, in active mode, the sparsity monitor periodically loops through the intermediate data to determine whether each intermediate data is being monitored using the hibernation monitoring period and whether the data is valid. i.e., has sparsity higher than an initial sparsity threshold (which may be a value selected, set, or configured, e.g., by software, processor, etc.). In other words, if the data has already been monitored at the slowest rate (ie, the longest/hibernation mode monitoring period) and has a sparsity that is changing by only a small amount. In such a case, the sparsity monitor increases the "hibernation ready valid count." The sparsity monitor also detects if the intermediate data is marked invalid, e.g. if a "fast" termination of the monitor is made on the intermediate data (i.e. due to having lower sparsity). (if monitoring of intermediate data ends before the end of the monitoring interval), the "hibernation ready invalid" count is increased. If all intermediate data are ready for hibernation, that is, the sum of the hibernation-ready valid count and the hibernation-ready invalid count is equal to the number of monitored intermediate data, the sparsity monitoring device enters hibernation mode 604. Transition to. However, it should be noted that in the initial state, ie, all intermediate data is marked as invalid, there is no need to transition to hibernation mode 604.

Continuing with the example of FIG. 8, the sparsity monitoring device transitions from hibernation mode 604 to active mode 602 if it is determined that the intermediate data has unstable sparsity after sufficient intermediate data has been acquired. . Part of the transition to active mode 602 is to reduce the monitoring period to an active mode initial value that is shorter than the hibernation mode monitoring period and may be a small portion of the hibernation mode monitoring period.

(process for monitoring sparsity)
FIG. 9 is a flowchart illustrating a process for monitoring sparsity in intermediate data during training iterations of a training process for a neural network, according to some embodiments. Note that the operations shown in FIG. 9 are shown as general examples of operations performed by some embodiments. Acts performed by other embodiments may include different acts, acts performed in a different order, and/or acts performed by different entities or functional blocks.

In FIG. 9, sparsity is monitored for "at least some" intermediate nodes of the neural network during the training process. A particular "at least some" of intermediate nodes may include a small number of intermediate nodes, such as one intermediate node, and a large number of intermediate nodes, such as all intermediate nodes. In some embodiments, a particular intermediate node that is monitored may achieve sparsity by responding to commands from an administrator, receiving requests from a processor (e.g., processor 202), running firmware, etc. Configured by a monitoring device (eg, sparsity monitoring device 206). Generally, in the described embodiments, the sparsity monitor is capable of monitoring sparsity for individual intermediate nodes and/or groups thereof.

The operations of FIG. 9 begin when the sparsity monitor obtains intermediate data output by at least some intermediate nodes of the neural network during training iterations occurring during the monitoring interval during the monitoring interval of the monitoring period. (step 900). As mentioned above, intermediate data is the result output by intermediate (or "hidden") nodes of a neural network while processing instances of training data, ie, during training iterations. How the intermediate data is obtained depends on the configuration of the processor that generates the intermediate data. For example, in some embodiments, intermediate data is obtained by being received by a sparsity monitor from a processor (eg, in one or more messages or communications). As another example, in some embodiments, intermediate data is obtained by being read by a sparsity monitor from memory (eg, memory 204).

Next, the sparsity monitor generates one or more values representing sparsity characteristics of the intermediate data based at least in part on the intermediate data (step 902). For example, the sparsity monitor may use the intermediate data to generate the number or percentage of intermediate nodes that generated zeros as a value representative of the sparsity characteristic. As another example, the sparsity monitor may use the intermediate data to generate a zero block ratio or another pattern identifier as a value representative of the sparsity characteristic. As another example, the sparsity monitor uses the intermediate data together with past/historical intermediate data from at least some of the intermediate nodes to generate averages, trends and/or other values representative of sparsity characteristics. Can be generated as a value. Generally, for this operation, the sparsity monitor uses sparsity of intermediate data for at least some intermediate nodes over one or more training iterations of one or more monitoring intervals by a processor and/or other entity. Generate values that can be used to evaluate

Next, the sparsity monitor sends one or more values representative of the sparsity characteristic to the processor (step 904). For example, a sparsity monitor may generate one or more messages containing values representative of sparsity characteristics and communicate the messages to a processor. As another example, the sparsity monitor may store one or more values representative of sparsity characteristics in a location in memory shared with the processor.

The processor then controls one or more aspects of performing subsequent training iterations based on the one or more values representing the sparsity characteristic (step 906). For example, a processor may use one or more values representative of sparsity characteristics to select a computational mechanism, such as a particular matrix math operation, to be used for subsequent training iterations. For example, the processor may select intermediate data if the value representing the sparsity property indicates that the sparsity is relatively high (i.e., more intermediate nodes are producing zero or other values). One may choose to use block-sparse matrix multiplication to compute. As another example, the processor uses one or more values representative of sparsity characteristics to apply to intermediate data before it is stored in memory (e.g., memory 204) for subsequent training iterations. You may select the data compression to be used for.

(Process for using active mode and hibernation mode)
10A-10B are flowcharts illustrating a process for using active mode and hibernation mode in a sparsity monitoring device (eg, sparsity monitoring device 206), according to some embodiments. Note that the operations shown in FIGS. 10A-10B are shown as general examples of operations performed by some embodiments. Acts performed by other embodiments may include different acts, acts performed in a different order, and/or acts performed by different entities or functional blocks.

For the operation of FIGS. 10A-10B, it is assumed that the sparsity monitor starts in active mode, but this is not a requirement. In these embodiments, the sparsity monitoring device starts in hibernation mode. As discussed above, and as shown in FIGS. 10A-10B, in active mode, the sparsity monitoring device can change the length of the monitoring period for individual intermediate nodes and/or groups thereof. However, in hibernation mode, the same fixed monitoring period is used on all intermediate nodes.

The operations of FIGS. 10A-10B begin when a sparsity monitor operating in active mode acquires intermediate data from one or more training iterations during a monitoring interval (step 1000). For this operation, the sparsity monitor transfers intermediate data generated by the processor when processing one or more corresponding instances of training data through the neural network to a memory (e.g., memory 204), a processor (e.g., , processor 202) or another source.

Next, the sparsity monitor determines whether the sparsity monitor can transition from active mode to hibernation mode. This includes whether all intermediate nodes are monitored using the hibernation monitoring mode monitoring period, i.e. a monitoring period equal to the longest allowed monitoring period, and whether the sparsity is stable (and therefore still stable). This includes determining whether there is a (Step 1002). For example, a sparsity monitor may compare the change in sparsity in each intermediate data from monitoring interval to monitoring interval (or from one training iteration to another) to a threshold to identify sparsity. It can be determined whether the amount has changed below the amount of and is therefore stable. If all intermediate nodes are monitored using a monitoring period equal to the monitoring period of the hibernation monitoring mode and the sparsity is stable enough (step 1002), the sparsity monitoring device transitions to the hibernation mode ( Step 1004).

Otherwise, if at least some intermediate nodes are not monitored using a monitoring period equal to the monitoring period of the hibernation monitoring mode, or if the sparsity is not stable enough (step 1002), then The monitoring device determines whether to increase (or possibly decrease) the monitoring period for each intermediate node. More specifically, for each intermediate node in turn (step 1006), the sparsity monitor determines whether the sparsity for that intermediate node is stable enough to increase the monitoring period for that intermediate node. (Step 1008). For example, the sparsity monitor may compare the difference between the current value representing the sparsity of the intermediate node and the historical/past value representing the sparsity of the intermediate node to different threshold values. If the sparsity is stable enough, the sparsity monitor increases the length of the monitoring period for that intermediate node (step 1010). For example, a sparsity monitor may double the monitoring period or add a certain amount (time, training repetitions, etc.) to the monitoring period. When increasing the monitoring period, the sparse monitoring device increases the monitoring period up to and without exceeding the hibernation mode monitoring period. For example, the active mode initial value of the monitoring period may be set to ⅛ of the length of the hibernation mode monitoring period. This allows three separate increases in the monitoring period for an intermediate node before reaching the hibernation mode monitoring period if the monitoring period is doubled (eg, during the corresponding monitoring period). In contrast, if the sparsity is not stable enough (step 1008), the sparsity monitor may change the monitoring period without changing the monitoring period (or as long as the monitoring period is still longer than the active mode initial value). may be decreased), the process returns to step 1006 so that the next intermediate node can be processed. If the last intermediate node has been processed (step 1006), the sparsity monitor returns to step 1000 to obtain intermediate data from the next training iteration.

Returning to step 1002, if all intermediate nodes are being monitored using a monitoring period equal to the monitoring period of hibernation monitoring mode and the sparsity is stable enough, the sparsity monitoring device transitions to hibernation mode. (step 1004). Recall that in hibernation mode, sparsity for all intermediate nodes is monitored using the hibernation mode monitoring period. In hibernation mode, the sparsity monitor obtains intermediate data from one or more training iterations during a monitoring period (step 1012). For this operation, the sparsity monitor transfers intermediate data generated by the processor when processing one or more corresponding instances of training data through the neural network to a memory (e.g., memory 204), a processor (e.g., , processor 202) or another source. Note that the monitoring interval of step 1012 is a different/later monitoring interval compared to the monitoring interval of step 1000.

Next, the sparsity monitor determines whether the sparsity monitor can transition from hibernation mode to active mode. This includes determining whether the sparsity is sufficiently unstable (step 1014). For example, the sparsity monitor compares the change in sparsity in each individual intermediate data from monitoring interval to monitoring interval (or from one training iteration to another) to a threshold to determine the sparsity. It can be determined whether it has changed by more than a certain amount and is therefore unstable. If the sparsity is unstable enough (step 1014), the sparsity monitor transitions to active mode (step 1016). This includes returning to step 1000 to obtain intermediate data from the next training iteration in active mode. Otherwise, if the sparsity is still stable, the sparsity monitor returns to step 1012 to obtain intermediate data from the next training iteration in hibernation mode.

(Fast end of monitoring period)
FIG. 11 is a flowchart illustrating a process for fast termination of a monitoring interval by a sparsity monitor, according to some embodiments. Note that the operations shown in FIG. 11 are shown as general examples of operations performed by some embodiments. Acts performed by other embodiments may include different acts, acts performed in a different order, and/or acts performed by different entities or functional blocks.

The process illustrated in FIG. 11 occurs when a sparsity monitor (e.g., sparsity monitor 206) obtains intermediate data from a first training iteration of the plurality of training iterations during a monitoring interval that includes multiple training iterations. (step 1100). For example, the sparsity monitor may obtain intermediate data from only the first training iteration (out of a number of training iterations that occur during the monitoring period) or from the first few training iterations. Obtaining intermediate data is described elsewhere in this specification.

Next, the sparsity monitoring device determines whether the intermediate data is less than a sparsity threshold (step 1102). For example, the sparsity monitor may count the number of intermediate nodes that output zero as intermediate data and calculate the proportion or proportion of intermediate data that is equal to zero. The sparsity monitor can then compare this percentage or ratio to a particular sparsity threshold.

The intermediate data is less than the sparsity threshold (step 1102), i.e., has a small number of zeros (or other values), such that the sparsity of the intermediate data is less than the threshold (i.e., the data is relatively full). ), the sparsity monitor provides fast termination of processing and acquisition of intermediate data during the monitoring interval. More specifically, the sparsity monitor does not acquire intermediate data for training iterations after the first training iteration by terminating sparsity monitoring early during the monitoring interval (step 1104). By terminating the monitoring during the monitoring interval in this manner, the sparsity monitoring device may Intermediate data acquisition and processing, thus avoiding power consumption, use of computational resources and communication system bandwidth, etc. The sparsity monitoring device also stores an indicator that the monitoring interval has ended early for the intermediate data (step 1106). For example, a sparsity monitor may mark intermediate data as invalid to indicate that typical acquisition and processing was not performed.

If the intermediate data exceeds the sparsity threshold (step 1102) and therefore the data is comparatively sparse/has more zeros (or other values), the sparsity monitor will Continue monitoring. Continuing includes obtaining intermediate data from training iterations after the first training iteration (step 1108). In this case, in other words, the sparsity monitor continues typical or normal sparsity monitoring and does not use fast termination of the monitoring interval.

In some embodiments, an electronic device (e.g., electronic device 200 and/or a portion thereof) performs operations described herein using code and/or data stored on a non-transitory computer-readable storage medium. Perform some or all of the operations described. More specifically, the electronic device reads code and/or data from a computer-readable storage medium and executes the code and/or uses the data in performing the described operations. A computer-readable storage medium may be any device, medium, or combination thereof that stores code and/or data for use by an electronic device. For example, the computer-readable storage medium may include flash memory, random access memory (e.g., eDRAM, RAM, SRAM, DRAM, DDR4 SDRAM, etc.), read-only memory (ROM), and/or magnetic or optical storage medium (e.g., disk The storage device may include volatile and/or non-volatile memory including, but not limited to, drives, magnetic tape, CDs, DVDs, etc.).

In some embodiments, one or more hardware modules perform the operations described herein. For example, a hardware module may include one or more processors/cores/central processing units (CPUs), application specific integrated circuit (ASIC) chips, neural network processors or accelerators, field programmable gate arrays (FPGAs), computational units, embedded May include, but are not limited to, processors, graphics processors (GPUs)/graphics cores, pipelines, accelerated processing units (APUs), sparsity supervisors, functional blocks, and/or other programmable logic devices. . When such a hardware module is activated, it performs some or all of its operations. In some embodiments, a hardware module includes one or more general-purpose circuits configured to perform operations by executing instructions (program code, firmware, etc.).

In some embodiments, data structures representing some or all of the structures and mechanisms described herein (e.g., electronic device 200, sparsity monitor 206, and/or portions thereof) are electronic stored on a non-transitory computer-readable storage medium containing a database or other data structure that can be read by a device and used directly or indirectly to manufacture hardware, including structures and mechanisms; There is. For example, the data structure may be a behavioral level description or a register transfer level (RTL) description of a hardware function in a high level design language (HDL) such as Verilog or VHDL. The description may be read by a synthesis tool, which synthesizes the description and generates a netlist containing a list of gates/circuit elements representing the functionality of the hardware, including the structures and mechanisms described above, using a synthesis library. It can be generated from. The netlist may then be placed and routed to generate a dataset describing the geometry applied to the mask. The mask may then be used in various semiconductor manufacturing steps to fabricate one or more semiconductor circuits (eg, integrated circuits) corresponding to the structures and mechanisms described above. Alternatively, the database on a computer-accessible storage medium may be a netlist (with or without a synthesis library) or a dataset or Graphics Data System (GDS) II data, as appropriate.

Variables or unspecified values (ie, a general description of a value without a specific instance of the value) are represented herein by a letter, such as N. Although similar letters may be used elsewhere in this description, as used herein, variables and nonspecific values in each case are not necessarily the same, i.e., common variables and nonspecific values. Different variable quantities and values may exist for some or all of the specified values. In other words, N and other letters used to represent variables and unspecified values in this description are not necessarily related to each other.

As used herein, the expression "etc." It is intended to present the equivalent of "one". For example, in the sentence "The electronic device performs a first action, a second action, etc.", the electronic device performs at least one of the first action, the second action, and the other action. Execute. Furthermore, the elements in the list that relate to "etc." are merely examples among a set of examples, and at least some of the examples may not appear in some embodiments.

The above description of embodiments has been presented for purposes of illustration and description only. They are not intended to be exhaustive or to limit the embodiments to the forms disclosed. Accordingly, many modifications and variations will be apparent to those skilled in the art. Furthermore, the above disclosure is not intended to limit the embodiments. The scope of the embodiments is defined by the following claims.

Claims (23)

a processor configured to perform training iterations during a neural network training process, each training iteration comprising processing a separate instance of training data through the neural network;
a sparsity monitoring device;
The sparsity monitoring device includes:
operating in active or hibernation mode, dynamically adjusting the length of the monitoring period based on sparsity characteristics of intermediate data output by at least some intermediate nodes of the neural network;
During monitoring intervals in each of one or more monitoring periods, obtaining intermediate data output by at least some intermediate nodes of the neural network during training iterations occurring during each monitoring interval;
generating one or more values representative of sparsity characteristics for the intermediate data based at least in part on the intermediate data;
one or more values representing the sparsity characteristic configured to control one or more aspects of performing subsequent training iterations based at least in part on the one or more values representing the sparsity characteristic. transmitting the information to the processor;
is configured to do
electronic device. While operating in the active mode, the sparsity monitoring device:
For each at least some of said intermediate nodes, said sparsity characteristic of said two or more monitoring intervals is determined based on one or more values representative of said sparsity characteristic from said two or more monitoring intervals. determining whether the active mode has changed by more than an active mode threshold amount between;
increasing the length of a monitoring period for a subsequent monitoring period for that intermediate node if the sparsity property has not changed by more than the threshold amount, such that the sparsity is stable; the length of the monitoring period is increased to a maximum length of a hibernation mode monitoring period;
is configured to do
The electronic device of claim 1. retaining the current length of the monitoring period if the sparsity characteristic has changed by more than the threshold amount for an intermediate node, thereby making the sparsity unstable; reducing the length of a monitoring period for a subsequent monitoring period for the node, wherein the length of the monitoring period is reduced to a minimum active mode initial value; ,
The electronic device of claim 2. Increasing the length of the monitoring period for said subsequent monitoring periods increases the monitoring period for individual intermediate nodes in said neural network such that different intermediate nodes can have different monitoring periods in said active mode. including individually increasing the length of
The electronic device of claim 2. While operating in the active mode, the sparsity monitoring device:
The length of the monitoring period is increased to the length of the hibernation mode monitoring period for all of the at least some of the intermediate nodes, and the intermediate data output by each of the at least some of the intermediate nodes is configured to cause the sparsity monitoring device to transition to hibernation mode when the intermediate data sparsity threshold is below;
In the hibernation mode, the sparsity monitoring device keeps the length of the monitoring period for all intermediate nodes to the length of the hibernation mode monitoring period.
The electronic device of claim 2. While operating in the hibernation mode, the sparsity monitoring device:
For each at least some of said intermediate nodes, said sparsity characteristic of said two or more monitoring intervals is determined based on one or more values representative of said sparsity characteristic from said two or more monitoring intervals. determining whether the active mode has changed by more than the active mode threshold amount between;
transitioning the sparsity monitor to the active mode if the sparsity characteristic changes by more than the threshold amount for a certain number of intermediate nodes, thereby making the sparsity unstable; and the transitioning includes reducing the length of the monitoring period to an active mode initial value that is shorter than the hibernation mode monitoring period;
is configured to do
The electronic device according to claim 5. The sparsity monitoring device includes:
For at least some of the intermediate nodes, one or more of the values representing the sparsity characteristic is less than the sparsity threshold, such that the intermediate data is not sparse. configured not to obtain the intermediate data output by the intermediate node during at least some of the training iterations occurring during the current monitoring interval by terminating the current monitoring interval early;
The electronic device of claim 1. Generating one or more values representative of sparsity characteristics for the intermediate data includes averaging intermediate data obtained from two or more training iterations occurring during one or more monitoring intervals. ,
The electronic device of claim 1. Controlling one or more aspects of performing the subsequent training iterations comprises:
configuring the subsequent training iterations to use simplified calculations for operations involving intermediate data when the one or more values representative of the sparsity characteristic indicate higher sparsity. ,
The electronic device of claim 1. Controlling one or more aspects of performing the subsequent training iterations comprises:
configuring the processor to use a reduced or compressed data format to store the intermediate data in memory during training iterations if one or more values representative of the sparsity characteristic indicate higher sparsity; including
The electronic device of claim 1. the one or more values representing sparsity characteristics of the intermediate data include one or more stages selected from a plurality of stages;
each stage of the plurality of stages indicates a corresponding different portion of a range of possible sparsity values for the intermediate data;
The electronic device of claim 1. the one or more values representative of the sparsity characteristic include trends or patterns in the sparsity of the intermediate data for two or more training instances;
The electronic device of claim 1. the one or more values representative of the sparsity characteristic include a zero block ratio of the intermediate data for one or more training instances;
The electronic device of claim 1. a processor configured to perform training iterations during a neural network training process, each training iteration comprising processing a separate instance of training data through the neural network; A method for monitoring sparsity in intermediate data in an electronic device comprising:
The sparsity monitoring device operates in an active mode or a hibernation mode, while adjusting the length of the monitoring period based on sparsity characteristics of intermediate data output by at least some intermediate nodes of the neural network. to adjust the
the sparsity monitoring device obtains, during monitoring intervals in each of one or more monitoring periods, intermediate data output by at least some intermediate nodes of the neural network during training iterations occurring during each monitoring interval; to do and
the sparsity monitor generating one or more values representative of sparsity characteristics for the intermediate data based at least in part on the intermediate data;
One or more aspects wherein the sparsity monitor performs subsequent training iterations based at least in part on the one or more values representing the sparsity characteristic. to the processor configured to control the
Method. While the sparsity monitoring device is operating in the active mode,
For each at least some of said intermediate nodes, said sparsity characteristic of said two or more monitoring intervals is determined based on one or more values representative of said sparsity characteristic from said two or more monitoring intervals. determining whether the active mode has changed by more than an active mode threshold amount between;
increasing the length of a monitoring period for a subsequent monitoring period for that intermediate node if the sparsity property has not changed by more than the threshold amount, such that the sparsity is stable; and the length of the monitoring period is increased to a maximum length of a hibernation mode monitoring period.
15. The method of claim 14. Increasing the length of the monitoring period for said subsequent monitoring periods increases the monitoring period for individual intermediate nodes in said neural network such that different intermediate nodes can have different monitoring periods in said active mode. including individually increasing the length of
16. The method of claim 15. While the sparsity monitoring device is operating in the active mode,
the length of the monitoring period is increased to the length of the hibernation mode monitoring period for all of the at least some of the intermediate nodes, and intermediate data output by each of the at least some of the intermediate nodes; is less than an intermediate data sparsity threshold, further comprising transitioning the sparsity monitoring device to a hibernation mode;
The sparsity monitoring device keeps the length of the monitoring period for all intermediate nodes to the length of the hibernation mode monitoring period.
16. The method of claim 15. While the sparsity monitoring device is operating in the hibernation mode,
For each at least some of said intermediate nodes, said sparsity characteristic of said two or more monitoring intervals is determined based on one or more values representative of said sparsity characteristic from said two or more monitoring intervals. determining whether the active mode has changed by more than the active mode threshold amount between;
transitioning the sparsity monitor to the active mode if the sparsity characteristic changes by more than the threshold amount for a certain number of intermediate nodes, thereby making the sparsity unstable; The transition further comprises: reducing the length of the monitoring period to an active mode initial value that is shorter than the hibernation mode monitoring period.
18. The method of claim 17. The sparsity monitoring device includes:
For at least some of the intermediate nodes, one or more of the values representing the sparsity characteristic is less than the sparsity threshold, such that the intermediate data is not sparse. further comprising not retrieving the intermediate data output by the intermediate node during at least some of the training iterations occurring during the current monitoring interval by terminating the current monitoring interval early;
15. The method of claim 14. Generating one or more values representative of sparsity characteristics for the intermediate data includes averaging intermediate data obtained from two or more training iterations occurring during one or more monitoring intervals. ,
15. The method of claim 14. Controlling one or more aspects of performing the subsequent training iterations comprises:
configuring the subsequent training iterations to use simplified calculations for operations involving intermediate data when the one or more values representative of the sparsity characteristic indicate higher sparsity. ,
15. The method of claim 14. Controlling one or more aspects of performing the subsequent training iterations comprises:
configuring the processor to use a reduced or compressed data format to store the intermediate data in memory during training iterations if one or more values representative of the sparsity characteristic indicate higher sparsity; including
15. The method of claim 14. the one or more values representing sparsity characteristics of the intermediate data include one or more stages selected from a plurality of stages;
each stage of the plurality of stages indicates a corresponding different portion of a range of possible sparsity values for the intermediate data;
15. The method of claim 14.

Images