JP2019114231A - Quantization of neural network parameter - Google Patents

Abstract

To provide a method of generating a neural network.SOLUTION: The method may include a step of applying a nonlinear quantization to a plurality of synapse weights of a neural network model. The method may further include a step of training the neural network model. The method may further include a step of generating a neural network output from the trained neural network model on the basis of one or more inputs received by the trained neural network model.SELECTED DRAWING: Figure 2

Description

  Embodiments described in the present disclosure relate to non-linear quantization of neural network parameters.

  Neural network analysis may include models of analysis produced by biological neural networks that attempt to model high level abstractions through multiple processing layers. Neural network analysis may consume a large amount of computing and / or network resources.

  The claimed subject matter in the present application is not limited to embodiments that solve any drawbacks or that operate only in the environment as described above. Rather, the description of this background is only provided to illustrate one exemplary technical area in which some embodiments described in the present disclosure may be practiced.

  One or more embodiments of the present disclosure may include a method of generating a neural network. The method may include applying non-linear quantization to a plurality of synapse weights of the neural network model. The method may also further include the step of training a neural network model. Additionally, the method may include generating a neural network output from the trained neural network model based on the one or more inputs received by the trained neural network model.

  The objects and advantages of the embodiments will be realized and attained by the elements, features, and combinations particularly pointed out in the claims. Both the foregoing general description and the following detailed description are exemplary and explanatory and not restrictive.

Exemplary embodiments will be described and explained more specifically and in detail using the attached drawings.
Graph of non-linear function. FIG. 1B is a graph showing the input-output characteristics of the non-linear function shown in FIG. 1A. 3 is a flowchart of an exemplary method of generating a neural network. FIG. 7 illustrates an exemplary weight distribution for a range of quantization levels. An exemplary graph showing functions and quantization levels. 6 is a flowchart of another exemplary method of generating a neural network. FIG. 7 is a flowchart of yet another exemplary method of generating a neural network. FIG. 2 is a block diagram of an exemplary computing device.

  The various embodiments disclosed in the present application relate to the quantization of neural networks. More particularly, according to some embodiments, a non-linear function may be applied to one or more parameters (eg, synapse weights) of one or more layers of one or more models of neural networks.

  A neural network may include a model having multiple layers, and neurons in each layer may be communicatively coupled to neurons in adjacent layers via coupling. More specifically, one neuron in one layer can be communicatively coupled to one neuron in one adjacent layer via one connection. Further, connections between neurons of adjacent layers may be assigned synapse weights (eg, 32-bit weights) that may indicate the strength or strength of the associated connection between the neurons.

  Neural networks are generally over-parameterized, and models in neural networks may include redundancy (eg, redundancy of training parameters such as synapse weights). Additionally, generating and / or training neural networks (eg, including one or more models) may require significant resources (eg, processing requirements and / or memory). There are various approaches such as fixed point implementation for synaptic weights and / or activation, network pruning, and sharing weights (eg via a hash function) to reduce redundancy.

  Although linear quantization may be used for neural network inference and / or training to reduce computational costs, this may reduce the numerical accuracy for the parameters. Furthermore, quantization with logarithmic coding may be used to reduce the computational complexity of converting multiplications to additions (eg, log (x * y) = log (x) + log (y)).

  Some conventional methods of generating neural networks involve increasing the number of bits in the neural network to provide a wider range and finer resolution. However, this may result in increased computational and memory requirements (e.g., due to processing elements with higher bits and increased amounts of data being stored).

  In addition, some conventional methods of generating neural networks may minimize signal-to-distribution quantization error (SQNR). For example, at least one conventional method minimizes the error for each single piece of data. While this can improve the efficiency of deep neural networks, it requires considerable use of random number generators, which are very computationally expensive. Furthermore, at least one other conventional method minimizes the total amount of error (e.g. E = Σ m * ε (m is the number of data to be quantized at a particular quantization level)). This method of requiring a trained model increases the cost of training the model.

  In addition, some conventional methods of generating neural networks that consider smaller learned weights that are not important compared to larger learned weights by pruning (eg, setting combinations with small weights to zero) Can improve efficiency, but these methods may require multiple loops of training, pruning, and tuning, thus increasing the total cost of training the neural network Be done. Other methods may use logarithmic quantization, which penalizes larger values (eg, fewer quantization levels).

  According to various embodiments, non-linear quantization can be used to improve the efficiency of the deep neural network. According to some embodiments, quantization may include coarse resolution (e.g., the number of lower levels) for low value inputs and fine resolution (e.g., the number of higher levels) for high value inputs. Furthermore, in various embodiments, non-linear functions that provide non-linear input-output characteristics may be used. Non-linear quantization (eg, via a non-linear function) can increase the diversity of neural network parameters (eg, synapse weights). Exemplary non-linear functions may include, but are not limited to cosh (x) (hyperbolic cosine function) and sinh (x) (hyperbolic sine function).

  In contrast to some conventional systems and methods that only reduce errors associated with data processed in neural networks, the various embodiments of the present disclosure are sufficient in classification as outputs of neural networks. The quantization characteristics (eg, range and / or resolution) can be improved to provide accuracy. For example, various embodiments modify (eg, increase or decrease) the range of weight values and / or change (eg, increase or decrease) the number of bits within the range (eg, number of quantization levels) ) To change (eg, increase or decrease) the resolution.

  Compared to conventional systems and methods, the various embodiments of the present disclosure reduce the number of bits (also referred to as "the number of quantization levels" in the present disclosure) while providing sufficient resolution and covering a sufficient range. be able to. Therefore, the computational cost can be reduced. Furthermore, instead of just minimizing the SQNR, various embodiments may use more levels (eg, more levels) to weights (eg, more important parameters) with larger values to improve the accuracy of the neural network. May involve allocating a large number of bits).

  Thus, the various embodiments of the present disclosure provide technical solutions to one or more problems arising from technology that can not reasonably be done by humans, as described in more detail in the present disclosure; The various embodiments disclosed in the present application are rooted in computer technology in order to overcome the problems and / or problems described above. Further, at least some embodiments disclosed in the present application can improve computer related technology by enabling computer execution of functions that could not previously be performed by the computer.

  Various embodiments of the present disclosure include internet and cloud applications (eg, image classification, speech recognition, language translation, language processing, emotion analysis recommendations, etc.), pharmacy and biology (eg, cancer cell detection, diabetes classification, wound creation Drugs etc), media and entertainment (eg video captioning, video search, real time translation etc), security and defense (eg face detection, video surveillance, satellite image etc), and autonomous machines (eg pedestrian detection, lanes It can be used in various applications such as tracking, traffic signal detection, etc.

  Embodiments of the present disclosure will now be described with reference to the attached figures.

  FIG. 1A is a graph showing an exemplary non-linear function (eg, cosh (x) -1 (x> = 0)). FIG. 1B is a graph showing the input-output characteristics for the function of FIG. 1A. As shown in FIG. 1B, the function of FIG. 1A provides non-linear input-output characteristics.

  FIG. 2 is a flowchart of an exemplary method 200 of generating a neural network, in accordance with at least one embodiment of the present disclosure. Method 200 may be performed by any suitable system, apparatus, or device. For example, one or more components of device 600 of FIG. 7, or components thereof, may perform one or more operations of the operations associated with method 200. In these and other embodiments, program instructions stored on computer readable media may be executed to perform one or more of the operations of method 200.

  At block 202, non-linear quantization may be applied to the neural network, and method 200 may proceed to block 204. More particularly, in at least some embodiments, a non-linear function such as, but not limited to cosh (x) and sinh (x) is applied to at least one parameter of the model of the neural network obtain. More particularly, in at least some embodiments, a non-linear function may be applied to one or more synaptic weights (eg, for neuron coupling) of a neural network. Thus, a non-linear relationship may exist between the initial synapse weight values and the quantized synapse weight values. For example, processor 610 of FIG. 7 may apply non-linear quantization to a neural network.

  At block 204, a neural network may be trained and the method 200 may proceed to block 206. For example, neural networks including models can be trained, for example, via conventional back propagation with random initialization and / or any other suitable training method. More specifically, one or more training parameters of each layer of the model may be trained. In some embodiments where a non-linear function is applied to one or more synaptic weights, one or more synaptic weights may be optimized via training. For example, processor 610 of FIG. 7 may train a neural network.

  At block 206, it may be determined if the required accuracy for the neural network is met. For example, the processor 610 of FIG. 7 compares the accuracy threshold with the determined (eg, measured) accuracy of the neural network to determine whether the neural network's accuracy meets the required accuracy. obtain. If the required accuracy is met, method 200 may proceed to block 208. If the required accuracy is not met, method 200 may return to block 204.

  At block 208, a neural network may be used. More specifically, for example, a neural network can receive one or more inputs (e.g., new data) and generate one or more outputs based on the trained model. For example, processor 610 of FIG. 7 may generate one or more outputs based on the trained model and the one or more inputs received.

  Modifications, additions, or omissions may be made to method 200 without departing from the scope of the present disclosure. For example, the acts of method 200 may be performed in a different order. Furthermore, the operations and steps described are merely provided as examples, and some of the operations and steps may be optional without compromising the essence of the disclosed embodiments. And may be combined into fewer operations and steps, and may be extended to additional operations and steps.

  FIG. 3 is a graph 300 illustrating an exemplary weight distribution for the range 301 of quantization level 302. The spacing between each quantization level 302 may indicate the resolution of the quantization operation. As shown in the graph 300, the resolution in the outer part of the range 301 is coarse (e.g. less quantization levels) and the resolution in the inner part of the range 301 is fine (e.g. more quantum) Level). Thus, as used in this disclosure, the phrase "coarse resolution" may be associated with a first number of bits (eg, bit number N (eg, fewer bits)), and the phrase "fine resolution" It may be associated with a more second number of bits (eg, number of bits M (eg, number of more bits)).

  Various embodiments of the present disclosure increase resolution for weights (eg, layer inputs) having higher values (eg, absolute values) as described in the present disclosure, and optionally lower values (eg, The accuracy of the neural network can be improved by reducing the resolution for weights (eg, layer inputs) that have absolute values. For example, according to various embodiments, coarse resolution quantization may be provided for lower valued inputs (e.g., weight values) (e.g., indicative of lower strength or strength of the associated coupling) and of finer resolution. Quantization may be provided for higher value inputs (e.g., weight values) (e.g., indicating high strength or strength of the associated coupling). For example, referring again to FIG. 1B, as the weight value (along the X axis) increases, the resolution (eg, the number of quantization levels) may also increase (eg, from coarse resolution to fine resolution).

For example, according to at least one embodiment, predefined quantization levels may be used. The function f (x) such as cosh (x), sinh (x) can be used to provide appropriate quantization levels via the following equation:
l _m = f ^-1 (m * d)
In the above equation, l _m is the m th quantization level, f ⁻¹ is the inverse function of f (x), m is an integer, and d represents the spacing between the quantized outputs, It may have the following relationship:
f (maximum range) = (maximum value of m) * d

In some embodiments, the spacing d may be a fixed value. For example, in at least some embodiments, the method 200 of FIG. 2 may be performed with predefined quantization levels (eg, fixed values and fixed ranges of intervals d). FIG. 4 is an exemplary graph 350 illustrating the function f (x). In this example, f (x) = cosh (x) -1 (x> = 0) or f (x) =-cosh (x) +1 (x <0). Graph 350 further includes a quantization level _{(e.g., l 1, l 2, l} 3 etc.), as described above, and a distance d is the spacing between the output (e.g., output of the quantizer). In this example, the spacing d is a fixed value.

  According to other embodiments, dynamic quantization levels may be used. For example, given a number of bits, the range and / or resolution may be enhanced by changing the spacing d to meet an accuracy threshold (eg, the required and / or desired accuracy).

  Furthermore, according to various embodiments, the resolution of quantization may be changed, for example, by changing the range and / or changing the number of bits (eg, the number of quantization levels). More specifically, the resolution may be increased by reducing the range and / or increasing the number of bits. In this example, the spacing between quantization levels (eg, spacing d) may be reduced. Furthermore, the resolution may be reduced by increasing the range and / or reducing the number of bits. In this example, the spacing between quantization levels (eg, spacing d) may be increased.

  FIG. 5 is a flowchart of an exemplary method 400 for generating a neural network, in accordance with at least one embodiment of the present disclosure. Method 400 may be performed by any suitable system, apparatus, or device. For example, one or more components of device 600 of FIG. 7, or components thereof, may perform one or more operations of the operations associated with method 400. In these and other embodiments, program instructions stored on computer readable media may be executed to perform one or more of the operations of method 400.

  At block 401, the spacing between quantized outputs may be set to a maximum value, and method 400 may proceed to block 402. For example, processor 610 of FIG. 7 may set the spacing between quantization outputs to a maximum value. For example, the range and / or resolution for parameters (eg, synapse weights) may be known, and the maximum value may be equal to the maximum spacing in the neural network.

  At block 402, non-linear quantization may be applied to the neural network, and the method 400 may proceed to block 404. More particularly, in at least some embodiments, a non-linear function such as, but not limited to cosh (x) and sinh (x) is applied to at least one parameter of the model of the neural network obtain. More particularly, in at least some embodiments, a non-linear function may be applied to one or more synaptic weights (eg, for coupling) of the neural network. For example, processor 610 of FIG. 7 may apply non-linear quantization to a neural network.

  At block 404, a neural network including a model may be trained, and method 400 may proceed to block 406. For example, neural networks may be trained, for example, via conventional back propagation with random initialization and / or any other suitable training method. More specifically, one or more training parameters of each layer of the model may be trained. In some embodiments where a non-linear function is applied to one or more synaptic weights, one or more synaptic weights may be optimized via training. For example, processor 610 of FIG. 7 may train a neural network.

  At block 406, it may be determined whether the required accuracy is met. For example, the processor 610 of FIG. 7 may compare the accuracy threshold to the measured accuracy of the neural network to determine if the neural network accuracy meets the required accuracy. If the required accuracy is met, method 400 may proceed to block 408. If the required accuracy is not met, method 400 may proceed to block 410.

  At block 408, a neural network may be used. More specifically, for example, a neural network can receive one or more inputs (e.g., new data) and generate one or more outputs based on the trained model. For example, processor 610 of FIG. 7 may generate one or more outputs based on the trained model and the one or more inputs received.

  At block 410, the spacing between the quantized outputs may be reduced, and the method 400 may return to block 402. For example, the spacing between quantization outputs (eg, spacing d) can be reduced by one unit, two units, three units, etc. to increase resolution (and hence, eg, neural network accuracy) . For example, processor 610 of FIG. 7 may reduce the spacing between quantization outputs.

  Modifications, additions, or omissions may be made to method 400 without departing from the scope of the present disclosure. For example, the acts of method 400 may be performed in a different order. Furthermore, the operations and steps described are merely provided as examples, and some of the operations and steps may be optional without compromising the essence of the disclosed embodiments. May be combined with fewer operations and steps, and may be extended with additional operations and steps.

  Furthermore, in some embodiments, a range of values for the spacing between quantization outputs may be used during neural network training. In addition, for each value of spacing (e.g. spacing d), the accuracy of the neural network can be measured and recorded. After each value for the interval is used, the determined optimal value for the interval (eg, the value providing the highest accuracy) may be used while the neural network is being used for inference.

  For example, FIG. 6 is a flow chart of an exemplary method 500 of generating a neural network, in accordance with at least one embodiment of the present disclosure. Method 500 may be performed by any suitable system, apparatus, or device. For example, one or more components of device 600 of FIG. 7, or components thereof, may perform one or more operations of the operations associated with method 500. In these and other embodiments, program instructions stored on computer readable media may be executed to perform one or more of the operations of method 500.

  At block 502, the spacing between quantization outputs may be set to a large value (eg, a maximum value), and the method 500 may proceed to block 504. For example, processor 610 of FIG. 7 may set the spacing between quantization outputs (eg, spacing d) to a large value (eg, maximum spacing in a neural network).

  At block 504, non-linear quantization may be applied to the neural network, and the method 500 may proceed to block 506. More particularly, in at least some embodiments, a non-linear function such as, but not limited to cosh (x) and sinh (x) is applied to at least one parameter of the model of the neural network obtain. More particularly, in at least some embodiments, a non-linear function may be applied to one or more synaptic weights (eg, for coupling) of the neural network. For example, processor 610 of FIG. 7 may apply non-linear quantization to a neural network.

  At block 506, a neural network including a model may be trained, and method 500 may proceed to block 508. For example, neural networks may be trained, for example, via conventional back propagation with random initialization and / or any other suitable training method. More specifically, one or more training parameters of each layer of the model may be trained. In some embodiments where a non-linear function is applied to one or more synaptic weights, one or more synaptic weights may be optimized via training. For example, processor 610 of FIG. 7 may train a neural network.

  At block 508, the spacing between the quantized outputs and the associated measured accuracy of the neural network may be recorded, and the method 500 may proceed to block 510. For example, processor 610 of FIG. 7 may measure the accuracy of the neural network using the spacing between quantization outputs (eg, spacing d), the value of the spacing between the measured accuracy and the quantization outputs being , (Eg, in memory 630 of FIG. 7).

  At block 510, it may be determined whether the spacing between quantization outputs is a minimum value. For example, processor 610 of FIG. 7 may determine if the spacing between quantization outputs (eg, spacing d) is a minimum value. For example, in a digital implementation using 8/16 bit integers (e.g. fixed point), the minimum value for the interval may be equal to one. If the spacing between quantized outputs is a minimum, method 500 may proceed to block 514. If the spacing between quantization outputs is greater than the minimum value, method 500 may proceed to block 512.

  At block 512, the spacing between the quantized outputs may be reduced, and the method 500 may return to block 504. For example, the spacing between quantization outputs may be reduced by one unit, two units, three units, etc. For example, processor 610 of FIG. 7 may reduce interval d.

  At block 514, a neural network may be used. More specifically, for example, a neural network can receive one or more inputs (e.g., new data) and generate one or more outputs based on the trained model. For example, processor 610 of FIG. 7 may generate one or more outputs based on the trained model and the one or more inputs received.

  Modifications, additions, or omissions may be made to method 500 without departing from the scope of the present disclosure. For example, the acts of method 500 may be performed in a different order. Furthermore, the operations and steps described are merely provided as examples, and some of the operations and steps may be optional without compromising the essence of the disclosed embodiments. And may be combined into fewer operations and steps, and may be extended to additional operations and steps.

  FIG. 7 is a block diagram of an exemplary computing device 600 in accordance with at least one embodiment of the present disclosure. The computing device 600 may be a desktop computer, laptop computer, server computer, tablet computer, mobile phone, smart phone, personal digital assistant (PDA), electronic reader device, network switch, network router, network hub, other networking device, or Other suitable computing devices may be included.

  Computing device 600 may include processor 610, storage device 620, memory 630, and communication device 640. Processor 610, storage device 620, memory 630, and / or communication device 640 may all be communicatively coupled such that each of these components may communicate with other components. The computing device 600 may perform any of the operations described in this disclosure.

  In general, processor 610 may include any suitable special purpose or general purpose computer, computing entity, or processing device including various computer hardware or software modules, and may be stored in any applicable computer readable storage medium May be configured to execute the instruction being For example, processor 610 may interpret and / or execute a microprocessor, microcontroller, digital signal processor (DSP), application specific integrated circuit (ASIC), field programmable gate array (FPGA), or program instruction and / or Or any other digital or analog circuit configured to process data. Although processor 610 is illustrated as one processor in FIG. 7, processor 610 may be any number of processors configured to perform any number of operations described in this disclosure, either individually or collectively. May be included.

  In some embodiments, processor 610 interprets and / or executes program instructions stored in storage device 620, memory 630, or both storage device 620 and memory 630 and / or storage device 620. , Data stored in memory 630, or both storage device 620 and memory 630 may be processed. In some embodiments, processor 610 may fetch program instructions from storage device 620 and load program instructions into memory 620. After program instructions are loaded into memory 630, processor 610 can execute the program instructions.

  For example, in some embodiments, one or more of the processing operations of generating and / or training a neural network may be included in storage device 620 as program instructions. Processor 610 fetches program instructions of one or more of the processing operations and loads program instructions of one or more of the processing operations into memory 630. Can. After program instructions of one or more of the processing operations are loaded into memory 630, processor 610 may compute operations associated with the processing operations as directed by the program instructions. Program instructions of one or more of such processing operations may be executed, as device 600 may implement.

  Storage device 620 and memory 630 may include computer readable storage media for carrying or storing computer executable instructions or data structures. Such computer readable storage media may include any available media that can be accessed by a general purpose or special purpose computer such as processor 610. By way of example and not limitation, such computer readable storage medium may be RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage device, flash memory device (eg solid state memory) Device) or any other storage medium that can be used to carry or store desired program code in the form of computer executable instructions or data structures, which can be accessed by a general purpose or special purpose computer And other storage media, including tangible or non-transitory computer readable storage media. Combinations of the above may also be included within the scope of computer readable storage media. Computer-executable instructions may include, for example, instructions and data configured to cause processor 610 to perform a certain operation or group of operations.

  In some embodiments, storage device 620 and / or memory 630 stores data associated with generating a neural network, and more particularly, data associated with quantization and / or training of neural network. be able to. For example, storage device 620 and / or memory 630 may store values for the spacing between quantization outputs (eg, spacing d), neural network precision measurements, neural networks (eg, models), etc.

  Communication device 640 may include any device, system, component, or collection of components configured to enable or facilitate communication between computing device 600 and another electronic device. For example, communication device 640 may be a modem, a network card (wireless or wired), an infrared communication device, an optical communication device, a wireless communication device (such as an antenna), and / or a chipset (a Bluetooth® device, an 802.6 device (such as It may include, but is not limited to, Metropolitan Area Network (MAN), Wi-Fi® devices, WiMAX® devices, cellular communication facilities, etc.) and / or the like. The communication device 640 may include and / or remote devices from any network, such as a cellular network, Wi-Fi network, MAN, optical network, etc., to name but a few. Data may be allowed to be exchanged with any other device described in.

  Modifications, additions, or omissions may be made to FIG. 7 without departing from the scope of the present disclosure. For example, computing device 600 may include more or fewer elements than those illustrated and described in this disclosure. For example, computing device 600 may include an integrated display device such as a tablet or a screen of a mobile phone, or an external monitor, projector, which may be separate from computing device 600 and communicatively connected to computing device 600. It may include a television or other suitable display device.

  As used in this disclosure, the terms "module" or "component" refer to a particular hardware implementation and / or general purpose hardware of a computing system that is configured to perform an action of the module or component For example, it may refer to a software object or software routine that may be stored on a computer readable medium or the like and / or may be executed by general purpose hardware (eg, a processing device or the like) of a computing system. In some embodiments, the different components, modules, engines, and services described in the present disclosure may be implemented as objects or processes that execute on a computing system (eg, as separate threads). Although some of the systems and methods described in this disclosure are generally described as being implemented by software (stored in general purpose hardware and / or executed by general purpose hardware), certain hardware may Combinations of implementations or software with specific hardware implementations are also possible and contemplated. In the present disclosure, a “computing entity” may be any computing system previously defined in the present disclosure, or any combination of modules or modules operating on the computing system.

  The terms used in the present disclosure and particularly in the claims (e.g. the body portion of the claims) are generally intended as being "open" terms (e.g. the term "comprising" comprises " Should be construed as not being limited to, the term "having" should be interpreted as "having at least", and the term "including" is , "Including but not limited to", etc.).

  Further, where a specific number of claiming items introduced is intended, such intent is explicitly stated in the claim, and where such a description is not present, such intent exists do not do. For example, as an aid to understanding, the claims may include the use of introductory phrases such as "at least one" and "one or more" to introduce claim language. However, the use of such an introductory phrase means that the introduction of claim contents by indefinite articles such as "a" or "an" is such that "one or more" or "at least one" in the same claim. Even if a phrase and an indefinite article such as "a" or "an" are included, the specific claim including the item recited in the introduced claim is limited to the embodiment including only the item recited in the claim Should not be interpreted as implying that (eg, “a” and / or “an” should be interpreted to mean “at least one” or “one or more”) . The same applies to the use of definite articles used to introduce claim recitations.

  Furthermore, even if a specific number of claiming items introduced is explicitly stated, such a description should at least be interpreted as meaning the stated number ( For example, one skilled in the art will appreciate that the mere mention of "two entries" without other modifiers means at least two entries or more than one entry). Further, where a notation similar to "at least one of A, B, and C, etc." or "one or more of A, B, C, etc." is used, generally, such a structure It is intended to include only A, only B, only C, both A and B, both A and C, both B and C, or all of A, B and C, and the like.

  Furthermore, any disjunction or disjunction phrase denoting two or more selectable terms, whether in the specification, claims, or drawings, one of those terms, that term It should be understood as intended the possibility of including combinations or all of those terms. For example, the phrase "A or B" should be understood as including the possibilities of "A" or "B" or "A and B."

  All examples and conditional language described in the present disclosure are intended for the concepts contributed by the inventor to promote the art and for educational purposes to help the reader understand the present invention. It should be construed as not being limited to such specifically described examples and conditions. Although the embodiments of the present disclosure have been described in detail, various changes, substitutions, and alterations to the embodiments are possible without departing from the spirit and scope of the present disclosure.

  Further, the following appendices will be disclosed regarding the above embodiment.

(Supplementary Note 1)
A method of operating a neural network,
Applying non-linear quantization to the plurality of synaptic weights of the neural network model by at least one processor;
Training the neural network model with the at least one processor;
Generating a neural network output from the trained neural network model based on the one or more inputs received by the trained neural network model by the at least one processor;
Method including.

(Supplementary Note 2)
Further comprising the step of determining the accuracy of said neural network model,
The method according to clause 1, wherein said generating a neural network output comprises generating said neural network output in response to the determined accuracy being greater than an accuracy threshold.

(Supplementary Note 3)
The method according to clause 1, wherein the applying non-linear quantization comprises applying a non-linear function to each synaptic weight of the plurality of synaptic weights of the neural network model.

(Supplementary Note 4)
The method according to clause 3, wherein the applying a non-linear function comprises applying one of a hyperbolic sine function and a hyperbolic cosine function to each synaptic weight of the plurality of synaptic weights.

(Supplementary Note 5)
Determining the accuracy of the neural network model;
Increasing the resolution by increasing the number of bits of the non-linear quantization in response to the determined accuracy being less than an accuracy threshold;
The method according to appendix 1, further comprising

(Supplementary Note 6)
Said application of nonlinear quantization is
Coarse resolution quantizing at least one synapse weight of the plurality of synapse weights having a first value;
Quantizing at least one other synapse weight of the plurality of synapse weights having a second value greater than the first value with fine resolution;
The method according to appendix 1, comprising

(Appendix 7)
The quantization with coarse resolution of the at least one synapse weight of the plurality of synapse weights having a first value may include the at least one synapse weight of the plurality of synapse weights as a number of bits. 7. The method of clause 6, comprising quantizing with N.

(Supplementary Note 8)
The quantization with fine resolution of the at least one other synapse weight of the plurality of synapse weights having a second value larger than the first value may be one of the plurality of synapse weights. The method according to clause 7, comprising quantizing the at least one other synapse weight with a bit number M, where M is greater than N.

(Appendix 9)
The method according to clause 1, wherein said applying non-linear quantization comprises applying a non-linear function to said plurality of synaptic weights of said neural network model so as to result in non-uniformly distributed quantization levels. .

(Supplementary Note 10)
The method according to clause 1, wherein the applying non-linear quantization comprises applying a non-linear function to the plurality of synaptic weights of the neural network model so as to provide uniformly distributed quantization levels.

(Supplementary Note 11)
One or more non-transitory computer readable media comprising instructions, wherein the instructions cause the one or more processors to perform a plurality of operations when executed by the one or more processors. The plurality of actions being configured
An operation of applying nonlinear quantization to a plurality of synapse weights of a neural network model;
An operation of training the neural network model;
Generating a neural network output from the trained neural network model based on the one or more inputs received by the trained neural network model;
And computer readable media.

(Supplementary Note 12)
The plurality of actions are
Further including an action of determining the accuracy of the neural network model,
Clause 12. The computer readable medium according to clause 11, wherein the generating a neural network output comprises generating the neural network output in response to the determined accuracy being greater than an accuracy threshold.

(Supplementary Note 13)
Clause 12. The computer readable medium according to clause 11, wherein the applying non-linear quantization comprises applying a non-linear function to each synaptic weight of the plurality of synaptic weights of the neural network model.

(Supplementary Note 14)
Clause 13. The computer readable medium according to clause 13, wherein said applying a non-linear function comprises applying one of a hyperbolic sine function and a hyperbolic cosine function to each synaptic weight of said plurality of synaptic weights Medium.

(Supplementary Note 15)
The plurality of actions are
An operation to determine the accuracy of the neural network model;
An operation of increasing resolution by increasing the number of bits of the non-linear quantization in response to the determined accuracy being smaller than an accuracy threshold;
The computer readable medium of clause 11, further comprising:

(Supplementary Note 16)
Said application of nonlinear quantization is
Coarse resolution quantizing at least one synapse weight of the plurality of synapse weights having a first value;
Quantizing at least one other synapse weight of the plurality of synapse weights having a second value greater than the first value with fine resolution;
Clause 12. The computer readable medium according to clause 11, comprising

(Supplementary Note 17)
The quantization with coarse resolution of the at least one synapse weight of the plurality of synapse weights having a first value may include the at least one synapse weight of the plurality of synapse weights as a number of bits. Clause 16: The computer readable medium according to clause 16, comprising quantizing with N.

(Appendix 18)
The quantization with fine resolution of the at least one other synapse weight of the plurality of synapse weights having a second value larger than the first value may be one of the plurality of synapse weights. 24. The computer readable medium according to clause 17, comprising quantizing the at least one other synapse weight with a bit number M, where M is greater than N.

(Appendix 19)
Clause 12. The computer of clause 11, wherein the applying non-linear quantization comprises applying a non-linear function to the plurality of synaptic weights of the neural network model to provide non-uniformly distributed quantization levels. Readable media.

(Supplementary Note 20)
The computer read of claim 11, wherein the applying non-linear quantization comprises applying a non-linear function to the plurality of synaptic weights of the neural network model to provide uniformly distributed quantization levels. Possible media.

600 computing device 610 processor 620 storage device 630 memory 640 communication device

Claims (20)

A method of operating a neural network,
Applying non-linear quantization to the plurality of synaptic weights of the neural network model by at least one processor;
Training the neural network model with the at least one processor;
Generating a neural network output from the trained neural network model based on the one or more inputs received by the trained neural network model by the at least one processor;
Method including. Further comprising the step of determining the accuracy of said neural network model,
The method of claim 1, wherein the generating a neural network output comprises generating the neural network output in response to the determined accuracy being greater than an accuracy threshold.   The method of claim 1, wherein the applying non-linear quantization comprises applying a non-linear function to each synaptic weight of the plurality of synaptic weights of the neural network model.   4. The method of claim 3, wherein the applying a non-linear function comprises applying one of a hyperbolic sine function and a hyperbolic cosine function to each synaptic weight of the plurality of synaptic weights. Determining the accuracy of the neural network model;
Increasing the resolution by increasing the number of bits of the non-linear quantization in response to the determined accuracy being less than an accuracy threshold;
The method of claim 1, further comprising Said application of nonlinear quantization is
Coarse resolution quantizing at least one synapse weight of the plurality of synapse weights having a first value;
Quantizing at least one other synapse weight of the plurality of synapse weights having a second value greater than the first value with fine resolution;
The method of claim 1, comprising:   The quantization with coarse resolution of the at least one synapse weight of the plurality of synapse weights having a first value may include the at least one synapse weight of the plurality of synapse weights as a number of bits. 7. The method of claim 6, comprising quantizing with N.   The quantization with fine resolution of the at least one other synapse weight of the plurality of synapse weights having a second value larger than the first value may be one of the plurality of synapse weights. The method of claim 7, comprising quantizing the at least one other synapse weight with a number of bits M, where M is greater than N.   2. The method of claim 1, wherein the applying non-linear quantization comprises applying a non-linear function to the plurality of synaptic weights of the neural network model to provide non-uniformly distributed quantization levels. Method.   The method of claim 1, wherein the applying non-linear quantization comprises applying a non-linear function to the plurality of synaptic weights of the neural network model to provide uniformly distributed quantization levels. . One or more non-transitory computer readable media comprising instructions, wherein the instructions cause the one or more processors to perform a plurality of operations when executed by the one or more processors. The plurality of actions being configured
An operation of applying nonlinear quantization to a plurality of synapse weights of a neural network model;
An operation of training the neural network model;
Generating a neural network output from the trained neural network model based on the one or more inputs received by the trained neural network model;
And computer readable media. The plurality of actions are
Further including an action of determining the accuracy of the neural network model,
The computer readable medium of claim 11, wherein the generating the neural network output comprises generating the neural network output in response to the determined accuracy being greater than an accuracy threshold.   The computer readable medium of claim 11, wherein the applying non-linear quantization comprises applying a non-linear function to each synaptic weight of the plurality of synaptic weights of the neural network model.   14. The computer readable medium of claim 13, wherein the applying a non-linear function comprises applying one of a hyperbolic sine function and a hyperbolic cosine function to each synaptic weight of the plurality of synaptic weights. Medium. The plurality of actions are
An operation to determine the accuracy of the neural network model;
An operation of increasing resolution by increasing the number of bits of the non-linear quantization in response to the determined accuracy being smaller than an accuracy threshold;
The computer readable medium of claim 11, further comprising: Said application of nonlinear quantization is
Coarse resolution quantizing at least one synapse weight of the plurality of synapse weights having a first value;
Quantizing at least one other synapse weight of the plurality of synapse weights having a second value greater than the first value with fine resolution;
A computer readable medium according to claim 11, comprising   The quantization with coarse resolution of the at least one synapse weight of the plurality of synapse weights having a first value may include the at least one synapse weight of the plurality of synapse weights as a number of bits. The computer readable medium according to claim 16, comprising quantizing with N.   The quantization with fine resolution of the at least one other synapse weight of the plurality of synapse weights having a second value larger than the first value may be one of the plurality of synapse weights. 18. The computer readable medium of claim 17, comprising quantizing the at least one other synapse weight with a number of bits M, where M is greater than N.   12. The method of claim 11, wherein the applying non-linear quantization comprises applying a non-linear function to the plurality of synaptic weights of the neural network model to provide non-uniformly distributed quantization levels. Computer readable medium.   12. The computer of claim 11, wherein the applying non-linear quantization comprises applying a non-linear function to the plurality of synaptic weights of the neural network model to provide uniformly distributed quantization levels. Readable media.

Images