CN108256642B - Circuit design method for non-precise calculation neural network - Google Patents
Circuit design method for non-precise calculation neural network Download PDFInfo
- Publication number
- CN108256642B CN108256642B CN201611242077.5A CN201611242077A CN108256642B CN 108256642 B CN108256642 B CN 108256642B CN 201611242077 A CN201611242077 A CN 201611242077A CN 108256642 B CN108256642 B CN 108256642B
- Authority
- CN
- China
- Prior art keywords
- precise
- multiplier
- bit width
- neural network
- design method
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
- G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F7/00—Methods or arrangements for processing data by operating upon the order or content of the data handled
- G06F7/38—Methods or arrangements for performing computations using exclusively denominational number representation, e.g. using binary, ternary, decimal representation
- G06F7/48—Methods or arrangements for performing computations using exclusively denominational number representation, e.g. using binary, ternary, decimal representation using non-contact-making devices, e.g. tube, solid state device; using unspecified devices
- G06F7/52—Multiplying; Dividing
- G06F7/523—Multiplying only
Abstract
The invention provides a circuit design method for a non-precise calculation neural network, which comprises the following steps: determining bit width for determining the bit width selection range of the imprecise multiplier; generating a circuit structure, and generating a plurality of specific circuits for each alternative bit width through a heuristic random algorithm which takes node importance ranking in the circuit as drive, wherein the specific circuits are all selectable non-precise multiplier structures; independent replacement, each precise multiplier in the neural network is independently replaced by a non-precise multiplier with a certain structure, and all obtained possible schemes form a solution space; and traversing and comparing, namely randomly selecting a part of schemes in the solution space, and traversing the training and testing effects of the non-precise neural network under each scheme to obtain the optimal solution. Under the condition that the accuracy of the operation result is not influenced greatly, the area and the power consumption are greatly reduced, and the operation speed is improved.
Description
Technical Field
The invention relates to the technical field of artificial neural networks, in particular to a circuit design method for a non-precise calculation neural network.
Background
The artificial neural network is a new popular algorithm in the field of machine learning, and achieves good achievement in numerous subdivision fields such as face recognition, target detection and the like. However, the method has great dependence on hardware performance, and often requires a CPU and a GPU at a server level to complete training and recognition, so that the method is extremely high in cost in volume and power consumption. Today, as smart devices are increasingly miniaturized, personal smart devices have to be always operated in a networked mode due to the characteristics of the artificial neural network, and further speed increase of the personal smart devices is hindered. If the accelerator chip is designed specially for the artificial neural network, the area and power consumption of the accelerator chip need to be very compact and saved.
The non-precise calculation is a widely applied method in the field of signal processing and scientific calculation, and the basic idea is to moderately simplify the calculation process when the requirement on the accuracy or precision of the calculation result is not high, so that the running time, the power consumption and other more intentional resources are saved. The artificial neural network naturally and very accords with the application condition of non-accurate calculation, and because a large amount of intermediate data with unknown individual significance exist in the operation process, the final result cannot be obviously influenced even if small errors exist. On the other hand, the repeated training process also makes it gradually adaptive to the calculation error. Therefore, the adoption of the non-precise operation unit in the execution of the artificial neural network is a very suitable means when the speed needs to be improved and the power consumption needs to be reduced.
However, the non-precise computing units are not of a specific structure, and it is difficult to determine which non-precise structure each of tens of thousands of computing units in the entire neural network can achieve the best effect. The specific application of the inaccurate calculation idea in the neural network operation also faces difficulties.
Disclosure of Invention
Technical problem to be solved
In view of the above technical problems, the present invention provides a circuit design method for a non-precision computation neural network.
(II) technical scheme
According to one aspect of the present invention, there is provided a circuit design method for a non-precision computational neural network, comprising the steps of: determining bit width for determining the bit width selection range of the imprecise multiplier; generating a circuit structure, and generating a plurality of specific circuits for each alternative bit width, wherein the specific circuits are all selectable non-precise multiplier structures; independent replacement, each precise multiplier in the neural network is independently replaced by a non-precise multiplier with a certain structure, and all obtained possible schemes form a solution space; and traversing and comparing, namely randomly selecting a part of schemes in the solution space, and traversing the training and testing effects of the non-precise neural network under each scheme to obtain the optimal solution.
In some embodiments of the present invention, the specific steps for determining the bit width are:
determining an upper bound; determining a lower bound; the step size is determined.
In some embodiments of the present invention, in the step of determining the upper bound, the bit width of the precision multiplier is taken as the upper bound; in the step of determining the lower bound, half of the bit width of the precision multiplier is taken as the lower bound.
In some embodiments of the invention, in the step of determining the step size, the step size is set to 2.
In some embodiments of the invention, in the step of generating the circuit structure, a heuristic stochastic algorithm driven by the importance ranking of nodes in the circuit is used to generate the circuit structure.
In some embodiments of the present invention, the heuristic stochastic algorithm specifically includes:
giving out a Carnot graph and a logic gate circuit of the precise multiplier under the current bit width;
calculating the hardware overhead of the logic gate circuit;
entering a traversal loop: turning any number of bits in the carnot diagram to obtain a new carnot diagram, calculating the result error and hardware overhead of a new logic gate circuit, and if the result error is within the tolerance and the hardware overhead is the current minimum, taking the current carnot diagram, the logic gate circuit, the result error and the hardware overhead as the current optimal solution;
and after traversing, obtaining a new circuit structure with the minimum hardware overhead within the tolerance of the error, namely a non-precise multiplier under the current bit width and the error tolerance.
In some embodiments of the present invention, the specific step of generating the circuit structure further comprises:
for each bit width, a plurality of non-precision multipliers at different error tolerances are generated.
In some embodiments of the present invention, the specific steps of the independent replacement are:
independently replacing each precise multiplier in the neural network with any non-precise multiplier to obtain a solution space;
a portion of the solution is filtered from the solution space.
In some embodiments of the invention, in the step of screening a portion of the solutions from the solution space,
randomly selecting; alternatively, the first and second electrodes may be,
the bit width is uniformly distributed according to the bit width, and is randomly selected according to the proportion of normal distribution of the error tolerance.
In some embodiments of the present invention, the specific steps of the traversal comparison are:
selecting an alternative scheme in a part of the screened schemes;
training and testing the non-precise neural network corresponding to the alternative scheme, if the test result is the best current solution, keeping the alternative scheme as the best current solution, otherwise, selecting another alternative scheme for training and testing until all the schemes are traversed and screened;
and the optimal solution after the traversal is finished is used as a final result.
(III) advantageous effects
According to the technical scheme, the circuit design method for the non-precise calculation neural network has at least one of the following beneficial effects:
(1) according to the neural network circuit designed and obtained by the invention, under the condition that the accuracy of the operation result is not influenced greatly, the area and the power consumption are greatly reduced, and the operation speed is improved;
(2) the invention compares various different schemes, and finally determines the adoption result according to the actual training and testing effect, thereby ensuring the design purpose as much as possible.
Drawings
FIG. 1 is a flow chart of steps of a circuit design method for a non-precision computational neural network according to an embodiment of the present invention;
FIG. 2 is a flowchart of a determine bit width step in a circuit design method for a non-precision computational neural network according to an embodiment of the present invention;
FIG. 3 is a flow chart of the steps of generating a circuit structure in a circuit design method for a non-precision computational neural network according to an embodiment of the present invention;
FIG. 4 is a flow chart of an independent alternative step in a circuit design method for a non-precision computational neural network according to an embodiment of the present invention;
FIG. 5 is a flowchart illustrating traversal comparison steps in a circuit design method for a non-precision computational neural network, according to an embodiment of the present invention.
Detailed Description
In order that the objects, technical solutions and advantages of the present invention will become more apparent, the present invention will be further described in detail with reference to the accompanying drawings in conjunction with the following specific embodiments. Other aspects, advantages, and salient features of the invention will become apparent to those skilled in the art from the following detailed description.
In this specification, the various embodiments described below which are meant to illustrate the principles of this invention are illustrative only and should not be construed in any way to limit the scope of the invention. The following description with reference to the accompanying drawings is provided to assist in a comprehensive understanding of exemplary embodiments of the invention as defined by the claims and their equivalents. The following description includes various specific details to aid understanding, but such details are to be regarded as illustrative only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Moreover, descriptions of well-known functions and constructions are omitted for clarity and conciseness.
In an embodiment of the present invention, a circuit design method for a non-precision computational neural network is provided. FIG. 1 is a flow chart of steps of a circuit design method for a non-precision computational neural network according to an embodiment of the present invention. As shown in fig. 1, the circuit design method for a non-precise computation neural network of the present invention includes the steps of: determining bit width; generating a circuit structure; independent replacement; and traversing and comparing.
In the step of determining the bit width, a bit width selection range of the non-precision multiplier is determined. FIG. 2 shows a flowchart of determining bit width steps in a circuit design method for a non-precision computational neural network, according to an embodiment of the present invention. The specific process for determining the bit width is as follows: determining an upper bound; determining a lower bound; the step size is determined. The step of determining the upper bound usually takes the bit width of the precise multiplier as the upper bound, and the upper bound is not included in the set under the condition that comparison is not needed; the determining a lower bound step would have as a lower bound one-half of the bit width of the precision multiplier and is contained within the set; the determine step tends to set the step size to 2 because firstly the bit width is typically even and secondly the granularity needs to be as fine as possible in order to try as many different choices as possible.
In the step of generating the circuit structure, a plurality of specific circuits are generated for each alternative bit width through a heuristic random algorithm driven by node importance ranking in the circuits, and the circuits are all selectable non-precise multiplier structures. FIG. 3 is a flow chart showing the steps of generating circuit structures in the circuit design method for the non-precision computing neural network according to the embodiment of the invention, which is essentially a heuristic random algorithm driven by the node importance ranking in the circuit. The specific process is as follows: firstly, a Carnot graph and a logic gate circuit of an accurate multiplier under the current bit width are given; then, calculating the hardware overhead of the logic gate circuit; and then, entering a traversal loop: turning any number of bits in the carnot diagram to obtain a new carnot diagram, calculating the result error and hardware overhead of a new logic gate circuit, and if the result error is within the tolerance and the hardware overhead is the current minimum, taking the current carnot diagram, the logic gate circuit, the result error and the hardware overhead as the current optimal solution; and finally, after traversing is completed, the new circuit structure with the minimum hardware overhead within the tolerance of the error is the non-precise multiplier under the current bit width and the error tolerance.
The heuristic random algorithm obtains a non-precise multiplier under a certain bit width and a certain error tolerance. There may be several different error tolerances for each bit width. Taking a 32-bit precise multiplier as an example, all possible bit widths are 8 types including 30 bits, 28 bits, 26 bits, 24 bits, 22 bits, 20 bits, 18 bits and 16 bits, and if 5 non-precise multipliers with different error tolerance degrees are designed for each bit width, the total is 8 × 5 to 40 types.
In the independent replacement step, each precise multiplier in the neural network is independently replaced by a non-precise multiplier with a certain structure, and all the obtained possible schemes form a solution space. FIG. 4 shows a flow diagram of individual alternative steps in a circuit design method for a non-precision computational neural network, according to an embodiment of the present invention. The specific process is as follows: firstly, independently replacing each precise multiplier in a neural network with any non-precise multiplier to obtain a solution space; then, a portion is randomly filtered in a solution space according to a certain algorithm.
The solution space is large because a neural network has many weighted summation operations, each synapse needs a multiplier, the number of which is used as an exponent, the base number is the number of the non-exact multiplier structure types obtained in the first two steps, and the size of the solution space is the power. For the example of the 40 non-exact multipliers, assuming 50000 multipliers in a neural network, the size of the solution space is 40^ 50000. Traversing such a large solution space is almost impossible and therefore requires choosing a part from it. The part can be selected randomly, or selected randomly according to bit width average distribution and the proportion of normal distribution according to error tolerance.
In the step of traversing comparison, firstly, a part of schemes are randomly selected in the solution space, and then the training and testing effects of the non-precise neural network under each scheme are traversed to obtain the optimal solution. FIG. 5 shows a flow chart of traversal comparison steps in a circuit design method for a non-precision computational neural network, according to an embodiment of the present invention. The specific process is as follows: firstly, selecting a replacement scheme from the small solution space screened in the last step, then training the non-precise neural network corresponding to the replacement scheme for a certain number of times, and then testing according to a certain standard. If the test result is the best current solution, the alternative scheme is reserved as the optimal current solution, otherwise, another alternative scheme is selected for training and testing until a smaller solution space is traversed. And the optimal solution after the traversal is finished is used as the final result of the method.
The circuit design method for the non-precise calculation neural network compares a plurality of different schemes, finally determines the adoption result according to the actual training and testing effect, and ensures the design purpose to the greatest extent. The obtained design scheme greatly reduces the area and the power consumption and improves the operation speed under the condition that the accuracy of the operation result is not greatly influenced.
The processes or methods depicted in the preceding figures may be performed by processing logic that comprises hardware (e.g., circuitry, dedicated logic, etc.), firmware, software (e.g., software carried on a non-transitory computer readable medium), or a combination of both. Although the processes or methods are described above in terms of some sequential operations, it should be understood that some of the operations described may be performed in a different order. Further, some operations may be performed in parallel rather than sequentially.
It is to be noted that, in the attached drawings or in the description, the implementation modes not shown or described are all the modes known by the ordinary skilled person in the field of technology, and are not described in detail. Further, the above definitions of the various elements and methods are not limited to the various specific structures, shapes or arrangements of parts mentioned in the examples, which may be easily modified or substituted by those of ordinary skill in the art.
The algorithms and displays presented herein are not inherently related to any particular computer, virtual machine, or other apparatus. Various general purpose systems may also be used with the teachings herein. The required structure for constructing such a system will be apparent from the description above. Moreover, the present invention is not directed to any particular programming language. It is appreciated that a variety of programming languages may be used to implement the teachings of the present invention as described herein, and any descriptions of specific languages are provided above to disclose the best mode of the invention.
In the description provided herein, numerous specific details are set forth. It is understood, however, that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.
Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, the disclosed method should not be interpreted as reflecting an intention that: that the invention as claimed requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.
The above-mentioned embodiments are intended to illustrate the objects, technical solutions and advantages of the present invention in further detail, and it should be understood that the above-mentioned embodiments are only exemplary embodiments of the present invention, and are not intended to limit the present invention, and any modifications, equivalents, improvements and the like made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (8)
1. A circuit design method for a non-precision computational neural network, comprising the steps of:
determining bit width for determining the bit width selection range of the imprecise multiplier;
generating a circuit structure, namely generating the circuit structure by adopting a heuristic random algorithm driven by node importance ranking in the circuit, and generating a plurality of specific circuits for each alternative bit width, wherein the specific circuits are all selectable non-precise multiplier structures; the alternative bit width is a bit width in the bit width selection range;
independent replacement, each precise multiplier in the neural network is independently replaced by a non-precise multiplier with a certain structure, and all obtained possible schemes form a solution space;
traversing and comparing, namely randomly selecting a part of schemes in the solution space, and traversing the training and testing effects of the non-precise neural network under each scheme to obtain an optimal solution; the non-precise neural network is obtained by independently replacing each precise multiplier in the neural network with a non-precise multiplier with a certain structure;
the heuristic random algorithm comprises the following specific steps:
giving out a Carnot graph and a logic gate circuit of the precise multiplier under the current bit width;
calculating the hardware overhead of the logic gate circuit;
entering a traversal loop: turning any number of bits in the carnot diagram to obtain a new carnot diagram, calculating the result error and hardware overhead of a new logic gate circuit, and if the result error is within the tolerance and the hardware overhead is the current minimum, taking the current carnot diagram, the logic gate circuit, the result error and the hardware overhead as the current optimal solution; wherein the new logic gate circuit is derived from the new carnot diagram;
and after traversing, obtaining a new circuit structure with the minimum hardware overhead within the tolerance of the error, namely a non-precise multiplier under the current bit width and the error tolerance.
2. The circuit design method according to claim 1, wherein the specific step of determining the bit width is:
determining an upper bound; determining a lower bound; the step size is determined.
3. The circuit design method of claim 2,
in the step of determining the upper bound, the bit width of the precision multiplier is taken as the upper bound;
in the step of determining the lower bound, half of the bit width of the precision multiplier is taken as the lower bound.
4. The circuit design method of claim 3,
in the step of determining the step size, the step size is set to 2.
5. The circuit design method of claim 1, wherein the step of generating the circuit structure further comprises:
for each bit width, a plurality of non-precision multipliers at different error tolerances are generated.
6. The circuit design method according to claim 1, wherein the specific steps of independent replacement are:
independently replacing each precise multiplier in the neural network with any non-precise multiplier to obtain a solution space;
a portion of the solution is filtered from the solution space.
7. The circuit design method of claim 6, wherein in the step of screening a portion of the solutions from the solution space,
randomly selecting; alternatively, the first and second electrodes may be,
the bit width is uniformly distributed according to the bit width, and is randomly selected according to the proportion of normal distribution of the error tolerance.
8. The circuit design method of claim 1, wherein the step of performing the traverse comparison comprises:
selecting an alternative scheme in a part of the screened schemes;
training and testing the non-precise neural network corresponding to the alternative scheme, if the test result is the best current solution, keeping the alternative scheme as the best current solution, otherwise, selecting another alternative scheme for training and testing until all the schemes are traversed and screened;
and the optimal solution after the traversal is finished is used as a final result.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611242077.5A CN108256642B (en) | 2016-12-29 | 2016-12-29 | Circuit design method for non-precise calculation neural network |
| PCT/CN2017/111872 WO2018121125A1 (en) | 2016-12-29 | 2017-11-20 | Method and device for circuit design, and computer-readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611242077.5A CN108256642B (en) | 2016-12-29 | 2016-12-29 | Circuit design method for non-precise calculation neural network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108256642A CN108256642A (en) | 2018-07-06 |
| CN108256642B true CN108256642B (en) | 2021-08-31 |
Family
ID=62707879
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611242077.5A Active CN108256642B (en) | 2016-12-29 | 2016-12-29 | Circuit design method for non-precise calculation neural network |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN108256642B (en) |
| WO (1) | WO2018121125A1 (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110728365B (en) * | 2019-09-12 | 2022-04-01 | 东南大学 | Method for selecting calculation bit width of multi-bit-width PE array and calculation precision control circuit |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6385268B1 (en) * | 1994-07-22 | 2002-05-07 | Aether-Wire & Technology | Spread spectrum localizers |
| CN105184365A (en) * | 2015-07-02 | 2015-12-23 | 清华大学 | Digital-analog mixed signal processing system for imprecise computation |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105426156B (en) * | 2015-10-28 | 2018-07-20 | 南京航空航天大学 | A kind of non-precision multiplier of high-performance and its application process |
-
2016
- 2016-12-29 CN CN201611242077.5A patent/CN108256642B/en active Active
-
2017
- 2017-11-20 WO PCT/CN2017/111872 patent/WO2018121125A1/en active Application Filing
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6385268B1 (en) * | 1994-07-22 | 2002-05-07 | Aether-Wire & Technology | Spread spectrum localizers |
| CN105184365A (en) * | 2015-07-02 | 2015-12-23 | 清华大学 | Digital-analog mixed signal processing system for imprecise computation |
Non-Patent Citations (1)
| Title |
|---|
| Design of Power-Efficient Approximate Multipliers for Approximate Artificial Neural Networks;Vojtech Mrazek等;《ICCAD》;20161110;第4.1节第1段；第4.2节第1段；第4.2.2节第1段；第3.4节第1段；第2.3节第3段；第4节第1段； * |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2018121125A1 (en) | 2018-07-05 |
| CN108256642A (en) | 2018-07-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3446260B1 (en) | Memory-efficient backpropagation through time | |
| EP3792840A1 (en) | Neural network method and apparatus | |
| CN109034365A (en) | The training method and device of deep learning model | |
| CN111587440A (en) | Neuromorphic chip for updating accurate synaptic weight values | |
| US20230196202A1 (en) | System and method for automatic building of learning machines using learning machines | |
| CN109586954B (en) | Network traffic prediction method and device and electronic equipment | |
| CN108805174A (en) | clustering method and device | |
| CN105718848A (en) | Quality evaluation method and apparatus of fingerprint images | |
| CN110909868A (en) | Node representation method and device based on graph neural network model | |
| CN111914994A (en) | Method and device for generating multilayer perceptron, electronic equipment and storage medium | |
| CN114781650B (en) | Data processing method, device, equipment and storage medium | |
| CN112420125A (en) | Molecular attribute prediction method and device, intelligent equipment and terminal | |
| CN107480771B (en) | Deep learning-based activation function realization method and device | |
| CN110019845B (en) | Community evolution analysis method and device based on knowledge graph | |
| CN108256642B (en) | Circuit design method for non-precise calculation neural network | |
| US11036980B2 (en) | Information processing method and information processing system | |
| US11461665B2 (en) | Systems and methods of a Boolean network development environment | |
| CN115862653A (en) | Audio denoising method and device, computer equipment and storage medium | |
| JP6233432B2 (en) | Method and apparatus for selecting mixed model | |
| JP6172315B2 (en) | Method and apparatus for mixed model selection | |
| CN114781619A (en) | Soft error detection method and device | |
| KR20210086233A (en) | Method and apparatus for processing matrix data through relaxed pruning | |
| EP3518153A1 (en) | Information processing method and information processing system | |
| JP2008226232A (en) | System and method for determining electrical characteristics of power distribution network using one-dimensional model | |
| CN114418063B (en) | Method and device for distributing network layer in neural network model |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |