JP6831990B2 - Hardware implementation method of neural network that does not require a random number generator and neural network that does not require a random number generator - Google Patents

Description

The present invention relates to a hardware implementation method of a neural network that does not require a random number generator used for deep learning, and a neural network that does not require a random number generator.

The restricted Boltzmann machine (see, for example, Non-Patent Document 1) is one of the neural networks used for deep learning, which has been actively studied in the field of AI (artificial intelligence) in recent years.
In training with a restricted Boltzmann machine, training data is input to multiple visible units that make up the visible layer and propagates to the hidden units that make up the hidden layer, and is formed by hidden units in the hidden layer. The coupling load (weight) when the data propagates from each visible unit (hidden unit) to each hidden unit (visible unit) by repeating the reverse learning in which the output data propagates from the hidden layer to the visible unit in the visible layer. The purpose is to match the output data from the hidden layer when the training data is input to the visible layer with the target output data within a certain error range.

Here, when learning a multi-layer neural network composed of a restricted Boltzmann machine on software, iterative learning of forward learning and reverse learning is executed a huge number of times, which requires a large amount of computational resources. There is a problem that a long calculation time is required and the power consumption is also increased.
Therefore, in order to enable learning of a multi-layer neural network by a restricted Boltzmann machine at higher speed and lower power consumption, research is being conducted on hardware (chip) of the restricted Boltzmann machine. (See, for example, Non-Patent Document 2).

Asja Fischer, Christian Igel, "An Introduction to Restricted Boltzmann Machines", Lecture Notes in Computer Science, Vol.7441, pp.14-36, 2012. Seongwook Park, Kyeongryeol Bong, Dongjoo Shin, Jinmook Lee, Sungpill Choi, Hoi-Jun Yoo, "A 1.93TOPS / W Scalable Deep Learning / Inference Processor with Tetra-Parallel MIMD Architecture for Big-Data Applications", 47th Annual IEEE / ACM International Symposium on Microarchitecture, 2015.

When the restricted Boltzmann machine is made into hardware, the arithmetic circuits for the visible units that execute the processing performed by the visible units are hidden by the number of visible units, and the arithmetic circuits for the hidden units that execute the processing performed by the hidden units are hidden. It is necessary to provide as many units as there are in the chip. Furthermore, in a restricted Boltzmann machine, the ignition or non-ignition state of each unit is randomly determined by the ignition probability of each unit, so it can be used as an arithmetic circuit for visible units and an arithmetic circuit for hidden units. It is necessary to provide a random number generator for each.
Here, when forming a random number generator, a shift register having a long bit length and a plurality of logic gates are required, so the required number of arithmetic circuits for the visible unit and the hidden unit equipped with the random number generator are chipped respectively. If formed inside, a large amount of circuit resources (hardware resources) will be consumed in the chip. For this reason, the scale of the restricted Boltzmann machine that can be formed in the chip is restricted, and a huge amount of hardware resources are required to realize a large-scale multi-layer neural network using the restricted Boltzmann machine. The problem arises that the tip must be prepared.

The present invention has been made in view of such circumstances, and a random number capable of reducing the hardware resources required for hardware implementation of a neural network and realizing hardwareization within a limited hardware resource. It is an object of the present invention to provide a method of hardware implementation of a neural network that does not require a generator and a neural network that does not require a random number generator.

The method of hardware implementation of a neural network that does not require a random number generator according to the first invention according to the above object is a method of hardware implementation of a neural network that includes a processing layer that does not require a random number generator used for deep learning. There,
The state value A indicating the ignition or non-ignition state of each of the plurality of units A forming the arbitrary treatment layer A excluding the most downstream treatment layer A is the treatment layer B on the downstream side adjacent to the treatment layer A. The state value B for each unit B, which is determined by being transmitted to each of the plurality of units B forming the above, is obtained by an operation using a fixed-point binary number, and the firing probability function with the state value B as a variable is used for each unit B. seeking firing probability P (B) and determines and outputs the state of the ignition or non-ignition, provided an arithmetic circuit unit that will be used as a hardware object, to the arithmetic circuit unit in determining said state value B Using a numerical value formed by using bits that exceed the bit width of the fixed-point binary number and being truncated as a random number, the ignition or non-ignition state of the unit B can be determined from the ignition probability P (B) for each unit B. Provide a unit state calculator to determine.

The neural network that does not require a random number generator according to the second invention according to the above object is a neural network that includes a processing layer that does not require a random number generator used for deep learning.
The state value A indicating the ignition or non-ignition state of each of the plurality of units A forming the arbitrary treatment layer A excluding the most downstream treatment layer A is the treatment layer B on the downstream side adjacent to the treatment layer A. The state value B for each unit B, which is determined by being transmitted to a plurality of units B forming the above, is obtained by a fixed-point binary number operation, and the firing probability function with the state value B as a variable is used for each unit B. ignition probability P (B) and to determine the state of the ignition or non-ignition asking output, an arithmetic circuit unit that will be used as a hardware object,
In the arithmetic circuit unit, a numerical value formed by using a bit that exceeds the bit width of the fixed-point binary number and is truncated when obtaining the state value B is used as a random number, and the ignition probability P for each unit B is used. From (B), a unit state calculator for determining the ignition or non-ignition state of the unit B is provided.

In the method of hardware implementation of a neural network that does not require a random number generator according to the first invention and the neural network that does not require a random number generator according to the second invention, a plurality of units forming the processing layer A of the neural network. the state value a indicating the state of a, provided by corresponds to the arithmetic circuit unit in each unit B determines and outputs the state of the plurality of unit B which forms the downstream side of the processing layer B adjacent to the processing layer a Therefore, the ignition or non-ignition states of the plurality of units B can be calculated and determined at the same time and at high speed. As a result, deep learning can be performed in a short time, and reduction in power consumption can be achieved.

Further, since the state value B of the unit B is obtained by the fixed-point binary number calculation, the required circuit resources can be reduced as compared with the case of the conventional floating-point binary number calculation. Further, as a random number required when determining the state of the unit B from the ignition probability P (B) of the unit B, a numerical value formed by using a bit to be truncated when obtaining the state value B of the unit B is used, which is conventionally necessary. The random number generator, which used to be, becomes unnecessary, and circuit resources can be saved.
As a result, in a neural network that does not require a random number generator, it becomes possible to realize a hardware implementation of a multi-layer neural network system that performs deep learning with more resource saving. Furthermore, in a neural network that does not require a random number generator, circuit resources for hardware conversion are reduced as compared with the conventional method, so that it is possible to reduce power consumption during operation.

(A) is an explanatory diagram of a hardware-software composite system to which a method of hardware implementation of a neural network that does not require a random number generator according to an embodiment of the present invention is applied, and (B) is a hardware calculation. It is explanatory drawing of the circuit part. (A) is an explanatory diagram showing the relationship between the visible layer and the hidden layer during forward learning, and (B) is an explanatory diagram showing the processing content of the hidden unit during forward learning. It is explanatory drawing of the method of determining the firing or non-firing state of a hidden unit from the firing probability of the hidden unit using a random number. (A) is an explanatory diagram showing the relationship between the hidden layer and the visible layer during reverse learning, and (B) is an explanatory diagram showing the processing content of the visible unit during reverse learning. It is a flow chart of learning of the restricted Boltzmann machine in Example 1. It is a graph which shows the relationship between the number of epochs and the cross entropy error in Example 1. FIG. It is a graph which shows the relationship between the maximum value of the cross entropy error in Example 1 and the bit width of the decimal part of a fixed-point binary number. It is explanatory drawing of the truncation bit generated by the operation of the fixed-point binary number in Example 2. In the second embodiment, (A) is a histogram of numerical values formed by using all the truncated bits of the decimal part, and (B) is a histogram of the numerical values formed by using all the truncated bits of the integer part. In the second embodiment, (A) is a numerical histogram formed by using the truncated bits of the decimal part created by omitting the data at the time of inputting the training data, and (B) is an integer created by omitting the data at the time of inputting the training data. It is a histogram of the numerical value formed by using the truncation bit of the part. 6 is a graph showing the relationship between the number of epochs and the cross entropy error in Example 3.

Subsequently, an embodiment embodying the present invention will be described with reference to the attached drawings, and the present invention will be understood.
The method of hardware implementation of a neural network that does not require a random number generator according to an embodiment of the present invention is a method of hardware implementation of a neural network that includes a processing layer that does not require a random number generator used for deep learning. The state value A indicating the ignition or non-ignition state of each of the plurality of units A forming the arbitrary treatment layer A excluding the most downstream treatment layer is the downstream treatment layer adjacent to the treatment layer A. The state value B for each unit B determined by being transmitted to a plurality of units B forming B is obtained by a fixed-point binary number calculation, and the firing probability for each unit B is calculated using the firing probability function with the state value B as a variable. An arithmetic circuit unit is provided which obtains P (B), determines a state of ignition or non-ignition, and outputs the output. Further, in the arithmetic circuit unit, a numerical value formed by using a bit that exceeds the bit width of the fixed-point binary number and is truncated when obtaining the state value B of the unit B is used as a random number, and the ignition probability for each unit B is used. A unit state calculator for determining the ignition or non-ignition state of the unit B from P (B) is provided.
Hereinafter, as an example of a neural network having a processing layer that does not require a random number generator, a method of hardware implementation of a restricted Boltzmann machine that does not require a random number generator will be specifically described.

As shown in FIG. 1A, in the method of mounting the hardware of the restricted Boltzmann machine, the hardware module 11 is connected to the main computer 10 on which the software of the restricted Boltzmann machine (hereinafter, also referred to as RBM) is installed. , Among the various processes executed by the RBM software, the software (CPU of the main computer 10) is in charge of the processes that are not suitable for hardware such as complicated condition separation, and for hardware such as bit processing and signal processing. The appropriate processing is assigned to a dedicated arithmetic circuit formed in the hardware module 11, and the hardware is configured as a hardware-software composite system 12.

Here, for the hardware module 11, for example, an FPGA (Field Programmable Gate Array) board is used. As a result, the main computer 10 and the hardware module 11 can be connected to each other via the bus 13 using some kind of interface (for example, a PCI interface, a PCIe interface, or an Ethernet interface), and the software can be connected on the CPU of the main computer 10. It is possible to use the arithmetic circuit as a hardware object in the same way that the processing by is used as a software object. The hardware objects include an arithmetic circuit unit (hw net) 14 and a data storage unit 15 (parts that store learning values and learning results used during deep learning), which are hardware realized on the hardware module 11. It is composed of input / output functions which are software realized on the CPU of the main computer 10.

In the method of mounting the hardware of the restricted Boltzmann machine according to the present embodiment, as shown in FIG. 2 (A), a plurality (n) of visible layers 16 (corresponding to the processing layer A) of the RBM are visible. status value indicating the status of each firing or non-firing of the unit 17 (corresponding to the unit _{a) v i (i = 1,2} , ···, n, corresponds to the state value a) binds visible layer 16 A state value for each hidden unit 19 determined by being transmitted to a plurality of (m) hidden units 19 (corresponding to unit B) forming a hidden layer 18 (corresponding to the processing layer B on the downstream side adjacent to the processing layer A). h _j (j = 1, 2, ..., m, corresponding to the state value B) is calculated by a fixed-point binary number, and every hidden unit 19 is calculated using the firing probability function f with _{the state value h j as a variable.} Hidden unit 19 is the first arithmetic circuit unit 20 that performs forward learning to determine the ignition or non-ignition state and output it by obtaining the ignition probability P _j (h _{j) (corresponding to the ignition probability P (B)).} It is provided in the arithmetic circuit unit (hw net) 14 so as to correspond to each case (see FIG. 1 (B)).

_{Since the state value h j} of the hidden unit 19 is obtained by an operation using a fixed-point binary number, the required circuit resources can be reduced as compared with the case of a conventional operation using a floating-point binary number.
Further, since the first arithmetic circuit unit 20 for obtaining the state value h _j of the hidden unit 19 exists as many as the number of hidden units 19, in the forward learning, the _{operation for obtaining} the state value h j of each hidden unit 19 is simultaneously performed. It will be possible to execute at high speed. Then, the circuit resources required for the first arithmetic circuit unit 20 are reduced, and the time required for the arithmetic is shortened, so that low power consumption can be reduced.

Here, the 1 state value v _i when i-th visible unit 16 is in the state of ignition, the state value v _i of when it is a non-excitation state is 0.
Further, as shown in FIG. 2 (B), the state value _{h j} of the j-th hidden unit 19, the state value _v i of each visible unit 17 (i = 1,2, ···, n) is the j-th _{The coupling load (weight) wij} (i = 1, 2, ..., n, j = 1, 2, ..., M) when transmitted to the hidden unit 19 and the j-th hidden unit 19 Using the bias value b _j (j = 1, 2, ..., M) of, b _j + w _1j v ₁ + w _2j v ₂ + w _3j v ₃ + ... + w _nj v _n = b _j + Σ _i w It becomes _ij v _i.
Then, for example, when a sigmoid function (logistic function) is used for the firing probability function f, the firing probability P _j (h _j ) of the _{j-th hidden unit 19 is P j} (h _j ) = 1 / (1 + exp (−h). _j / T)) (T is a parameter).
As the firing probability function f, a tanh function or a Piecewise linear function can be used instead of the sigmoid function.

As shown in FIG. 1 (B), the first arithmetic circuit unit 20 receives _{a fixed-point binary number when obtaining a state value h j} (j = 1, 2, ..., M) in forward learning. Bits that exceed the bit width of (the upper bit of the integer part and the lower bit of the decimal part) are truncated. Therefore, as a unit state calculator, a numerical value formed by using the lower bits of the decimal part among the bits to be truncated is used as a random number h (used instead of a random number), and the firing probability P for each hidden unit 19 is used. _{From j} (h _j ), a first unit state calculator 21 for determining the firing or non-firing state of the j-th hidden unit 19 is provided. Since the random number generator is not used, the hardware resources required for the first unit state calculator 21 can be reduced, and the RBM can be implemented in hardware in a more resource-saving manner.

Here, as shown in FIG. 3, the first unit state calculator 21 hides the jth position when the numerical value formed from the lower bits of the truncated decimal part is equal to or less than the _{firing probability P j} (h _j). The unit 19 determines that it is in a firing state, and outputs 1 from the hidden unit 19. _{Therefore, the state value h j} of the j-th hidden unit 19 is 1.
On the other hand, when the numerical value formed from the lower bits of the truncated decimal part _{exceeds the ignition probability P j} (h _j ), it is determined that the j-th hidden unit 19 is in the non-ignition state, and the hidden units 19 to 0 are set. Output. _{Therefore, the state value h j} of the j-th hidden unit 19 becomes 0.

_{The first arithmetic circuit unit 20 receives the initial state value vi0} (i = 1, 2, ..., N) for each visible unit 17 in the first forward learning, and the visible unit 17 receives the learning value (2). A unit initial state calculator 22 having a function of obtaining by inputting 1 or 0) for a base number is provided.

Here, since the initial state value v _i0 is 1 or 0, the truncated bit when obtaining the status value h _j0 for each hidden unit 19 when the initial state value v _i0 is respectively transmitted to the hidden unit 19 occurs do not do. Therefore, in the unit initial state calculator 22, each initial state value _vi0 is transmitted to the hidden unit 19, and the ignition probability for each hidden unit 19 is determined by the ignition probability P _j (h _j0 ) for each hidden unit 19 to be ignited or not ignited. _{The random number h 0} used to determine the state is _{not truncated when the state value h 0} for each hidden unit 19 determined by transmitting each initial state value v ₀ to the hidden unit 19 is calculated by a fixed-point binary number. It also has a function of making a numerical value formed by using all the bits of the decimal part in the bit.
Then, when the random number h ₀ exceeds the firing probability P _j (h ₀ ), the hidden unit 19 is set to the firing state, and when the random number h ₀ exceeds the firing probability P _j (h ₀ ), the hidden unit 19 is set to the non-firing state.

Further, in the hardware mounting method of the restricted Boltzmann machine, as shown in FIG. 4A, the state value h _j (j = 1, 2, ...) For each hidden unit 19 determined by the forward learning. , m) is obtained by computation according to each state value of each visible unit 17 determined is transmitted respectively to the visible unit _{17 v i (i = 1,2,} ···, n) fixed-point binary number, status value _{v i} the using ignition probability function f to variable visible the second arithmetic circuit 23 for performing a reverse learning to determine the status of ignition or non-ignition seeking firing probability P _i for each visible unit 17 _{(v i)} Each unit 17 is provided in the arithmetic circuit unit (hw net) 14 (see FIG. 1 (B)).

As shown in FIG. 4A, in the reverse direction learning, the processing direction is from the hidden layer 18 to the visible layer 16, so that the hidden layer 18 corresponds to the processing layer A and the visible layer 16 is the processing layer A. It corresponds to the adjacent downstream processing layer B. Similarly, the hidden unit 19 corresponds to the unit A and the visible unit 17 corresponds to the unit B. Further, firing the probability P _{(v i)} corresponds to the ignition probability P (B).

Since obtaining a state value v _i of the visible unit 17 in operation with a fixed point binary number, as compared with the case of the operation according to the conventional floating-point binary number, it is possible to reduce the circuit resources required.
Further, since the second arithmetic circuit 23 for obtaining a state value v _i of the visible unit 17 is present by the number of visible unit 17, and the reverse learning, the operation for obtaining a state value v _i of each visible unit 17 at the same time It will be possible to execute at high speed. Then, the circuit resources required for the second arithmetic circuit unit 23 are reduced, and the time required for the arithmetic is shortened, so that low power consumption can be reduced.

Here, the 1 state value h _j when j th hidden unit 19 is in the state of ignition, and 0 the state value h _j when a non-excitation state.
Further, FIG. 4 (B), the state value _{v i} of the i-th visible unit 17, the state value _h j of the hidden units 19 (j = 1,2, ···, m) is the i-th _{The coupling load (weight) wij} (i = 1, 2, ..., n, j = 1, 2, ..., M) when transmitted to the visible unit 17 of the above, and the i-th visible unit 17 Using the bias value a _i (i = 1, 2, ..., n) of, a _i + w _i1 h ₁ + w _i2 h ₂ + w _i3 h ₃ + ... + w _im h _m = a _i + Σ _j w It becomes _ij h _j.
Then, using a sigmoidal function to the ignition probability function f, ignition probability _P i of the i-th visible unit 17 _{(v j) becomes} P i = 1 / (1 + exp (-v j / T)).

As shown in FIG. 1 (B), the second arithmetic circuit 23, the state value _v i in the opposite direction learning (i = 1,2, ···, n ) in determining a fixed point binary number Bits that exceed the bit width of (the upper bit of the integer part and the lower bit of the decimal part) are truncated. Therefore, as a unit state calculator, among the bits to be truncated, a numerical value formed by using the lower bits of the decimal part is used as a random number v (used instead of a random number), and the firing probability P for each visible unit 17 is used. from _{i (v i),} the second unit state calculator 24 which determines the state of the ignition or non-ignition of the i-th visible unit 17 is provided. Since the random number generator is not used, the hardware resources required for the second unit state calculator 24 can be reduced, and the RBM can be implemented in hardware in a more resource-saving manner.

Here, the second unit state calculator 24, as shown in FIG. 3, in the case numbers formed from the lower side bit of the fraction obtained by truncating the following firing probability P _{i (v i), i} th The visible unit 17 determines that it is in an ignited state, and outputs 1 from the visible unit 17. Therefore, the state value _{v i} of the i-th visible unit 17 is one.
On the other hand, when the numerical value formed from the lower side bits of the decimal part truncated exceeds the firing probability P _{i (v i), i-th} visible unit 17 determines that the non-excitation state, the 0 from the visible unit 17 Output. Therefore, the state value _{v i} of the i-th visible unit 17 becomes zero.

The neural network that does not require a random number generator according to an embodiment of the present invention is a restricted Boltzmann machine that is an example of a neural network that includes a processing layer that does not require a random number generator used for deep learning (FIGS. 2 and 4). (See), and as shown in FIG. 1 (B), forward learning is performed, that is, a plurality of visible units 17 (corresponding to unit A) forming the upstream visible layer 16 (corresponding to the processing layer A). The state value v (corresponding to the state value A) indicating each ignition or non-ignition state of) forms a hidden layer 18 (corresponding to the treatment layer B on the downstream side adjacent to the treatment layer A) connected to the visible layer 16. The state value h (corresponding to the state value B) of each hidden unit 19 determined by being transmitted to each of the plurality of hidden units 19 (corresponding to the unit B) to be formed is obtained from the calculation by the fixed decimal binary number, and the state value h is used as a variable. The first arithmetic circuit unit that obtains the ignition probability P (h) (corresponding to the ignition probability P (B)) for each hidden unit 19 using the ignition probability function f to determine the ignition or non-ignition state and outputs it. 20 is provided in the arithmetic circuit unit 14.

Further, as shown in FIG. 1 (B), the restricted Boltzmann machine performs reverse direction learning, that is, corresponds to the hidden layer 18 determined by the forward learning (because the data processing direction is reversed, it corresponds to the processing layer A). , And so on), the state value h (corresponding to the state value A) for each hidden unit 19 (corresponding to the unit A) forms a plurality of visible units 17 (corresponding to the processing layer B) forming the visible layer 16 (corresponding to the processing layer B). The state value v (corresponding to the state value B) for each visible unit determined by being transmitted to each (corresponding) is obtained from the calculation by the fixed decimal binary number, and the firing probability function f with the state value v as a variable is used for each visible unit 17. The arithmetic circuit unit 14 has a second arithmetic circuit unit 23 for determining the ignition or non-ignition state by obtaining the ignition probability P (v) (corresponding to the ignition probability P (B)).

By providing the first arithmetic circuit unit 20, it becomes possible to simultaneously and at high speed execute the arithmetic for obtaining the state value h of each hidden unit 19 in the forward learning. Then, since the first arithmetic circuit unit 20 executes the calculation by the fixed-point binary number, the circuit resources required for the first arithmetic circuit unit 20 are reduced, and the time required for the calculation is shortened, so that the power consumption is reduced. Reduction is possible.
Further, by providing the second arithmetic circuit unit 23, it is possible to simultaneously and at high speed execute the arithmetic for obtaining the state value v of each visible unit 17 in the reverse learning. Then, since the second arithmetic circuit unit 23 executes the calculation by the fixed-point binary number, the circuit resources required for the second arithmetic circuit unit 23 are reduced, and the time required for the calculation is shortened, so that the power consumption is low. Can be reduced.

Then, in the forward learning, each state value v of the plurality of visible units 17 is transmitted to the plurality of hidden units 19 of the hidden layer 18 to the first arithmetic circuit unit 20, and the state value for each hidden unit 19 is determined. When h is calculated, a numerical value formed by using bits that exceed the bit width of the fixed-point binary number and are truncated is used as the random number h, and the firing probability P (h) of each hidden unit 19 is used to ignite the visible unit 19. A first unit state calculator 21 for determining a non-ignition state is provided.

Further, in the second arithmetic circuit unit 23, in the reverse direction learning, each state value h of the plurality of hidden units 19 is transmitted to each of the plurality of visible units 17 of the visible layer 16, and the state value for each visible unit 17 is determined. The firing probability P (v) of each visible unit 17 is used to ignite the visible unit 17 or by using a numerical value formed by using bits that exceed the bit width of the fixed-point binary number and are truncated when calculating v as the random number v. A second unit state calculator 24 is provided to determine the non-ignition state.

Since the conventional random number generator is not used, the hardware resources required for the first and second unit state calculators 21 and 24 can be reduced, and the Boltzmann machine with more limited resource saving can be implemented in hardware. It will be possible.

Here, since the initial state value for each visible unit 17 becomes 1 or 0 when the forward learning is started, the state value for each hidden unit 19 when each initial state value is transmitted to the hidden unit 19 is used. No truncation bit is generated when calculating. Therefore, as shown in FIG. 1 (B), the first arithmetic circuit unit 20 is subjected to the first forward learning by inputting the learning values (1 or 0 because of the binary number) into the visible units 17, respectively. A unit initial state calculator 22 for obtaining an initial state value for each visible unit 17 is provided. In the unit initial state calculator 22, each initial state value of the visible unit 17 is transmitted to the hidden unit 19, and the ignition probability of each hidden unit 19 determines the ignition or non-ignition state of each hidden unit 19. The numerical value used as a random number is determined by transmitting each initial state value to the hidden unit 19, and the state value of each hidden unit 19 is calculated by a fixed-point binary number operation using all the decimal bits of the non-truncated bits. Is formed.

(Example 1)
In the operation using fixed-point binary numbers, the bit width of the integer part is set to 8 bits, and the bit width of the fraction part is set to 5 steps of 4 bits, 6 bits, 8 bits, 10 bits, and 12 bits. The effect of width was examined.
Several images selected from SIDBA (Standard Image Data-Base) (image size is 1024 pixels of 32 pixels in height x 32 pixels in width) are displayed in RBM (the number of visible layer nodes is 1024 and the number of hidden layer nodes is 16). It was trained by using it as a training image. The flow of learning is shown in FIG.

As shown in FIG. 5, the RBM captures all the trained images and converts them into binary images. Then, for each training image, a value of 1 or 0 of each pixel of the binary image is input to each visible unit of the visible layer (each visible unit of the visible layer corresponds to the value of each pixel of the learning image, respectively). become) was determined and forward learning toward the hidden layer from the visible layer, the initial value of the parameter by performing a pre-learning repeated one reverse learning toward the visible layer from the hidden layer. Then, using all of the learning image, the forward learning and reverse learning parameters cooperation learning performed entire learning repeated 800 times (referred to 1 epoch) of traveling from the hidden layer to the visible layer toward the hidden layer from the visible layer Was converged to the optimum value.

Here, in the pre-learning and the whole learning, in the first forward learning, the state value indicating each state of the plurality of visible units is input, and the value (learning value) of each pixel of the learning image is input to the visible unit. The initial state value obtained was used. In addition, a random number used to determine the firing or non-firing state of each hidden unit was formed using the state value of each hidden unit determined by transmitting the initial state value of each visible unit to a plurality of hidden units. It was a numerical value.

The value of the cross entropy error between the image obtained by recalling the learning image during training and the training image was observed. When the value at the time of calculation exceeds the maximum value or the minimum value of the value that can be expressed in binary, it is fixed to a constant value that can be expressed. In addition, the value after the decimal point that cannot be expressed due to bit precision was set to 0. The change of the cross entropy error for each epoch is shown in FIG. When the bit width of the decimal part is 4 bits, the value of the cross entropy error diverges from the middle, so the value of the cross entropy error is not described after the range of the divergent epoch number.

FIG. 7 shows the relationship between the maximum value of the cross entropy error obtained from FIG. 6 and the bit width of the decimal part. From FIG. 7, it can be seen that the larger the bit width of the decimal part, the larger the value of the cross entropy error (approaching the maximum value 0), and the better the result tends to be. As a result, it is considered preferable to set the bit width of the decimal part to at least 8.

(Example 2)
The validity of using a numerical value formed by using bits that exceed the bit width of a fixed-point binary number and being truncated is examined as a random number.
Using three learning images selected from SIDBA (image size is 1024 pixels of 32 pixels in height x 32 pixels in width), RBM (number of visible layer nodes is 1024, number of hidden layer nodes is 16, coupling load and state value). 8 bits the bit width of the integer part by the fixed-point binary number, the bit width of the fractional part set to 8 bits), the update of the parameters while repeatedly forward learning and the two learning phases reverse learning 5 times Pre-learning was done.

In the fixed-point binary number of the combined load and the state value, both the integer part and the decimal part are 8 bits. Therefore, as shown in FIG. 8, the product of the combined load and the state value is 8 for the integer part and the decimal part, respectively. Since it is the multiplication of the bit values, the result is 16 bits for the integer part and 16 bits for the decimal part. Subsequently, in the operation of adding the value of the product of the state value and the weight by the number of all visible units, the number of visible nodes is 1024, so the increment of the bit after the addition is 10 bits (log ₂ 1024 = log _2). Since it is ²¹⁰ ), the bit width of the value after addition is 26 bits for the integer part and 16 bits for the fraction part. Here, in order to return the increased bit width to the original 8 bits of the integer part and 8 bits of the fractional part, 18 bits on the upper side of the 26 bits of the integer part and 16 bits on the lower side of the 16 bits of the fractional part are lower. Each 8 bit will be truncated.

A histogram of the numerical value (decimal number display) formed from the lower 8 bits of the decimal part that was truncated when the fifth pre-learning was performed is formed in FIG. 9 (A) from the 18 bits of the upper side of the integer part. A histogram of the numerical values (decimal number display) is shown in FIG. 9 (B). The number of data is 153,600 in both cases.
As shown in FIG. 9A, it can be seen that the numerical value formed from the lower 8 bits of the decimal part has an extremely large number of 0s. In the first forward training, the state value of the visible unit is fixed to either the binary value of 1 or 0, which is the pixel value of the input training image, only when the training data is input, and becomes a decimal point that becomes a truncated bit. This is because a value of 9 bits or less does not appear. Further, as shown in FIG. 9B, only the maximum value or the minimum value appears as the numerical value formed from the 18 bits on the upper side of the integer part.

Therefore, a histogram of the numerical value formed from the lower 8 bits of the decimal part created by omitting the data (initial value) at the time of inputting the training data is formed in FIG. 10 (A) from the upper 18 bits of the integer part. A histogram of the numerical values to be obtained is shown in FIG. 10 (B). As shown in FIG. 10A, with respect to the numerical values formed from the truncated bits of the decimal part, each numerical value appears at substantially the same frequency, and it was confirmed that the numerical value has a certain degree of whiteness. .. On the other hand, the numerical value formed from the truncated bit of the integer part was only the minimum value or the maximum value.
As a result, when the state value h for each hidden unit is calculated by a fixed-point binary number operation, a numerical value formed from the lower bits of the decimal part that exceeds the bit width of the fixed-point binary number and is truncated is used as a random number. It was confirmed that it can be used.

(Example 3)
In the first step of starting learning, that is, in the first forward learning, each hidden unit is fired from the firing probability P (h _{0 ) of each hidden unit, which is determined by transmitting each initial state value v 0} to each hidden unit. _{Or, when the random number h 0} used when determining the non-ignition state is _{obtained by calculating the state value h 0} for each hidden unit determined by transmitting each initial state value v ₀ to each hidden unit by a fixed-point binary number calculation. It is a numerical value formed by using all the bits of the decimal part in the non-truncated bits.
Therefore, the validity of using a numerical value formed by using all 8 decimal bits in the non-truncated bits when calculating the state value h _{0 for each hidden unit by a fixed-point binary number calculation is examined.} Therefore, the RBM having the same structure as that used in example 2, after obtaining the initial value of the parameter by prior learning once by the learning image used in example 1, performs overall learning of 800 times epochs, The cross entropy error was calculated for each epoch. The result is shown in FIG. 11 as Example 3.

11 shows, as a comparative example 1, the RBM the same structure as Example 3, while determined by the random number generator on all the random numbers software for use in determining the state of the ignition or non-ignition of hidden units, after determining the initial value of the parameter by one prior learning performs overall learning of 800 times epoch, shows the result of obtaining cross-entropy error per epoch.
Further, in FIG. 11, as a comparative example 2, a random number to be used only in determining the ignition or non-ignition state of the hidden units when the initial state value v ₀ of the visible unit is transmitted, the software while determined by the random number generator, after obtaining the initial value of the parameter by prior learning once performs overall learning of 800 times epoch, the result of obtaining the cross-entropy error per epoch, as Comparative example 3 , One advance while using the numerical value formed by using the lower 4 bits of the fractional part in the non-truncated bits when calculating the state value h _{0 for each hidden unit by a fixed-point binary number operation as a random number.} after determining the initial value of the parameter by learning performs overall learning of 800 times epoch, shows the result of obtaining cross-entropy error per epoch, respectively.

From FIG. 11, the change in the cross entropy error of Example 3 shows the same result as the change in the cross entropy error of Comparative Example 1 in which all random numbers were obtained by a random number generator on the software. As a result, in the first forward learning, the initial state value v ₀ of the visible unit is transmitted to each hidden unit and determined as a random number used when determining the firing or non-firing state of each hidden unit. It was confirmed that it is valid to use a numerical value formed by using all 8 decimal bits in the non-truncated bits when the state value h _{0 is calculated by a fixed-point binary number operation.}

Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the configuration described in the above-described embodiments, and the matters described in the claims. It also includes other embodiments and variations that may be considered within the scope.
Further, the present invention also includes a combination of the components included in the present embodiment and other embodiments and modifications.
For example, in the present embodiment and the embodiment, as a numerical value formed by using bits that exceed the bit width of the fixed-point binary number and are truncated, a numerical value formed from the lower bits of the decimal part of the fixed-point binary number is used. Although it was used, a numerical value formed from the upper bits of the integer part of the fixed-point binary number can also be used. Further, a numerical value formed from the lower bit of the decimal part of the fixed-point binary number and a numerical value formed from the upper bit of the integer part can be used in combination.

10: Main computer, 11: Hardware module, 12: Hardware and software composite system, 13: Bus, 14: Arithmetic circuit unit (hp net), 15: Data storage unit, 16: Visible layer, 17: Visible unit , 18: hidden layer, 19: hidden unit, 20: first arithmetic circuit unit, 21: first unit state arithmetic unit, 22: unit initial state arithmetic unit, 23: second arithmetic circuit unit, 24: first 2 unit state calculator

Claims (8)

It is a hardware implementation method of a neural network equipped with a processing layer that does not require a random number generator used for deep learning.
The state value A indicating the ignition or non-ignition state of each of the plurality of units A forming the arbitrary treatment layer A excluding the most downstream treatment layer A is the treatment layer B on the downstream side adjacent to the treatment layer A. The state value B for each unit B, which is determined by being transmitted to each of the plurality of units B forming the above, is obtained by an operation using a fixed-point binary number, and the firing probability function with the state value B as a variable is used for each unit B. seeking firing probability P (B) and determines and outputs the state of the ignition or non-ignition, provided an arithmetic circuit unit that will be used as a hardware object, to the arithmetic circuit unit in determining said state value B Using a numerical value formed by using bits that exceed the bit width of the fixed-point binary number and being truncated as a random number, the ignition or non-ignition state of the unit B can be determined from the ignition probability P (B) for each unit B. A method of hardware implementation of a neural network that does not require a random number generator, which is characterized by providing a unit state calculator for determining. In the method of hardware implementation of a neural network that does not require a random number generator according to claim 1, the neural network is a restricted Boltzmann machine.
The state value v indicating the ignition or non-ignition state of each of the plurality of visible units forming the visible layer of the restricted Boltzmann machine is transmitted to the plurality of hidden units forming the hidden layer connected to the visible layer. The state value h for each hidden unit determined by the above is obtained from an operation using a fixed-point binary number, and the firing probability P (h) for each hidden unit is obtained using the firing probability function with the state value h as a variable. A first arithmetic circuit unit for performing forward learning that determines and outputs a non-ignition state is provided for each hidden unit, and a plurality of state values h for each hidden unit determined by the forward learning are provided. The state value v for each visible unit, which is determined by being transmitted to the visible units, is obtained from an operation using a fixed-point binary number, and the firing probability function for each visible unit is used with the state value v as a variable. A second arithmetic circuit unit that performs reverse-direction learning that obtains P (v) and determines a state of ignition or non-ignition is provided for each visible unit, and the first arithmetic circuit unit and the second arithmetic circuit unit are provided. The arithmetic circuit unit is composed of the arithmetic circuit unit of
In the first arithmetic circuit unit, a numerical value formed by using a bit that exceeds the bit width of the fixed-point binary number and is truncated when obtaining the state value h in the forward learning is used as a random number h. A first unit state arithmetic unit for determining the ignition or non-ignition state of the hidden unit from the ignition probability P (h) for each hidden unit is provided, and the second arithmetic circuit unit is provided with the reverse direction learning. In, when the state value v is obtained, a numerical value formed by using a bit that exceeds the bit width of the fixed-point binary number and is truncated is used as a random number v, and the ignition probability P (v) for each visible unit is used as described above. A second unit state calculator for determining the ignition or non-ignition state of the visible unit is provided, and the unit state calculator is configured by the first unit state calculator and the second unit state calculator. A method of hardware implementation of a neural network that does not require a characteristic random number generator. In the method of hardware implementation of a neural network that does not require the random number generator according to claim 2, the bits that exceed the bit width of the fixed-point binary number and are truncated are the lower bits of the decimal part or the fixed-point binary number. A method of hardware implementation of a neural network that does not require a random number generator, which is characterized by being the upper bit of the integer part of. In the method of hardware implementation of a neural network that does not require the random number generator according to claim 2 or 3, when the random number h is equal to or less than the firing probability P (h), the hidden unit is set to the firing state, and the random number h is the firing. When the probability P (h) is exceeded, the hidden unit is placed in a non-firing state, and when the random number v is equal to or lower than the firing probability P (v), the visible unit is placed in a firing state, and the random number v sets the firing probability P (v). A method of hardware implementation of a neural network that does not require a random number generator, which comprises putting the visible unit in a non-firing state when it exceeds. In the method of hardware implementation of a neural network that does not require a random number generator according to claim 2, the initial state value v ₀ for each visible unit in the first forward learning is input to each of the visible units. A random number used to determine the ignition or non-ignition state of each hidden unit from the ignition probability P (h ₀ ) of each hidden unit, which is determined by transmitting each initial state value v _{0 to the hidden unit.} h ₀ is the decimal part of the non-truncated bit when the _{state value h 0} for each hidden unit, which is determined by transmitting each initial state value v _{0 to the hidden unit, is obtained by the calculation by the fixed decimal number binary number.} A method of hardware implementation of a neural network that does not require a random number generator, which is characterized by being formed using all bits. In the method of hardware implementation of a neural network that does not require a random number generator according to claim 5, when the random number h ₀ is equal to or less than the firing probability P (h ₀ ), the hidden unit is set to a firing state, and the _{random number h 0 is the random number h 0.} A method of hardware implementation of a neural network that does not require a random number generator, characterized in that the hidden unit is put into a non-ignition state when the ignition probability P (h _{0) is exceeded.} A neural network with a processing layer that does not require a random number generator used for deep learning.
The state value A indicating the ignition or non-ignition state of each of the plurality of units A forming the arbitrary treatment layer A excluding the most downstream treatment layer A is the treatment layer B on the downstream side adjacent to the treatment layer A. The state value B for each unit B, which is determined by being transmitted to a plurality of units B forming the above, is obtained by a fixed-point binary number operation, and the firing probability function with the state value B as a variable is used for each unit B. ignition probability P (B) and to determine the state of the ignition or non-ignition asking output, an arithmetic circuit unit that will be used as a hardware object,
In the arithmetic circuit unit, a numerical value formed by using a bit that exceeds the bit width of the fixed-point binary number and is truncated when obtaining the state value B is used as a random number, and the ignition probability P for each unit B is used. A neural network that does not require a random number generator, characterized in that a unit state calculator for determining a firing or non-firing state of the unit B is provided from (B). In the neural network that does not require the random number generator according to claim 7, the processing layer A is a visible layer of the restricted Boltzmann machine, and the processing layer B is a hidden layer of the restricted Boltzmann machine.
The arithmetic circuit unit, the state value v indicating the state of each of the firing or non-firing of the plurality of visible units forming the visible layer, each of a plurality of hidden units forming the hidden layer to bind to said visible layer The state value h for each hidden unit, which is transmitted and determined, is obtained from an operation using a fixed-point binary number, and the firing probability P (h) for each hidden unit is obtained using the firing probability function with the state value h as a variable. The state value h for each of the hidden units determined by the forward learning and the first arithmetic circuit unit that performs forward learning that determines and outputs the ignition or non-ignition state are transmitted to the plurality of visible units. The state value v for each visible unit determined by the above is obtained from a fixed-point binary number calculation, and the firing probability P (v) for each visible unit is obtained using the firing probability function with the state value v as a variable. It has a second arithmetic circuit unit that performs reverse direction learning to determine the state of ignition or non-ignition.
The unit state arithmetic unit is provided in the first arithmetic circuit unit, and is formed by using bits that exceed the bit width of the fixed-point binary number and are truncated when the state value h is obtained in the forward learning. A first unit state calculator that determines the ignition or non-ignition state of the hidden unit from the ignition probability P (h) of each hidden unit, and the second arithmetic circuit unit, using the numerical value to be generated as a random number h. A numerical value formed by using a bit that exceeds the bit width of the fixed-point binary number and is truncated when the state value v is obtained in the reverse direction learning is used as a random number v for each visible unit. A neural network that does not require a random number generator, characterized in that it has a second unit state calculator that determines the firing or non-firing state of the visible unit from the firing probability P (v).

Images