JP7040168B2 - Learning identification device and learning identification method - Google Patents

Description

The present invention relates to a learning identification device and a learning identification method.

In recent years, attempts to replace human functions based on a large amount of data by using machine learning, which has become generally known in relation to AI (Artificial Intelligence), are spreading in each field. .. This field is still developing significantly day by day, but there are currently some challenges. Typical examples are the limit of accuracy including generalization performance for extracting general-purpose knowledge from data, and the limit of processing speed due to the large computational load. In addition, as well-known high-performance machine learning algorithms, there are Deep learning (DL) and Resolution Natural Network (CNN) in which the input vector is limited only to the periphery. In comparison, at present, GBDT (Gradient Boosting Decision Tree) is inferior in accuracy to input data such as images, sounds, and languages because it is difficult to extract features, but other than that. It is known that structured data will give better performance. In fact, in Kaggle, a data scientist competition, GBDT is the most standard algorithm. It is said that 70% of the problems that we want to solve by machine learning in the real world are structured data other than images, sounds and languages, and GBDT is an important algorithm for solving problems in the real world. There is no doubt. Furthermore, in recent years, a method for extracting features of data such as images and sounds using a decision tree has begun to be proposed.

As a technique for realizing high-speed identification processing using such a decision tree, the effect of the cache memory is enhanced by appropriately adjusting the threshold value of the search processing for the node data of the decision tree, and the identification processing is performed. A technique for increasing the speed is disclosed (see Patent Document 1).

However, in the technique described in Patent Document 1, sample data is copied to a new memory area for each branch, so that there is a problem that a memory capacity is required as the node hierarchy becomes deeper.

The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to provide a learning identification device and a learning identification method capable of reducing the amount of memory used.

In order to solve the above-mentioned problems and achieve the object, the present invention has a data memory for storing training data for learning a determination tree, and each feature amount of the training data is read from the data memory. The learning unit that learns the determination tree by deriving the data of the node based on each feature amount, and the learning data read from the data memory by the data of the node derived by the learning unit is obtained from the node. The data memory includes at least two bank areas for storing the addresses of the learning data, and the at least two bank areas are learning targets. Each time the hierarchy of the node is switched, the bank area for reading and the bank area for writing are switched, and the learning unit uses the address of the learning data branched by the node as the bank area for reading. The learning data is read from the area of the data memory indicated by the address, and the identification unit writes the address of the learning data branched at the node to the bank area for writing.

According to the present invention, the amount of memory used can be reduced.

FIG. 1 is a diagram showing an example of a decision tree model. FIG. 2 is a diagram showing an example of a module configuration of the learning identification device according to the embodiment. FIG. 3 is a diagram showing an example of the configuration of the pointer memory. FIG. 4 is a diagram showing an example of the module configuration of the learning module. FIG. 5 is a diagram showing the operation of the module at the time of initialization of the learning identification device according to the embodiment. FIG. 6 is a diagram showing the operation of the module when determining the node parameters of the depth 0 and the node 0 of the learning identification device according to the embodiment. FIG. 7 is a diagram showing the operation of the module at the time of branching of the depth 0 and the node 0 of the learning identification device according to the embodiment. FIG. 8 is a diagram showing the operation of the module when determining the node parameters of the depth 1 and the node 0 of the learning identification device according to the embodiment. FIG. 9 is a diagram showing the operation of the module at the time of branching of the depth 1 and the node 0 of the learning identification device according to the embodiment. FIG. 10 is a diagram showing the operation of the module when determining the node parameters of the depth 1 and the node 1 of the learning identification device according to the embodiment. FIG. 11 is a diagram showing the operation of the module at the time of branching of the depth 1 and the node 1 of the learning identification device according to the embodiment. FIG. 12 is a diagram showing the operation of the module when the node parameters of the depth 1 and the node 1 of the learning identification device according to the embodiment are determined and the modules are not branched. FIG. 13 is a diagram showing the operation of the module when updating the state information of all sample data when the learning of the decision tree is completed in the learning identification device according to the embodiment. FIG. 14 is a diagram showing an example of the configuration of the model memory of the learning identification device according to the modified example. FIG. 15 is a diagram showing an example of the configuration of the classification module of the learning identification device according to the modified example.

Hereinafter, embodiments of the learning identification device and the learning identification method according to the present invention will be described in detail with reference to FIGS. 1 to 15. Further, the present invention is not limited to the following embodiments, and the components in the following embodiments include those easily conceived by those skilled in the art, substantially the same, and so-called equivalent ranges. Is included. Further, various omissions, substitutions, changes and combinations of components can be made without departing from the gist of the following embodiments.

(About GBDT logic)
In DL as a high-performance machine learning algorithm, the classifier has been tried to be implemented by various hard logics, and it has been found that the power efficiency is higher than the processing by GPU (Graphics Processing Unit). However, among DLs, especially in the case of CNN, the GPU architecture matches very well, so in terms of speed, FPGA (Field-Programmable Gate Array) with logic implementation is said to be faster to identify than GPU. Do not mean. On the other hand, for decision tree algorithms such as GBDT, implementation of hard logic by FPGA has been tried, and results faster than GPU have been reported. This is because, as will be described later, the decision tree algorithm is not suitable for the GPU architecture due to the characteristics of the data array.

In addition, regarding learning, the examination of the world is behind the identification, and there are almost no reports on the current situation in DL, and there are few reports in decision trees. Among them, GBDT learning has not been reported from anywhere at present, and is considered to be an undeveloped field at present. In order to obtain an accurate discriminative model, a huge number of trials are required to select and design features and hyperparameters of the learning algorithm during training, especially when there is a large amount of training data. In reality, the high speed of the learning process greatly affects the accuracy of the final model. Furthermore, in fields such as robotics, HFT (High Frequency Trading), and RTB (Real-Time Bidding) where real-time performance of tracking environmental changes is required, speed is directly linked to performance. Therefore, if high-speed learning processing can be performed in GBDT with high accuracy, it is considered that the performance of the system using it can be greatly improved as a result.

(Affinity of GBDT for FPGA)
The reason why the decision tree or GBDT is not faster on the GPU and why it is faster on the FPGA will be described in terms of the affinity of the GBDT for the FPGA.

First, it will be described from the viewpoint that GBDT is an algorithm using boosting. Among the decision trees, in the case of Random Forest (RF) using ensemble learning, it is easy to parallelize even with GPU because there is no dependency between trees, but GBDT is a method of connecting many trees using boosting. Yes, if the result of the previous tree is not obtained, the learning of the next tree cannot be started. Therefore, it is a serial process, and the key is how quickly you can learn each tree one by one. On the other hand, in RF, even if each tree is slow, the option of speeding up the learning of the whole tree can be taken by speeding up the learning of many trees in parallel. Therefore, even when the GPU is used, it is considered possible to conceal the problem of access latency of the DRAM (Dynamic Random Access Memory) described below to some extent.

Next, it will be described from the viewpoint of the limit of the access speed (particularly random access) of the GPU device to the RAM (Random Access Memory). The SRAM (Static Random Access Memory) built into the FPGA can greatly increase the bus width of the RAM in the FPGA, so even when using a middle-range FPGA, for example, the XC7k325T manufactured by Xilinx, as follows. It reaches 3.2 [TB / sec]. The capacity of the built-in RAM is 16 [Mb].

BRAM 445 pieces x 36bit x 100MHz x 2 ports = 445 * 36 * 2 * 100 * 10 ^ 6/10 ^ 12 = 3.2TB / sec

Further, when VU9P manufactured by Xilinx, which is a high-end FPGA, is used, it is 6.9 [TB / sec]. The capacity of the built-in RAM is 270 [Mb].

URAM 960 pieces x 36bit x 100MHz x 2 ports = 960 * 36 * 2 * 100 * 10 ^ 6/10 ^ 12 = 6.9TB / sec

These values are for the case where the clock frequency is 100 [MHz], but in reality, if the circuit configuration is devised, operation at about 200 to 500 [MHz] can be considered, and the limit band is several times. Become. On the other hand, the RAM connected to the CPU (Central Processing Unit) is DDR4 (Double-Data-Rate4) in the current generation, but the band with one DIMM (Dual Inline Memory Module) as shown below. Stays at 25.6 [GB / sec]. Even with an interleaved configuration (256 bit width) of four sheets, it is about 100 [GB / sec]. When the chip standard of DDR4 is DDR4-3200 (bus width 64 bits, DIMM 1 sheet), it is as follows.

200MHz x 2 (DDR) x 64 = 200 * 10 ^ 6 * 2 * 64/10 ^ 9 = 25.6GB / sec

In GDDR5 (Graphics Double-Data-Rate5) mounted on the GPU, it is about four times larger than the band of DDR4, but it is still about 400 [GB / sec] at the maximum.

As described above, there is a large difference in bandwidth between the RAM in the FPGA and the external memory in the GPU and the CPU. Furthermore, although the case of sequential access to the address has been described so far, the access time at the time of random access is more effective than this. Since the RAM with built-in FPGA is SRAM, the access latency is 1 clock for both sequential access and random access, but DDR4 and GDDR5 are DRAMs, and for the convenience of the sense amplifier, the latency when accessing different columns. Becomes larger. For example, in a DDR4 RAM, a typical CAS latency is 16 clocks, which is simply a calculation that produces only 1/16 of the throughput as compared with sequential access.

In the case of CNN, the data of adjacent pixels is processed, so the latency of random access does not matter much, but in the case of a decision tree, if you continue branching, the original data will be displayed for each branch. The addresses become more and more discontinuous, and basically random access. Therefore, when the data is placed in the DRAM, the throughput becomes a bottleneck and the speed is greatly deteriorated. There is a cache in the GPU in order to suppress the performance deterioration in such a case, but since the decision tree is basically an algorithm that swipes the data, there is no locality in the data access and the effect of the cache is very effective. Hateful. In the GPU structure, the GPU has a shared memory consisting of SRAM allocated for each arithmetic core (SM), and if this is used, high-speed processing may be possible, but one SM. In the case of a small amount of 16 to 48 [kB] per unit and access across SMs, a large latency occurs. The following is a trial calculation of the capacity of the shared memory in the case of NVIDIA K80, which is the current expensive and large-scale GPU.

K80 = 2 × 13 SMX = 26 SMX = 4992 CUDA core
26 × 48 × 8 = 9Mb

In this way, even with a large-scale GPU that costs hundreds of thousands of yen, there is only 9 [Mb] of shared memory, and the capacity is too small. Further, in the case of GPU, as described above, the SM that performs processing cannot directly access the shared memory of other SMs, and when used for learning a decision tree, high-speed coding is difficult. Also exists.

As described above, on the premise that the data is stored in the SRAM on the FPGA, it is considered that the FPGA can implement the GBDT learning algorithm at a higher speed than the GPU.

(GBDT algorithm)
FIG. 1 is a diagram showing an example of a decision tree model. Hereinafter, the basic logic of GBDT will be described with reference to Equations (1) to (22) and FIG.

GBDT is a method of supervised learning, and supervised learning uses the loss function L (θ), which represents the goodness of fitting to the training data, and the complexity of the learned model, as shown in the following equation (1). It is a process of optimizing the objective function obj (θ) consisting of the regularization term Ω (θ) to be expressed by some scale. The regularization term Ω (θ) has the role of preventing the model (decision tree) from becoming too complicated, that is, improving the generalization performance.

The loss function of the first term of the equation (1) is, for example, the sum of the losses calculated by the error function l for each sample data (learning data) as shown in the following equation (2). Here, n is the number of sample data, i is the sample number, y is the label, and y (hat) of the model is a predicted value.

Here, as the error function l, for example, a square error function or a logistic loss function as shown in the following equations (3) and (4) is used.

Further, as the regularization term Ω (θ) of the second term of the equation (1), for example, the squared norm of the parameter θ as shown in the following equation (5) is used. Here, λ is a hyperparameter representing the weight of regularization.

Here, consider the case of GBDT. First, the predicted value for the i-th sample data x _i of GBDT can be expressed by the following equation (6).

Here, K is the total number of decision trees, k is the number of the decision tree, f _k () is the output of the kth decision tree, and _xi is the feature amount of the input sample data. From this, it can be seen that the final output of GBDT is the sum of the outputs of each decision tree, as in RF and the like. Further, the parameter θ is θ = {f ₁ , f ₂ , ..., F _K }. From the above, the objective function of GBDT is expressed by the following equation (7).

Although learning is performed on the above objective function, a method such as SGD (Stochastic Gradient Descent) used in learning a neural network or the like cannot be used in the decision tree model. Therefore, learning is performed using Adaptive Training. In Adaptive Training, the predicted value in a certain round (number of learnings, number of decision tree models) t is expressed by the following equation (8).

From equation (8), it can be seen that it is necessary to obtain the decision tree (output) ft (x _i ) in a certain round _t . Conversely, in one round t there is no need to think about another round. Therefore, in the following, the round t will be considered. The objective function in round t is expressed by the following equation (9).

Here, the Taylor expansion of the objective function in round t (censored up to the quadratic term) is as shown in the following equation (10).

Here, in the formula (10), _gi and _hi are represented by the following formula (11).

Ignoring the constant term in equation (10), the objective function in round t is as shown in equation (12) below.

According to this equation (12), the objective function in round t is expressed by the first-order derivative and the second-order derivative of the error function with the predicted value one round before, and the regularization term, so that the first-order derivative and the second derivative are expressed. It can be seen that it can be applied if it is an error function for which the derivative can be obtained.

Now consider the decision tree model. FIG. 1 shows an example of a decision tree model. The decision tree model consists of a node and a leaf. A node inputs an input to the next node or leaf based on a branch condition, and the leaf has a leaf weight, which is the output to the input. For example, FIG. 1 shows that the leaf weight W2 of the “leaf 2” is “-1”.

Further, the decision tree model is formulated as shown in the following equation (13).

In equation (13), w represents a leaf weight and q represents a tree structure. That is, the input (sample data x) is assigned to any leaf by the structure q of the tree, and the leaf weight of that leaf is output.

Here, the complexity of the decision tree model is defined as the following equation (14).

In equation (14), the first term is the complexity of the number of leaves, and the second term is the squared norm of the leaf weights. Gamma is a hyperparameter that controls the importance of the regularization term. From the above, the objective function in round t is organized as shown in the following equation (15).

However, in the formula (15), I _j , G _j , and H _j are expressed as the following formula (16).

From the equation (15), the objective function in a certain round t is a quadratic function with respect to the leaf weight w, and the minimum value of the quadratic function and the conditions at that time are generally expressed by the following equation (17).

That is, when the structure q of the decision tree of a certain round t is determined, its objective function and leaf weight are as shown in the following equation (18).

Up to this point, it has become possible to calculate the leaf weight when the structure of the decision tree is decided in a certain round. The procedure for learning the structure of the decision tree will be described below.

Greedy Algorithm is one of the methods for learning the structure of a decision tree. The greedy algorithm is an algorithm that learns the structure of a decision tree by starting the tree structure from a depth of 0, calculating a branch score (Gain) at each node, and determining whether or not to branch. The branch score is calculated by the following equation (19).

Here, _GL and _HL are the gradient information of the sample branched to the left node, _GR and _HR are the gradient information of the sample branched to the right node, and γ is a regularization term. The first term in [] of the equation (19) is the score of the sample data branched to the left node (objective function), the second term is the score of the sample data branched to the right node, and the third term is the score when no branch is made. It represents the degree of improvement of the objective function by branching.

The branching score shown in the above equation (19) represents the goodness when branching at a certain threshold value with a certain feature amount, but it is not possible to determine what kind of condition is optimal by itself. Therefore, in the greedy method, the branch score is obtained from all the threshold candidates of all the features, and the condition that maximizes the branch score is searched. As described above, the greedy algorithm is very simple as an algorithm, but the calculation cost is high because the branch score is obtained for all the threshold candidates of all the features. Therefore, in a library such as XGBoost, which will be described later, a device is made to reduce the calculation cost while maintaining the performance.

(About XGBoost)
Hereinafter, the XGBoost, which is well known as a GBDT library, will be described. In the XGBoost learning algorithm, two points are devised: reduction of threshold value candidates and handling of missing values.

First, the reduction of threshold candidates will be described. The above-mentioned greedy method has a problem that the calculation cost is high. In XGBoost, the number of threshold candidates is reduced by a method called Weighted Quantile Sketch. This is because, in the calculation of the branch score (Gain), the sum of the gradient information of the sample data separated into left and right is important, and only the threshold value at which the sum of the gradient information changes by a certain percentage is used as a search candidate. Specifically, h, which is a quadratic gradient of the sample, is used. Assuming that the dimension of the feature amount is f, the set of the quadratic gradient h of the feature amount and the sample data is expressed by the following equation (20).

Further, the rank function r _f is defined as the following equation (21).

Here, z is a threshold candidate. The rank function r _f shown in the equation (21) means the ratio of the sum of the quadratic gradients of the sample data smaller than a certain threshold candidate to the sum of the quadratic gradients of all the sample data. Finally, for the feature quantity represented by the dimension f, it is necessary to obtain a set of certain threshold candidates {s _f1 , s _f2 , ..., _Sfl }, which is expressed by the following equation (22). Ask.

Here, ε is a parameter that determines the degree of reduction of the threshold value candidates, and approximately 1 / ε threshold value candidates can be obtained.

The Weighted Quantile Sketch can be considered in two patterns: global, which is performed at the first node of the decision tree (collectively for all sample data), and local, which is performed for each node (every time for the sample assigned to the node). Since the results show that local is better in terms of generalization performance, XGBoost adopts local.

Next, the handling of missing values will be described. The handling of missing values in the input sample data is not limited to GBDT and decision trees, and there is no generally effective method in the field of machine learning. There are methods of supplementing missing values with an average value, a median value, a cooperative filter, etc., or a method of excluding features with many missing values, but they are not successful in many cases in terms of performance. However, structured data often contains missing values, and practical measures are required.

XGBoost has a learning algorithm devised so that it can directly handle sample data including missing values. This is a method of obtaining the score when all the missing value data is assigned to either the left or right node when obtaining the branch score of the node. Further, in the case of performing the above-mentioned Weighted Quantile Sketch, the threshold value candidate may be obtained for the set excluding the sample data including the missing value.

(About LightGBM)
Next, LightGBM, which is a library of GBDT, will be described. LightGBM adopts the quantization of the feature quantity called binning for the preprocessing, and adopts the high-speed algorithm using GPU for the calculation of the branch score. LightGBM has the same performance as XGBoost, and the learning speed is several times faster, and the number of users has been increasing in recent years.

First, the quantization of the feature quantity will be described. The branch score needs to be calculated for a large number of threshold candidates if the dataset is large. LightGBM reduces the number of threshold candidates by quantizing the features as a pre-processing for learning. Further, by quantization, the value and the number of threshold candidates do not change for each node unlike XGBoost, which is an indispensable process when using the GPU.

Various studies have been conducted on the quantization of features under the name of binning. In LightGBM, the features are divided into k bins, and the threshold candidates are only k. k is 255, 63, 15, etc., and the performance or learning speed differs depending on the data set.

In addition, the calculation of the branch score is simplified by quantizing the features. Specifically, the threshold candidates are simply quantized values. Therefore, it is sufficient to create histograms of the first-order gradient and the second-order gradient for each feature quantity and obtain the branch score for each bin (quantized value). This is called a feature histogram.

Next, the calculation of the branch score using the GPU will be described. The calculation of the branch score itself has a maximum of 256 patterns because the features are quantized, but since the number of sample data exceeds tens of thousands depending on the data set, histogram creation becomes dominant with respect to the learning time. .. As mentioned above, in the calculation of the branch score, it is necessary to obtain the feature histogram. When GPU is used, it is necessary for a plurality of threads to update the same histogram, but at this time, the same bin may be updated. Therefore, it is necessary to use atomic arithmetic, and if the ratio of updating the same bin is high, the performance will decrease. Therefore, in LightGBM, when creating a histogram, which of the histograms of the primary gradient and the secondary gradient is used to update the value is divided for each thread, thereby reducing the frequency of updating the same bin.

(Configuration of learning identification device)
FIG. 2 is a diagram showing an example of a module configuration of the learning identification device according to the embodiment. FIG. 3 is a diagram showing an example of the configuration of the pointer memory. FIG. 4 is a diagram showing an example of the module configuration of the learning module. The module configuration of the learning identification device 1 according to the present embodiment will be described with reference to FIGS. 2 to 4.

As shown in FIG. 2, the learning identification device 1 according to the present embodiment includes a CPU 10, a learning module 20 (learning unit), a data memory 30, a model memory 40, and a classification module 50 (identification unit). , Is equipped. Of these, the learning module 20, the data memory 30, the model memory 40, and the classification module 50 are composed of FPGAs. Data communication is possible between the CPU 10 and the FPGA via a bus. The learning identification device 1 includes not only each component shown in FIG. 2, but also other components, for example, a RAM that serves as a work area of the CPU 10, a ROM (Read Only Memory) that stores a program executed by the CPU 10, and the like. It may be provided with an auxiliary storage device that stores various data (programs and the like), a communication I / F that communicates with an external device, and the like.

The CPU 10 is an arithmetic unit that controls GBDT learning as a whole. The CPU 10 has a control unit 11. The control unit 11 controls each module of the learning module 20, the data memory 30, the model memory 40, and the classification module 50. The control unit 11 is realized by a program executed by the CPU 10.

The learning module 20 calculates the optimum feature quantity number (hereinafter, may be referred to as “feature quantity number”) and the threshold value for each node constituting the decision tree, and when the node is a leaf, the learning module 20 calculates the optimum feature quantity number (hereinafter, may be referred to as “feature quantity number”). This is a hardware module that calculates leaf weights and writes them to the model memory 40. Further, as shown in FIG. 4, the learning module 20 includes a gain calculation module 21_1, 21_2, ..., 21_n (gain calculation unit) and an optimum condition derivation module 22 (derivation unit). Here, n is at least the number of feature quantities of sample data (including both learning data and identification data). The gain calculation modules 21_1, 21_2, ..., 21_n are simply referred to as "gain calculation module 21" when any gain calculation module is indicated or generically referred to.

The gain calculation module 21 is a module that calculates the branch score at each threshold value for the corresponding feature amount among the feature amounts included in the input sample data by using the above equation (19). Here, among the sample data, the training data includes a label (true value) in addition to the feature amount, and the identification data among the sample data includes the feature amount but does not include the label. Further, each gain calculation module 21 has a memory for calculating and storing a histogram of all the features input once (1 clock), and calculates all the features in parallel. From the result of the histogram, the gain of each feature is calculated in parallel. As a result, processing for all features can be performed at once or at the same time, so that the speed of learning processing can be dramatically improved. In this way, a method of reading out all the features in parallel and processing them is called a feature parallel. In order to realize this method, the data memory needs to be able to read all the features at once (1 clock). Therefore, it cannot be realized with a memory having a data width of a normal 32-bit or 256-bit width. Further, in software, the number of bits of data that can be normally handled by the CPU at one time is limited to 64 bits, and even if the number of feature quantities is 100 and the number of bits of each feature quantity is 8 bits, 8000 bits are required. , I can't handle it at all. Therefore, conventionally, a method has been adopted in which a different feature amount is stored for each memory address (for example, a 64-bit width that can be handled by the CPU), and all the feature amounts are saved across a plurality of addresses. .. On the other hand, in this method, the new technical content is that all the features are stored in one address of the memory and all the features are read by one access.

As mentioned above, GBDT cannot parallelize the learning of decision trees. Therefore, how fast the decision trees are learned one by one is dominant in terms of the speed of the learning process. On the other hand, in an RF that performs ensemble learning, since there is no dependency between decision trees at the time of learning, it is easy to parallelize the learning process for each decision tree, but the accuracy is generally inferior to GBDT. As described above, the speed of the training process of the decision tree can be improved by applying the above-mentioned feature parallel (Feature Parallel) to the learning of GBDT having higher accuracy than RF.

The gain calculation module 21 outputs the calculated branch score to the optimum condition derivation module 22.

The optimum condition derivation module 22 inputs each branch score corresponding to each feature amount output by each gain calculation module 21, and derives the feature amount number (feature amount number) and the threshold value that maximize the branch score. It is a module to do. The optimal condition derivation module 22 writes the derived feature quantity number and the threshold value into the model memory 40 as branch condition data (an example of node data) of the corresponding node.

The data memory 30 is an SRAM that stores various data. The data memory 30 includes a pointer memory 31, a feature memory 32, and a state memory 33.

The pointer memory 31 is a memory for storing the storage destination address of the sample data stored in the feature memory 32. As shown in FIG. 3, the pointer memory 31 has a bank A (bank area) and a bank B (bank area). The details of the operation of dividing into two banks, bank A and bank B, and storing the storage destination address of the sample data will be described later with reference to FIGS. 5 to 13. The pointer memory 31 does not limit the possession of three or more banks.

The feature memory 32 is a memory for storing sample data (including learning data and identification data).

The state memory 33 is a memory for storing state information (w, g, h described above) and label information.

The model memory 40 includes branch condition data (feature quantity number, threshold value) for each node of the decision tree, a leaf flag indicating whether or not the node is a leaf (flag information, an example of node data), and A SRAM that stores leaf weights when the node is a leaf.

The classification module 50 is a hardware module that distributes sample data for each node and each decision tree. Further, the classification module 50 calculates the state information (w, g, h) and writes it to the state memory 33.

The classification module 50 has the same module configuration not only in the identification (branch) of the sample data (learning data) in the learning process but also in the identification process for the sample data (identification data) as described above. It is possible to identify the identification data. In addition, even during the identification process, the process by the classification module 50 can be pipelined by reading all the features at once, and the process speed is high up to the identification of one sample data for each clock. It becomes possible to change. On the other hand, if it is not possible to read all at once as described above, it is not possible to know which feature amount is required until branching to each node, so a pipe is used to access the address of the corresponding feature amount each time. It will not be possible to make a line.

Further, it is assumed that a plurality of the above-mentioned classification modules 50 are provided, and a plurality of identification data are divided (data parallel) and distributed to each classification module 50 for identification processing. , It is also possible to speed up the identification process.

(Learning process of learning identification device)
Hereinafter, the learning process of the learning identification device 1 will be specifically described with reference to FIGS. 5 to 13.

<Initialization>
FIG. 5 is a diagram showing the operation of the module at the time of initialization of the learning identification device according to the embodiment. As shown in FIG. 5, first, the control unit 11 initializes the pointer memory 31. For example, as shown in FIG. 5, the control unit 11 assigns the addresses of the sample data (learning data) in the feature memory 32 to the bank A of the pointer memory 31 in order by the number of training data (for example, of the addresses). Write (from lowest to lowest).

It should be noted that the learning data is not limited to using all of the training data (writing all addresses), and learning data randomly selected based on the probability according to a predetermined random number is used by so-called data subsampling (the learning data is randomly selected based on the probability according to a predetermined random number). The address of the selected learning data may be written). For example, when the data subsampling is 0.5, half of all the addresses of the training data may be written to the pointer memory 31 (here, bank A) with a half probability according to the random number. Pseudo-random numbers created by LFSR (Linear Feedback Shift Register) can be used to generate random numbers.

Also, the training data used for training is not limited to using all the features, and is randomly selected based on the probability according to the same random number as described above by the so-called feature subsample (for example,). It is also possible to use only the feature amount selected (half selected). In this case, for example, as the data of the feature amount other than the feature amount selected by the feature subsample, a constant may be output from the feature memory 32. This has the effect of improving the generalization performance for unknown data (identification data).

<Determination of branch condition data for depth 0 and node 0>
FIG. 6 is a diagram showing the operation of the module when determining the node parameters of the depth 0 and the node 0 of the learning identification device according to the embodiment. The top layer of the decision tree is called "depth 0", and the layers below it are called "depth 1", "depth 2", and so on, and the leftmost node of a specific layer. Is referred to as "node 0", and the node to the right thereof is referred to as "node 1", "node 2", ...

As shown in FIG. 6, first, the control unit 11 transmits a start address and an end address to the learning module 20, and triggers the learning module 20 to start processing. The learning module 20 specifies the address of the target learning data from the pointer memory 31 (bank A) based on the start address and the end address, and reads the training data (feature amount) from the feature memory 32 according to the address. State information (w, g, h) is read from the state memory 33.

In this case, as described above, each gain calculation module 21 of the learning module 20 calculates a histogram of the corresponding feature amount, stores it in its own SRAM, and calculates the branch score at each threshold value based on the result. calculate. Then, the optimum condition derivation module 22 of the learning module 20 inputs each branch score corresponding to each feature amount output by each gain calculation module 21, and the feature amount number (feature amount number) at which the branch score becomes maximum is input. And derive the threshold. Then, the optimal condition derivation module 22 writes the derived feature quantity number and the threshold value into the model memory 40 as branch condition data of the corresponding nodes (depth 0, node 0). At this time, the optimal condition derivation module 22 may set the leaf flag to "0" to indicate that the node (depth 0, node 0) is further branched, and may be a part of the data (branch condition data) of the node. ) Is written to the model memory 40.

For the above operation, the learning module 20 designates the addresses of the learning data written in the bank A in order, and reads out each learning data from the feature memory 32 by the addresses.

<Data branch processing at depth 0 / node 0>
FIG. 7 is a diagram showing the operation of the module at the time of branching of the depth 0 and the node 0 of the learning identification device according to the embodiment.

As shown in FIG. 7, the control unit 11 transmits a start address and an end address to the classification module 50, and causes the classification module 50 to start processing by a trigger. The classification module 50 specifies the address of the target learning data from the pointer memory 31 (bank A) based on the start address and the end address, and the learning data (feature amount) is obtained from the feature memory 32 by the address. read out. Further, the classification module 50 reads branch condition data (feature quantity number, threshold value) of the corresponding node (depth 0, node 0) from the model memory 40. Then, the classification module 50 determines whether to branch the read sample data to the left side or the right side of the node (depth 0, node 0) according to the branch condition data, and based on the determination result, the said The address in the feature memory 32 of the training data is different from the read bank (here, bank A) (bank area for reading) of the pointer memory 31 and the other bank (write bank) (here, bank B) (bank area for writing). ).

At this time, when the classification module 50 determines that the branch is to the left side of the node, the address of the learning data is written in order from the lowest address of the bank B as shown in FIG. 7, and the address of the learning data is written in order from the lower side to the right side of the node. If it is determined that the learning data is branched to, the address of the learning data is written in order from the highest address of the bank B. As a result, in the write bank (bank B), the address of the learning data branched to the left side of the node is written to the lower address, and the address of the learning data branched to the right side of the node is written to the higher address. be able to. In the write bank, the address of the learning data branched to the left side of the node may be written to the higher address, and the address of the learning data branched to the right side of the node may be written to the lower address.

As described above, in the pointer memory 31, as described above, two banks A and B are configured, and by alternately reading and writing, the capacity of the SRAM in the FPGA is limited, and the efficiency is increased. Memory can be used. There is also a method of simply configuring two banks each of the feature memory 32 and the state memory 33, but in general, the data indicating the address in the feature memory 32 is smaller than the sample data, so this embodiment. As described above, the method of preparing the pointer memory 31 and indirectly specifying the address makes it possible to save the memory capacity.

For the above operation, the classification module 50 performs branch processing on all the training data. However, since the same number of training data is not divided into the left side and the right side of the node (depth 0, node 0) after the branch processing is completed, the classification module 50 has the address of the training data branched to the left side. And the address (intermediate address) in the write bank (bank B) corresponding to the boundary with the address of the learning data branched to the right side is returned to the control unit 11. The intermediate address is used in the next branch processing.

<Determination of branch condition data for depth 1 and node 0>
FIG. 8 is a diagram showing the operation of the module when determining the node parameters of the depth 1 and the node 0 of the learning identification device according to the embodiment. Basically, it is the same as the process of determining the branch condition data of depth 0 and node 0 shown in FIG. 6, but since the hierarchy of the target node changes (from depth 0 to depth 1), a pointer. The roles of bank A and bank B of the memory 31 are reversed. Specifically, bank B is a read bank and bank A is a write bank (see FIG. 9).

As shown in FIG. 8, the control unit 11 transmits the start address and the end address to the learning module 20 based on the intermediate address received from the classification module 50 in the processing at depth 0, and the learning module 20 is triggered. Starts processing by. The learning module 20 specifies the address of the target learning data from the pointer memory 31 (bank B) based on the start address and the end address, and reads the training data (feature amount) from the feature memory 32 according to the address. State information (w, g, h) is read from the state memory 33. Specifically, as shown in FIG. 8, the learning module 20 designates addresses in order from the left side (lower address) of bank B to the intermediate address.

In this case, as described above, each gain calculation module 21 of the learning module 20 stores each feature amount of the read learning data in its own SRAM and calculates the branch score at each threshold value. Then, the optimum condition derivation module 22 of the learning module 20 inputs each branch score corresponding to each feature amount output by each gain calculation module 21, and the feature amount number (feature amount number) at which the branch score becomes maximum is input. And derive the threshold. Then, the optimal condition derivation module 22 writes the derived feature quantity number and the threshold value into the model memory 40 as branch condition data of the corresponding nodes (depth 1, node 0). At this time, the optimal condition derivation module 22 may set the leaf flag to "0" to indicate that the node (depth 1, node 0) is further branched, and may be a part of the data (branch condition data) of the node. ) Is written to the model memory 40.

The learning module 20 designates the above operations in order from the left side of the bank B (the one with the lower address) to the intermediate address, and reads out each learning data from the feature memory 32 according to the address.

<Data branch processing at depth 1 and node 0>
FIG. 9 is a diagram showing the operation of the module at the time of branching of the depth 1 and the node 0 of the learning identification device according to the embodiment.

As shown in FIG. 9, the control unit 11 transmits a start address and an end address to the classification module 50 based on the intermediate address received from the classification module 50 in the processing at depth 0, and the class is triggered by a trigger. The processing by the fiction module 50 is started. The classification module 50 specifies the address of the target learning data from the left side of the pointer memory 31 (bank B) based on the start address and the end address, and the learning data (feature amount) from the feature memory 32 by the address. ) Is read. Further, the classification module 50 reads branch condition data (feature quantity number, threshold value) of the corresponding node (depth 1, node 0) from the model memory 40. Then, the classification module 50 determines whether to branch the read sample data to the left side or the right side of the node (depth 1, node 0) according to the branch condition data, and based on the determination result, the said The address in the feature memory 32 of the training data is different from the read bank (here, bank B) (bank area for reading) of the pointer memory 31 and the other bank (write bank) (here, bank A) (bank area for writing). ).

At this time, when the classification module 50 determines that it branches to the left side of the node, the address of the learning data is written in order from the lowest address of the bank A as shown in FIG. 9, and the address of the learning data is written in order from the lower side to the right side of the node. If it is determined that the learning data is branched to, the address of the learning data is written in order from the highest address of the bank A. As a result, in the write bank (bank A), the address of the learning data branched to the left side of the node is written to the lower address, and the address of the learning data branched to the right side of the node is written to the higher address. be able to. In the write bank, the address of the learning data branched to the left side of the node may be written to the higher address, and the address of the learning data branched to the right side of the node may be written to the lower address.

For the above operation, the classification module 50 performs branch processing on the learning data designated by the address written on the left side of the intermediate address of the bank B among all the learning data. However, since the same number of training data is not divided into the left side and the right side of the node (depth 1, node 0) after the branch processing is completed, the classification module 50 has the address of the training data branched to the left side. And the address (intermediate address) in the write bank (bank A) corresponding to the middle of the address of the learning data branched to the right side is returned to the control unit 11. The intermediate address is used in the next branch processing.

<Determination of branch condition data for depth 1 and node 1>
FIG. 10 is a diagram showing the operation of the module when determining the node parameters of the depth 1 and the node 1 of the learning identification device according to the embodiment. As in the case of FIG. 8, since the hierarchy is the same as that of the nodes of depth 1 and node 0, bank B is a read bank and bank A is a write bank (see FIG. 11).

As shown in FIG. 10, the control unit 11 transmits the start address and the end address to the learning module 20 based on the intermediate address received from the classification module 50 in the processing at the depth 0, and the learning module 20 is triggered. Starts processing by. The learning module 20 specifies the address of the target learning data from the pointer memory 31 (bank B) based on the start address and the end address, and reads the training data (feature amount) from the feature memory 32 according to the address. State information (w, g, h) is read from the state memory 33. Specifically, as shown in FIG. 10, the learning module 20 designates addresses in order from the right side (higher address) of bank B to the intermediate address.

In this case, as described above, each gain calculation module 21 of the learning module 20 stores each feature amount of the read learning data in its own SRAM and calculates the branch score at each threshold value. Then, the optimum condition derivation module 22 of the learning module 20 inputs each branch score corresponding to each feature amount output by each gain calculation module 21, and the feature amount number (feature amount number) at which the branch score becomes maximum is input. And derive the threshold. Then, the optimal condition derivation module 22 writes the derived feature quantity number and the threshold value into the model memory 40 as branch condition data of the corresponding nodes (depth 1, node 1). At this time, the optimal condition derivation module 22 may set the leaf flag to "0" to indicate that the node (depth 1, node 1) is further branched, and may be a part of the data (branch condition data) of the node. ) Is written to the model memory 40.

The learning module 20 designates the above operation in order from the right side (higher address) of the bank B to the intermediate address, and reads out each learning data from the feature memory 32 according to the address.

<Data branch processing at depth 1 and node 1>
FIG. 11 is a diagram showing the operation of the module at the time of branching of the depth 1 and the node 1 of the learning identification device according to the embodiment.

As shown in FIG. 11, the control unit 11 transmits a start address and an end address to the classification module 50 based on the intermediate address received from the classification module 50 in the processing at depth 0, and the class is triggered by a trigger. The processing by the fiction module 50 is started. The classification module 50 specifies the address of the target learning data from the right side of the pointer memory 31 (bank B) based on the start address and the end address, and the learning data (feature amount) from the feature memory 32 by the address. ) Is read. Further, the classification module 50 reads branch condition data (feature quantity number, threshold value) of the corresponding node (depth 1, node 1) from the model memory 40. Then, the classification module 50 determines whether to branch the read sample data to the left side or the right side of the node (depth 1, node 1) according to the branch condition data, and based on the determination result, the said The address in the feature memory 32 of the training data is different from the read bank (here, bank B) (bank area for reading) of the pointer memory 31 and the other bank (write bank) (here, bank A) (bank area for writing). ).

At this time, when the classification module 50 determines that it branches to the left side of the node, the address of the learning data is written in order from the lowest address of the bank A as shown in FIG. 11, and the address of the learning data is written in order from the lower side to the right side of the node. If it is determined that the learning data is branched to, the address of the learning data is written in order from the highest address of the bank A. As a result, in the write bank (bank A), the address of the learning data branched to the left side of the node is written to the lower address, and the address of the learning data branched to the right side of the node is written to the higher address. be able to. In the write bank, the address of the learning data branched to the left side of the node may be written to the higher address, and the address of the learning data branched to the right side of the node may be written to the lower address. In this case, it is necessary to match the operation in FIG.

For the above operation, the classification module 50 performs branch processing on the learning data designated by the address written on the right side of the intermediate address of the bank B among all the learning data. However, since the same number of training data is not divided into the left side and the right side of the node (depth 1, node 1) after the branch processing is completed, the classification module 50 has the address of the training data branched to the left side. And the address (intermediate address) in the write bank (bank A) corresponding to the middle of the address of the learning data branched to the right side is returned to the control unit 11. The intermediate address is used in the next branch processing.

<When branching does not occur when determining the branching condition data for depth 1 and node 1>
FIG. 12 is a diagram showing the operation of the module when the node parameters of the depth 1 and the node 1 of the learning identification device according to the embodiment are determined and the modules are not branched. As in the case of FIG. 8, since the hierarchy is the same as that of the nodes of depth 1 and node 0, bank B is a read bank.

As shown in FIG. 12, the control unit 11 transmits the start address and the end address to the learning module 20 based on the intermediate address received from the classification module 50 in the processing at the depth 0, and the learning module 20 is triggered. Starts processing by. The learning module 20 specifies the address of the target learning data from the pointer memory 31 (bank B) based on the start address and the end address, and reads the training data (feature amount) from the feature memory 32 according to the address. State information (w, g, h) is read from the state memory 33. Specifically, as shown in FIG. 12, the learning module 20 designates addresses in order from the right side (higher address) of bank B to the intermediate address.

When the learning module 20 determines from the calculated branch score and the like that it will no longer branch from the node (depth 1, node 1), the leaf flag is set to "1" and the data of the node (as part of the branch condition data). It may be written) to the model memory 40, and also transmits that the leaf flag of the node is "1" to the control unit 11. As a result, it is recognized that the node (depth 1, node 1) does not branch to the lower hierarchy. Further, when the leaf flag of the node (depth 1, node 1) is "1", the learning module 20 uses the leaf weight (w) (as part of the branch condition data) instead of the feature quantity number and the threshold value. May be good) is written in the model memory 40. As a result, the capacity of the model memory 40 can be made smaller than that of having them separately.

By proceeding with the processes shown in FIGS. 6 to 12 for each layer (depth), the entire decision tree is completed (decision tree is learned).

<When learning of decision tree is completed>
FIG. 13 is a diagram showing the operation of the module when updating the state information of all sample data when the learning of the decision tree is completed in the learning identification device according to the embodiment.

When the training of one decision tree constituting GBDT is completed, the linear gradient g corresponding to the error function of each training data is used for boosting to the next decision tree (here, gradient boosting). It is necessary to calculate the quadratic gradient h and the leaf weight w for each training data. As shown in FIG. 13, the control unit 11 initiates the above-mentioned calculation by the classification module 50 by a trigger. The classification module 50 performs branch determination processing for all depth (hierarchy) nodes for all learning data, and calculates leaf weights corresponding to each learning data. Then, the classification module 50 calculates the state information (w, g, h) based on the label information for the calculated leaf weight, and writes it back to the address of the original state memory 33. In this way, the next decision tree is learned using the updated state information.

As described above, in the learning identification device 1 according to the present embodiment, the learning module 20 is provided with a memory (for example, SRAM) for reading each feature amount of the input sample data. As a result, all the features of the sample data can be read out with one access, and each gain calculation module 21 can process all the features at once, which dramatically improves the speed of learning processing of the decision tree. It is possible to make it.

Further, in the learning identification device 1 according to the present embodiment, the pointer memory 31 is composed of two banks A and B, and is supposed to read and write alternately. This makes it possible to use the memory efficiently. There is also a method of simply configuring two banks each of the feature memory 32 and the state memory 33, but in general, the data indicating the address in the feature memory 32 is smaller than the sample data, so this embodiment. As described above, the method of preparing the pointer memory 31 and indirectly specifying the address makes it possible to reduce the amount of memory used. Further, when the classification module 50 determines that the learning data branch is to the left side of the node, the learning data address is written in order from the lower write bank address of the two banks, and the classification module 50 determines that the learning data address is branched to the right side of the node. In this case, the address of the learning data is written in order from the highest address of the write bank. As a result, in the write bank, the address of the learning data branched to the left side of the node can be written to the lower address, and the address of the learning data branched to the right side of the node can be written to the higher address.

(Modification example)
FIG. 14 is a diagram showing an example of the configuration of the model memory of the learning identification device according to the modified example. With reference to FIG. 14, a configuration in which a memory is provided for each depth (layer) of the decision tree in the model memory 40 in the learning identification device 1 according to the present modification will be described.

As shown in FIG. 14, the model memory 40 of the learning identification device 1 according to this modification stores data (specifically, branch condition data) for each depth (hierarchy) of the model data of the learned decision tree. It has a depth 0 memory 41_1, a depth 1 memory 41_2, ..., And a depth (m-1) memory 41_m. Here, m is at least a number equal to or greater than the number of depths (hierarchies) of the model of the decision tree. That is, the model memory 40 simultaneously extracts data (depth 0 node data, depth 1 node data, ..., Depth (m-1) node data) for each depth (hierarchy) of the learned model data of the decision tree. It means that it has an independent port for. As a result, the classification module 50 reads the data (branch condition data) corresponding to the next node in parallel at all depths (hierarchy) based on the branch result at the first node in the decision tree. It is possible to execute branch processing (pipeline processing) at each depth (hierarchy) at the same time for one sample data (identification data) at one clock without going through a memory on the way. As a result, the identification process in the classification module 50 only takes time for the number of sample data, and the speed of the identification process can be dramatically improved. On the other hand, in the conventional technology, since the sample data is copied to a new memory area for each node, the speed is affected only by the time for reading and writing the memory, and the identification (number of sample data x number of depths) is identified. Since it is the processing time, the identification processing according to this modification is significantly superior as described above.

FIG. 15 is a diagram showing an example of the configuration of the classification module of the learning identification device according to the modified example. As shown in FIG. 15, the classification module 50 has a node 0 discriminator 51_1, a node 1 discriminator 51_2, a node discriminator 51_3, and so on. From the feature memory 32, one sample data is supplied as a feature amount per clock. As shown in FIG. 15, the feature amount is first input to the node 0 discriminator 51_1, and the node 0 discriminator 51_1 is the data of the node (depth 0 node data) from the depth 0 memory 41_1 of the corresponding model memory 40. Receive (conditions for going to the right or to the left, and the feature number to use). The node 0 discriminator 51_1 determines whether the corresponding sample data goes to the right or to the left according to the condition. Here, it is assumed that each of the depth memories (depth 0 memory 41_1, depth 1 memory 41_2, depth 2 memory 41_3, ...) Has a latency of 1 clock. Based on the result of the node 0 discriminator 51_1, the number of the node to go to in the next depth 1 memory 41_1 is specified, the data of the corresponding node (depth 1 node data) is extracted, and the node 1 discriminator is used. It is input to 51_2.

Since the latency of the memory 41_1 for depth 0 is one clock, the feature amount is also input to the node 1 discriminator 51_2 with a delay of one clock. Further, the feature amount of the next sample data is input to the node 0 discriminator 51_1 at the same clock. In this way, by performing identification by pipeline processing, it is possible to identify one sample data in one clock as a whole decision tree on the premise that the memory is output at the same time for each depth. .. Since the memory 41_1 for depth 0 has only one node in depth 0, only one address is required, and the memory 41_1 for depth 1 needs two addresses because there are two nodes in depth 1. Yes, similarly, the depth 2 memory 41_3 requires four addresses, and the depth 3 memory (not shown) requires eight addresses. The classification module 50 identifies the entire tree, but when learning a node, the same circuit is diverted by learning using only the node 0 discriminator 51_1 to reduce the circuit scale. can do.

Hereinafter, the prediction result of the speed of the learning process in the learning identification device 1 according to the above-described embodiment will be described.

First, for comparison, the learning speeds of the above-mentioned XGBoost and LightGBM, which are representative libraries of GBDT, were evaluated. As of December 2017, the speed was high when GPU was used in LightGBM, and this was actually measured.

The processing time was calculated from the clock of the hardware configuration. In the hardware logic implemented this time, three main processes are learning processing by the learning module 20, identification processing by the classification module 50 (node unit), and identification processing by the classification module 50 (tree unit). ..

<About the processing of the learning module>
Here, the creation of the gradient histogram and the calculation of the branch score from each feature of the sample data are dominant. In creating a gradient histogram from each feature of the sample data, it is necessary to read all the sample data for each depth (hierarchy). This estimate is the maximum, as some sample data completes learning when the depth of the tree is shallow. Since the calculation of the branch score refers to all the bins of the gradient histogram, a clock of the number of bins (dimension dimension) is required. From the above, the clock number _learning of the processing of the learning module 20 is expressed by the following equation (23).

Here, n _{sample_train} is the number of sample data used for learning the decision tree, and is generally a set subsampled from all the sample data. Further, maxdepth is the maximum depth of the decision tree, _nfature is the number of bins (dimension of the feature amount), and _nnode is the number of nodes.

<Processing of classification module (node unit)>
Here, the result of the learned node is used to process whether the sample data is assigned to the lower node on the left or right. Since the total number of sample data to be processed does not change for each depth, the clock number C _{Classification_node} is expressed by the following equation (24). Actually, there is a node where learning ends in the middle, so the following estimate is the maximum value.

<Processing of classification module (tree unit)>
Here, after the learning of one decision tree is completed, the gradient information is updated for each sample data for the learning of the next decision tree. Therefore, it is necessary to make predictions for all sample data using the learned decision tree. In the processing of each tree, a delay occurs by the depth. In this case, the clock number C _{Cclassification_tree} is expressed by the following equation (25).

Here, the total sample data is the total number of all training sample data before the subsample and all validation sample data.

From the above, the number of clocks C _tree (maximum value) required for the learning process for one decision tree is expressed by the following equation (26).

Since GBDT is composed of a large number of decision trees, if the number of decision trees is n _tree , the clock number C _gbdt of the entire GBDT model is expressed by the following equation (27).

The above is a trial calculation in the case of the above-mentioned feature parallel (Fature Parallel). In the so-called data parallel (Data Parallel) in which a large number of modules are arranged in parallel and divided by data, each module is used at each node. If there is no bias in the number of data, it is basically possible to increase the speed by several times the number of modules. Since the degree of bias depends on the sample data and the method of dividing the sample data into each module, this overhead will be examined using actual data in the future. As a prediction, even if this overhead is taken into consideration, it is estimated that the efficiency will be 50% or more.

<About usage data>
As the sample data for the test, learning data and identification data (evaluation data) were randomly selected from about 100,000 cases. The outline of the data set is shown below.

・ Number of classes: 2
・ Feature dimension: 129
・ Number of learning data: 63415
・ Number of evaluation data: 31707

The speed measurement conditions are shown in (Table 1) below. The clock frequency of the FPGA is assumed to be 100 [MHz] (actually, there is a high possibility that it will be higher).

<Estimation of hardware logic>
The following (Table 2) shows the trial calculation of the learning speed in the above-mentioned architecture using the above-mentioned speed calculation formula. However, this trial calculation is the worst value because it is a trial calculation when all the sample data goes to the end branch.

<Comparison results including actual measurement with CPU / GPU>
The actual measurement results on the CPU / GPU are shown in (Table 3) below. For comparison, the hard logic calculation results are also included. Since the estimation up to this point is only for feature parallel (Fature Parallel), the estimation result when data parallel (Data Parallel) is also used is added as a reference.

Regarding this data, it can be seen that the speed is lower than that of the CPU even when the GPU is used. Microsoft, the developer of LightGBM, says that when using GPU, it will be 3 to 10 times faster, but it depends heavily on data, and for this data, the speed on GPU did not go well. I understand. This result also shows that the GBDT algorithm is not as easy to speed up the GPU as CNN. The result on the CPU is about 10 times faster in the later LightGBM than in the XGBoost, which is the most basic library. Even with the hardware logic of only the feature parallel (Fature Parallel), the speed is about 2.3 times faster than that of the fastest CPU (LightGBM) in the PC (Personal Computer). In addition, when 15 parallel data parallels (Data Parallels) are also used, even if the efficiency of the data parallels (Data Parallels) is 75%, it is 25 times or more, 240 parallels when considering AWS f1.16xlage instances. Assuming that the efficiency in the case of is 50%, it is estimated that the speed will be 275 times or more. However, this calculation is a calculation when the memory bandwidth is the limit, and it is necessary to consider whether or not this amount of logic fits in the FPGA in the future.

Regarding power consumption, it is predicted to be a number [W] in FPGA, and considering that it is 100 [W] or more in CPU and GPU, power consumption differs by 2 digits in addition to speed, so power efficiency is 3 digits. The above difference may occur.

1 Learning identification device 10 CPU
11 Control unit 20 Learning module 21, 21_1, 21_1 Gain calculation module 22 Optimal condition derivation module 30 Data memory 31 Pointer memory 32 Feature memory 33 State memory 40 Model memory 41_1 Depth 0 memory 41_1 Depth 1 memory 41_3 Depth 2 memory 50 Classification module 51_1 node 0 discriminator 51_1 node 1 discriminator 51_3 node 2 discriminator

Japanese Patent No. 5032602

Claims (14)

A data memory that stores learning data for learning decision trees, and
A learning unit that learns the decision tree by reading each feature amount of the learning data from the data memory and deriving node data based on the feature amount.
An identification unit that determines to which the learning data read from the data memory is branched from the node based on the data of the node derived by the learning unit.
Equipped with
The data memory has at least two bank areas for storing the addresses of the learning data.
The at least two bank areas are switched between a read bank area and a write bank area each time the hierarchy of the node to be learned is switched.
The learning unit reads the address of the learning data branched by the node from the bank area for reading, reads the learning data from the area of the data memory indicated by the address, and reads the learning data.
The identification unit is a learning identification device that writes the address of the learning data branched at the node to the bank area for writing. The identification unit is
The address of the learning data branched from the node to one of the lower nodes is written to the write bank area in order from the smallest address of the write bank area.
The learning identification according to claim 1, wherein the address of the learning data branched from the node to the other lower node is written in order from the bank area for writing in descending order of the address of the bank area for writing. Device. The learning unit learns the decision tree by reading all the feature quantities contained in one of the learning data from the data memory by one access and deriving the data of the node based on each feature quantity. The learning identification device according to claim 1 or 2. The learning unit
It has at least as many memories as the feature amount of the training data.
The learning identification device according to claim 3, wherein a histogram of the feature amount of the learning data read from the data memory is stored in each of the memory, and processing based on each feature amount is performed at once. The learning unit
A gain calculation unit that calculates a branch score for each threshold value based on the histogram of the feature amount provided in each memory and stored in each memory.
A derivation unit for deriving the feature quantity number and the threshold value for which each branch score is optimal as data of the node, and a derivation unit.
The learning identification device according to claim 4. The learning identification device according to any one of claims 1 to 5, wherein the identification unit reads out the label information of the learning data together with the feature amount of the learning data from the data memory. The learning identification device according to any one of claims 1 to 6, wherein the learning unit learns the next decision tree by gradient boosting based on the learning result of the learned decision tree. In order to allow the learning unit to learn the next decision tree by the gradient boosting, the identification unit has a primary gradient, a secondary gradient, and each learning corresponding to the error function of each training data. The learning identification device according to claim 7, wherein a leaf weight for data is calculated and written to the data memory. The learning identification device according to any one of claims 1 to 8, wherein the identification unit shares a configuration for performing an operation of identifying the learning data at the time of learning, and performs an operation of identifying the identification data at the time of identification. .. Further, a model memory for storing the data of the node of the decision tree is provided.
The learning unit
When the node to be learned is to be further branched, flag information indicating that fact is written to the model memory as data of the node.
The learning according to any one of claims 1 to 9, wherein when the node to be learned is not branched any more, flag information indicating that fact is written to the model memory as data of the node. Identification device. The learning identification device according to any one of claims 1 to 10, wherein the learning unit performs learning using some of the feature quantities of all the feature quantities of the learning data. The learning identification device according to any one of claims 1 to 10, wherein the learning unit performs learning using a part of the learning data among all the learning data. The learning identification device according to any one of claims 1 to 12, wherein at least the data memory, the learning unit, and the identification unit are configured on an FPGA (Field-Programmable Gate Array). A learning step of learning the decision tree by reading each feature amount of the learning data from a data memory that stores learning data for learning the decision tree and deriving node data based on each feature amount. ,
An identification step for determining to which the learning data read from the data memory is branched from the node based on the derived data of the node.
At least two bank areas for storing the addresses of the learning data possessed by the data memory are switched between a bank area for reading and a bank area for writing each time the hierarchy of the node to be learned is switched. Steps and
A step of reading the address of the learning data branched at the node from the bank area for reading and reading the learning data from the area of the data memory indicated by the address.
A step of writing the address of the learning data branched at the node to the bank area for writing, and
Learning identification method having.

Images