JP7021010B2 - Machine learning system - Google Patents

Description

One aspect of the present invention relates to a machine learning system.

Conventionally, attempts have been made to execute machine learning using a neural network at high speed. For example, Patent Document 1 describes a method of reducing the dimension of a feature space by learning a quadratic function by a polynomial neural network and selecting a subspace for storing a main component of the quadratic function. This technique involves selecting one or more vectors that are principal components from an eigenvector and a coefficient vector and generating a subspace generated by the selected vector as a new feature space.

Japanese Unexamined Patent Publication No. 2010-39778

As the number of dimensions of the output layer of the neural network increases, the amount of calculation for obtaining the vector of the output layer becomes enormous, and the matrix operation in the output layer can greatly affect the speed of machine learning. Therefore, it is desired to execute machine learning at high speed even when the number of dimensions of the output layer is large.

The machine learning system according to one aspect of the present invention calculates an output vector, which is a vector of the output layer of the neural network, by using the intermediate vector obtained in the intermediate layer of the neural network and the transformation matrix A, and the output vector. It is provided with a prediction unit that predicts an event based on Yes, the matrix Σ is a diagonal matrix, a temporary vector is calculated based on the intermediate vector and the matrix V, and the first column of the matrix UΣ is used by using the value k indicating the division position of each of the matrix UΣ and the temporary vector. The prematrix defined using the kth column from to and the prematrix defined using the first to kth elements of the temporary vector are obtained, and the approximation vector is obtained based on the prematrix and the prematrix. Is calculated, and the approximation vector is set as the output vector.

In such an aspect, the singular value decomposition of A = UΣV is executed for the transformation matrix A for obtaining the output vector from the intermediate vector. Then, an approximate vector can be obtained by using a part of the matrix UΣ (prematrix) without using the entire matrix UΣ. This approximation vector can be said to be an approximation of the vector of the output layer. By considering this approximation vector as an output vector, the output vector can be obtained with a smaller amount of calculation than when the entire matrix UΣ (that is, the transformation matrix A itself) is used, so that machine learning can be executed at high speed.

According to one aspect of the present invention, machine learning can be executed at high speed even when the number of dimensions of the output layer of the neural network is large.

It is a figure which shows an example of the functional structure of the machine learning system which concerns on embodiment. It is a figure which shows an example of the neural network used in the machine learning system which concerns on embodiment. It is a figure which shows the conventional calculation method for obtaining an output vector. It is a figure which shows the calculation method in this embodiment for obtaining an output vector. It is a figure which shows the calculation method in this embodiment for obtaining an output vector. It is a flowchart which shows an example of the operation of the machine learning system which concerns on embodiment. It is a flowchart which shows an example of the operation of the machine learning system which concerns on embodiment. It is a figure which shows the application example of the machine learning system which concerns on embodiment. It is a figure which shows an example of the hardware composition of the computer used for the machine learning system which concerns on embodiment.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the description of the drawings, the same or equivalent elements are designated by the same reference numerals, and duplicate description will be omitted.

The machine learning system 10 according to the embodiment is a computer system that predicts an arbitrary event. Machine learning is a method of autonomously finding a law or rule by iteratively learning based on given information. The machine learning system 10 predicts an event by executing machine learning using a neural network. A neural network is a model of information processing that imitates the mechanism of the human cranial nerve system.

An event is a thing that appears in an observable form, in other words, a thing that can be expressed by data in any format. The events predicted by the machine learning system 10 are not limited in any way, and therefore the machine learning system 10 may be used for any purpose. The machine learning system 10 may predict an event in the real world or may predict an event in the virtual world. The machine learning system 10 may be used for a classification problem (discrimination problem) for determining which classification the data to be processed belongs to, or a regression problem for predicting unknown data (new data) from the data to be processed. May be used for. The machine learning system 10 can be used for various information processing such as natural language processing, image processing (image recognition), voice processing (voice recognition), and data prediction. For example, the machine learning system 10 is applied to various technical fields such as machine translation, automatic dialogue, optical character recognition (OCR), automatic driving, medical diagnosis, fraud detection, face detection, product recommendation, customer analysis, and financial transactions. can do. Therefore, there are various prediction results (predicted events) output as machine learning processing results, such as translations, texts recognized from voice or images, driving control instructions, diagnostic results, and detections. It can be fraudulent, recommended merchandise, financial transaction instructions, etc.

The machine learning system 10 may be composed of one computer or a plurality of computers. When a plurality of computers are used, one machine learning system 10 is logically constructed by connecting these computers via a communication network such as the Internet or an intranet.

FIG. 1 is a diagram showing an example of the functional configuration of the machine learning system 10. As shown in FIG. 1, the machine learning system 10 includes a prediction unit 11 as a functional element.

The prediction unit 11 is a functional element that predicts an event using a neural network. The prediction unit 11 acquires the input data to be processed, inputs the input data to the neural network, executes machine learning, and obtains output data (processing result).

The method of acquiring the input data is not limited. For example, the prediction unit 11 may read data stored in an arbitrary database as input data, or may receive data transmitted from another computer system as input data. Alternatively, the prediction unit 11 may acquire data processed by another functional element (not shown) in the machine learning system 10 as input data.

The processing method of output data (processing result) is also not limited. For example, the prediction unit 11 may display the output data on a monitor, store it in an arbitrary database, or transmit it to another computer system. Alternatively, another functional element (not shown) within the machine learning system 10 may further process the output data.

In this embodiment, it is premised that the prediction unit 11 uses a trained neural network (so-called trained model). The trained model can be said to be the best neural network estimated to have the highest prediction accuracy. Keep in mind, however, that the trained model is not always the “best in reality”. Generally, a training data set (corpus) containing one or more training samples is prepared to generate a trained model. The trained model can be obtained by performing machine learning while sequentially inputting individual training samples into the neural network to be trained. The trained model can be generated using any prior art. The trained model may be generated by the machine learning system 10 or another computer system.

It can be said that the trained model is a combination of a computer program and parameters. Alternatively, the trained model can be said to be a combination of the structure of the neural network and a parameter (weighting coefficient) which is the strength of the connection between the individual neurons of the neural network. Alternatively, the trained model can be said to be a computer program configured to obtain one result (perform a predetermined process).

FIG. 2 is a diagram schematically showing an example of a neural network (trained model) used in the machine learning system 10. It can be said that this neural network 12 is a part of the prediction unit 11. The neural network 12 is composed of a first layer which is an input layer, a second layer, a third layer, and a fourth layer which are intermediate layers (hidden layers), and a fifth layer which is an output layer. The first layer outputs the input vector x = (x ₀ , x ₁ , x ₂ , ... X _p ) having p parameters as elements to the second layer as it is. Each of the second layer, the third layer, and the fourth layer converts the total input into an output by the activation function and passes the output to the next layer. The fifth layer also converts the total input into an output by the activation function, and this output is the output vector y = (y ₀ , y ₁ , ..., Y _q ) of the neural network having q parameters as elements. The number of nodes (elements) in each layer is not limited, and may be set according to, for example, the characteristics of the data to be processed and the characteristics of the data to be obtained.

The neural network 12 has 5 layers (4 layers when the input layer is excluded), but the number of layers of the neural network constituting the machine learning system 10 (prediction unit 11) is not limited at all. For example, the machine learning system 10 may use a neural network having an arbitrary number of layers of 3 or more, which means that a neural network having an arbitrary number of layers of 1 or more may be used. ..

One of the features of the prediction unit 11 is a calculation method for obtaining a vector (output vector) of the output layer from a vector showing the result of the last intermediate layer. In the neural network 12, the fourth layer is the last intermediate layer. In the following, the vector showing the result of the last intermediate layer is referred to as "intermediate vector". The prediction unit 11 does not obtain an accurate output vector from the beginning, but first calculates an approximate value of the output vector. Then, the prediction unit 11 determines whether or not to use the approximate value as the final result. When it is determined that the approximate value is used, the prediction unit 11 sets the approximate value as an output vector. On the other hand, when it is determined that the approximate value is not adopted, the prediction unit 11 obtains an accurate output vector.

3 to 5 are diagrams for explaining the calculation of the output vector by the prediction unit 11. FIG. 3 is a diagram showing a conventional calculation method. 4 and 5 are diagrams showing a method of obtaining an approximate value of an output vector in the present embodiment.

The prediction unit 11 uses a transformation matrix of n rows and m columns (this is referred to as "n × m transformation matrix") in order to obtain an n-dimensional output vector from an m-dimensional intermediate vector. Assuming that the intermediate vector is x, the output vector is y, and the transformation matrix is A, an accurate output vector can be obtained with y = Ax, as shown in FIG. Since a matrix operation of n × m is performed in order to obtain the output vector y, the amount of calculation becomes enormous if the dimension of the output layer is large. For example, even if the dimension of the intermediate vector is 500, if the dimension of the output vector is 50,000, a matrix operation of 500 × 50,000 is required to obtain the output vector. When the dimension of the output layer becomes large due to a large number of classification candidates in the classification problem (discrimination problem), the matrix operation for obtaining the output layer tends to dominate the calculation of the neural network.

In the present embodiment, the prediction unit 11 uses singular value decomposition (SVD), which is a method of matrix factorization, in order to execute the matrix operation at high speed. As shown in FIG. 4, the prediction unit 11 decomposes the transformation matrix A into the matrix UΣ and the matrix V by executing the singular value decomposition represented by A = UΣV on the transformation matrix A. Both the matrix U and the matrix V are orthogonal matrices. The matrix Σ is a diagonal matrix, and more specifically, the off-diagonal component is 0 and the diagonal component ((i, i) element) is composed of a singular value (singular value of the transformation matrix A). It is a matrix. The matrix UΣ is the product of the matrix U and the matrix Σ. The n × m transformation matrix A is decomposed into an n × n matrix U, an n × m matrix Σ, and an m × m matrix V. Since the transformation matrix A that constitutes a part of the neural network (trained model) is given in advance, the prediction unit 11 can acquire the matrix UΣ and the matrix V by executing the singular value decomposition in advance.

In the prediction unit 11, the diagonal component of the matrix Σ is located so that the important element (element that affects the calculation) is located in the front column of the matrix Σ and the element that does not affect the calculation is located in the rear column. Line up. Specifically, the prediction unit 11 determines the diagonal of the matrix Σ so that each diagonal component included in the front column is equal to or larger than the maximum value of the diagonal component in the rear column. Arrange the ingredients. In short, the prediction unit 11 arranges the diagonal components of the matrix Σ so that the diagonal components having large values are gathered in the front row. The "front column" means the first column to the kth column of the matrix Σ, and the "rear column" means the (k + 1) column to the last column of the matrix Σ. The value k is 1 or more and smaller than the number of columns of the determinant Σ. For example, the prediction unit 11 may generate the matrix Σ so that the diagonal components are arranged in descending order from the first column to the last column.

The prediction unit 11 obtains an approximate value of the output vector using the matrix UΣ and the matrix V. As shown in FIG. 5, the prediction unit 11 divides the matrix UΣ into a front matrix L and a back matrix R. The prematrix L is a matrix defined using the first to kth columns of the matrix UΣ (that is, the column in front of the matrix UΣ), and is therefore an n × k matrix. The back matrix R is a matrix composed of the remaining columns of the matrix UΣ (that is, the columns behind the matrix UΣ). More specifically, the back matrix R is a matrix defined by using the (k + 1) th column to the last column of the matrix UΣ, and is therefore an n × (m−k) matrix. It can be said that the value k is a value indicating the division position of the matrix UΣ.

Further, the prediction unit 11 obtains an m-dimensional temporary vector x'based on the matrix V and the intermediate vector x. Specifically, the prediction unit 11 obtains the product of the matrix V and the intermediate vector x as a temporary vector x'. That is, x'= Vx. The prediction unit 11 divides this temporary vector x ′ into a front vector x _L and a rear vector x _R. The prevector x _L is a vector defined by using the first to kth elements of the temporary vector x', and is therefore a k-dimensional vector. The posterior vector x _R is a vector composed of the remaining elements of the temporary vector x'. More specifically, the posterior vector x _R is a matrix defined using the (k + 1) th element to the last element (m th element) of the temporary vector x', and therefore (mk + 1). ) It is a vector of dimensions. It can be said that the value k is also a value indicating the division position of the temporary vector x'.

Subsequently, the prediction unit 11 obtains an n-dimensional approximate vector ya _a based on the front matrix L and the front vector x _L. Specifically, the prediction unit 11 obtains the product of the front matrix L and the front vector _{x L} as the approximation vector ya. That is, ya ₌ Lx _L. The approximation vector y _a is an approximate value of the accurate output vector y obtained by y = Ax.

Since the approximation vector y _a is obtained by using only the important element (prematrix L) of the transformation matrix A, it can be expected that the accurate output vector y is approximated with high accuracy. Specifically, it is highly probable that the index of the maximum element of the approximation vector y _a is the same as the index of the maximum element of the accurate output vector y. Here, the maximum element means the element having the largest value. Further, the index means an element number indicating the position of the element. For example, in a classification problem (discrimination problem), it is sufficient to know the index of the maximum element. Therefore, if the index of the maximum element does not change even in the approximate calculation of ya ₌ Lx _L , the classification result (discrimination result) is the same as the case where y = Ax is calculated. Since the approximate calculation means that only a part of the matrix operation of y = Ax is calculated, the execution time of machine learning can be shortened by considering the approximate vector y _a as the output vector y.

On the other hand, when the approximation vector y _a does not approximate the output vector y, the omitted data (back matrix R and back vector x _R ) can be further used to obtain an accurate output vector y (correct classification result). Obtainable.

The operation of the machine learning system 10 will be described with reference to FIGS. 6 and 7. FIG. 6 is a flowchart showing an example of the process executed when the trained model is acquired. FIG. 7 is a flowchart showing an example of a process of obtaining an output vector from an intermediate vector.

The process executed when the trained model is acquired will be described with reference to FIG. In step S11, the prediction unit 11 acquires the trained model. As described above, this trained model is a neural network composed of the transformation matrix A.

In step S12, the prediction unit 11 decomposes the transformation matrix A into a matrix UΣ and a matrix V by singular value decomposition. That is, the prediction unit 11 calculates A = UΣ × V. If the transformation matrix A is an n × m matrix, the matrix UΣ is an n × m matrix, and the matrix V is an m × m matrix.

In step S13, the prediction unit 11 divides the matrix UΣ into a front matrix L and a back matrix R. In the prediction unit 11, each diagonal component included in the front column (first column to kth column) is the same as or the maximum value of the diagonal component in the remaining column ((k + 1) column to last column). Arrange the diagonal components of the matrix Σ so that they are larger than the maximum value. For example, the prediction unit 11 may arrange the diagonal components in descending order. The method for determining the value k indicating the division position of the matrix UΣ is not limited. The value k may be predetermined or may be dynamically (that is, automatically) determined by the prediction unit 11.

For example, the value k may be half the number of dimensions (the number of columns of the matrix Σ) m of the intermediate vector. For example, if the number of dimensions m is an even number, k = m / 2. If the number of dimensions m is an odd number, k = (m-1) / 2 or k = (m + 1) / 2 may be used. In the present embodiment, even in this case where the dimension number m is an odd number, it is included in the example in which the value k is half the dimension number of the intermediate vector.

Alternatively, the prediction unit 11 arranges the diagonal components of the matrix Σ in descending order from the first column to the last column, and then the column number of the last column satisfying that the diagonal components are equal to or higher than the predetermined threshold value Ta. May be set as k. For example, assuming that the diagonal components are arranged in descending order, m = 100, and Ta = 1, the diagonal components in the 60th column are 1 or more, and the diagonal components in the 61st column are less than 1. In some cases, the prediction unit 11 sets k to 60. The specific value of the threshold value Ta is not limited, and may be set in consideration of various factors such as the characteristics of the trained model and the characteristics of the event to be predicted.

Alternatively, the prediction unit 11 arranges the diagonal components of the matrix Σ in descending order from the first column to the last column, and then the last column satisfying that the deviation value of the diagonal components is equal to or higher than the predetermined threshold value Tb. The column number of may be set as k. For example, assuming that the diagonal components are arranged in descending order, m = 100, and Tb = 50, the deviation value of the diagonal components in the 40th column is 50 or more, and the diagonal components in the 41st column If the deviation value is less than 50, the prediction unit 11 sets k to 40. The deviation value of each diagonal component can be determined using the mean and variance of all diagonal components. The specific value of the threshold value Tb is not limited, and may be set in consideration of various factors such as the characteristics of the trained model and the characteristics of the event to be predicted.

In this way, the method of determining the value k is not limited, but in any case, the prediction unit 11 arranges the diagonal components of the matrix Σ so that the diagonal components having a large value gather in the front column, and then arranges the diagonal components of the matrix Σ, and then the matrix UΣ. Is divided into a front matrix L and a back matrix R. If the matrix UΣ is an n × m matrix, the front matrix L is an n × k matrix and the rear matrix R is an n × (m−k) matrix.

The calculation of the output vector will be described with reference to FIG. 7. FIG. 7 shows a process of obtaining one output vector. It may be necessary to obtain the output vector multiple times in order to solve one problem using the neural network 12. In this case, a series of processes shown in FIG. 7 is executed a plurality of times in order to process the one problem.

In step S21, the prediction unit 11 obtains a temporary vector x'based on the matrix V and the intermediate vector x. Specifically, the prediction unit 11 obtains the product of the matrix V and the intermediate vector x as a temporary vector x'.

In step S22, the prediction unit 11 divides the temporary vector x ′ into a front vector x _L and a rear vector x _R. The prediction unit 11 generates a front vector x _L using the first to kth elements of the temporary vector x', and the last element (m pieces) from the (k + 1) th element of the temporary vector x'. The rear vector x _R is generated using the eye element). The value k for dividing the temporary vector x'in this way is the same as the value k used when dividing the matrix UΣ into the front matrix L and the back matrix R. Therefore, the prediction unit 11 also uses the value k set in step S13 above for the division of the temporary vector x'.

In step S23, the prediction unit 11 obtains the approximation vector ya _a based on the front matrix L and the front vector x _L. Specifically, the prediction unit 11 obtains the product of the front matrix L and the front vector _{x L} as the approximation vector ya.

In step S24, the prediction unit 11 has the maximum element (maximum value of the element of the approximation _{vector ya} ) and at least one other element (at least one of the elements of the approximation vector _ya other than the maximum element) in the approximation vector ya. Calculate the degree of deviation from one). In step S25, the prediction unit 11 compares the degree of deviation with a predetermined threshold value. The degree of divergence is an index showing how far the maximum element of the approximation vector _ya is from the values of other elements of the approximation vector _ya . It can be said that the larger the degree of divergence, the larger the maximum value of the element of the approximation vector _ya is from the values of other elements. If the degree of divergence is greater than a certain level, it can be said that there is a significant difference between the maximum element and other elements.

The type of deviation degree used in steps S24 and S25 is not limited. _For example, the prediction unit 11 obtains the difference between the maximum element of the approximation vector _ya and the second largest element in the approximation vector ya as the degree of divergence, and determines whether or not the degree of divergence is larger than the threshold value Tc. You may. Alternatively, the prediction unit 11 may obtain the deviation value of the maximum element of the approximation vector _ya as the degree of deviation and determine whether or not the degree of deviation is larger than the threshold value Td. This deviation value can be obtained by using the mean and variance of all the elements of the approximation vector _ya . The specific values of the threshold values Tc and Td are not limited, and may be set in consideration of various factors such as the characteristics of the trained model and the characteristics of the event to be predicted.

If the degree of deviation is larger than the threshold value (YES in step S25), the process proceeds to step S26, and the prediction unit 11 sets the approximation vector y _a as the output vector y. If the degree of deviation is larger than the threshold value, it is highly probable that the index of the maximum element of the approximation vector y _a is the same as the index of the maximum element of the accurate output vector y. For example, in the classification problem (discrimination problem), it is sufficient to know the index of the maximum element, so if the degree of deviation is larger than the threshold value, the classification result (discrimination result) by the approximation vector _ya is different from the classification result by the accurate output vector y. You can expect not to.

If the degree of deviation is equal to or less than the threshold value (NO in step S25), the process proceeds to step S27. In step S27, the prediction unit 11 calculates the output vector y by further using the back matrix R and the back vector _{x R} in addition to the approximation vector ya. Specifically, the prediction unit 11 obtains an accurate output vector y by adding the product of the back matrix R and the back vector x _R to the approximation vector y _a . That is, the prediction unit 11 calculates y ₌ ya + Rx _R. If the degree of deviation is equal to or less than the threshold value, it is highly probable that the index of the maximum element of the approximation vector y _a is different from the index of the maximum element of the accurate output vector y. In this case, the accuracy of the prediction is higher when the output vector y is calculated accurately instead of adopting the approximate vector y _a as the final result. The prediction unit 11 further obtains an accurate output vector y by further using the omitted back matrix R and the back vector x _R.

The prediction unit 11 calculates or generates the final result of the neural network based on the output vector y obtained by these series of processes. There are no restrictions on how the final result is calculated or generated.

For example, the prediction unit 11 may obtain the final result by using the Softmax function represented by the following equation (1).

式（１）において、ｙ_ｉは出力ベクトルｙのｉ番目の要素を表し、ｎは出力ベクトルｙの要素数を表す。

In the equation (1), y _i represents the i-th element of the output vector y, and n represents the number of elements of the output vector y.

This softmax function transforms the elements of the output vector into a probability distribution. By the softmax function, each element of the output vector takes a value between 0 and 1, and the sum of all the elements of the output vector is 1. Generally, this softmax function is often used to solve a classification problem (discrimination problem).

Alternatively, the prediction unit 11 may set the output vector y as it is as the final result. For example, the prediction unit 11 may output the output vector y as it is as the final result when solving the regression problem.

Since the transformation matrix A is fixed in the trained model, the prediction unit 11 may perform the singular value decomposition only once when the trained model is read to obtain the front matrix L and the back matrix. Therefore, it is not necessary to obtain the front matrix L and the back matrix R each time an individual output vector is to be obtained.

In the present embodiment, the machine learning system 10 (prediction unit 11) acquires the matrix UΣ and the matrix V by singularly decomposing the transformation matrix A, but the machine learning system 10 (prediction unit 11) is a different computer system. The calculated matrix UΣ and the matrix V may be acquired. That is, the other computer system may decompose the transformation matrix A into singular values.

In the present embodiment, the machine learning system 10 (prediction unit 11) sets the approximation vector y _a as the output vector y based on the degree of deviation with respect to the approximation vector y _a , or the accurate output vector y ₌ ya. Find + Rx _R. However, this branching process based on the degree of divergence is not essential. Therefore, the machine learning system 10 (prediction unit 11) may set the approximation vector y _a as the output vector y without obtaining the degree of deviation.

When comparing the magnitude relations of two numerical values in the machine learning system 10, either of the two criteria of "greater than or equal to" and "greater than or equal to" may be used, and the two criteria of "less than or equal to" and "less than" are used. Either of the criteria may be used. The selection of such criteria does not change the technical significance of the process of comparing the magnitude relations of two numbers.

By using the approximate value of the output vector as in the present embodiment, it is possible to execute machine learning at high speed even when the dimension of the output layer is enormous. This technical effect will be described with reference to FIG. FIG. 8 is a diagram schematically showing an actual example in which this embodiment is applied (applied) to machine translation using a neural network 20 called LSTM (Long Short-Term Memory).

In natural language processing such as translation, the number of vocabularies can be the number of output candidates, so the number of dimensions of the vector of the output layer is tens of thousands or more (for example, about 50,000) corresponding to the number of vocabularies. As a result, the amount of calculation of the output vector becomes enormous. For example, even if the number of dimensions of the intermediate vector is about 500, a matrix operation of 500 × 50,000 is required to calculate the output vector, and this matrix operation can be dominant in the calculation of the neural network.

In the example of FIG. 8, the neural network (LSTM) 20 translates a Japanese sentence into English. In this neural network 20, it is assumed that the number of dimensions of the intermediate layer and the output layer are 500 and 50,000, respectively. In the example of FIG. 8, the Japanese sentence "I am Japanese." Is translated as "I am Japanese." Since the matrix operation for obtaining the output vector from the intermediate vector is executed for each word constituting the English sentence (including the sentence ending symbol <EOS>), the matrix operation is executed four times in the example of FIG. To. The number of times that the approximation vector y _a is set as the output vector y in the four matrix operations is between 0 and 4.

In the neural network 20 having a 500-dimensional intermediate layer and a 50,000-dimensional output layer, the value k for dividing the 500-column matrix UΣ into the front matrix L and the rear matrix R is set to a fixed value of 300. Therefore, the number of columns in the front matrix L and the back matrix R was 300 and 200, respectively. As an example, when the Japanese sentence "If you go to work by bicycle, you will get exercise." Is translated into English by this neural network 20 to which this embodiment is applied, "If you go to work by bicycle, you will get exercise." I was able to obtain the correct English translation in 268 ms (milliseconds).

For comparison, the Japanese translation was always performed using all the columns without dividing the matrix UΣ, and the correct English translation was obtained in 302 ms. When the Japanese sentence was intentionally translated into English using only the front matrix L without using the back matrix R, the result changed according to the number of columns k of the front matrix L. Specifically, at k = 400, a correct English translation was obtained in 277 ms. At k = 300, a result close to the correct answer, "If you go to work by bicycle, you will exercise.", Was obtained in 229 ms. In the case of k = 200, an incomplete result of "When you go to work by bicycle, you can exercise." Was obtained in 192 ms. In the case of k = 100, the mistranslation "To go to work on a bike is a sport." Was obtained in 173 ms.

As can be seen from the experiment using the neural network 20, by adopting the machine learning system 10 according to the present embodiment, it is possible to obtain highly accurate results at high speed. In the above translation example, only when the discrimination result is ambiguous in the calculation using only the front matrix L having 300 columns, the calculation is further performed using the back matrix R having the remaining 200 columns. Therefore, the correct answer could be obtained in a short time of 268 ms.

The block diagram used in the description of the above embodiment shows a block of functional units. These functional blocks (components) are realized by any combination of hardware and / or software. Further, the means for realizing each functional block is not particularly limited. That is, each functional block may be realized by one physically and / or logically coupled device, or directly and / or indirectly by two or more physically and / or logically separated devices. It may be connected specifically (eg, wired and / or wireless) and realized by these plurality of devices.

For example, the machine learning system 10 in one embodiment of the present invention may function as a computer for processing the present embodiment. FIG. 9 is a diagram showing an example of a hardware configuration of a computer 100 that functions as a machine learning system 10. The computer 100 may physically include a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, and the like.

In the following description, the word "device" can be read as a circuit, a device, a unit, or the like. The hardware configuration of the machine learning system 10 may be configured to include one or more of the devices shown in the figure, or may be configured not to include some of the devices.

For each function in the machine learning system 10, by loading predetermined software (program) on hardware such as the processor 1001 and the memory 1002, the processor 1001 performs an calculation, and communication by the communication device 1004, the memory 1002, and the storage are performed. It is realized by controlling the reading and / or writing of data in 1003.

Processor 1001 operates, for example, an operating system to control the entire computer. The processor 1001 may be composed of a central processing unit (CPU) including an interface with a peripheral device, a control device, an arithmetic unit, a register, and the like. For example, at least a part of the functional elements of the machine learning system 10 may be realized by the processor 1001.

Further, the processor 1001 reads a program (program code), a software module and data from the storage 1003 and / or the communication device 1004 into the memory 1002, and executes various processes according to these. As the program, a program that causes a computer to execute at least a part of the operations described in the above-described embodiment is used. For example, at least a part of the functional elements of the machine learning system 10 may be realized by a control program stored in the memory 1002 and operated by the processor 1001, and may be realized for other functional blocks as well. Although it has been described that the above-mentioned various processes are executed by one processor 1001, they may be executed simultaneously or sequentially by two or more processors 1001. Processor 1001 may be mounted on one or more chips. The program may be transmitted from the network via a telecommunication line.

The memory 1002 is a computer-readable recording medium, and is composed of at least one such as a ROM (Read Only Memory), an EPROM (Erasable Programmable ROM), an EEPROM (Electrically Erasable Programmable ROM), and a RAM (Random Access Memory). May be done. The memory 1002 may be referred to as a register, a cache, a main memory (main storage device), or the like. The memory 1002 can store a program (program code), a software module, and the like that can be executed to implement the wireless communication method according to the embodiment of the present invention.

The storage 1003 is a computer-readable recording medium, and is, for example, an optical disk such as a CDROM (Compact Disc ROM), a hard disk drive, a flexible disk, a photomagnetic disk (for example, a compact disk, a digital versatile disk, or a Blu-ray (registration)). It may consist of at least one such as a (trademark) disk), a smart card, a flash memory (eg, a card, stick, key drive), a floppy (registered trademark) disk, a magnetic strip, and the like. The storage 1003 may be referred to as an auxiliary storage device. The storage medium described above may be, for example, a table, server or other suitable medium containing memory 1002 and / or storage 1003.

The communication device 1004 is hardware (transmission / reception device) for communicating between computers via a wired and / or wireless network, and is also referred to as, for example, a network device, a network controller, a network card, a communication module, or the like. For example, at least a part of the functional elements of the machine learning system 10 may be realized by the communication device 1004.

The input device 1005 is an input device (for example, a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that receives an input from the outside. The output device 1006 is an output device (for example, a display, a speaker, an LED lamp, etc.) that performs output to the outside. The input device 1005 and the output device 1006 may have an integrated configuration (for example, a touch panel).

Further, each device such as the processor 1001 and the memory 1002 is connected by a bus 1007 for communicating information. The bus 1007 may be composed of a single bus or may be composed of different buses between the devices.

Further, the computer 100 is configured to include hardware such as a microprocessor, a digital signal processor (DSP: Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (ProgrammableLogic Device), and an FPGA (Field Programmable Gate Array). However, the hardware may realize a part or all of each functional block. For example, the processor 1001 may be implemented on at least one of these hardware.

As described above, the machine learning system according to one aspect of the present invention uses the intermediate vector obtained in the intermediate layer of the neural network and the transformation matrix A to obtain an output vector which is a vector of the output layer of the neural network. A prediction unit that calculates and predicts an event based on the output vector is provided, and the prediction unit acquires the matrix UΣ and the matrix V obtained by decomposing the transformation matrix A into singular values, and here, the matrix U and the matrix V are obtained. The matrix V is an orthogonal matrix, the matrix Σ is a diagonal matrix, a temporary vector is calculated based on the intermediate vector and the matrix V, and the values k indicating the respective division positions of the matrix UΣ and the temporary vector are used. Obtain the prematrix defined using the first to kth columns of the matrix UΣ and the prematrix defined using the first to kth elements of the temporary vector, and obtain the prematrix and the prematrix. Calculate the approximation vector based on the vector and set the approximation vector as the output vector.

In such an aspect, the singular value decomposition of A = UΣV is executed for the transformation matrix A for obtaining the output vector from the intermediate vector. Then, an approximate vector can be obtained by using a part of the matrix UΣ (prematrix) without using the entire matrix UΣ. This approximation vector can be said to be an approximation of the vector of the output layer. By considering this approximation vector as an output vector, the output vector can be obtained with a smaller amount of calculation than when the entire matrix UΣ (that is, the transformation matrix A itself) is used, so that machine learning can be executed at high speed.

In the machine learning system related to the other aspect, the prediction unit calculates the degree of deviation between the maximum element in the approximation vector and at least one other element, and sets the approximation vector as an output vector when the degree of deviation is larger than the threshold value. You may. If the degree of divergence is large, it can be said that the difference between the maximum element and other elements is large, and therefore it is highly probable that the index of the maximum element is the same between the approximate vector and the accurate output vector. By setting the approximation vector as the output vector when the degree of deviation is large, highly accurate machine learning can be executed at high speed.

In the machine learning system according to the other aspect, when the degree of deviation is less than or equal to the threshold value, the predictor is composed of the posterior matrix R composed of the remaining columns of the matrix UΣ and the posterior matrix composed of the remaining elements of the temporary vector. The output vector may be calculated based on the vector and the approximation vector. When the degree of divergence is small, the difference between the maximum element and other elements is not so large, so it is highly probable that the index of the maximum element differs between the approximate vector and the accurate output vector. By accurately obtaining the output vector only when the degree of deviation is small, highly accurate machine learning can be executed at high speed.

In the machine learning system according to the other aspect, the prediction unit may calculate the deviation value of the maximum element of the approximation vector as the degree of deviation. By using the deviation value, which is a kind of statistical value, as the degree of deviation, it is possible to correctly estimate how far the maximum element is from other elements.

In the machine learning system according to the other aspect, the prediction unit may calculate the difference between the maximum element of the approximation vector and the second largest element in the approximation vector as the degree of deviation. By using the difference between the maximum element and the second largest element as the degree of divergence, the degree of divergence can be easily obtained.

In the machine learning system according to the other aspect, the prediction unit may set a value of half the number of dimensions of the temporary vector as the value k. By setting the division position in this way, the prematrix and the prevector can be easily obtained from the matrix UΣ and the temporary vector.

In the machine learning system according to the other aspect, the prediction unit may set the column number of the last column satisfying that the diagonal component of the matrix Σ is equal to or greater than the threshold value as the value k. By setting the division positions of the matrix UΣ and the temporary vector in this way, important elements that affect the calculation are gathered in the prematrix, so that a highly accurate approximate vector can be obtained.

In the machine learning system according to the other aspect, the prediction unit may set the column number of the last column satisfying that the deviation value of the diagonal component of the matrix Σ is equal to or greater than the threshold value as the value k. By setting the division positions of the matrix UΣ and the temporary vector in this way, important elements that affect the calculation are gathered in the prematrix, so that a highly accurate approximate vector can be obtained.

Although the present embodiment has been described in detail above, it is clear to those skilled in the art that the present embodiment is not limited to the embodiments described in the present specification. This embodiment can be implemented as an amendment or modification without departing from the spirit and scope of the present invention as determined by the description of the scope of claims. Therefore, the description herein is for purposes of illustration only and has no limiting implications for this embodiment.

Notification of information is not limited to the embodiments and embodiments described herein, and may be performed by other methods. For example, information notification includes physical layer signaling (eg, DCI (Downlink Control Information), UCI (Uplink Control Information)), higher layer signaling (eg, RRC (Radio Resource Control) signaling, MAC (Medium Access Control) signaling, etc. It may be carried out by broadcast information (MIB (Master Information Block), SIB (System Information Block)), other signals, or a combination thereof. Further, the RRC signaling may be referred to as an RRC message, and may be, for example, an RRC Connection Setup message, an RRC Connection Reconfiguration message, or the like.

Each aspect / embodiment described herein includes LTE (Long Term Evolution), LTE-A (LTE-Advanced), SUPER 3G, IMT-Advanced, 4G, 5G, FRA (Future Radio Access), W-CDMA. (Registered Trademarks), GSM (Registered Trademarks), CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi), IEEE 802.16 (WiMAX), IEEE 802.20, UWB (Ultra-Wideband), It may be applied to Bluetooth®, other systems that utilize suitable systems and / or next-generation systems that are extended based on them.

The processing procedures, sequences, flowcharts, and the like of each aspect / embodiment described in the present specification may be rearranged in order as long as there is no contradiction. For example, the methods described herein present elements of various steps in an exemplary order and are not limited to the particular order presented.

Information and the like can be output from the upper layer (or lower layer) to the lower layer (or upper layer). Input / output may be performed via a plurality of network nodes.

The input / output information and the like may be stored in a specific place (for example, a memory) or may be managed by a management table. Information to be input / output may be overwritten, updated, or added. The output information and the like may be deleted. The input information or the like may be transmitted to another device.

The determination may be made by a value represented by 1 bit (0 or 1), by a boolean value (Boolean: true or false), or by comparing numerical values (for example, a predetermined value). It may be done by comparison with the value).

Each aspect / embodiment described in the present specification may be used alone, in combination, or may be switched and used according to the execution. Further, the notification of predetermined information (for example, the notification of "being X") is not limited to the explicit one, but is performed implicitly (for example, the notification of the predetermined information is not performed). May be good.

Software, whether referred to as software, firmware, middleware, microcode, hardware description language, or other names, is an instruction, instruction set, code, code segment, program code, program, subprogram, software module. , Applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, features, etc. should be broadly interpreted.

Further, software, instructions, and the like may be transmitted and received via a transmission medium. For example, the software uses wired technology such as coaxial cables, fiber optic cables, twisted pair and digital subscriber lines (DSL) and / or wireless technologies such as infrared, wireless and microwave to websites, servers, or other When transmitted from a remote source, these wired and / or wireless technologies are included within the definition of transmission medium.

The information, signals, etc. described herein may be represented using any of a variety of different techniques. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description are voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, light fields or photons, or any of these. It may be represented by a combination of.

The terms described herein and / or the terms necessary for understanding the present specification may be replaced with terms having the same or similar meanings.

The terms "system" and "network" used herein are used interchangeably.

Further, the information, parameters, etc. described in the present specification may be represented by an absolute value, a relative value from a predetermined value, or another corresponding information. .. For example, the radio resource may be indexed.

The names used for the parameters described above are not limited in any way. Further, mathematical formulas and the like using these parameters may differ from those expressly disclosed herein. Since the various channels (eg, PUCCH, PDCCH, etc.) and information elements (eg, TPC, etc.) can be identified by any suitable name, the various names assigned to these various channels and information elements are in any respect. However, it is not limited.

User terminals and mobile communication terminals may be used by those skilled in the art as subscriber stations, mobile units, subscriber units, wireless units, remote units, mobile devices, wireless devices, wireless communication devices, remote devices, mobile subscriber stations, access terminals, etc. It may also be referred to as a mobile device, wireless device, remote device, handset, user agent, mobile client, client, or some other suitable term.

As used herein, the terms "determining" and "determining" may include a wide variety of actions. “Judgment” and “decision” are, for example, judgment, calculation, computing, processing, deriving, investigating, and looking up (for example, a table). , Searching in a table or another data structure), ascertaining can be regarded as "judgment" or "decision". Also, "judgment" and "decision" are receiving (for example, receiving information), transmitting (for example, transmitting information), input (input), output (output), and access. (Accessing) (for example, accessing data in memory) may be regarded as "judgment" or "decision". In addition, "judgment" and "decision" are considered to be "judgment" and "decision" when the things such as solving, selecting, selecting, establishing, and comparing are regarded as "judgment" and "decision". Can include. That is, "judgment" and "decision" may include considering some action as "judgment" and "decision".

The terms "connected", "coupled", or any variation thereof, mean any direct or indirect connection or connection between two or more elements and each other. It can include the presence of one or more intermediate elements between two "connected" or "joined" elements. The connection or connection between the elements may be physical, logical, or a combination thereof. As used herein, the two elements are by using one or more wires, cables and / or printed electrical connections, and, as some non-limiting and non-comprehensive examples, radio frequencies. By using electromagnetic energies such as electromagnetic energies with wavelengths in the region, microwave region and light (both visible and invisible) regions, they can be considered to be "connected" or "coupled" to each other.

The phrase "based on" as used herein does not mean "based on" unless otherwise stated. In other words, the statement "based on" means both "based only" and "at least based on".

As used herein by designations such as "first", "second", etc., any reference to that element does not generally limit the quantity or order of those elements. These designations can be used herein as a convenient way to distinguish between two or more elements. Thus, references to the first and second elements do not mean that only two elements can be adopted there, or that the first element must somehow precede the second element.

As long as "include", "including", and variations thereof are used herein or within the scope of the claims, these terms are similar to the term "comprising". In addition, it is intended to be inclusive. Moreover, the term "or" as used herein or in the claims is intended to be non-exclusive.

In the present specification, a plurality of devices shall be included unless the device has only one device apparently in context or technically.

10 ... Machine learning system, 11 ... Predictor, 12 ... Neural network.

Claims (8)

Using the intermediate vector obtained in the intermediate layer of the neural network and the transformation matrix A, an output vector which is a vector of the output layer of the neural network is calculated, and a prediction unit for predicting an event based on the output vector is provided. ,
The prediction unit
The matrix UΣ and the matrix V obtained by decomposing the transformation matrix A by singular values are obtained, where the matrix U and the matrix V are orthogonal matrices, and the matrix Σ is a diagonal matrix.
A temporary vector is calculated based on the intermediate vector and the matrix V,
Using the values k indicating the division positions of the matrix UΣ and the temporary vector, the prematrix defined using the first to kth columns of the matrix UΣ and the first to k of the temporary vector. Get the pre-vector defined using the elements up to the th item,
Calculate the approximation vector based on the prematrix and the prevector,
Setting the approximation vector as the output vector,
Machine learning system. The prediction unit
The degree of divergence between the maximum element in the approximation vector and at least one other element is calculated.
When the degree of deviation is larger than the threshold value, the approximation vector is set as the output vector.
The machine learning system according to claim 1. When the degree of deviation is equal to or less than the threshold value, the prediction unit approximates the posterior matrix R composed of the remaining columns of the matrix UΣ and the posterior vector composed of the remaining elements of the temporary vector. Calculate the output vector based on the vector,
The machine learning system according to claim 2. The prediction unit calculates the deviation value of the maximum element of the approximation vector as the degree of deviation.
The machine learning system according to claim 2 or 3. The prediction unit calculates the difference between the maximum element of the approximation vector and the second largest element in the approximation vector as the degree of deviation.
The machine learning system according to claim 2 or 3. The prediction unit sets a value that is half the number of dimensions of the temporary vector as the value k.
The machine learning system according to any one of claims 1 to 5. The prediction unit sets the column number of the last column satisfying that the diagonal component of the matrix Σ is equal to or greater than the threshold value as the value k.
The machine learning system according to any one of claims 1 to 5. The prediction unit sets the column number of the last column satisfying that the deviation value of the diagonal component of the matrix Σ is equal to or greater than the threshold value as the value k.
The machine learning system according to any one of claims 1 to 5.

Images