KR20180120061A - Artificial neural network model learning method and deep learning system - Google Patents

Abstract

According to the present invention, disclosed are an artificial neural network model learning method and a deep learning system which allow a user to efficiently learn specific data without defined maximum and minimum values. According to one aspect of the present invention, the deep learning system comprises: an acquisition module which acquires at least one learning data used for learning an artificial neural network model wherein each learning data comprises M feature values corresponding to each of first to M^th features (M is an integer of two or more); and a learning module which learns the artificial neural network model by using the at least one learning data. The artificial neural network model comprises an input layer, a modification function layer, at least one inner product layer, and an output layer. The modification function layer includes K modification function nodes (K is an integer of two or more). A k^th modification function node (k is an arbitrary natural number which satisfies 1 <= k <= K) receives an M-dimensional vector corresponding to the learning data from the input layer, applies a specific k^th modification function to each feature value included in the input M-dimensional vector, generates an M-dimensional modified value vector, and outputs the generated M-dimensional modified value vector to each node of the uppermost inner product layer.

Description

{Artificial neural network model learning method and deep learning system}

The present invention relates to an artificial neural network model learning method and a deep learning system, and more particularly, to an artificial neural network model learning method and a deep learning system capable of efficiently learning specific data for which a minimum value and a maximum value are not defined.

Hereinafter, the basic theory related to machine learning, the conventional machine learning algorithm, and the deep running will be described in order to facilitate understanding of the invention.

1. Classification of Machine Learning

1.1 Map Learning

Supervised learning is a type of machine learning that obtains a reasoning function to obtain a desired value from learning data. The learning data has input data and a corresponding correct label (label). The Supervised Learning algorithm generates a function that can analyze and guess the learning data. This function is called Classifier (when the output value is discontinuous) or Regression (when the output value is continuous, it is regression reference).

Fig. 1 is a diagram relating to classification of map learning.

For example, in Fig. 1, when a vector x is input, predicting a hamburger using a reasoning function f (x) is called Supervised Learning.

1.1.1 Classification

Classification means that y _n takes a discrete value when finding the reasoning function f: R ^D → R based on the learning data (x _n , y _n ). Generally, learning data consists of input object x _n ∈ R ^D and output value y _n ∈ R. The classification problem is divided into binary classification and multi-variable classification. In the case of binary classification, the output value has two kinds, and the multi-variable classification has two or more kinds of output values.

2 is a diagram showing an example of classification.

1.1.2 Regression Reasoning

The regression model has continuous values of y _n when the inference function f: R ^D → R is obtained with the learning data (x _n , y _n ). In general, the learning data consists of the input object x _n ∈ R ^D and the output value y _n ∈ R, and a loss function representing the difference between the predicted value and the actual value is required. In general problem, the following squared loss function L is mainly used .

The regression problem is divided into linear and nonlinear, in which linear functions are used for linear problems and nonlinear functions are used for inferring the results.

Figure 3 is an example of linear regression and nonlinear regression.

1.2 Bid map learning

Unsupervised learning is a technique that learns how to classify a particular input pattern in such a way that the model learns the statistical structure of the input pattern. Unlike Supervised Learning and Reinforcement Learning, there is no input for compensating for the predicted value or the correct answer to the problem. Thus, non-instructional learning is used to find the structural characteristics of the input or the relationship between the different inputs rather than finding the answer to the input.

4 is a diagram of an example of Unsupervised Learning.

For example, as shown in FIG. 4, hamburger photographs without labels in a model are provided as learning data. At this time, f (x) is a model for non-map learning, and the model finds a common set of burgers with only bread and patty, a hamburger with various toppings, and a hamburger set.

1.2.1 Clustering

5 is a diagram of an example of clustering. Clustering finds similarities between each input value, as shown in Figure 5, and groups input values based on this similarity.

Similar to Classification, it is used to classify each input value, but there is a difference in classifying inputs into groups that have not been defined before. Representative examples of clustering include k-Means Clustering and EM Clustering.

2. Traditional Machine Learning Algorithm

2.1 Classification Algorithm

Typical classification algorithms include k-nearest neighbors (k-NN), neural networks, and support vector machines (SVM) [43]. This section briefly describes the operation principle of the representative classification algorithm.

2.1.1 k-Nearest Neighbor

The k-Nearest Neighbor (k-NN) is one of the simple and effective Classification learning algorithms, and it uses a Euclidean distance between training data and input data (or a deformation algorithm using other kinds of metrics) An algorithm that finds the nearest k data and votes with k labels of data to infer the result of the currently input data.

6 is a diagram illustrating an example of a K-Nearest Neighbor.

Assuming that the learning data is divided into a group of squares and triangles, the distance between all the learning data and the value to be predicted (indicated by a question mark in FIG. 6) is obtained and then k pieces of data having the closest distance are extracted, The method of voting with the label values of the k-NN is as follows. In the above example, when k is 3, the predicted value becomes red triangle.

The advantage of k-NN is that it estimates the answer through distance or similarity measurement with learning data, so it is likely to perform well when there are many learning data. It is also relatively stable because it has fewer parameters than other algorithms.

On the other hand, there is a disadvantage in that the computation cost is large because it must be compared with all the learning data, and if the local cluster is formed, the prediction is likely to fail. Setting the value of the parameter k well has a great effect on performance.

2.1.2 Neural network

It is also called artificial neural network, and refers to a structure that forms a network by using artificial neurons that mimic neurons of biological neural networks.

The Deep Learning algorithm, which is currently in the spotlight, is ultimately a kind of neural network, but it has much more hidden layers than conventional neural networks.

FIG. 7 is a diagram illustrating an example of a neural network, and FIG. 8 is a diagram illustrating a weighted sum process (Perceptron).

A neural network consists of three layers: an input layer that accepts input, a hidden layer that actually learns (also called a black box), and an output layer that returns the result of the operation. In FIG. 7, each node means a neuron. When data is input, a weighted sum is obtained as shown in FIG. 8, and an operation result is passed to the next node through an activation function. In this case, weights are learned for each node.

The inputs of each neuron are weighted into the results of the previous neuron and added to bias to compute the sum. The activation function uses sigmoid, tanh, ReLU, and so on.

9A is a diagram of an Activation Function: Sigmoid, FIG. 9B is a diagram of an Activation Function: tanh, and FIG. 9C is a diagram of an Activation Function: ReLU.

Learning in the neural network structure composed of several neurons proceeds through two steps: forward propagation and back propagation.

FIG. 10 is a diagram for a forward propagation process of a neural network, and FIG. 11 is a diagram for a back propagation process of a neural network.

In Forward propagation, the weighted sum of each node is obtained, and the computation value is transmitted to the next node through the activation function. Using the resulting values, we calculate the loss as follows.

Based on this loss, we calculate the slope of the loss obtained by forward propagation to find the optimal point of the loss function. In other words, the variation is calculated to reduce the error value, and then reflected on each weight. Obtain the slope using the partial derivatives and the chain rule as follows.

Using the chain rule and partial derivatives, we obtain the slopes of the weights at the corresponding node, and this slope is used again to correct the weights.

Where α is the learning rate of the optimizer function. We correct the weights by partially differentiating each of the weights in the opposite direction to the direction in which we learned the above procedures in each perceptron. This whole process is repeated until the optimal point is approximated and the neural network is learned.

2.1.3. Support Vector Machine

Support Vector Machine (SVM) is commonly used as a representative model of supervised binary classification. The SVM assumes that the learning data is located in the vector space as the name implies. That is, the vector space is the space where the learning data is located in the rectangular coordinate system, and the factor that determines the dimension is a feature possessed by the data.

12 is a diagram showing a two-dimensional vector having two characteristic values in a two-dimensional space. The goal of the SVM is to find a straight line y = w ^T x + b separating the two groups (red and blue in the figure). Where w is a normal vector perpendicular to the straight line. b is a scalar constant, and the straight line moves in parallel up and down and left and right according to b value.

W ^T x + b> 0 if it is in the blue region, and w ^T x + b <0 if it is in the red region. The problem is that it is difficult to clearly predict the position of x if the input vector x is close to the boundary. SVM is the solution to this problem. That is, the SVM finds a line expression that separates the two groups, with a straight line between the lines represented by the line form, and possibly two groups farther apart. It is Linear SVM to obtain this linear equation.

The distance between two straight lines, that is, the margin, is expressed as follows when X1 is the value of the red region and X2 is the value of the blue region.

를 구하여야 한다. 결국 위의 조건을 만족시키면서 마진을 최대로 하기 위해서는 아래와 같은 식이 도출되며,To maximize the margin, || w || must be minimized to make it an optimization problem.

. In order to maximize the margin while satisfying the above conditions, the following equation is derived,

The above optimization problem must be solved. In this case, a _i is a Lagrangian multiplier, which is always greater than or equal to 0 in the optimization problem of finding the minimum value.

To solve this Lagrangian problem, we first take differentials for w and b, and assign them to the existing formula, which is called dual lagranian.

이 되고 이 행렬을 가지고 위의 공식을 한번 더 치환하면,If we convert the above equation into a problem with N learning data,

If we replace this formula with this matrix,

The solution of the two equations can be obtained and the maximum margin can be obtained to obtain the boundary equation separating the two groups.

2.1.4 Decision trees

Decision Tree is one of the simplest and most successful non-parametric supervised learning techniques of the map classification learning algorithm. The goal of this algorithm is to create a model that predicts the values of the target variables by learning simple decision rules deduced from the example. Decision trees can be used in both regression analysis and classification analysis depending on the type of dependent variable. The internal node of the Decision Tree represents a series of branching rules for observations, and this series of partitioning rules can be visualized in an abstract structure for easy understanding, making observation easy. FIG. 13 shows an example of a vehicle purchase by a decision tree. Each node corresponds to a conditional statement in the programming language, and the branch for each condition searches the tree until it reaches a node that represents the end result based on all accumulated decisions. This example represents a decision tree that produces a decision of 'BUY' or 'DO NOT BUY'.

The Decision Tree has three advantages:

First, it is easy to understand and easy to interpret, and the decision tree can be visualized.

Second, if a white box model is used and the given situation is observable in the model, the description of the condition can be easily explained by Boolean logic. In contrast, results in black box models (ex: artificial neural networks, deep runs, etc.) are very difficult to interpret.

Third, the validity of the model can be verified using the statistical test, and the reliability of the model can be judged.

The algorithm that constructs the decision tree is mainly Top-Down (top-down) technique. At each stage, a variable is selected which divides a given data set into the most appropriate criteria. Different algorithms have their own criteria for measuring the "fitness of division", and these criteria usually measure the homogeneity of the target variable in a subset.

The ID3 algorithm uses the greedy search methodology to create nodes and create a tree to determine the attributes used for branching. The search in this tree is top-down (root → leaf). The process of creating and searching the tree is repeated until the classification results are satisfactory.

The metric for selecting the attributes to be used for branching each node is calculated using Information Gain and Entropy.

The following is an algorithm for constructing a tree.

Algorithm 1. ID3 tree configuration algorithm Find the branch attribute of the Root Node and call it A
Create a new descendant node for each branch of A
Sorts examples into corresponding leaf nodes
If the example is perfectly categorized stop, otherwise loop the algorithm on the new leaf node.

The Entropy function E is as follows.

The function argument S represents an example, p represents the ratio of class + learning examples, and q = 1 - p represents an example ratio of classes.

Information gain can also be obtained by using this.

Where Gain represents the expected decrease in entropy due to alignment in A.

14 is a diagram illustrating an example of a classifier. In Fig. 14, the higher the humidity of humidity and the higher the humidity, the better the branch property. In this way, we use top-down techniques to add new nodes until the classification results are satisfactory.

15 is a diagram relating to the example of C4.5.

The C4.5 classification algorithm is an algorithm that complements some of the shortcomings of the ID3 algorithm. The problems of the ID3 algorithm that the C4.5 algorithm tries to complement are as follows.

Handling continuous attribute - The ID3 algorithm suggests a way to create a tree only for categorical attributes. Therefore, numerical attributes have limitations that can not be applied to model creation. In C4.5, we propose a method of using up to numerical attributes.

Binary segmentation is performed to process attributes with successive values. Once the attribute values are sorted, a gain is computed for all breakpoints of the attribute. When selected as the best separation point (h), the A property is divided into A? H and A> h. For example, if we look at the glu attribute, the first node in Figure 15, we can see that the point with the lowest gain is computed as 123 and the point automatically selected as the split point (h). Similarly, bmi, age, dp, and ped can automatically select each breakpoint h in the same way and finally predict the label of the input data.

Tree Depth Problem - When creating a tree model with the ID3 algorithm, there is a problem that the depth of the tree becomes too deep. The C4.5 algorithm limits the depth to solve this problem.

Missing values - The missing values of the processing problem for data for which the value of a particular attribute is not partially entered in the data is usually "?" . Dealing with missing values involves substitution, which means that the substitution can be estimated from the available data if the major function is missing. Distribution-based imputation is performed when an example is divided into multiple instances with different values for missing functionality. A weight corresponding to the estimated probability is assigned to a specific missing value, and the maximum value of the weight is 1.

The Classification and Regression Tree (CART) classification algorithm is similar to the existing C4.5 algorithm. The difference is as follows.

- Limit the number of child nodes to 2

- Gain information gain using Gini impurity instead of Entropy

2.1.5 Random Forest

Random Forest is a machine learning algorithm that learns multiple decision trees (DT) using bagging technique.

Bias-variance trade off means that learning errors have a sort of zero sum game appearance, as the variance increases as bias decreases in machine learning algorithms. Bagging is used as a solution to this problem.

Bagging is an abbreviation of Bootstrap Aggregation and is used to overcome the bias-variance trade off. Here, bias and variance are two factors that constitute a learning error. If the bias is high, the prediction result is often inaccurate compared to the actual result. If the variance is high, the prediction result is good in some learning examples, In the learning example, the performance is greatly degraded and the stability of the prediction result is degraded.

Thus, Bagging refers to solving the bias-variance trade-off problem by determining the prediction results by voting the results of the individual prediction models by sampling N subsets randomly in a given learning sample N times to generate N prediction models.

The learning process of the random forest through bagging consists of three stages as follows.

Algorithm 2. Random forest learning process through bagging Through the bootstrap method, N learning examples are generated.
N DTs are learned.
Combine the DT into a classifier (ensemble), then use the average or majority voting method to predict the outcome.

Since DT has a small bias and large variance, it experiences overfitting problems in very deep DTs. One DT is very sensitive to the noise in the training example, but if different DTs are not related, averaging the DTs can reduce the sensitivity to noise. However, when training with the same learning example, each DT correlation is larger, so we can use different random learning examples to reduce the correlation between DTs. The algorithm of Random Forest is as follows.

Algorithm 3. Random Forest

, features F, and number of trees in forest B.
function RANDOMFOREST(S,F)

for

do

A boostrap sample from

RANDOMIZEDTREELEARN

end for
return H
end function
function RANDOMIZEDTREELEARN(S,F)
At each node:

very small subset of

Split on best feature in

Return The learned tree
end function Precondition: A training set

, features F, and number of trees in forest B.
function RANDOMFOREST (S, F)

for

do

A boostrap sample from

RANDOMIZEDTREELEARN

end for
return H
end function
function RANDOMIZEDTREELEARN (S, F)
At each node:

very small subset of

Split on best feature in

Return The learned tree
end function

There are two main differences between DT and Random Forest. First, the Random Forest trains multiple DTs by repeatedly sampling the training samples using bootstrap, and second, it considers only m arbitrarily selected variables for each partition.

Out-of-bag (OOB) errors are used to estimate the generalization error. The OOB error is obtained by learning each DT as learning data, estimating each result using test data, estimating the final result by voting it, and measuring the error with the actual value. This OOB is important because in Breiman's (1996b) invention of measuring errors in the classifier of the bogged classifiers, we give empirical evidence that the OOB prediction method is as accurate as the test set, Because.

To measure the significance of the jth feature after training, we calculate the OOB error of the random forest using data excluding the jth feature. The significance score of the jth variable has a higher order of importance than that of a feature with a small OOB error value in the original data set for all trees.

2.1.6 Gradient Boosted Tree

In Gradient Boosted Tree, the meaning of Gradient Boosted can be interpreted as 'suitable for optimizer function'. Here, Gradient is a Gradient Descent Optimizer, and Gradient Boosted Tree is a model that learns by reducing errors between predicted and actual values using Gradient Descent Optimizer. First, define the objective function as follows.

은 loss function이며,

는 regularization,

는 다음과 같다.here

Is a loss function,

Regularization,

Is as follows.

path id in the structure of

's tree,

weightwhere,

path id in the structure of

's tree,

weight

Here you can find the Hypothesis function using the rule that the model has to learn what you need to learn and then add only one tree per step through the following steps.

and

space of all possible decision trees.where,

and

space of all possible decision trees.

If logistic loss is used as a loss function of an objective function, the equation is as follows.

라 할 수 있다. 이때

가 미리 학습된 모델이라 가정하고

가 새로 만들 모델이라 가정했을 때, 그 과정은 다음과 같다.The aforementioned rule is hypothesis here

. At this time

Is a pre-learned model

Assuming the model is a new model, the process is as follows.

Substituting this result into the Objective function,

Gain function is calculated by assuming the above-mentioned logistic loss,

는 왼쪽 가지의 스코어이고,

는 오른쪽 가지의 스토어이며,

는 원래 가지의 스코어이고,

는 regularizer이다.Can be obtained. From here

Is the score of the left branch,

Is the store on the right branch,

Is the original branch's score,

Is a regularizer.

보다 작다면 새로운 가지를 추가하지 않는게 더 낫다는 해석을 할 수 있다.Here,

If it is smaller, it can be interpreted that it is better not to add a new branch.

2.1.7 Ensemble Boosted Tree

Ensemble Boosted Tree is an ensemble of the Gradient Boosted Tree mentioned above. In order to make an ensemble, the correlation between the models to be ensemble should be low, so different learning data is put into each model. If less training data is available, resampling is performed.

The results of the models created through the learning results of each Boosted Tree are used as the final results by voting as follows.

and K is total number of Model,

is Classifier.where,

and K is the total number of Model,

is Classifier.

2.2 Clustering (Clustering)

3.2.1 k-means clustering

k-Means Clustering is a method of classifying k clusters by assuming that the total number of inputs is n.

개의 집단으로 군집한다. 이때 기준점은

가 되며

는 입력(데이터)이고 다음을 최소화 하는 방향으로 학습된다.Each input value is adjusted so that the Euclidean distance is minimized around the reference point.

Groups. At this time,

And

Is the input (data) and is learned in a direction that minimizes the following.

k-Means Clustering is fast and simple, but if it does not find an appropriate k value, it will degrade performance, be sensitive to outliers, and can not distinguish between different clusters.

3.2.2 Expectation-Maximum Clustering

의 Maximum likelihood 나 Maximum a posteriori 를 찾기 위한 방법이다.EM Clustering (Expectation-Maximum Clustering) is an unknown parameter,

And the maximum likelihood or Maximum a posteriori.

는 우도(likelihood) 함수이며here

Is a likelihood function

It can be seen that it is derived to the right side of equation (1) by using the above log and the definition of the likelihood function. (Here, the reason for the expansion of the distribution Z is because it is the distribution to be clustered.

은 우도 함수를 이루는 항중 하나로 우도 함수를 직접적으로 증가시키는게 아니라

을 증가시켜 최대값을 찾는다.

Does not directly increase the likelihood function to one of the points making up the likelihood function

To find the maximum value.

에서의 변수

에서 주어진

와

의 조건부 확률 분포(Conditional Distribution)

에 대한 log likelihood의 기대 값을 구한다.Given arbitrary moment

Variables in

Given in

Wow

Conditional Distribution of Conditional Distribution

The expected value of log likelihood is obtained.

Learning in the direction of maximizing this. However, the maximum likelihood method uses the Maximum a posteriori method to solve this problem because the output value changes too sensitively according to the observed input value.

3. Deep Learning

16 is a diagram of an example of a Convolutional Neural Network (Alex Net).

3.1 Convolution

Convolution is an operation that is used mainly in the field of signal processing.

Convolution is the operation used to learn arbitrary filters and extract the appropriate features from a given matrix or vector. It is mainly used in image and speech recognition.

3.2 Local Connectivity

When dealing with high-dimensional inputs, connecting all the neurons of the previous volume to each other is impractical in terms of memory or computation, so each neuron is connected only to the local area of the input volume. The spatial extent of this connection is then a hyper parameter called the acceptance field of the neuron. The extent to which the depth axis is connected is always equal to the depth of the input volume, and it is important to re-emphasize this asymmetry in how spatial dimensions (width and height) and depth dimensions are handled. The connection is local (in width and height) in space, but always equal to the total depth of the input volume.

3.3 Shared Weights

example:

Using the neuron with the acceptance field size F = 11, stride S = 4 and zero padding P = 0 in the first convolution layer, we assume that the depth of the convolution layer is K = 96, (227-11) / 4 + 1 = 55, the output volume of the convolution layer is [55x55x96]. Each of the 55 * 55 * 96 neurons of this volume is connected to an area of size [11x11x3] in the input volume. In addition, all 96 neurons in each depth row are connected to the same [11x11x3] region of the input, and are learned and connected to different nodes, respectively.

The way weights are shared is used in Convolution to control the number of parameters. The actual example above shows that there are 55 * 55 * 96 = 290,400 neurons in the first transition layer and 11 * 11 * 3 = 363 weights and one bias, respectively. This adds 290400 * 364 = 105,705,600 parameters to the first layer of the Convolution Network. This number is a very large number in terms of cost. If it is useful to calculate one feature at some spatial location (x, y), calculating at another location (x2, y2) can dramatically reduce the number of parameters It turned out. That is, if there are [55 * 55 * 96] neurons, the spatial location information of [55 * 55] can be shared, and the depth dimension can be learned separately.

3.4 Pooling Layers

The function of the pooling is to gradually reduce the spatial size of the representation, thereby reducing the parameters and computation of the network and preventing overfitting. The pooling layer works independently on all layers and spatially reduces it by taking the maximum or mean value within the specified filter size. The most common form is a pooling layer with a 2x2 filter. Down-sampling the depth slice of the input twice, twice by width and height, to inactivate 75%. In addition to Max pooling, the pooling device may perform other functions such as mean pooling or L2-norm pooling.

17 is a diagram illustrating an example of Max-pooling.

The pooling layer of FIG. 17 spatially down-samples the volume independently for each depth slice of the input volume.

In the example on the left side of FIG. 17, the input volume of size [224x224x64] is pooled to filter size 2, the size is 0.5 times the output volume of [112x112x64], and the volume depth is maintained. The right side of Figure 17 is an example of max pooling, the most common down sampling operation.

3.5 ReLU

18 is a diagram illustrating an example of the ReLU Function.

를 계산한다. 즉, 활성화는 도 18에서처럼 단순히 0으로 임계화 한다. ReLU를 사용하는 데는 몇 가지 장단점이 있다. Rectified Linear Unit Functions

. That is, activation is simply thresholded to zero as in Fig. There are several advantages and disadvantages to using ReLU.

The first advantage is that when estimated from the experimental results, the convergence of the stochastic gradient descent is significantly increased compared to the Sigmoid and tanh functions because it is a linear, non-saturating form. Second, ReLU can be implemented by simply thresholding the activation matrix to zero, compared to the computationally intensive tanh and sigmoid neurons.

On the other hand, the disadvantage is that when the ReLU unit receives a large gradient value during learning, the weight is fixed at zero so that the neuron is not reactivated at all data points and may not be learned again. For example, if the learning rate is set too high, about 40% of the network may not be updated, but the problem can be solved by setting the learning rate appropriately.

3.6 Inner Product

The inner product can be defined algebraically or geometrically. The geometric definition is based on the concept of angles and distances (the size of the vector) and is the same as the shape of the hidden layer in Neural Network.

3.7 Softmax

와 같이 binary라고 가정한다면, Softmax regression는 다음과 같이 인

개의 클래스를 예측할 수 있게 해준다. Softmax function은 다음과 같다.Softmax regression (or multinomial logistic regression) is a method for solving multi-class classification. Generally, in logistic regression,

Assuming binary as follows, Softmax regression is sign

It allows you to predict the classes of The softmax function is as follows.

The cross-entropy between the actual data distribution p and the estimated distribution q in the information theory is defined as

Thus, Softmax minimizes cross-entropy between estimated class probabilities.

3.8 Dropout

Dropout is an effective and simple normalization technique, which is a complementary method to normalization methods such as L1, L2, and max-norm, which prevents overfitting of neural networks. Dropout is a technique to normalize information between all nodes by disconnecting the nodes by keeping the neurons active or deactivating them with a small probability. However, all neurons are activated during the test. 19 is a diagram related to an example of Dropout.

3.9 Finetuning

It is a technique to quickly fine-tune the weights of some layers by using already learned weights. It is a technique to maximize the learning effect by initializing the weights that have already been learned in other similar data sets when there is insufficient data. At this point, fine tuning can produce good results for a generalized problem.

In the case of specific data (for example, financial statement data) in which the minimum value and the maximum value are not defined, unlike the data used for signal processing, the value may be too large to be applied to the known machine learning techniques. As a result, the loss also has to be large, and such a problem can not be avoided even in the conventional deep running.

SUMMARY OF THE INVENTION Accordingly, an object of the present invention is to provide an artificial neural network model capable of efficiently learning specific data (for example, financial statement data) in which a minimum value and a maximum value are not defined, and a machine learning method using the artificial neural network model.

According to an aspect of the invention, there is provided a deep learning system, comprising: an acquisition module for acquiring at least one learning data used to learn an artificial neural network model, wherein each of said learning data comprises first to Mth features, And an learning module for learning the artificial neural network model using the at least one learning data, wherein the artificial neural network model includes an input layer, A transform function layer, at least one inner product layer, and an output layer, wherein the transform function layer comprises K (K is a natural number of 2 or more) , Wherein the kth transform function node (k is an arbitrary natural number satisfying 1 < = k < = K) Dimensional transformed value vector is generated by applying a predetermined k-th transform function to each feature value included in the input M-dimensional vector, and the generated M-dimensional transformed value vector is generated A deep running system is provided that is configured to output to each node of the top-most inner product layer.

In one embodiment, the first transform function to the K-th transform function are all functions having an S-shaped curve property or a function having a property of increasing a function value and decreasing a derivative value in an interval (0,?) .

In one embodiment, each of the first to K-th transform functions may be expressed by a sigmoid function, a hyperbolic tangent function,

[Formula] tf (x) = log ( α h × x + β h) ( here, h is 1 <= h <= H in each of the integer (H is an integer of 1 or greater), and α _h and β _h is pre-defined Lt; / RTI &gt;

According to another aspect of the present invention, there is provided an artificial neural network model learning method, comprising: acquiring at least one learning data used for learning an artificial neural network model, wherein each of the learning data includes: M is an integer greater than or equal to 2), and learning the artificial neural network model using the at least one learning data, wherein the artificial neural network model comprises an input layer wherein the transform function layer comprises K (K is an integer greater than or equal to 2), a transform function layer, at least one inner product layer, and an output layer, (K is an arbitrary natural number satisfying 1 < = k < = K) is included in the input layer Dimensional transformed value vector by applying a predetermined k-th transform function to each feature value included in the input M-dimensional vector, and generating an M-dimensional transformed value vector by using the generated M-dimensional transformed value An artificial neural network model learning method is provided that is configured to output vectors to each node of the uppermost inner product layer.

According to another aspect of the present invention, there is provided a computer program installed in a data processing apparatus for performing the above-described method.

According to another aspect of the present invention there is provided a deep running system comprising: a processor and a memory storing a computer program, wherein the computer program, when executed by the processor, causes the deep learning system to perform the method A deep running system is provided.

It is possible to provide an artificial neural network model capable of efficiently learning specific data (for example, financial statement data) in which the minimum value and the maximum value are not defined, and a machine learning method using the artificial neural network model.

BRIEF DESCRIPTION OF THE DRAWINGS A brief description of each drawing is provided to more fully understand the drawings recited in the description of the invention.
Fig. 1 is a diagram relating to classification of map learning.
2 is a diagram showing an example of classification.
Figure 3 is an example of linear regression and nonlinear regression.
4 is a diagram of an example of Unsupervised Learning.
5 is a diagram of an example of clustering.
6 is a diagram illustrating an example of a K-Nearest Neighbor.
FIG. 7 is a diagram illustrating an example of a neural network, and FIG. 8 is a diagram illustrating a weighted sum process (Perceptron).
9A is a diagram of an Activation Function: Sigmoid, FIG. 9B is a diagram of an Activation Function: tanh, and FIG. 9C is a diagram of an Activation Function: ReLU.
FIG. 10 is a diagram for a forward propagation process of a neural network, and FIG. 11 is a diagram for a back propagation process of a neural network.
12 is a diagram showing a two-dimensional vector having two characteristic values in a two-dimensional space.
FIG. 13 shows an example of a vehicle purchase by a decision tree.
14 is a diagram illustrating an example of a classifier.
15 is a diagram relating to the example of C4.5.
16 is a diagram of an example of a Convolutional Neural Network (Alex Net).
17 is a diagram illustrating an example of Max-pooling.
18 is a diagram illustrating an example of the ReLU Function.
19 is a diagram related to an example of Dropout.
20 is a diagram showing an example of a Normal Ensemble Model.
21 is a diagram showing an example of a regression model.
22 is a diagram showing an example of a Function Concatenate Model.
23 is a diagram showing an example of the LSTM Model.
24 is a block diagram showing a schematic structure of a deep learning system according to an embodiment of the present invention.
25 is a diagram showing an example of basic learning data.
26 is a diagram illustrating an example of an artificial neural network model according to an embodiment of the present invention.
27 is a flowchart schematically illustrating an artificial neural network model learning method according to an embodiment of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS The present invention is capable of various modifications and various embodiments, and specific embodiments are illustrated in the drawings and described in detail in the detailed description. It is to be understood, however, that the invention is not to be limited to the specific embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

The terms first, second, etc. may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another.

The terminology used in this application is used only to describe a specific embodiment and is not intended to limit the invention. The singular expressions include plural expressions unless the context clearly dictates otherwise.

In this specification, the terms "comprises" or "having" and the like refer to the presence of stated features, integers, steps, operations, elements, components, or combinations thereof, But do not preclude the presence or addition of features, numbers, steps, operations, components, parts, or combinations thereof.

Also, in this specification, when any one element 'transmits' data to another element, the element may transmit the data directly to the other element, or may be transmitted through at least one other element And may transmit the data to the other component. Conversely, when one element 'directly transmits' data to another element, it means that the data is transmitted to the other element without passing through another element in the element.

Hereinafter, the details of the research performed to derive the claimed invention and the algorithms presented therefrom will be described first.

1. Characteristics of data

The main goal of this study is to enable financial statement data to be effectively applied to machine learning. Therefore, the learning data used in this study is the financial data such as the amount and the ratio, and unlike the signal data such as image or voice, which is mainly dealt with by the existing deep learning, the range of the value is very large, so the influence of the data normalization There are many characteristics to receive.

2. Data preprocessing technique

2.1 Data normalization technique

In the case of financial statement data, which is mainly covered in this study, since the minimum value and the maximum value are not determined unlike the data used for signal processing, the value is too large to be applied to the known machine learning techniques. I can not help it. So it takes a lot of time if learning is never done or learning is done. Therefore, it is essential to apply data normalization.

Hereinafter, three normalization methods widely used in the field of statistics are described.

(1) Feature scaling (zero to one)

 The feature scaling method, which is most commonly used in data normalization, converts data from 0 to 1 according to the minimum and maximum values.

This method has the disadvantage of causing considerable information loss because it converts all numbers from 0 to 1.

(2) z-score normalization

In this normalization method, the value of x is normalized according to the mean and standard deviation, and its formula is as follows.

는

의 평균값이고,

는 표준편차이다. From here

The

&Lt; / RTI &gt;

Is the standard deviation.

This method is useful when the minimum and maximum values of the entire data can not be known.

(3) Median and Median Absolute Deviation

Median and median absolute deviation (MAD) are important measures of variability in the univariate sample. MAD is a statistical variance measure, which can be applied more flexibly to data outliers than standard deviations. The formula is as follows.

2.2 Automatic selection method of normalization technique for each feature

Choosing the most suitable normalization algorithm for each feature requires a lot of time and effort. It is very difficult to find an optimal normalization algorithm especially when there are many kinds of features such as financial statement data. Therefore, a method of simultaneously applying various normalization techniques and learning all the data at once can be used. However, there is a limitation in the amount of computation and memory, and noise data are simultaneously generated.

To overcome this problem, we propose an algorithm that automatically finds the best normalization method for each feature. The algorithm is as described below.

Algorithm 4. Normalization technique selection algorithm suitable for each feature

(Feature scaling)

(z-score normalization)

(Median and Median Absolute Deviation)
2. 1의 결과에 각각 다음의 함수를 적용한다. (a는 1의 결과들의 각 instance)

3. 2의 결과를 label을 기준으로 bad class와 good class로 분류한다.

4. 3에서 분류한 데이터 각각의 feature histogram 작성
두 그룹별 히스토그램 사이의 Euclidean distance 측정

5. 각 feature별로 d가 큰 3의 결과를 취함

1. Apply the following equations to the train data.

(Feature scaling)

(z-score normalization)

(Median and Median Absolute Deviation)
2. Apply the following function to the result of 1, respectively. (a is each instance of the results of 1)

3. Classify the result of 2 as bad class and good class based on label.

4. Create feature histogram for each of the data classified in 3
Euclidean distance measurement between histograms of two groups

5. For each feature, d takes the result of large 3

The above algorithm applies to each individual feature.

2.3 Data filtering techniques

Among the features of the data used for learning, there is a noise feature that interferes with solving the binary classification problem, and the label assigned to the learning data may be derived based on data other than the learning data (for example, For example, in the case of financial statement data, the label may not be derived from the corresponding financial statement data), in which case the labeling may be considered incorrect. Thus, mislabeled data can cause confusion in learning.

To solve these problems, various data filtering algorithms have been proposed. The proposed algorithms are as follows.

2.3.1 Grouping of similar companies using k-means clustering

Although the present invention is based on the manufacturing industry only, if the types of business or the scale of the business are different, the type of financial statement is different, and the learning is biased toward many companies of the similar type. Therefore, the data filtering is carried out in such a way that the companies are divided into detailed and re-learning.

The k-means clustering method is used to divide the group into k companies, and then the final results are obtained through learning algorithms using only a number of groups that can be learned.

2.3.2 Removing meaningless features using histogram distance

All the features of the learning data may not be distinguishable in solving binary classification problems (for example, the distinction between good and bad classes), and there may be some disturbing elements. Such data consumes a lot of memory and computation time, and it interferes with learning.

Therefore, a method of automatically removing unnecessary features is proposed as follows.

Algorithm 5. Elimination of meaningless feature using Histogram distance

Euclidean distance가 threshold 이상인 feature만을 남기고, 이하인 것은 noise feature로 간주하여 제거한다.
유의미하다고 판단된 feature만을 이용하여 학습 알고리즘에 적용한다.For each feature, calculate good class and bad class on the same condition.
Euclidean distance measurement between two histograms

Euclidean leaves only those features with a distance equal to or greater than the threshold, and removes them as noise features.
And apply it to the learning algorithm using only the features that are judged to be significant.

2.3.3 Removing noise data using k-NN

Among the instances that are currently held, instances with bad classes are created through data other than the actual learning data, which causes many obstacles to learning because it is normal data based on the current data. Therefore, only the data having the label derived based on the actual learning data can be used for correct learning. However, it is impossible to know what is the normal data based on the current information alone, and an algorithm for randomly estimating it is needed. Therefore, in the present invention, the learning data satisfying a predetermined classification criterion is labeled as a good class and the remaining data is labeled as a bad glass (the classification criterion is that the data classified as a good class by the classification criterion are all normal , It is suggested to measure the similarity between the data labeled with good class and the data labeled with bad class and regard it as noise data if the similarity is high.

Algorithm 6. Remove noise data using k-NN

,

: label이 0인

번째 instance,

: label이 1인 instance,

번째 feature값

The label of the instance that satisfies a predetermined division criterion is divided into 0 and the rest is divided into 1.
Obtain the instances of Label 0 and the distance of instance 1.

,

: label is 0

Th instance,

: instance with label 1,

Th feature value

2.3.4 Removing noise data using Random Forest

In this paper, we propose a method to extract features with high feature importance derived by using the previously known Random Forest algorithm. At this time, in order to compensate the variance of the experiment result, there are parts to randomly select the data in the Random Forest. After extracting the feature importance several times using the same data, only the features coming in the Top 10 are merged, use.

3.5 Removing noise data using SVM

Among the most widely used feature selection methods, the three most intuitive and widely used methods are forward selection, backward selection, and stepwise selection. First, the forward selection constructs a classifier with a specific classification methodology for each individual feature, selects the feature with the highest accuracy, and then for each model paired with the previously selected feature and the non-selected feature, To develop the feature combination that has the best performance. The feature combination is extended until the performance of the classification model based on the feature combination obtained by gradually adding the features is no longer improved (generally, n candidate repetitions are performed when n candidate features are present). In the case of backward selection, a classification model is constructed based on all the first features, the performance is evaluated, the individual features are removed one by one, and a classification model is constructed again. Gradually, It is a way to create a feature set. Finally, the Stepwise selection method is a mixture of forward selection and backward selection. First, the classifier is constructed based on the first feature by forward selection. Next, the second feature is included through the forward method and the backward selection method is used to determine whether or not to remove the first feature that has already been selected. For features that are not already selected, forward selection is applied to select features one by one, and features already included are removed by backward method. In the present invention, the backward selection is applied to the SVM to select the main features and then applied to the learning algorithm.

2.4 Semantic Data Processing Techniques

The method described above extracts only meaningful features among the features that are presently present, and aims at extracting features that are deemed important in the original state without considering correlation between the features. In this case, when using the Deep Learning algorithm, it is possible to judge a certain degree of correlation because weighted sum is performed in the network, but there is a disadvantage that the multiplication or division can not be performed automatically and the ratio data can not be automatically generated. Therefore, we have created data by merging all the data into sales, total assets, and tangible assets that are considered to be valid when a person thinks it. If this problem can be automatically generated through the algorithm, then unnecessary features And does not generate a job. Therefore, in this study, we propose a method to apply synthetic feature extraction technique using Ensemble Boosted Tree (hereinafter referred to as 'conventional research').

2.4.1 Synthetic feature extraction using EBT

In the previous research, an algorithm for extracting synthetic features using Ensemble Boosted Tree has been proposed. The main idea used here is to randomly select one feature for each feature, then randomly select one of the operators to increase the feature dimension, and then learn again to remove the features whose feature importance is below the threshold K It is a way to find the relationship between features repeatedly.

However, because the characteristics of data covered in this study should be applied to all data, the ratio of sales, total assets, and tangible assets should be applied to all data. All possible feature combinations and all arithmetic operations were applied to extract synthetic features. The details are as follows.

Algorithm 7. Synthetic feature extraction algorithm

: training set,

: number of synthetic features

: number of base learners

: features acceptance threshold
Output:

:set of base learners
for

do
Train

using

;
Remove features from

for which

;
Estimate

from model

;
for

do
for

do
Sample features

and

;
Generate new features

; operation

from {+, -, *, /}
Extend

with new values of

end
end
end
return

Input :

: training set,

: number of synthetic features

: number of base learners

: features acceptance threshold
Output :

: set of base learners
for

do
Train

using

;
Remove features from

for which

;
Estimate

from model

;
for

do
for

do
Sample features

and

;
Generate new features

; operation

from {+, -, *, /}
Extend

with new values of

end
end
end
return

3. Deep Learning Model

The following three suggestions are made as a deep learning model suitable for financial statement data.

3.1 Normal Ensemble Model

Using the Ensemble model, its performance has increased to some extent already in previous inventions. Therefore, we propose an Ensemble model of voting by learning multiple Deep Learning models at the same time. The structure is shown in Fig. 20 is a diagram showing an example of a Normal Ensemble Model.

3.2 Regression Model

In the case of the Normal Ensemble Model, we predicted the results as good and bad classes. We propose a regression mode that estimates the degree of bad to some degree between 0 and 1 by further subdivision. The model is shown in Fig. 21 is a diagram showing a regression model.

3.3 Function Concatenate Model

In previous models, sigmoid, tanh, and other functions were selected randomly and then input to the deep learning model so that learning can be performed well for normalized data. However, since the function that maximizes each learning effect differs from feature to feature, we proposed a function concatenate model that allows the user to select the deeper learning model while learning all functions.

Function Concatenate Model is a deep learning model that improves the learning effect by automatically correcting the outliers of various features of financial data.

The algorithm of the model is as follows.

Function Concatenate Model can increase the weight on a certain feature by itself, but it also has the effect of maximizing the effect by accepting all functions in case of a particularly important feature. The overall model is shown in FIG. 22, and this model is also improved by using an ensemble technique. 22 is a diagram showing an example of a Function Concatenate Model.

3.4 LSTM Model

Since the financial statement data is time series data having information and labels inputted every year, it proposes a LSTM (Long Short-term Memory) model as shown in FIG. LSTM is a network that temporarily memorizes previous results and helps predict the next data.

4. Conclusion

In this study, we first applied various data normalization methods to apply this data to traditional machine learning methods. Based on these results, we propose an algorithm that automatically selects the optimal normalization method for each feature of learning data .

We also proposed an algorithm to remove unnecessary information / noise features.

If the amount of surplus is given by other items other than the financial statement data that you have, the data should be labeled after being labeled 'normal'. However, it is difficult to deduce the companies that are labeled by other information using only the data that they have. Therefore, we have proposed a technology to automatically detect and remove companies whose labels are misidentified using only current data.

 In addition, we proposed a synthetic data processing algorithm to study the prediction model with more various types of relationship information, not limited to the proportional and inverse relation of the correlation between data.

Finally, we propose a deep learning network modeling the data preprocessing to be automatically processed by the deep learning model.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. Like reference symbols in the drawings denote like elements.

The deep learning system according to an embodiment of the present invention can learn an artificial neural network model using a plurality of learning data.

24 is a block diagram illustrating a schematic structure of a deep learning system 100 according to an embodiment of the present invention.

Referring to FIG. 24, the deep learning system 100 may include an acquisition module 110, a control module 120, and the like. Depending on the embodiment of the present invention, some of the components described above may not necessarily correspond to components that are essential to the implementation of the present invention, and in some embodiments the deep learning system 100 may be more But may include more components. For example, the deep learning system 100 may further include a preprocessing module (not shown) for performing a learning data preprocessing process as described above.

The deep learning system 100 may include hardware resources and / or software necessary for realizing the technical idea of the present invention, and it means one physical component or one device no. That is, the deep learning system 100 may mean a logical combination of hardware and / or software provided to implement the technical idea of the present invention. If necessary, the deep learning system 100 may be installed in a separate apparatus, The present invention may be embodied as a set of logical structures for realizing the technical idea of the present invention. Also, the deep learning system 100 may mean a set of configurations separately implemented for each function or role to implement the technical idea of the present invention. The modules that make up the deep learning system 100 may be located in different physical devices or may be located in the same physical device. In addition, according to an embodiment, the software and / or hardware constituting each module constituting the deep learning system 100 are also located in different physical devices, and configurations located in different physical devices are organically coupled to each other The functions performed by the modules of FIG.

In this specification, a module may mean a functional and structural combination of hardware for carrying out the technical idea of the present invention and software for driving the hardware. For example, the module may mean a logical unit of a predetermined code and a hardware resource for executing the predetermined code, and it does not necessarily mean a physically connected code or a kind of hardware. It can easily be deduced to a technician.

The control module 120 may be coupled to other components (e.g., the acquisition module 110) included in the deep learning system 100 to control their functions and / or resources. Also, the control module 120 may control the deep learning system 100 to perform the artificial neural network learning method described above.

The acquisition module 110 may obtain at least one learning data used to learn an artificial neural network model. Here, each of the learning data may include M feature values corresponding to the first to Mth features (M is an integer of 2 or more).

For example, the acquisition module 110 may read a file storing N basic learning data. Alternatively, the acquisition module 110 may receive N basic learning data through a predetermined input device. Or the acquisition module 110 may receive N basic learning data via the network.

FIG. 25 is a diagram showing an example of N pieces of learning data composed of M feature values and labels (N is an integer of 2 or more).

As shown in FIG. 25, each learning data D ₁ to D _N may include M feature values and labels. For example, the learning data D ₁ includes M feature values V ₁₁ , V ₁₂ , V ₁₃ , ... , V _1M, and label L ₁ , and may include feature values V ₁₁ , V ₁₂ , V ₁₃ , ... , V _1M may in turn correspond to features F ₁ , F ₂ , F ₃ to F _M , respectively. Similarly, the learning data D _n includes M feature values V _n1 , V _n2 , V _n3 , ... , V _NM, and label L _n , and the feature values V _n1 , V _n2 , V _n3 , ... , V _nM may in turn correspond to features F ₁ , F ₂ , F ₃ to F _M , respectively.

Referring again to FIG. 24, the control module 120 may learn the artificial neural network model using the at least one learning data.

26 is a diagram for explaining a structure of an artificial neural network model that can be learned by the deep learning system 100 according to an embodiment of the present invention.

26, the artificial neural network model includes an input layer 10, a modification function layer 20, at least one inner product layer 30-1 to 30-L, , And an output layer (30).

The transform function layer 20 may be configured to include K (K is a natural number of 2 or more) transform function nodes. Although FIG. 26 illustrates four nodes, the artificial neural network model may include more or fewer number of transform function nodes.

All the nodes included in the transform function layer 20 can receive the respective learning data from the input layer 10. As described above, each learning data may be composed of M feature values.

That is, the k-th transform function node (k is an arbitrary natural number satisfying 1 <= k <= K) can receive the M-dimensional vector corresponding to the learning data from the input layer 10.

The k-th transform function node (k is an arbitrary natural number satisfying 1 < = k < = K) applies a predetermined k-th transform function to each feature value included in the input M- You can create a vector. For example, a k-th function conversion node, if f _k (), the first transformation function _{Dn = [V n1, V n2} , V n3, ... , V _nM ] are input from the input layer 10, the k-th transform function node can be expressed as [f _k (V _n1 ), f _k (V _n2 ), f _k (V _n3 ), ... , f _k (V _nM )].

The k-th transformation function node (k is an arbitrary natural number satisfying 1 < = k < = K) transforms the generated M-dimensional transformation value vector to the uppermost inner product layer 30-1, Lt; RTI ID = 0.0 &gt; node &lt; / RTI &gt;

The first to K-th transform functions may all be different functions.

Meanwhile, the first transform function to the K-th transform function may all have a function having an S-shaped curve property or a function having a property of increasing a function value and decreasing a derivative value in an interval (0,?). That is, the first to K-th transformation functions may all be functions that the increase of the function value decreases as the variable value increases.

For example, the first transform function to the K-th transform function may be expressed by a sigmoid function expressed by [Expression 1], a hyperbolic tangent function expressed by Expression 2, And may be any one of tf (x).

[Equation 1] 1 / (1 + e ^-x )

[Equation 2] tanh (x)

[Formula 3] tf (x) = log (α h × x + β h) ( here, h is 1 <= h <= H in each of the integer (H is an integer of 1 or greater), and α _h and β _h in advance Defined constants)

On the other hand, in a particular embodiment, expression of [Formula 3] tf (x) α _h is 10, and the like 100, 1000, 10000 on the, β _h may be one day. That is, in the present embodiment, the plurality of conversion functions are log (10 x x + 1), log (100 x x + 1), log (1000 x x + .

On the other hand, at least one inner product layer (30-1 to 30-L) may have the same properties as the hidden layer of a typical neural network model. Each of the at least one inner product layers 30-1 to 30-L may have a weight, and the weight of each node may be optimized according to learning data as learning proceeds. Each inner product layer may include at least one node, and at least some of the nodes included in the upper layer may be connected to at least some of the nodes included in the next layer.

Although nodes are not shown in the second inner product layer 30-2 to the Lth inner product layer 30-L for ease of understanding in FIG. 26, in an actual neural network model, at least one inner product layer Node.

22 shows that the transform function layer 20 includes six transform function nodes, and the transform function nodes are in turn a sigmoid function, a hyperbolic tangent function, log (10000 x x + 1), log (1000 x x + 1) , log (100 x x + 1), and log (10 x x + 1).

FIG. 27 is a flowchart illustrating an artificial neural network model learning method according to another embodiment of the present invention.

Referring to FIG. 27, the deep learning system 100 may acquire at least one learning data used for learning an artificial neural network model (S100, S110). Here, each of the learning data may include M feature values corresponding to the first to Mth features (M is an integer of 2 or more).

Also, the deep learning system 100 may learn an artificial neural network model using the at least one learning data (S100, S120).

Meanwhile, according to an embodiment, the deep learning system 100 may include a processor and a memory for storing a program executed by the processor. The processor may include a processor such as a CPU, a GPU, an MCU, a microprocessor, and the like, and may include a single-core CPU or a multi-core CPU. The memory may include volatile memory and non-volatile memory. The memory may include, for example, a flash memory, a ROM, a RAM, an EEROM, an EPROM, an EEPROM, a hard disk, and a register. Or the memory may include a file system, a database, and an embedded database. Access to the memory by the processor and other components can be controlled by the memory controller.

Meanwhile, the method according to an embodiment of the present invention may be implemented in the form of computer-readable program instructions and stored in a computer-readable recording medium. A computer-readable recording medium includes all kinds of recording apparatuses in which data that can be read by a computer system is stored.

Program instructions to be recorded on a recording medium may be those specially designed and constructed for the present invention or may be available to those skilled in the art of software.

Examples of the computer-readable recording medium include magnetic media such as a hard disk, a floppy disk and a magnetic tape, optical media such as CD-ROM and DVD, a floptical disk, And hardware devices that are specially configured to store and execute program instructions such as magneto-optical media and ROM, RAM, flash memory, and the like. The computer readable recording medium may also be distributed over a networked computer system so that computer readable code can be stored and executed in a distributed manner.

Examples of program instructions include machine language code such as those produced by a compiler, as well as devices for processing information electronically using an interpreter or the like, for example, a high-level language code that can be executed by a computer.

The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

It will be understood by those skilled in the art that the foregoing description of the present invention is for illustrative purposes only and that those of ordinary skill in the art can readily understand that various changes and modifications may be made without departing from the spirit or essential characteristics of the present invention. will be.

It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive. For example, each component described as a single entity may be distributed and implemented, and components described as being distributed may also be implemented in a combined form.

It is intended that the present invention covers the modifications and variations of this invention provided they come within the scope of the appended claims and their equivalents. .

Claims (7)

As a deep running system,
An acquisition module for acquiring at least one learning data used for learning an artificial neural network model, wherein each of said learning data includes M feature values corresponding to respective first to Mth features (M is an integer of 2 or more) box; And
And a control module for learning the artificial neural network model using the at least one learning data,
Wherein the artificial neural network model comprises:
An apparatus, comprising: an input layer; a modification function layer; at least one inner product layer; and an output layer,
Wherein the transform function layer comprises:
K < / RTI &gt; (K is a natural number greater than or equal to 2) transform function nodes,
The kth transform function node (k is an arbitrary natural number satisfying 1 &lt; = k &lt; = K)
Dimensional vector corresponding to learning data from the input layer, generates a M-dimensional transformed value vector by applying a predetermined k-th transform function to each feature value included in the input M-dimensional vector, Dimensional transform value vector to each node of the uppermost inner product layer,
Wherein the first transform function to the K-th transform function is a function having an S-shaped curve property or a function having a property of increasing a function value and decreasing a differential value at an interval (0,?) .
The method according to claim 1,
Wherein each of the first to K-
sigmoid function, a hyperbolic tangent function, and at least one tf (x) represented by the following equation.
[Formula] tf (x) = log ( α h × x + β h) ( here, h is 1 <= h <= H in each of the integer (H is an integer of 1 or greater), and α _h and β _h is pre-defined Lt; / RTI &gt;
As a deep running system,
An acquisition module for acquiring at least one learning data used for learning an artificial neural network model, wherein each of said learning data includes M feature values corresponding to respective first to Mth features (M is an integer of 2 or more) box; And
And a control module for learning the artificial neural network model using the at least one learning data,
Wherein the artificial neural network model comprises:
An apparatus, comprising: an input layer; a modification function layer; at least one inner product layer; and an output layer,
Wherein the transform function layer comprises:
K < / RTI &gt; (K is a natural number greater than or equal to 2) transform function nodes,
The kth transform function node (k is an arbitrary natural number satisfying 1 < = k < = K)
Dimensional vector corresponding to learning data from the input layer, generates a M-dimensional transformed value vector by applying a predetermined k-th transform function to each feature value included in the input M-dimensional vector, Dimensional transform value vector to each node of the uppermost inner product layer,
Wherein each of the first to K-
sigmoid function, a hyperbolic tangent function, and at least one tf (x) represented by the following equation.
[Formula] tf (x) = log ( α h × x + β h) ( here, h is 1 <= h <= H in each of the integer (H is an integer of 1 or greater), and α _h and β _h is pre-defined Lt; / RTI &gt;
As an artificial neural network model learning method,
An acquisition step of acquiring at least one learning data used for learning an artificial neural network model, wherein each of the learning data includes M feature values corresponding to each of the first feature to the M feature (M is an integer of 2 or more) box; And
And a learning step of learning an artificial neural network model using the at least one learning data,
Wherein the artificial neural network model comprises:
An apparatus, comprising: an input layer; a modification function layer; at least one inner product layer; and an output layer,
Wherein the transform function layer comprises:
K < / RTI &gt; (K is a natural number greater than or equal to 2) transform function nodes,
The kth transform function node (k is an arbitrary natural number satisfying 1 &lt; = k &lt; = K)
Dimensional vector corresponding to learning data from the input layer, generates a M-dimensional transformed value vector by applying a predetermined k-th transform function to each feature value included in the input M-dimensional vector, Dimensional transform value vector to each node of the uppermost inner product layer,
Wherein the first transform function to the K-th transform function are functions having an S-shaped curve property or a function having a property of increasing a function value and decreasing a derivative value in an interval (0,?) Learning method.
As an artificial neural network model learning method,
An acquisition step of acquiring at least one learning data used for learning an artificial neural network model, wherein each of the learning data includes M feature values corresponding to each of the first feature to the M feature (M is an integer of 2 or more) box; And
And a learning step of learning an artificial neural network model using the at least one learning data,
Wherein the artificial neural network model comprises:
An apparatus, comprising: an input layer; a modification function layer; at least one inner product layer; and an output layer,
Wherein the transform function layer comprises:
K < / RTI &gt; (K is a natural number greater than or equal to 2) transform function nodes,
The kth transform function node (k is an arbitrary natural number satisfying 1 < = k < = K)
Dimensional vector corresponding to learning data from the input layer, generates a M-dimensional transformed value vector by applying a predetermined k-th transform function to each feature value included in the input M-dimensional vector, Dimensional transform value vector to each node of the uppermost inner product layer,
Wherein each of the first to K-
sigmoid function, hyperbolic tangent function, and at least one tf (x) expressed by the following equation.
[Formula] tf (x) = log ( α h × x + β h) ( here, h is 1 <= h <= H in each of the integer (H is an integer of 1 or greater), and α _h and β _h is pre-defined Lt; / RTI &gt;
A computer program for performing the method according to claim 4 or 5, installed in a data processing apparatus.
As a deep running system,
A processor; And
A memory for storing a computer program,
Wherein the computer program causes the deep learning system to perform the method of claim 4 or 5 when executed by the processor.

Images