KR101887267B1 - Neural Networks Training Method Using Subnetwork Training algorithms for multi-class prediction neural networks on high level image recognition - Google Patents

Abstract

A partial structure learning technique and a neural network learning method and apparatus using image data are disclosed. The neural network learning method according to an embodiment of the present invention includes: calculating neural network output values for each of multi-dimensional information set in advance based on weights constituting a neural network, with respect to input learning data; Calculating a learning error value based on the neural network output value; Dividing the weights constituting the neural network into a plurality of clusters; And updating the weights included in each of the divided clusters based on the calculated learning error value.

Description

[0001] The present invention relates to a neural network learning method and a neural network learning method and apparatus using image data,

The present invention relates to a neural network learning technique equipped with an image recognition function, and more particularly, to a neural network learning technique in which a neural network, for example, a CNN (Convolutional Neural Network) receives image data and performs multidimensional recognition, And to a neural network learning method and apparatus capable of extracting multi-dimensional information.

It is known that certain problems with computers, such as character recognition and image recognition, can be handled well by machine-learning techniques. The most important of these techniques is the use of neural networks. Neural networks are a class of algorithms based on the idea of interconnected "neurons".

In a typical neural network, a neuron contains data values, including pre-defiend strength for each connection, and whether or not the sum of connections for each particular neuron exceeds a pre-defined threshold, The value affects the value of the connected neuron. By determining appropriate connection strengths and thresholds (also referred to as "training"), the neural network can effectively recognize images and characters. Neurons are grouped mainly into "layers" so as to make connections between the groups more definite and for each operation of the value.

The learning time of the circuit neural network requires a lot of time and calculation amount. Recently, it is possible to shorten the learning time by parallelizing the calculation amount by using GPGPU (General Purpose Graphic Processing Unit) It is incomprehensible to learn information (output data). In order to extract multi-dimensional information from one image data, many neural networks are needed, and the time required to learn them is proportional to the number of neural networks.

Therefore, there is a need for a method for rapidly learning object recognition function and multidimensional information of an object of a circuit neural network.

Embodiments of the present invention provide a neural network learning method and apparatus capable of rapidly learning object recognition functions of a circuit neural network and multidimensional information of objects using partial structure learning techniques and image data.

Embodiments of the present invention provide a neural network learning method and apparatus capable of improving learning speed by reducing the amount of computation of a circuit neural network required for learning and recognizing multidimensional image information.

The neural network learning method according to an embodiment of the present invention includes: calculating neural network output values for each of multi-dimensional information set in advance based on weights constituting a neural network, with respect to input learning data; Calculating a learning error value based on the neural network output value; Dividing the weights constituting the neural network into a plurality of clusters; And updating the weights included in each of the divided clusters based on the calculated learning error value.

The updating may sequentially update the weights included in each of the divided clusters for each of the divided clusters.

The updating may determine whether the computed learning error value is greater than or equal to a predefined reference error value and update the weights included in each of the divided clusters if the learning error value is greater than or equal to a reference error value.

The updating may update the weights included in each of the divided clusters using a slope descent technique.

The step of calculating the neural network output value may calculate a neural network output value for each of the multi-dimensional information based on weights constituting a convolutional neural network (CNN).

According to another aspect of the present invention, there is provided a neural network learning method comprising: dividing weights constituting a neural network into a plurality of clusters; Updating the weights included in each of the plurality of clusters; And learning the neural network through learning for each of the plurality of clusters with respect to input learning data.

The neural network learning apparatus according to an embodiment of the present invention includes a neural network operation unit for calculating a neural network output value for each of multi-dimensional information set in advance based on weights constituting a neural network, with respect to input learning data; An error calculation unit for calculating a learning error value based on the neural network output value; And a weight control unit for dividing the weights constituting the neural network into a plurality of clusters, and updating weights included in each of the divided clusters based on the calculated learning error value.

The weight control unit may update the weights included in each of the divided clusters sequentially for each of the divided clusters.

The weight control unit may determine whether the computed learning error value is greater than or equal to a predetermined reference error value and update the weights included in the divided clusters if the learning error value is greater than or equal to the reference error value.

The weight control unit may update the weights included in each of the divided clusters using a slope descent technique.

The neural network operation unit may calculate a neural network output value for each of the multi-dimensional information based on weights constituting a Convolutional Neural Network (CNN).

According to embodiments of the present invention, it is possible to rapidly learn the object recognition function of the circuit neural network and the multidimensional information of the object using the partial structure learning technique and the image data, and to improve the learning speed by reducing the computational amount of the circuit neural network required for learning .

Also, according to embodiments of the present invention, it is possible to extract multi-dimensional information using a combination of a biologically plausible-neighbor cluster technique and a technique that imitates brain perception, and when applied to an intelligent system, Accuracy can be improved.

In addition, according to the embodiments of the present invention, it is possible to apply to an intelligent robot, and when applied to an intelligent robot, it is possible to maximize the accuracy of a task to be solved by a robot as well as a vision recognition problem, which is an aspect of an intelligent robot.

1 is a flowchart illustrating an operation of a neural network learning method according to an embodiment of the present invention.
2 is a diagram illustrating an example of a partial learning using a cluster in a neural network for extracting multi-dimensional information.
3 illustrates an exemplary diagram for explaining a neighbor cluster in the present invention.
Fig. 4 shows examples of learning data
Figure 5 shows an example of learning error by the method according to the invention.
FIG. 6 is a diagram illustrating an example of comparison of learning time between the existing learning method and the learning method of the present invention.
FIG. 7 illustrates a configuration of a neural network learning apparatus according to an embodiment of the present invention.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to or limited by the embodiments. In addition, the same reference numerals shown in the drawings denote the same members.

Embodiments of the present invention are based on the fact that the object recognition function of the circuit neural network and the multi-dimensional information of the object are rapidly learned using the partial structure learning technique and the image data.

Here, the present invention can extract multidimensional information using a combination of a biologically plausible-neighbor cluster technique and a technique that mimics brain perception.

The learning method and apparatus according to the present invention divides the weights constituting the neural network into a plurality of clusters and sequentially updates the weights included in the divided clusters for each of the clusters, thereby reducing the amount of computation required to update the weights , Which can improve learning speed.

와 그에 대응되는 클래스 벡터

, k번째 자료의 q 번째 값은 1 나머지는 0인 벡터를 하나의 학습자료로 설정하고 여러 개의 인스턴스(

)를 가지는 학습데이터베이스를 사용한다.In the learning process, the kth RGB camera pixel data

And its corresponding class vector

, the q-th value of the k-th data is 1, and the rest is 0 (zero).

) Is used.

은 DB의 3번째 학습자료는 1번째 라벨에서 1번째 물체 종류 및 2번째 라벨에서 2번째 물체 종류에 포함된다는 것을 의미할 수 있다.The class vector is based on a multiple one-hot encoding technique. E.g,

May mean that the third learning material of the DB is included in the first object type in the first label and in the second object type in the second label.

The object type and the class vector, for example, the supervised label (target vector) used for learning, can be set by the user. For example, when an input RGB image wants to distinguish between two types of objects As shown in FIG.

의 경우 5번째 학습자료는 자동차를 의미할 수 있고,

의 경우 10번째 학습자료는 비행기를 의미할 수 있다.E.g,

In the case of the 5th learning material, it could mean a car,

, The tenth learning material can mean an airplane.

Since these two objects belong to the machine, they can be set as follows to represent the machine.

의 경우 5번째 학습자료는 기계를 의미할 수 있고,

의 경우 10번째 학습자료는 기계를 의미할 수 있다.E.g,

, The fifth learning material may be a machine,

, The tenth learning material may mean a machine.

The method and apparatus of the present invention may include a process of calculating an output value of a neural network, a process of calculating a learning error value, and a process of controlling a weight, and the method and apparatus according to the present invention will be described with reference to Figs. 1 to 7 The following will be described.

1 is a flowchart illustrating an operation of a neural network learning method according to an embodiment of the present invention.

Referring to FIG. 1, a neural network learning method according to an embodiment of the present invention initializes weights constituting a neural network, for example, a circuit neural network, and inputs learning data to be learned, for example, a learning image (S110, S120 ).

Here, the weights constituting the neural network are weights connecting the layers constituting the neural network, for example, the input layer, the hidden layer and the output layer. In step S110, these weights are initialized to predetermined values , The learning data input in step S120 may be various types of image data such as an airplane, an automobile, and a ship, as shown in an example shown in FIG. Of course, the learning data input in the present invention is not limited to the input image shown in FIG. 4, and may include all kinds of images to be learned.

The learning labels to be learned from the input learning image of FIG. 4 are learned by the types of objects included in the image and the purpose of use thereof, for example, in the case of airplanes, for civilian use or military use. In the case of automobiles, . In the case of ship, it can be learned by cargo, civilian, military, etc. In other words, it is aimed to learn learning level including multidimensional information from learning data.

When the input learning data is received in step S120, a feed forward calculation process for the neural network, a process for calculating a learning error value, and a process for dividing or dividing the weights constituting the neural network into a plurality of clusters, The neural network is learned (S130 to S160).

Here, steps S120 to S160 may be repeated until the calculated learning error value is smaller than a predefined reference error value.

Hereinafter, steps S130 to S160 will be described as follows.

The feedforward operation process (S130) for the neural network is a process of calculating a neural network output value for each of the multidimensional information set in advance based on the weights constituting the neural network with respect to the input learning data.

The lth line layer from the RGB image I (or the color space of HSV and CMKY) received from the camera device can be calculated as shown in Equation (1) below.

[Equation 1]

와

은 l-1번째 계층의 m번째 특징맵과 l 번째 계층의 n 번째 특징맵을 연결하는 회선 가중치 행렬을 의미하며,

은 n 번째 계층의 l번째 바이어스를 의미할 수 있다.Here, f means a neural network activation function,

Wow

Denotes a circuit weighting matrix connecting the m-th feature map of the (l-1) -th layer and the n-th feature map of the l-th layer,

May denote the lth bias of the nth layer.

The calculation of the lth sub-sampling layer can be calculated as Equation (2) below using the output value of the circuit layer.

&Quot; (2) "

를 의미하고, i와 j는 회선 계층의 픽셀의 인덱스를 의미하며,

와

은 인덱스와 연결된 가중치와 바이어스를 각각 의미할 수 있다.here,

I & j denotes an index of a pixel in the line layer,

Wow

May mean a weight and a bias associated with the index, respectively.

The output value of the last layer of the circuit neural network can be calculated as Equation (3) below.

&Quot; (3) "

와

는 각각 가중치와 바이어스를 의미하고,

는 서브 샘플링 계층 계산으로부터 획득되는 값을 의미할 수 있다.here,

Wow

Quot; means a weight and a bias, respectively,

May mean a value obtained from the subsampling layer computation.

또는

를 가질 수 있고, C는 영상을 분류하고자 하는 클래스의 개수 즉, 출력 노드의 개수를 의미한다.The unit of the subsampling layer of the last layer, for example the neural network node, is fully connected. The activation function of the output layer of the circuit neural network is

or

And C denotes the number of classes to classify images, that is, the number of output nodes.

The process S140 of calculating the learning error value is a process of calculating a learning error value based on the neural network output value calculated in step S130. Lt; / RTI >

That is, in the step of calculating the learning error value (S140), the degree of the weight to be updated is calculated, and the value of the objective function is calculated. The objective function can be defined as Equation (4) below .

&Quot; (4) "

는 q 번째 클래스 라벨의 k 번째 자료의 목적 벡터의 j 번째 값을 의미하며,

은 신경망의 출력 노드의 활성화 값을 의미하는 것으로, k 번째 자료를 신경망에 입력했을 때의 값을 의미할 수 있고, k는

일 수 있으며, N은 학습 자료의 개수일 수 있고, j는

일 수 있으며, C는 영상을 분류하고자 하는 클래스의 개수일 수 있다. Here, Q denotes the number of multi-dimensional classes,

Denotes the j-th value of the k-th data of the q-th class label,

Is the activation value of the output node of the neural network, which can be the value when the kth data is input to the neural network,

, N may be the number of learning materials, j may be

And C may be the number of classes to classify the image.

The activation value of the output node of the neural network can be calculated as shown in Equation (5) below.

&Quot; (5) "

In step S150, a process of dividing or dividing the weights constituting the neural network into a plurality of clusters and sequentially updating weights for each of the divided clusters will be described. In step S150, the neural network is configured based on a predetermined criterion or rule. Dividing the weights into a plurality of clusters, and step S160 may be a process of updating the weights included in each of the clusters for each of the clusters divided by step S150.

For example, the step S150 may divide the weights into clusters in units of 10 to 20%.

의 학습 규칙은 상기 수학식 4의 목적 함수를 최소화시키며, 가중치 업데이트는 아래 <수학식 6>과 같은 기울기 강하 기법을 사용할 수 있다.Describing these steps S150 to S160,

The learning rule of Equation (4) minimizes the objective function of Equation (4), and the gradient updating method such as Equation (6) below can be used for weight updating.

&Quot; (6) &quot;

는 학습 계수를 의미하는 것으로, 0과 1 사이의 값을 가지며 사용자의 실험 결과에 따른 변수일 수 있으며,

은 아래 <수학식 7>과 같이 계산될 수 있다.here,

Is a learning coefficient, which has a value between 0 and 1 and can be a variable according to the user's experimental result,

Can be calculated as shown in Equation (7) below.

&Quot; (7) &quot;

를 의미하고, L은 출력층을 의미하고, f는 신경망 활성화 함수를 의미할 수 있다.Here, s _j ^(k) is the k-th and the mean value of the j-th label of the learning data, o _j ^{(k, q)} is the q th label or class j-th neural networks when the k-th learning data is input to the neural network Output value, qi _{, j} denotes a weight value

, L denotes the output layer, and f denotes the neural network activation function.

Also, step S150 may define a neighbor cluster and update only the neural network weights in the defined neighbor cluster.

In this case, the portion of the neighbor cluster in which the weight is updated is the neighbor nodes of the lower layer connected to the nodes of the upper layer, and the node of another higher layer and the neighbor nodes of the lower layer connected thereto can be selected in the neighbor cluster per learning epoch.

For example, the neighbor cluster between the input data and the line weights shown in FIG. 3A shows neighboring clusters between the input data units connected to the unit of the line layer, the bold line and the thin line represent the connection weight, and the bold line represents the weight Is updated.

3B shows a neighbor cluster between the subsampling layer and the output layer. Likewise, the bold line and thin line represent connection weights, and the bold line represents a portion where weights are updated in neighboring clusters.

As shown in FIG. 2A, the present invention recognizes multidimensional information such as a first classification label, a second classification label, and a third classification label through learning of a neural network. .

At this time, the multidimensional information, for example, the first classification label, the second classification label, and the third classification label may be a facial expression for identification, smile or sod or happy, man or woman for Alice or Bob, Gender, etc., and such multidimensional information can be recognized through partial learning using clusters (bold lines in FIG. 2B).

As described above, the neural network learning method according to the present invention can extract and recognize desired multi-dimensional information by learning only the partial weight of the relevant part of the image data using the clusters. The weight of each of the plurality of divided clusters can be divided into clusters Since each of them is sequentially updated, the learning speed can be improved by reducing the amount of computation of the neural network required for learning.

As shown in the learning error according to the epoch of FIG. 5, the method according to the present invention shows that the error mean value MSE (blue line) and the error variance value (orange line) decrease with the epoch. Here, FIG. 5 shows learning errors according to epochs in the circuit neural network structure.

The method according to the present invention is similar to the conventional backpropagation method, as shown in Table 1 below. For each data set of MNIST, small NORB, and cifar-10, . &Lt; / RTI &gt;

7, the learning time for MNIST, small NORB, and cifar-10, which are three image data sets, is compared with learning time of the learning method of the present invention. It can be seen that the learning rate is decreased by 25 ~ 30 [%] compared with the conventional backpropagation technique.

As described above, the learning speed can be improved while maintaining the learning accuracy by reducing the amount of computation of the neural network with the accuracy similar to that of the existing learning method and reducing the learning time through the partial learning of the present invention.

FIG. 7 illustrates a configuration of a neural network learning apparatus according to an embodiment of the present invention, and shows a configuration of an apparatus for performing the operations of FIGS. 1 to 6 described above.

7, a neural network learning apparatus 700 according to an embodiment of the present invention includes a neural network operation unit 710, an error calculation unit 720, and a weight control unit 730.

When the input learning data is received, the neural network operation unit 710 calculates a neural network output value for each of the multidimensional information set in advance based on the weights constituting the neural network with respect to the input learning data.

In this case, the neural network operation unit 710 may initialize the weights constituting the CNN, and then calculate the neural network output values for each of the multi-dimensional information using the weights constituting the CNN with respect to the input learning data .

The error calculator 720 calculates a learning error value based on the computed neural network output value.

Here, the error calculator 720 may calculate the objective function value of Equation (4) as a constituent means for calculating how much weight to update.

The weight control unit 730 divides the weights constituting the neural network into a plurality of clusters and updates weights included in each of the clusters divided based on the calculated learning error value.

In this case, the weight control unit 730 may sequentially update the weight values included in the divided clusters for each of the divided clusters, and determine whether the calculated learning error value is equal to or greater than a predetermined reference error value, And update the weights included in each of the divided clusters if the error value is greater than or equal to the reference error value.

The weight control unit 730 may update the weights included in each of the clusters divided using the gradient descent method. Of course, the weight control unit 730 is not limited to updating the weights by the slope descending method, and various methods of updating the weights constituting the neural network may be applied.

Although not illustrated in FIG. 7, the apparatus of FIG. 7 may perform all of the operations of FIGS. 1 through 6 described above and may include all of the contents of FIGS.

The system or apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components. For example, the systems, devices, and components described in the embodiments may be implemented in various forms such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array ), A programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For ease of understanding, the processing apparatus may be described as being used singly, but those skilled in the art will recognize that the processing apparatus may have a plurality of processing elements and / As shown in FIG. For example, the processing unit may comprise a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of the foregoing, and may be configured to configure the processing device to operate as desired or to process it collectively or collectively Device can be commanded. The software and / or data may be in the form of any type of machine, component, physical device, virtual equipment, computer storage media, or device , Or may be permanently or temporarily embodied in a transmitted signal wave. The software may be distributed over a networked computer system and stored or executed in a distributed manner. The software and data may be stored on one or more computer readable recording media.

The method according to embodiments may be implemented in the form of a program instruction that may be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions to be recorded on the medium may be those specially designed and configured for the embodiments or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI &gt; or equivalents, even if it is replaced or replaced.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (11)

Calculating a neural network output value for each of the preset multi-dimensional information based on weights constituting the neural network with respect to the input learning data in the neural network operation unit;
Calculating a learning error value based on the neural network output value in an error calculation unit;
Dividing the weights constituting the neural network into a plurality of clusters in a weight control unit; And
Updating the weights included in each of the divided clusters based on the calculated learning error value in the weight control unit
Lt; / RTI &gt;
The updating step
Defining neighbors of a lower layer connected to a node of an upper layer as a neighbor cluster for each of the clusters and updating only the weights included in the defined neighbor cluster.
The method according to claim 1,
The updating step
Wherein the weights included in each of the divided clusters are sequentially updated for each of the divided clusters.
The method according to claim 1,
The updating step
Determining whether the computed learning error value is greater than or equal to a predefined reference error value, and updating the weights included in each of the divided clusters if the learning error value is greater than or equal to a reference error value.
The method according to claim 1,
The updating step
And the weights included in each of the divided clusters are updated using a slope descent technique.
The method according to claim 1,
The step of computing the neural network output value
Wherein the neural network output value for each of the multi-dimensional information is calculated based on weights constituting a convolutional neural network (CNN).
Dividing weights constituting a neural network into a plurality of clusters in a neural network learning apparatus;
Updating the weights included in each of the plurality of clusters in the neural network learning apparatus; And
Learning the neural network through learning for each of the plurality of clusters with respect to input learning data in the neural network learning apparatus
Lt; / RTI &gt;
The updating step
Defining neighbors of a lower layer connected to a node of an upper layer as a neighbor cluster for each of the clusters and updating only the weights included in the defined neighbor cluster.
A neural network operation unit for calculating a neural network output value for each of the multidimensional information set in advance based on the weights constituting the neural network with respect to the input learning data;
An error calculation unit for calculating a learning error value based on the neural network output value; And
Dividing the weights constituting the neural network into a plurality of clusters and updating the weights included in each of the divided clusters based on the calculated learning error value,
Lt; / RTI &gt;
The weight control unit
Defining neighbors of the lower layer connected to the nodes of the upper layer as neighboring clusters and updating only the weights included in the neighboring cluster defined for each of the clusters.
8. The method of claim 7,
The weight control unit
And sequentially updates the weights included in each of the divided clusters for each of the divided clusters.
8. The method of claim 7,
The weight control unit
Wherein the determining unit determines whether the calculated learning error value is equal to or greater than a predetermined reference error value and updates weight values included in each of the divided clusters when the learning error value is equal to or greater than a reference error value.
8. The method of claim 7,
The weight control unit
And the weights included in each of the divided clusters are updated using a slope descent technique.
8. The method of claim 7,
The neural network operation unit
And calculates a neural network output value for each of the multi-dimensional information based on weights constituting a convolutional neural network (CNN).

Images