KR102100973B1 - Machine learning system using joint learning and the method thereof - Google Patents

Abstract

The present invention relates to a technology for jointly learning a teacher learning module and a student learning module, between a teacher learning module and a student learning module branched in a shared engine that extracts features of input data using a loss function. Minimizing errors and characteristic differences and jointly learning the teacher learning module and the student learning module by the loss function.

Description

MACHINE LEARNING SYSTEM USING JOINT LEARNING AND THE METHOD THEREOF}

The present invention relates to a machine learning system using co-learning and a method thereof, and more particularly, to a technique for jointly learning a teacher learning module and a student learning module.

Facial Landmark Detection (FLD) is a method of localizing (locating) key points around the face around the face, such as eyes, nose, nose, mouth, eyebrows and facial contours. Since FLD plays an important role in various face-related applications, it has attracted much attention in face analysis. For example, face markings are used for facial verification and postural composition in facial recognition and head posture evaluation. In addition, a recently studied GAN (Generative Adversarial Network) model generates a face image through a face marker.

Most of the previous studies were based on hand-crafted FLDs. For example, in a research paper by Liang (Liang, L., Xiao, R., Wen, F., Sun, J .: Face alignment via componentbased discriminative search. Computer VisionECCV 2008 7285 (2008)), face marker points Has been improved through iterative optimization. In addition, a paper from BurgosArtizzu (BurgosArtizzu, XP, Perona, P., Doll Pr, P .: Robust face landmark estimation under occlusion.In: Proceedings of the IEEE International Conference on Computer Vision, pp. 15131520. (2013)) Proposed a regression-based approach (referred to as RCPR).

In addition, recent FLD research has focused on convolutional neural networks (CNNs). For example, Sun (Sun, Y., Wang, X., Tang, X .: Deep convolutional network cascade for facial point detection.In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 34763483. ( 2013)) proposed a three-step CNN for FLD through the paper, Zhang (Zhang, Z., Luo, P., Loy, CC, Tang, X .: Learning deep representation for face alignment with auxiliary attributes.IEEE transactions on pattern analysis and machine intelligence 38,918930 (2016)) proposed a multi-task learning method (called TCDCN) through a paper. However, most CNN-based FLD methods require a large number of network parameters and related computational costs. Therefore, these methods are not suitable for mobile applications. Accordingly, a compact neural network having few parameters in a limited environment is required.

Recent research has focused on addressing CNN memory and computational cost issues. For example, Howard (Howard, AG, Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., Adam, H .: Mobilenets: Efficient convolutional neural networks for mobile vision applications.arXiv preprint arXiv: 1704.04861 (2017)) proposed MobileNets using convolution that allows for depthwise separation through the paper. The above paper proposed to reduce the number of convolution filter parameters and computational complexity by using depth-wise separable convolution and 1 × 1 point-wise convolution.

In addition, Lee (Lee, TK, Baddar, WJ, Kim, ST, Ro, YM: Convolution with Logarithmic Filter Groups for Efficient Shallow CNN.arXiv preprint arXiv: 1707.09855 (2017)) is a logarithmic group filter A compact convolutional layer was proposed. In addition, Lin (Lin, M., Chen, Q., Yan, S .: Network in network. ArXiv preprint arXiv: 1312.4400 (2013)) proposed global average pooling to replace the fully connected layer, Decreased.

In addition, as described above, not only a convolution technique that replaces the fully connected layer with global average pooling, but also a knowledge distillation method is useful for reducing network parameters.

In the knowledge distillation method, the 'student' network is trained using the knowledge acquired about the original 'teacher' network. At this time, the student network was trained to mimic a network of pre-trained teachers. Accordingly, this method has limitations because a network of pre-trained teachers is required. Furthermore, if there is no teacher network previously trained, there is a problem that too much time is consumed when training the teacher network and the student network.

An object of the present invention is to provide a system and method for jointly learning a teacher learning module and a student learning module to reduce the size of a machine learning model.

In addition, an object of the present invention is to propose a new compact machine learning model targeting a mobile application, and to provide a new teacher and student co-learning method that can be applied to a compact machine learning model.

In the machine learning method using joint learning according to an embodiment of the present invention, errors and characteristic differences between a teacher learning module and a student learning module branched from a sharing engine extracting features of input data using a loss function And minimizing and jointly learning the teacher learning module and the student learning module by the loss function.

In addition, the machine learning method using the joint learning according to the embodiment of the present invention classifies and regresses the input data using the network parameters of the learned student learning module and the sharing engine. It may further include the step of reasoning.

The step of minimizing the error and characteristic difference between the branched teacher learning module and the student learning module is a marking error of the teacher learning module and the student learning module using the first loss function and the second loss function among the three loss functions. And minimize the characteristic difference between the teacher learning module and the student learning module using a third loss function.

The step of jointly learning the teacher learning module and the student learning module by the loss function may train the student learning module imitating the teacher learning module by the final loss function among the three loss functions.

The step of jointly learning the teacher learning module and the student learning module by the loss function includes the student learning module, which is compact architectures, and the teacher learning module including a fully connected layer. It can be learned jointly and emulate each other.

The step of inferring classification and regression values for the input data may infer classification and regression values for the input data through a compact machine learning model according to the network parameter received from the sharing engine and the student learning module. have.

The machine learning model may remove the teacher learning module and include a compact learning module constructed from the student learning module and the sharing engine.

In the machine learning system using the joint learning according to the embodiment of the present invention, errors and characteristic differences between the teacher learning module and the student learning module branched from the shared engine extracting the characteristics of the input data using the loss function And a joint learning unit for jointly learning the teacher learning module and the student learning module by a minimizing processing unit and the loss function.

In addition, the machine learning system using joint learning according to an embodiment of the present invention classifies and regresses values of the input data by using the network parameters of each of the learned student learning module and the sharing engine. The reasoning unit may further include a reasoning unit.

The processing unit minimizes the marking error of the teacher learning module and the student learning module by using a first loss function and a second loss function among the three loss functions, and the teacher learning module and the student by using a third loss function. Differences in characteristics between learning modules can be minimized.

The co-learning unit may train the student learning module imitating the teacher learning module by a final loss function among the three loss functions.

The co-learning unit may jointly learn the student learning module, which is compact architectures, and the teacher learning module including a fully connected layer, to emulate each other.

The inference unit may infer classification and regression values for the input data through a compact machine learning model according to the network parameter received from the sharing engine and the student learning module.

The machine learning system using the co-learning according to another embodiment of the present invention co-learns the teacher learning module and the student learning module branched in a sharing engine that extracts features of the input data, thereby mimicking the teacher learning module. It includes a training unit for training the student learning module and a test unit for constructing a compact machine learning model including the shared engine and a compact learning module to infer classification and regression values for the input data. do.

The training unit equally supplies the features encoded by the shared layer in the sharing engine to the teacher learning module and the student learning module, and the encoded features through the teacher learning module and the student learning module. Can be converted to classification and regression values.

The training unit is characterized by training to jointly train the teacher learning module and the student learning module using three loss functions, and the teacher uses the first loss function and the second loss function among the three loss functions. Marking errors of the learning module and the student learning module may be minimized, and a difference in characteristics between the teacher learning module and the student learning module may be minimized using a third loss function.

The test unit may remove the teacher learning module and infer classification and regression values for the input data through the compact machine learning model constructed by constructing the compact learning module from the student learning module.

According to an embodiment of the present invention, the compact architecture of the student learning module can be applied to the compact machine learning model by co-learning with the fully connected layer of the teacher learning module and imitating each other.

1 is a block diagram of a detailed configuration of a machine learning system using joint learning according to an embodiment of the present invention.
Figure 2 shows an application example of the proposed teacher and student co-learning method for a compact machine learning model according to another embodiment of the present invention.
3 shows details of a joint learning method between a teacher learning module and a student learning module according to an embodiment of the present invention.
4A and 4B show experimental results verifying the performance of a machine learning system using joint learning according to an embodiment of the present invention.
5 is a flowchart of a machine learning method using joint learning according to an embodiment of the present invention.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings. However, the present invention is not limited or limited by the embodiments. In addition, the same reference numerals shown in each drawing denote the same members.

In addition, terms used in the present specification (terminology) are terms used to properly express a preferred embodiment of the present invention, which may vary depending on viewers, operator's intentions, or customs in the field to which the present invention pertains. Therefore, definitions of these terms should be made based on the contents throughout the present specification.

Recently, a widely used mobile application requires a compact neural network with limited memory and computation. Thus, as an example of using a machine learning system and method using a joint learning according to an embodiment of the present invention, by constructing a compact facial landmark detection network (compact Facial Landmark Detection; compact FLD) that is a compact neural network for face mark detection , It solves the problems and limitations of the prior art, provides a neural network with a small number of parameters in a limited environment, and makes the size of the machine learning model lighter.

Face mark detection is a necessary frontal module for face analysis applications. The present invention proposes a new teacher and student co-learning method that can be applied to a compact face mark detection network.

However, hereinafter, focusing on the detection of landmarks of 'face mark detection', the teacher and student co-learning method proposed in the present invention includes not only landmark detection, but also all inference models, for example, object recognition, Since it can be applied to deep network technologies such as location inference and emotion recognition, it is not limited to face mark detection.

Hereinafter, a machine learning system and method using the joint learning according to an embodiment of the present invention will be described in more detail with reference to FIGS. 1 to 5.

1 is a block diagram showing a detailed configuration of a machine learning system using joint learning according to an embodiment of the present invention, and FIG. 5 is a flowchart of a machine learning method using joint learning according to an embodiment of the present invention. It is done.

The machine learning system 100 using joint learning according to an embodiment of the present invention jointly trains a teacher learning module and a student learning module.

To this end, the machine learning system 100 using joint learning according to an embodiment of the present invention includes a processing unit 110 and a joint learning unit 120. At this time, the processing unit 110 and the joint learning unit 120 may be configured to perform the steps 510 to 520 of FIG. 5.

In step 510, the processor 110 uses a loss function to extract a feature of the input data, a teacher learning module (or a teacher regression network) branched from a shared engine (or a shared convolution network). Minimize error and character differences between the teacher regression network and the student learning module (or student regression network). In this case, the input data may be various data such as image data, audio data, content data, and the like, and is not limited.

The processing unit 110 may minimize errors and characteristic differences between the teacher learning module and the student learning module using three loss functions. For example, among the three types of loss functions, the first learning function and the second loss function are used to minimize the marking errors of the teacher learning module and the student learning module, and the third loss function is used to learn the teacher learning module and the student learning. Differences in characteristics between modules can be minimized. At this time, the student learning module may mimic the teacher learning module by the final loss function among the three loss functions.

For example, two of the three loss functions may minimize marking errors in the teacher learning module and the student learning module, and another loss function may minimize the characteristic difference between the teacher learning module and the student learning module.

In step 520, the joint learning unit 120 jointly trains the teacher learning module and the student learning module by a loss function.

For example, the co-learning unit 120 may train a student learning module that mimics a teacher learning module by a final loss function among three loss functions.

In the machine learning system 100 using the joint learning according to the embodiment of the present invention, the teacher learning module and the student learning module are connected to each other through a sharing engine, and then branched. At this time, the student learning module is a compact architecture, while the teacher learning module includes a fully connected layer including large-sized parameters, and the teacher learning module's fully connected layer and the student learning module. Its compact architecture can be learned jointly and imitate each other. That is, in the machine learning system 100 using the joint learning according to the embodiment of the present invention, the teacher learning module and the student learning module are simultaneously learned with the sharing engine. This differs from traditional teacher and student learning, which includes a network of pre-trained teachers and a student network that is learned separately.

In addition, the sharing engine extracts features for the input data, and each of the teacher learning module and the student learning module can estimate points using features encoded by the sharing engine. Accordingly, the features encoded by the shared layer in the sharing engine are supplied to the teacher learning module and the student learning module equally, so that the teacher learning module and the student learning module can convert the encoded features into classification and regression values. have.

1 and 5, the machine learning system 100 using joint learning according to an embodiment of the present invention may further include a reasoning unit 130, and the reasoning unit 130 may include step 530 of FIG. 5. It can be configured to perform.

The inference unit 130 may infer classification and regression values of the input data using the network parameters of the learned student learning module and the sharing engine.

For example, the reasoning unit 130 may infer classification and regression values for input data through a compact machine learning model including a sharing engine and a compact regression network. According to an embodiment, the machine learning model may be a compact facial landmark detection (compact FLD) network.

The machine learning model is composed of a shared engine and a compact learning module, the teacher learning module is removed, and the student learning module can transmit network parameters to the compact learning module. At this time, each of the shared engine and the student learning module may transmit network parameters in a compact machine learning model. Accordingly, the inference unit 130 may infer classification and regression values for input data through a compact machine learning model.

That is, since all the frameworks in the machine learning system 100 using the joint learning according to the embodiment of the present invention are capable of learning, there is no need for a pre-trained existing teacher network, and the student learning module of the present invention is a teacher Performance similar to the learning module can be achieved.

Figure 2 shows an application example of the proposed teacher and student co-learning method for a compact machine learning model according to another embodiment of the present invention.

Referring to FIG. 2, a machine learning system using joint learning according to another embodiment of the present invention includes a training unit (or training step 210) and a testing unit (or test step, 220).

In the training unit 210, the network is composed of three sub-modules 211, 212, and 213.

The first network is a sharing engine (or shared convolution network, 211) that extracts the characteristics of the input data. The second is a teacher learning module (or teacher regression network, 212), and the third is a student learning module (or student regression network, 213). Accordingly, the sharing engine 211 extracts the feature map for the input data, and the training unit 210 jointly trains the teacher learning module 212 and the student learning module 213 branched from the sharing engine 211. , Train a student learning module 213 that mimics the teacher learning module 212.

Referring to FIG. 2, for example, the sharing engine 211 may extract a feature map of facial components in input data (or an input image), and use the feature map of the sharing engine 211 The teacher learning module 212 and the student learning module 213 may estimate facial landmarks coordinates. In the teacher learning module 212 and the student learning module 213, facial landmark points (green dots on an image in FIG. 2) may be estimated through features encoded by the sharing engine 211. have. However, the input data may be various data such as image data, audio data, and content data, and thus is not limited to images.

In the training unit 210, the teacher learning module 212 and the student learning module 213 are simultaneously learned with the sharing engine 211. At this time, the teacher learning module 212 and the student learning module 213 may be co-learned with three loss functions. For example, the estimated learning error between the teacher learning module 212 and the student learning module 213 is minimized using two of the three loss functions, and the teacher learning module 212 using another loss function. ) And the difference between the features between the student learning module 213 can be minimized. Further, by the final loss function, the student learning module 213 can mimic the teacher learning module 212.

That is, in the training unit 210, the teacher learning module 212 and the student learning module 213 use the same features of the sharing engine 211, so that the student learning module 213 can easily make the teacher learning module 212 easy. You can imitate. Through such a learning process, the compact learning module 221 may maintain excellent performance with a small number of parameters, and the student learning module 213 mimics the teacher learning module 212 so that a pre-trained teacher network (or teacher learning module) It does not need.

Referring to FIG. 2 again, in the test unit 220, the compact machine learning model is composed of two sub-modules 211 and 221. In this case, the machine learning model may be a compact facial landmark detection network (compact FLD).

The first network is a sharing engine 211, and the second network is a compact learning module (or compact regression network, 221). At this time, the sharing engine 211 receives network parameters from the sharing engine 211 in the training unit 210, and the compact learning module 221 is a student learning module 213 in the training unit 210. ).

Hereinafter, through the following [Table 1] to look at the occupancy (Occupation) of the parameters in the existing face mark detection network (FLD).

[Table 1]

[Table 1] shows the number of parameters of the recently reported face mark detection network (FLD). One is TCDCN and the other is DCNN­I / C. At this time, TCDCN is Zhang, Z., Luo, P., Loy, C.C., Tang, X .: Learning deep representation for face alignment with auxiliary attributes. Reported by IEEE transactions on pattern analysis and machine intelligence 38,918930 (2016), DCNNI / C is Baddar, WJ, Son, J., Kim, DH, Kim, ST, Ro, YM: A deep facial landmarks detection with facial contour and facial components constraint. In: Image Processing (ICIP), 2016 IEEE International Conference on, pp. 3209­3213. IEEE, (2016).

TCDCN is a simple structure with 4 convolutional layers including one fully connected layer and one output layer. DCNN­I / C is an FLD network composed of two sub-networks. At this time, in DCNN (I / C, one sub-network (eg, DCNN­C) can be used to detect the face contour marker, and the other (eg, DCNN­I) can be used to detect the face composition. At this time, the DCNNC includes four convolutional layers including one fully connected layer and one output layer, and DCNNI includes three common convolutional layers and five internal facial component convolutional layers, each The inner facial composition convolution layer itself has a fully connected layer and an output layer.

That is, referring to [Table 1], more than half of all parameters are included in the complete connection layer. In this regard, the machine learning system using joint learning according to an embodiment of the present invention can minimize the number of parameters in the fully connected layer by constructing a compact face mark detection network.

Referring back to FIG. 2, in the test unit 220 of the machine learning system using the joint learning according to the embodiment of the present invention, the student learning module 213 has a convolution of 1 × 1 instead of a flat layer and a fully connected layer. And global average pooling, which can significantly reduce network parameters. In addition, since the student learning module 213 is trained to imitate the teacher learning module 212, it can provide performance comparable to the teacher learning module 212. Accordingly, the compact face mark detection network (compact FLD) constructed with the sharing engine 211 and the compact learning module 221 may significantly reduce the number of network parameters.

3 shows details of a joint learning method between a teacher learning module and a student learning module according to an embodiment of the present invention.

Referring to FIG. 3, the two networks branch to the teacher learning module 320 and the student learning module 330 at the end of the shared layer (or shared convolutional layers) 310 in the sharing engine. )do. At this time, the fully connected layer of the teacher learning module 320 is referred to as the TFC 322, the corresponding layer of the student learning module 330 is referred to as the STDFC 332, and the STDFC layer 332 is a 1 × 1 convolution. It consists of a solution (1x1 convolution) and global average pooling (GAP, 333). The dimensions of the STD­FC 332 and T­FC 322 are the same.

Features encoded by the shared layer 310 are simultaneously supplied to the teacher learning module 320 and the student learning module 330. The encoded features are converted into classification and regression values through the teacher learning module 320 and the student learning module 330. At this time, the input of the teacher learning module 320 and the student learning module 330 is characterized in that it is the same as that encoded by the shared layer 310.

Since the fully connected layer converts the CNN (Convolutional Neural Networks) feature map into marker coordinates, a compact machine learning model according to an embodiment of the present invention (e.g., a compact facial marker detection network) Facial Landmark Detection (compact FLD)) plays an essential role in machine learning systems.

Accordingly, a loss function designated for joint learning is required so that the STD_FC layer 332 mimics the T_FC layer 322. To this end, three loss functions have been devised in a machine learning system using joint learning according to an embodiment of the present invention.

Of the three loss functions, the first loss function L ¹ serves to minimize the estimated error of the teacher regression output 321, that is, the detection of the face mark by the teacher learning module 320. The second loss function L ² plays a role of minimizing the detection of the face mark by the student learning module 330, that is, the estimated error of the student regression output 331. The third loss function L ³ represents the difference between the output vectors of the TFC layer 322 and the STDFC layer 332. The three loss conditions L ¹ , L ² , and L ³ can be formulated as [Equation 1] below.

[Equation 1]

는 실측 자료(ground truth) 얼굴 표식 좌표를 나타내고,

는 입력 이미지를 나타낸다. In Equation 1, N represents the number of training images, i represents the index of the input image,

Indicates the ground truth face marker coordinates,

Indicates an input image.

는 T­FC(322)의 출력 벡터이며,

는 STD­FC(332)의 출력 벡터이다. In addition, h (.) Means the function of the shared engine parameterized by W _CNN , f (.) Means the function of the teacher learning module 320 parameterized by W _T , and g (.) _Denotes a function of the student learning module 330 parameterized by W _STD . In addition,

Is the output vector of the TFC 322,

Is the output vector of STDFC 332.

와 같이 정의된다. 이 때,

은 각 손실의 중요도(정도)를 나타낸다. 모든 손실은 모든 네트워크와 공동으로 영향을 미치기 때문에, 파라미터를 적절하게 설정한다. The total loss from the forked point is

Is defined as At this time,

Indicates the importance (degree) of each loss. Since all losses affect all networks jointly, set the parameters accordingly.

로 고정하였다. 그런 다음 교사 학습 모듈(320), 학생 학습 모듈(330) 및 공유 엔진(또는 공유 컨볼루션 네트워크, Shared convolution network)에 공동으로 영향을 미치는

를 0부터 1까지 점진적으로 증가시킨다(즉, t번째 반복에서

). 실시예에 따라서, 초기에

로 설정하는 경우, 교사 학습 모듈(320)의 성능은 최상의 성능을 얻기 전에 포화 상태에 도달한다. 따라서, 트레이닝이 시작될 때에는 공유 엔진과 교사 학습 모듈(320)이 중점적으로 트레이닝되고, 이후 학생 학습 모듈(330)이 점차적으로 트레이닝된다. For example, in the training phase, the teacher learning module 320 must be continuously trained because it must achieve the best performance. If the teacher learning module 320 does not achieve the best performance, the student learning module 330 also cannot achieve the best performance. To achieve this best performance, from start to finish

Was fixed with. Then jointly affect the teacher learning module 320, the student learning module 330, and the sharing engine (or shared convolution network).

Incrementally from 0 to 1 (i.e. at the t iteration

). Depending on the embodiment, initially

If set to, the performance of the teacher learning module 320 reaches saturation before obtaining the best performance. Therefore, when training starts, the sharing engine and the teacher learning module 320 are intensively trained, and then the student learning module 330 is gradually trained.

[Algorithm 1] below describes the proposed teacher and student co-learning method as described above.

[Algorithm 1]

4A and 4B show experimental results verifying the performance of a machine learning system using joint learning according to an embodiment of the present invention.

Before explaining the experimental results of FIGS. 4A and 4B, the face mark detection network (FLD) of the machine learning model used in the experiment and the experimental setup will be described.

In order to verify the effectiveness of the machine learning system using joint learning through the compact face mark detection network (compact FLD) proposed in the present invention, it was applied to an existing FLD network. At this time, two FLD networks, TCDCN and modified DCNN­I / C (M­DCNN­I / C), were used.

In TCDCN, four convolutional layers are considered as shared engines, and the features of the last convolutional layer are provided in each of the teacher learning module and the student learning module. The proposed teacher learning module consists of a flat layer, fully connected layer, and teacher regression output layer, and the student learning module consists of a 1 × 1 convolution layer, a power average pooling layer, and a student regression output layer.

M­DCNN­I / C integrates the inner part of the face and the contour of the face. At this time, in the DCNN­I / C architecture, the convolutional layers of the eye and eyebrow are separated into the left eye / right eye and the left eyebrow / right eyebrow convolution layer. M­DCNN­I / C consists of three convolutional layers and eight facial component convolutional layers (left eyebrow, right eyebrow, left eye, right eye, nose, nose, mouth and face outline). At this time, in M­DCNN­I / C, the sharing engine is three convolutional layers and a facial convolutional layer. The convolutional layer for each facial component is divided into its own teacher learning module and student learning module.

In the experiments of FIGS. 4A and 4B, an experiment was performed on a 300 dBW, a benchmark data set widely used in recent studies. The data set provides 68 face marker points and a face bounding box. Datasets for training of the present invention were collected from a subset of AFW, HELEN, LFPW and IBUG. In particular, 337 training images were collected from AFW, 2000 training images from HELEN, and 811 training images from LFPW. Therefore, the total number of training images is 3,148, and the total number of test images is 689.

At this time, the test data set is composed of two types of test sets. One is the common test set of LFPW (224 test images) and HELEN (330 test images), and the other test data set is the task set from IBUG (135 test images). The task set includes task conditions such as occlusions, illumination deformation, head posture and expression. The present invention has been performed to increase data to prevent overfitting and to increase training data transformation, and to increase the number of training images by performing image translation, rotation, and zooming.

Experiments were conducted based on the above-described settings, and network performance was evaluated to verify the performance of the compact facial landmark detection (compact FLD) network of the present invention. At this time, the mean error (Mean error) is calculated as a measure of the distance between the ground truth and the marker estimated through [Equation 2] below.

[Equation 2]

In Equation 2, N represents the number of images, M represents the number of landmarks, and j represents the index of the landmark points. In addition, o represents the output, g represents the ground truth, and l and r represent the coordinates of the left eye and the right eye, respectively.

In order to evaluate the performance of the compact face mark detection network (compact FLD) proposed in the present invention, the teacher-student co-learning method through compact FLD according to an embodiment of the present invention is based on a manual method and deep learning based Compared to various existing FLD methods, including methods.

Table 2 below shows a comparison of the average error between the existing FLD method and the compact face mark detection network according to an embodiment of the present invention of a 300 ­W data set with 68 face mark points.

[Table 2]

As shown in [Table 2], the average error of hand craft methods is more than 6.3%, and the deep learning based methods achieve a lower average error than the manual method of 300W data set. Did.

In Table 2, TCDCNC is a compact version of the TCDCN network trained by the teacher and student co-learning method through the compact FLD according to an embodiment of the present invention, and MDCNNI / CC is a compact FLD according to an embodiment of the present invention. It is a compact version of MDCNNI / C trained in a joint learning method for teachers and students through. At this time, TCDCN­C performance (5.40%) shows similar performance to TCDCN (5.54%). In addition, while M­DCNN­I / C­C (4.95%) achieved performance comparable to M­DCNN­I / C (4.92%), the number of parameters was significantly reduced (see Table 3 below).

[Table 3]

Therefore, according to the above, it is shown that the teacher and student co-learning method through compact FLD according to an embodiment of the present invention can be well applied to various networks.

When evaluating the total number of network parameters with reference to Table 3, TCDCN reduced the total number of network parameters from 312,244 to 148,404 (parameters were reduced by about 52%). In particular, while MDCNNI / C has a large number of network parameters (4,331,444), the total number of network parameters after applying the teacher and student co-learning method through compact FLD according to an embodiment of the present invention to MDCNNI / C is 4,331,444 It can be seen that the number of dogs was significantly reduced from 399,284 dogs (parameters are reduced by about 90%). Furthermore, MDCNNI / CC not only has fewer parameters than the number of DCNNI / C parameters (3,298,152 in [Table 1]), but also an error rate less than the error rate of DCNNI / C (6.12% in [Table 2]). Indicates.

Hereinafter, evaluation of the teacher and student co-learning method through the compact FLD according to the embodiment of the present invention will be described based on the experiment setup and performance comparison as described above.

로 설정되었다. 이 때,

와

는 0에서 1로 기하 급수적으로 증가하였다. In the machine learning system using the joint learning according to the embodiment of the present invention, the student learning module is intended to mimic the teacher learning module by the joint learning of the teacher and the student without the prior training model of the teacher network. To confirm this, all average errors of the test image were measured for each epoch in the training phase. The loss balance weight during the training phase

Was set to. At this time,

Wow

Increased exponentially from 0 to 1.

In addition, at the beginning of the training phase, the teacher learning module loss (L ¹ ) was again predominantly propagated, and after a certain period (epoch) the student learning module was gradually trained into L ² and L ³ . In the second half of the training phase, all loss functions were performed to train the entire network.

4A and 4B show the experimental results for the average error of two network structures, TCDCN and M­DCNN­I / C. FIG. 4A is an experimental result of TCDCN, and FIG. 4B is an experimental result of M­DCNN­I / C.

4A and 4B, it can be seen that four FLD networks are compared. 'Compact FLD with the conventional teacher student learning' is a compact network (student model) trained with existing teacher and student learning methods. The teacher models (TCDCN, M­DCNN­I / C) and student models (composed of shared engines and student learning modules) were trained as separate, ie, student models that mimic the pre-trained teacher model.

On the other hand, 'FLD with teacher regression network (FLD) using a teacher learning module' is a FLD network composed of a sharing engine and a teacher learning module, and is trained jointly with a student learning module.

4A and 4B, it can be seen that the student learning module worked well to create a compact Facial Landmark Detection (compact FLD) network in both TCDCN and M­DCNN­I / C. 'FLD with teacher regression network (FLD) using teacher learning module' provided excellent performance in both TCDCN and M­DCNN­I / C. It can be seen that the performance of TCDCNC follows the performance of 'FLD with teacher regression network (FLD) using teacher learning module', and in the case of MDCNNI / C, the performance of MDCNNI / CC is' FLD (FLD using teacher learning module) with teacher regression network) 'and MDCNNI / C.

That is, looking at the experimental results according to FIGS. 4A and 4B, the machine learning system using the joint learning according to the embodiment of the present invention was effective in constructing a compact facial landmark detection (compact FLD) network. Can be confirmed. Furthermore, the experimental results indicate that the compact FLD proposed in the present invention significantly reduced the number of network parameters, and achieved a much lower error rate than the existing teacher and student networks. These results indicate that the proposed compact FLD of the present invention can provide high face mark detection network (FLD) performance with a small number of network parameters in a mobile application.

The device described above may be implemented with hardware components, software components, and / or combinations of hardware components and software components. For example, the devices and components described in the embodiments include, for example, processors, controllers, arithmetic logic units (ALUs), digital signal processors (micro signal processors), microcomputers, field programmable arrays (FPAs), It may be implemented using one or more general purpose computers or special purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may perform an operating system (OS) and one or more software applications running on the operating system. In addition, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For convenience of understanding, a processing device may be described as one being used, but a person having ordinary skill in the art, the processing device may include a plurality of processing elements and / or a plurality of types of processing elements. It can be seen that may include. For example, the processing device may include a plurality of processors or a processor and a controller. In addition, other processing configurations, such as parallel processors, are possible.

The software may include a computer program, code, instruction, or a combination of one or more of these, and configure the processing device to operate as desired, or process independently or collectively You can command the device. Software and / or data may be interpreted by a processing device, or to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. , Or may be permanently or temporarily embodied in the transmitted signal wave. The software may be distributed on networked computer systems, and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer-readable medium may include program instructions, data files, data structures, or the like alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiments or may be known and usable by those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CDROMs, DVDs, and magnetic optical media such as floptical disks. (magnetooptical media), and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language code that can be executed by a computer using an interpreter, etc., as well as machine language codes produced by a compiler. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described by a limited embodiment and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques are performed in a different order than the described method, and / or the components of the described system, structure, device, circuit, etc. are combined or combined in a different form from the described method, or other components Alternatively, even if replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

100: machine learning system using joint learning
210: training unit (or training phase)
220: test unit (or test phase)
211: sharing engine
212, 320: Teacher Learning Module
213, 330: Student Learning Module
221: compact learning module
310: shared layer (or shared convolutional layer)
321: Teacher learning module output
322: TFC layer
323: Flatten layer
331: Student learning module output
332: STDFC layer
333: global average polling

Claims (18)

In the machine learning method using the joint learning of the machine learning system including a processing unit and a joint learning unit,
Minimizing errors and characteristic differences between a teacher learning module and a student learning module branched from a sharing engine that extracts features of the input data using the loss function; And
The co-learning unit jointly learning the teacher learning module and the student learning module by the loss function.
Machine learning method using a joint learning that includes. According to claim 1,
Inferring the classification and regression values for the input data using the network parameters of the learned student learning module and the sharing engine, which are further included in the machine learning system.
Machine learning method using a joint learning further comprising. According to claim 1,
The step of the processing unit to minimize errors and characteristic differences between the branched teacher learning module and the student learning module is
Among the three loss functions, a first loss function and a second loss function are used to minimize the marking error of the teacher learning module and the student learning module, and a third loss function is used between the teacher learning module and the student learning module. Machine learning method using co-learning to minimize the difference in characteristics. According to claim 3,
The co-learning unit jointly learning the teacher learning module and the student learning module by the loss function is
A machine learning method using joint learning, characterized by learning the student learning module imitating the teacher learning module by a final loss function among the three loss functions. According to claim 1,
The co-learning unit jointly learning the teacher learning module and the student learning module by the loss function is
A machine learning method using co-learning such that the student learning modules, which are compact architectures, and the teacher learning modules including a fully connected layer are jointly learned and imitated with each other. According to claim 2,
The inferring unit may infer classification and regression values for the input data.
A machine learning method using joint learning to infer classification and regression values for the input data through a compact machine learning model according to the network parameters received from the sharing engine and the student learning module. The method of claim 6,
The machine learning model
A machine learning method using a joint learning that removes the teacher learning module and includes a compact learning module and the sharing engine constructed from the student learning module. A computer program stored in a computer-readable recording medium to carry out the method of claim 1. A processing unit for minimizing error and characteristic differences between a teacher learning module and a student learning module branched from a sharing engine that extracts features of input data using a loss function; And
Joint learning unit for jointly learning the teacher learning module and the student learning module by the loss function
Machine learning system using a joint learning that includes. The method of claim 9,
Inference unit for inferring classification and regression values for the input data using the network parameters of each of the learned student learning module and the sharing engine
Machine learning system using a joint learning further comprising. The method of claim 9,
The processing unit
Among the three loss functions, a first loss function and a second loss function are used to minimize the marking error of the teacher learning module and the student learning module, and a third loss function is used between the teacher learning module and the student learning module. Machine learning system using co-learning to minimize the difference in characteristics. The method of claim 11,
The joint learning department
A machine learning system using joint learning, characterized by learning the student learning module imitating the teacher learning module by a final loss function among the three loss functions. The method of claim 9,
The joint learning department
A machine learning system using co-learning that enables the student learning modules, which are compact architectures, and the teacher learning modules, including a fully connected layer, to be jointly learned and imitated with each other. The method of claim 10,
The reasoning part
A machine learning system using joint learning to infer classification and regression values for the input data through a compact machine learning model according to the network parameters received from the sharing engine and the student learning module. A training unit for training the student learning module imitating the teacher learning module by jointly learning a teacher learning module and a student learning module branched from a sharing engine extracting features of input data; And
A test unit for constructing a compact machine learning model including the shared engine and a compact learning module to infer classification and regression values for the input data
Machine learning system using a joint learning that includes. The method of claim 15,
The training unit
The features encoded by the shared layer in the sharing engine are equally supplied to the teacher learning module and the student learning module, and the encoded features are classified and regressed through the teacher learning module and the student learning module. Machine learning system using co-learning to convert to value. The method of claim 15,
The training unit
Characterized in that the training to jointly learn the teacher learning module and the student learning module using three loss functions,
Among the three loss functions, a first loss function and a second loss function are used to minimize the marking error of the teacher learning module and the student learning module, and a third loss function is used between the teacher learning module and the student learning module. Machine learning system using co-learning to minimize the difference in characteristics. The method of claim 15,
The test section
A machine learning system using joint learning to remove the teacher learning module and infer classification and regression values for the input data through the compact machine learning model constructed by constructing the compact learning module from the student learning module.

Images