CN110263939A

CN110263939A - A kind of appraisal procedure, device, equipment and medium indicating learning model

Info

Publication number: CN110263939A
Application number: CN201910549544.6A
Authority: CN
Inventors: 周晋; 李超; 王翔
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2019-06-24
Filing date: 2019-06-24
Publication date: 2019-09-20

Abstract

This application discloses a kind of appraisal procedures for indicating learning model, it include: to generate Performance Evaluating Indexes for the expression learning model being trained based on unsupervised mode, it includes at least one of the first index and the second index, wherein, first index is the expression vector of each sample in the first sample subset learnt in the training process based on expression learning model, generate for measuring the quantizating index that similar sample is close and inhomogeneity sample is mutually become estranged, second index is the corresponding similarity vector of expression vector of each sample in the second sample set learnt in the training process based on expression learning model, what is generated indicates the quantizating index of stability for measuring sample, according to the Performance Evaluating Indexes, determine the training for indicating learning model.By above-mentioned quantizating index, so that being no longer dependent on subsequent machine learning task, the training iterative process that whole table dendrography is practised is greatly speeded up.Disclosed herein as well is corresponding device, equipment and media.

Description

Evaluation method, device, equipment and medium for representing learning model

Technical Field

The present application relates to the field of computer technologies, and in particular, to an evaluation method, an evaluation device, an evaluation apparatus, and a computer storage medium for representing a learning model.

Background

Representation learning refers to a task of converting raw data into a form that can be efficiently developed by machine learning by learning a representation of the data, so that useful information is more easily extracted when a classifier or other prediction task is subsequently constructed. Colloquially, it is to convert data into a vector representation, while making the vector contain as much data information as possible that is useful for subsequent tasks. In recent years, learning has attracted attention in the fields of voice, images, and the like.

Unsupervised representation learning refers to training a representation learning model on unlabeled training data. It is difficult to evaluate unsupervised learning models because without known tags, the results of unsupervised learning cannot be compared to actual tags.

Generally, the evaluation of the representation learning model trained in an unsupervised manner depends on the evaluation result of a subsequent machine learning task, which results in that the period of training and optimization iteration of the unsupervised representation learning model is prolonged, the time cost of model training is increased, the iteration speed of the model is slowed down, and the loss of practical application is caused.

Disclosure of Invention

The application provides an evaluation method for representing a learning model, which provides two quantitative indexes for evaluating training quality to measure the training condition of an unsupervised representation learning model, so that abnormal conditions in the training process are found in time, the training period is prevented from being prolonged, the training speed is prevented from being slowed down, the training time cost is prevented from being increased, and further, the damage to practical application is avoided. Corresponding apparatus, devices, media and computer program products are also provided.

A first aspect of the present application provides an evaluation method of a representation learning model, the method comprising:

generating a performance evaluation index of a representation learning model trained based on an unsupervised mode, wherein the performance evaluation index comprises at least one of a first index and a second index;

the first index is a quantization index which is generated for measuring similar samples and distant samples of different types based on the expression vector of each sample in the first sample subset which is learned by the expression learning model in the training process; the first subset of samples is generated by labeling a first subset of the training sample set representing the learning model, wherein the first subset comprises samples of different classes;

the second index is a quantitative index which is generated for measuring the representation stability of the samples based on the similar vectors corresponding to the representation vectors of the samples in the second sample subset which are learned by the representation learning model in the training process; the second subset of samples is a second subset of the set of training samples;

and determining the training condition of the representation learning model according to the performance evaluation index.

A second aspect of the present application provides an evaluation apparatus representing a learning model, the apparatus comprising:

the index generation module is used for generating a performance evaluation index of the representation learning model aiming at the representation learning model trained on an unsupervised mode, wherein the performance evaluation index comprises at least one of a first index and a second index;

and the evaluation module is used for determining the training condition of the representation learning model according to the performance evaluation index.

Optionally, the index generating module includes:

the first obtaining submodule is used for obtaining a representation vector obtained by the representation learning model aiming at the learning of each sample of the first sample subset in the training process;

a generation submodule, configured to determine inter-class distances and intra-class distances of various types of samples according to the expression vectors and labels of the samples in the first sample subset, and generate a composition ratio according to a ratio of the inter-class distances to the intra-class distances;

and the first determining submodule is used for taking the combination ratio as a first index.

Optionally, the performance evaluation index includes a first index;

the evaluation module is specifically configured to:

and when a plurality of the split-combination ratios determined based on a plurality of iteration turns within a preset time period are in a convergence state and the convergence value is greater than a first reference threshold value, determining that the training condition of the representation learning model tends to be stable.

Optionally, the index generating module includes:

the second obtaining submodule is used for obtaining a representation vector obtained by the representation learning model in a training process in multiple iteration rounds aiming at the learning of each sample of the training sample set;

an adding submodule, configured to select, according to a representation vector of each sample in the training sample set learned in each iteration round, a preset number of most similar samples for each sample in the second sample subset, and add the preset number of similar samples selected for the sample to a similar sample set corresponding to each sample in the second sample subset and the iteration round;

and a second determining submodule, configured to generate a jacobian index corresponding to each sample in the second sample subset for a plurality of similar sample sets corresponding to each sample in the second sample subset, where the jacobian index is used as a second index.

Optionally, the performance evaluation index includes a second index;

the evaluation module is specifically configured to:

and when the proportion of the samples in the second sample subset which are larger than the Jacard exponent threshold exceeds a preset proportion, determining that the training condition of the representation learning model tends to be stable.

Optionally, the performance evaluation index includes a first index and a second index;

the evaluation module comprises:

the weighting submodule is used for weighting the first index and the second index;

and the evaluation submodule is used for determining the training condition of the representation learning model according to the weighting processing result.

Optionally, the apparatus further comprises:

the first display module is used for drawing and displaying a training effect curve representing the learning model according to the performance evaluation indexes generated by different iteration turns of the representing learning model, and the training effect curve represents the change situation of the performance of the representing learning model along with the training process.

Optionally, the index generating module is specifically configured to:

generating the performance evaluation indexes of different iteration rounds aiming at the representation learning model configured with different hyper-parameters;

the device further comprises:

and the second display module is used for drawing and displaying the comparison effect graph representing the learning model, the comparison effect graph is used for representing the respective training effect curves of the representing learning model based on different hyper-parameters, and the training effect curves represent the change situation of the performance of the representing learning model along with the training process.

Optionally, the representation learning model is a word vector representation learning model.

A third aspect of the present application provides a terminal device, comprising a processor and a memory:

the memory is used for storing a computer program;

the processor is configured to execute the method for evaluating a representation learning model according to the first aspect of the present application.

A fourth aspect of the present application provides a computer-readable storage medium for storing a computer program for executing the method for evaluating a representation learning model according to the first aspect.

A fifth aspect of the present application provides a computer program product comprising instructions which, when run on a computer, cause the computer to perform the method of evaluating a representation learning model as described in the first aspect above.

According to the technical scheme, the embodiment of the application has the following advantages:

the embodiment of the application provides an evaluation method of a representation learning model, aiming at the representation learning model trained in an unsupervised mode, the method provides two quantitative indexes, the training condition of the model can be measured through at least one of the two quantitative indexes, specifically, the first index is used for marking partial data in a training sample set, in the training process, the intra-group distance and the inter-group distance of various samples are calculated based on the marked data, the quantitative indexes which are used for measuring similar samples and distant samples of the same type are generated according to the intra-group distance and the inter-group distance, the more similar the quantitative indexes represent the similar samples, the more distant the different samples, the better the classification capability of the model is indicated, thus, the training condition of the representation learning model can be determined based on the first index, the second index is used for determining partial samples from the training sample set, in the training process, the similarity degree of the expression vector and the sample expression in the whole training sample set is periodically calculated, the similar vector corresponding to the expression vector in each period is determined, and further, a quantization index for measuring the sample expression stability is generated, the more stable the quantization index is, the more stable the model is, and thus, the training condition of the expression learning model can be determined based on the second index.

Through the quantitative indexes, a user can timely master the model training condition, namely whether the model is gradually improved, whether the training can be stopped and the like, and does not depend on a subsequent machine learning task, so that the whole training iterative process of the representation learning is greatly accelerated, and the time consumed by subsequently adding a model for training, adjusting and evaluating is saved. Moreover, the method provides a quantitative evaluation result, and a subsequent adjustment method of the model can be determined through the performance of different parameter combinations, so that the problem of error easiness caused by dependence on historical experience and strong subjectivity is avoided, and automatic adjustment of the hyper-parameters becomes possible.

Drawings

FIG. 1 is a diagram illustrating a scene architecture of an evaluation method of a learning model according to an embodiment of the present application;

FIG. 2 is a flow chart illustrating a method for evaluating a learning model in an embodiment of the present application;

FIG. 3 is a graph showing a comparison of training time of a learning model in an embodiment of the present application;

FIG. 4 is a graph of training effects of different representation learning models in the embodiment of the present application;

FIG. 5A is a diagram illustrating a scenario of an evaluation method of a learning model in an embodiment of the present application;

FIG. 5B is a flowchart illustrating a method for evaluating a learning model according to an embodiment of the present application;

FIG. 6 is a schematic structural diagram of an evaluation apparatus showing a learning model in an embodiment of the present application;

FIG. 7 is a schematic structural diagram of an evaluation apparatus showing a learning model in an embodiment of the present application;

FIG. 8 is a schematic structural diagram of an evaluation apparatus showing a learning model in an embodiment of the present application;

FIG. 9 is a schematic structural diagram of an evaluation apparatus showing a learning model in an embodiment of the present application;

FIG. 10 is a schematic structural diagram of an evaluation apparatus showing a learning model in an embodiment of the present application;

FIG. 11 is a schematic structural diagram of an evaluation apparatus showing a learning model in an embodiment of the present application;

fig. 12 is a schematic structural diagram of a terminal in an embodiment of the present application.

Detailed Description

In order to make the technical solutions of the present application better understood, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

The terms "first," "second," "third," "fourth," and the like in the description and in the claims of the present application and in the drawings described above, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the application described herein are, for example, capable of operation in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

Aiming at the problems that the evaluation of the representation learning model trained in an unsupervised mode depends on the evaluation result of a subsequent machine learning task, so that the training period of the unsupervised representation learning model is prolonged, the training speed is slowed down and the training time cost is increased, the method can realize quantitative evaluation in the training process.

The method ensures that the user can acquire the training condition without relying on the subsequent machine learning task, the whole training iterative process of the representation learning is greatly accelerated, and the time consumed by subsequently adding a model for training, adjusting and evaluating is saved. Moreover, the method provides a quantitative evaluation result, and a subsequent adjustment method of the model can be determined through the performance of different parameter combinations, so that the problem of error easiness caused by dependence on historical experience and strong subjectivity is avoided, and automatic adjustment of the hyper-parameters becomes possible.

It is understood that the evaluation method for representing the learning model provided by the present application can be applied to any processing device with data processing capability, and the processing device can be a terminal or a server. The terminal can be a desktop, a portable mobile terminal device such as a tablet computer and a smart phone, or a mainframe. A server refers to a device that provides computing services, and may be a stand-alone computing device or a computing cluster composed of multiple computing devices.

The evaluation method of the representation learning model provided by the present application may be stored in a processing device in the form of a computer program, and the processing device implements the above-described evaluation method of the representation learning model by executing the computer program. The computer program may be a stand-alone computer program, or may be a functional module, a plug-in, an applet, or the like running on another program.

In practical applications, the evaluation method representing the learning model provided by the present application can be applied to, but is not limited to, the application environment shown in fig. 1.

As shown in fig. 1, a terminal 102 is connected to a database 104, a training sample set is stored in the database 104, the terminal 102 obtains samples from the training sample set in the database 104, and trains a representation learning model in an unsupervised manner, during the training process, the terminal 102 generates a first index for measuring similar samples and distant samples of different types based on a representation vector of each sample in a first sample subset, generates a second index for measuring representation stability of the sample based on a similar vector corresponding to a representation vector of each sample in a second sample subset, and the terminal 102 determines a training condition representing the learning model based on at least one of the first index and the second index.

Next, each step of the evaluation method representing the learning model provided in the embodiment of the present application will be described in detail from the perspective of the terminal.

Referring to fig. 2, a flow chart illustrating a method of evaluating a learning model, the method comprising:

s201: and generating a performance evaluation index of the representation learning model for the representation learning model trained in an unsupervised mode.

In this embodiment, the terminal uses samples in the training sample set to train and represent the learning model in an unsupervised manner. In order to evaluate the training situation of the representation learning model, the terminal generates a performance evaluation index for the representation learning model for evaluating the representation learning model.

Wherein the performance evaluation index includes at least one of the first index and the second index. The first index is a quantitative index obtained by real-time calculation based on a pre-labeled sample, and the quantitative index can represent the classification capability of the model, namely the capability of expressing the same type of sample as a similar result and expressing different types of samples as distant results, in other words, the quantitative index can measure the degree of similarity of the same type of sample and distant degree of the different types of samples; the second index is a quantitative index obtained by calculation based on a small number of samples tracked in the whole process, and the quantitative index quantitatively evaluates the stability of the output result of the model based on the similarity of the small number of samples tracked in the whole process and the whole number of results represented by the small number of samples tracked in the whole process.

The first index is generated based on a representative vector representing each sample in the first subset of samples learned by the learning model during the training process. Wherein the first subset of samples is generated based on a first subset of the set of training samples. Specifically, a small number of samples are determined from the training sample set to form a first subset, the first subset includes samples of different classes, so as to determine the classification capability of the model for the samples of different classes, and labeling the samples in the first subset can obtain the first sample subset. That is, the first subset of samples includes the first subset and the labels corresponding to the samples in the first subset.

During specific implementation, the terminal obtains a representation vector obtained by a representation learning model in a training process aiming at each sample of the first sample subset, then determines the inter-class distance and the intra-class distance of each sample according to the representation vector and the label of each sample of the first sample subset, generates a composition ratio according to the ratio of the inter-class distance and the intra-class distance, and takes the composition ratio as a first index. The calculation of the first index can be seen in the following formula:

wherein Λ represents a split-combination ratio, d_ABDenotes the distance between the classes AB, d_A、d_BRespectively represent the sample distance within class A and the sample distance within class B, mean represents the mean value, based on which mean (d)_AB) Mean (d) representing the average distance between classes AB_A)、mean(d_B) The mean distance of samples within class a and the mean distance of samples within class B are indicated, respectively.

The sample distance has multiple expression forms, and in some possible implementation manners, the cosine distance may be used for characterization, which may specifically be referred to as the following formula:

where d is a cosine distance, X and Y respectively represent corresponding representation vectors of different samples, | X | and | Y | respectively represent the respective lengths of X and Y.

Of course, in practical applications, the sample distance may also be calculated in other manners, such as using an euclidean distance, a manhattan distance, a chebyshev distance, and the like, which is not limited in this embodiment. In addition, in some cases, the terminal may also use the median of the distance instead of the average distance value to calculate the score-sum ratio, which is only an example of the present application and does not limit the technical solution of the present application.

And for the second index, the quantization index is generated for measuring the sample representation stability based on the similar vector corresponding to the representation vector of each sample in the second sample subset learned by the representation learning model in the training process. Wherein the second subset of samples is a second subset of the set of training samples.

It can be understood that the representation learning model is iteratively trained according to rounds, in order to calculate a second index for measuring the representation stability of the samples, the terminal may obtain a representation vector obtained by learning the representation learning model for each sample of the training sample set in a plurality of iteration rounds, select, according to the representation vector of each sample in the training sample set learned in each iteration round, the most similar N samples for each sample in the second sample subset, add the N similar samples selected for the sample to the similar sample set corresponding to each sample and iteration round in the second sample subset, generate a jacobian index corresponding to each sample in the second sample subset for the plurality of similar sample sets corresponding to each sample in the second sample subset, and use the jacobian index as the second index. Where N is a preset number, which may be set according to actual requirements, for example, a positive integer greater than 1.

The Jaccard Index (Jaccard Index), also called Jaccard similarity coefficient, is a probability value for comparing similarity or dispersion in a sample set, and is equal to a ratio of a sample set intersection to a sample set union, referred to as an intersection ratio for short, and specifically, in the present application, it can be calculated by the following formula:

wherein,characterizing a set of similar samples for the ith sample in the second subset of samples at step t (i.e. round t) of the model iteration value,characterizing a similar sample set of the ith sample in the second sample subset at the t + n step (i.e. the t + n round) of the iteration value of the model, wherein i is a positive integer between 1 and k, k is the number of elements in the second sample subset,the number of intersection elements of the two similar sample sets is represented,the number of union elements characterizing the two similar sample sets.

S202: and determining the training condition of the representation learning model according to the performance evaluation index.

In practical applications, the performance evaluation index includes at least one of a score and a jacor index, based on which the terminal may determine a training situation representing a learning model through several implementation manners as described in detail below.

A first implementation is to determine the training case representing the learning model based only on the split ratio. Specifically, when a plurality of the split-join ratios determined based on a plurality of iteration turns within a preset time period are in a convergence state and the convergence value is greater than a first reference threshold value, it is determined that the training condition of the representation learning model tends to be stable.

It can be understood that the benchmark reference value of the conjunction Λ is 1, which means that the average distance inside the two classes is not different from the average distance between the classes, and based on this, in an effective representation learning training process, the value of the conjunction should be larger than 1 and gradually increase until the stability. Based on the convergence condition and the convergence value of the split-combination ratio determined by a plurality of iteration turns in the preset time period, the terminal can determine whether the training condition of the representation learning model tends to be stable.

The terminal can be realized as follows for the convergence status and the convergence value of the split ratio. Specifically, for each round of iteration process in a preset time period, the inter-class distance and the intra-class distance of each type of sample are calculated according to the expression vector of each sample in the first sample subset and the corresponding label, and a split ratio is calculated based on the inter-class distance and the intra-class distance, wherein the split ratio before iteration is recorded as a first split ratio, the split ratio after iteration is recorded as a second split ratio, and when the first split ratio and the second split ratio are both greater than a first reference threshold and the absolute value of the difference between the second split ratio and the first split ratio is smaller than a second reference threshold, the split ratio is determined to be converged, and the convergence value is greater than the first reference threshold.

It should be noted that the preset time period, the first reference threshold and the second reference threshold may be set according to actual requirements, and as an example of the present application, the preset time period may be one day, the first reference threshold may be 2, and the second reference threshold may be 0.01.

For the combination ratio, the larger the numerical value is, the more obvious the difference between the categories is, the more similar the interior of each category is, the result of the unsupervised representation learning is reflected to accord with the classification standard on the small sample, the representation vector output by the representation learning model carries valuable information, and the effectiveness of the whole model training is guaranteed.

A second implementation is to determine the training case representing the learning model based on the jacobian index only. Specifically, for the jacobian index, when the proportion of the samples in the second sample subset which are larger than the jacobian index threshold exceeds a preset proportion, the training condition of the representation learning model is determined to tend to be stable.

It can be understood that in an effective representation learning training process, as the representation learning model is trained further, the similar sample set corresponding to each sample in the second sample subset should exhibit high correlation and have a gradually fixed trend, in other words, the similar sample set corresponding to each sample does not change greatly. Based on the above, the terminal can determine whether the training condition of the representation learning model tends to be stable or not based on the size of the Jacobian index in the preset time period. And aiming at the samples in the second sample subset, if the Jacobian index of the samples is greater than the Jacobian index threshold value, the similar sample sets of the samples are almost consistent before and after iteration, and if the Jacobian indexes of the samples exceeding the preset proportion are all greater than the Jacobian index threshold value, the learning model tends to be stable.

It should be noted that the preset ratio and the jacobian index threshold may be set according to actual requirements, as an example of the present application, the jacobian index threshold may be set to 70%, and the preset ratio may be set to 80%.

A third implementation is that the training condition representing the learning model is jointly determined based on the split-ratio and the Jacobian index. When the training condition is determined based on the split-combination ratio and the Jacore index, whether the split-combination ratio and the Jacore index meet respective corresponding standards can be respectively judged, so that the training condition representing the learning model is determined; the combination ratio and the Jacobian index may be weighted, and the training condition of the representation learning model may be determined based on the result of the weighting.

The conjunctive ratio and the jacarat index belong to different dimensions, and when the conjunctive ratio and the jacarat index are weighted, the conjunctive ratio and the jacarat index may be normalized first, and then the weighted processing may be performed based on the normalized index. Wherein, the respective weights of the split ratio and the Jacobian index can be set according to actual needs.

It should be further noted that, in the above three implementation manners, the first index is used as the split-combination ratio, and the second index is used as the jacarat index, in other possible implementation manners of the embodiment of the present application, when the first index and the second index are other parameters, the training condition representing the learning model may be determined by referring to at least one of the other parameters.

From the above, an embodiment of the present application provides an evaluation method for a representation learning model, which provides two quantization indexes, and a training status of the model can be measured by at least one of the two quantization indexes, specifically, a first index is used to label part of data in a training sample set, and in a training process, an intra-group distance and an inter-group distance of each type of sample are calculated based on the labeled data, and a quantization index for measuring similar samples and distant samples of the same type is generated according to the intra-group distance and the inter-group distance, and the quantization index represents that the samples of the same type are closer and the samples of different type are farther, and the classification capability of the model is better, so the training status representing the learning model can be determined based on the first index, and a second index is used to determine part of samples from the training sample set, in the training process, the similarity degree of the expression vector and the sample expression in the whole training sample set is periodically calculated, the similar vector corresponding to the expression vector in each period is determined, and further, a quantization index for measuring the sample expression stability is generated, the more stable the quantization index is, the more stable the model is, and thus, the training condition of the expression learning model can be determined based on the second index.

Through the quantitative indexes, a user can timely master the model training condition, namely whether the model is gradually improved, whether the training can be stopped and the like, and does not depend on a subsequent machine learning task, so that the whole training iterative process of the representation learning is greatly accelerated, and the time consumed by subsequently adding a model for training, adjusting and evaluating is saved.

The application provides a comparison graph of time consumed by unsupervised representation learning and time consumed by traditional unsupervised representation learning based on the evaluation method of the representation learning model, as shown in fig. 3, the traditional machine learning time consumption is 7 days, the unsupervised representation learning time (2 days) and the time consumed by determining the learning condition of unsupervised representation learning based on a subsequent machine learning task (5 days) are included, in the unsupervised representation learning process, the learning condition can be determined based on the performance evaluation index, the current learning condition of unsupervised representation learning does not need to be determined through the subsequent machine learning, the time consumed by the subsequent machine learning task is directly saved, and the training progress is accelerated.

Considering that the performance evaluation indexes are variable, the terminal can also draw and display a training effect curve representing the learning model according to the performance evaluation indexes generated by different iteration turns of the learning model, and the training effect curve represents the change condition of the performance of the learning model along with the training process, so that the process and the effect of model training are visualized, and a user can intuitively find whether the model training is effective or not.

Furthermore, the terminal can also generate the performance evaluation indexes of different iteration turns aiming at the representation learning model configured with different hyper-parameters, draw and display a comparison effect graph of the representation learning model, wherein the comparison effect graph is used for expressing respective training effect curves of the representation learning model based on different hyper-parameters, and the comparison effect graph can help a user to determine the model selection and parameter adjustment directions, so that the problem of error proneness caused by dependence on historical experience and strong subjectivity is avoided, and automatic adjustment of the hyper-parameters becomes possible.

For the convenience of understanding, the application also provides a specific example of a comparison effect graph. As shown in fig. 4, there are shown training effect curves 41 to 45 representing learning models corresponding to 5 different sets of hyper-parameter combinations, where the combination/combination ratio of the learning models corresponding to the curve 41 and the curve 42 converges to a high value, the combination/combination ratio of the learning models corresponding to the curve 43, the curve 44 and the curve 45 converges to a high value, and the learning model corresponding to the curve 43 reaches a steady state first.

The evaluation method of the representation learning model provided by the application can be applied to various unsupervised representation learning tasks, such as T-distributed random neighborhood Embedding learning (T-SNE), manifold learning and word vector representation learning, and is suitable for various Loss functions for representation learning, including but not limited to Noise-contrast estimation Loss function (NCE Loss).

In order to make the technical solution of the present application clearer and easier to understand, the following describes an evaluation method of a representation learning model of the present application with reference to a specific scenario of "vectorization representation of a domain name visited by a user in one month".

Referring to an application scenario diagram of the evaluation method representing the learning model shown in fig. 5A and a flowchart of the evaluation method representing the learning shown in fig. 5B, the application scenario includes the terminal 102, the terminal 102 obtains the access record of the terminal in one month from the local cache, extracts the domain name from the access record, then generates a training sample set by using the domain name as a sample, and trains the representation learning model based on the training sample set by using an unsupervised learning manner, so as to implement domain name vectorization.

During the training process, the evaluation of the representation learning model is also realized by the following steps:

the method comprises the following steps: and manually selecting partial domain names from the training sample set to form a first subset, and labeling the samples in the first subset to generate a first sample subset.

Step two: and selecting a part of domain names from the training sample set to form a second subset, and taking the second subset as a second sample subset.

The first subset includes at least two types of domain names with larger difference, for example, the domain name of a blackout-related website and the domain name of a long video-on-demand website. In practical applications, about 50 samples of each class are taken for subsequent calculation of the score-to-average ratio. For the second subset, the number of domain names may be set according to actual requirements, and in this embodiment, the second subset includes 9 domain names, and these 9 domain names are added to the attention list, so as to calculate a second index representing the stability of the model based on the domain names in the attention list in the following.

It should be noted that, step one and step two may be executed in parallel, or may be executed sequentially according to a set sequence, which is not limited in this embodiment.

Step three: a representation learning model for the domain name is trained based on a training sample set.

In this embodiment, the representation learning model for the domain name may be a word2vec model, where the model takes the domain name as input and takes the representation vector corresponding to the domain name as output. Inputting the domain names in the training sample set into a word2vec model, wherein the model can extract corresponding features from the domain names based on a learning algorithm and convert the features into vectors, so that the expression vectors for the domain names are generated.

Step four: in the model training process, the corresponding split ratio of each iteration turn is synchronously calculated based on the identification vector and the label of each sample of the first sample subset.

And acquiring a representation vector obtained by the word2vec model by learning for each sample of the first sample subset aiming at each iteration turn, determining the distance between classes and the distance in the classes of each sample based on the representation vector and the label of each sample, and calculating the ratio of the distance between the classes and the distance in the classes to obtain the score combination ratio. In this way, the score-sum ratio corresponding to each iteration turn can be obtained.

Step five: in the model training process, the domain names in the attention list and the full training samples (namely all samples in the training sample set) are subjected to cosine similarity calculation periodically to obtain the similar domain names corresponding to the domain names in the attention list, a similar sample set is formed, and the Jacard index corresponding to each domain name in the second sample subset is generated aiming at the similar sample set.

In a specific implementation, the number of the selected similar domain names can be set according to requirements. As an example, the present embodiment selects the 8 domain names with the closest proximity in the list of interest to form a similar sample set.

The fourth step and the fifth step can be executed in parallel or sequentially according to a set sequence.

Step six: and judging whether the split-combination ratio and the stability meet preset conditions, if so, executing the seventh step, and if not, returning to the third step.

Step seven: a training condition representing the learning model is determined.

Specifically, whether the convergence value is converged or not can be judged according to the convergence ratio, and whether the convergence value is larger than a first reference threshold value or not, if so, the model has a good representation effect on the same type of samples and different types of samples, and the model tends to a stable state.

For the jacobian index, whether the proportion of the samples in the second sample subset which are larger than the jacobian index threshold exceeds a preset proportion or not can be judged, and if yes, the training condition of the characterization model tends to be stable. In some cases, the preset ratio may be set to 100%, that is, it is determined whether the jacobian index of each sample in the first sample subset is greater than the jacobian index threshold, and if each sample satisfies the jacobian index thresholdAnd if p is the Jacard index threshold, the training result of the word2vec word vectorization representation learning tends to be stable.

When the split-combination ratio and the Jacore index both meet the conditions, determining that the word2vec model tends to a stable state, and when at least one of the split-combination ratio and the Jacore index does not meet the conditions, indicating that the model also has an optimization space, and optimizing and adjusting the model.

Based on the above specific implementation manners of the evaluation method representing the learning model provided in the embodiments of the present application, the embodiments of the present application also provide corresponding apparatuses, which are described below from the perspective of function modularization.

Referring to fig. 6, an evaluation apparatus for representing a learning model, the apparatus 600 includes:

an index generating module 610, configured to generate, for a representation learning model trained in an unsupervised manner, a performance evaluation index of the representation learning model, where the performance evaluation index includes at least one of a first index and a second index;

and the evaluation module 620 is configured to determine the training condition of the representation learning model according to the performance evaluation index.

Optionally, referring to fig. 7, fig. 7 is a schematic structural diagram of an evaluation apparatus representing a learning model according to an embodiment of the present application, and on the basis of the structure shown in fig. 6, the index generating module 610 includes:

the first obtaining submodule 611, configured to obtain a representation vector obtained by the representation learning model in a training process for each sample of the first sample subset;

a generating submodule 612, configured to determine inter-class distances and intra-class distances of each class of samples according to the expression vectors and labels of each sample of the first sample subset, and generate a composition ratio according to a ratio of the inter-class distances to the intra-class distances;

a first determining submodule 613, configured to use the integration ratio as a first indicator.

Optionally, the performance evaluation index includes a first index;

the evaluation module 620 is specifically configured to:

Optionally, referring to fig. 8, fig. 8 is a schematic structural diagram of an evaluation apparatus representing a learning model according to an embodiment of the present application, and on the basis of the structure shown in fig. 6, the index generating module 610 includes:

a second obtaining submodule 614, configured to obtain a representation vector obtained by learning, in a training process, the representation learning model for each sample of the training sample set in multiple iteration rounds;

an adding submodule 615, configured to select, according to a representation vector of each sample in the training sample set learned in each iteration round, a preset number of most similar samples for each sample in the second sample subset, and add the preset number of similar samples selected for the sample to a similar sample set corresponding to each sample in the second sample subset and the iteration round;

a second determining submodule 616, configured to generate a jacobian index corresponding to each sample in the second sample subset for a plurality of similar sample sets corresponding to each sample in the second sample subset, where the jacobian index is used as a second indicator.

Optionally, the performance evaluation index includes a second index;

the evaluation module 620 is specifically configured to:

Alternatively, referring to fig. 9, fig. 9 is a schematic structural diagram of an evaluation apparatus representing a learning model according to an embodiment of the present application, where on the basis of the structure shown in fig. 6, the performance evaluation index includes a first index and a second index;

the evaluation module 620 includes:

a weighting submodule 621, configured to weight the first indicator and the second indicator;

and the evaluation submodule 622 is used for determining the training condition of the representation learning model according to the weighting processing result.

It should be noted that fig. 9 may also include the weighting sub-module and the evaluation sub-module on the basis of the descriptions in fig. 7 or fig. 8.

Alternatively, referring to fig. 10, fig. 10 is a schematic structural diagram of an evaluation apparatus representing a learning model provided in an embodiment of the present application, and on the basis of the structure shown in fig. 6, the performance evaluation index includes a first index and a second index;

the apparatus 600 further comprises:

the first display module 630 is configured to draw and display a training effect curve of the representation learning model according to the performance evaluation indexes generated by different iteration rounds of the representation learning model, where the training effect curve represents a change situation of the performance of the representation learning model along with a training process.

Of course, fig. 10 may also include the first display module on the basis of fig. 6 to 9.

Alternatively, referring to fig. 11, fig. 11 is a schematic structural diagram of an evaluation apparatus representing a learning model according to an embodiment of the present application, where on the basis of the structure shown in fig. 6, the performance evaluation index includes a first index and a second index;

the index generating module 610 is specifically configured to:

the apparatus 600 further comprises:

the second display module 640 is configured to draw and display a comparison effect graph of the representation learning model, where the comparison effect graph is used to represent respective training effect curves of the representation learning model based on different hyper-parameters, and the training effect curves represent changes of performance of the representation learning model along with a training process.

Fig. 11 may also include a second display module on the basis of fig. 6 to 9.

An embodiment of the present application further provides an apparatus, which may be specifically a terminal, as shown in fig. 12, for convenience of description, only a portion related to the embodiment of the present application is shown, and details of the specific technology are not disclosed, please refer to a method portion in the embodiment of the present application. The terminal may be any terminal device including a mobile phone, a tablet computer, a Personal digital assistant (PDA, for short, in general: Personal digital assistant), a Sales terminal (POS, for short, in general: Point of Sales), a vehicle-mounted computer, etc., and the terminal is taken as a mobile phone as an example:

fig. 12 is a block diagram illustrating a partial structure of a mobile phone related to a terminal provided in an embodiment of the present application. Referring to fig. 12, the cellular phone includes: radio Frequency (RF) circuit 1210, memory 1220, input unit 1230, display unit 1240, sensor 1250, audio circuit 1260, wireless fidelity (WiFi) module 1270, processor 1280, and power supply 1290. Those skilled in the art will appreciate that the handset configuration shown in fig. 12 is not intended to be limiting and may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components.

The following describes each component of the mobile phone in detail with reference to fig. 12:

the RF circuit 1210 is configured to receive and transmit signals during information transmission and reception or during a call, and in particular, receive downlink information of a base station and then process the received downlink information to the processor 1280; in addition, the data for designing uplink is transmitted to the base station. In general, RF circuit 1210 includes, but is not limited to, an antenna, at least one Amplifier, a transceiver, a coupler, a low noise Amplifier (Lownoise Amplifier; LNA), a duplexer, and the like. In addition, the RF circuit 1210 may also communicate with networks and other devices via wireless communication. The wireless communication may use any communication standard or protocol, including but not limited to Global System for Mobile communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), e-mail, Short Message Service (SMS), and so on.

The memory 1220 may be used to store software programs and modules, and the processor 1280 executes various functional applications and data processing of the mobile phone by operating the software programs and modules stored in the memory 1220. The memory 1220 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data (such as audio data, a phonebook, etc.) created according to the use of the cellular phone, and the like. Further, the memory 1220 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device.

The input unit 1230 may be used to receive input numeric or character information and generate key signal inputs related to user settings and function control of the cellular phone. Specifically, the input unit 1230 may include a touch panel 1231 and other input devices 1232. The touch panel 1231, also referred to as a touch screen, can collect touch operations of a user (e.g., operations of the user on or near the touch panel 1231 using any suitable object or accessory such as a finger, a stylus, etc.) thereon or nearby, and drive the corresponding connection device according to a preset program. Alternatively, the touch panel 1231 may include two portions, a touch detection device and a touch controller. The touch detection device detects the touch direction of a user, detects a signal brought by touch operation and transmits the signal to the touch controller; the touch controller receives touch information from the touch sensing device, converts the touch information into touch point coordinates, and sends the touch point coordinates to the processor 1280, and can receive and execute commands sent by the processor 1280. In addition, the touch panel 1231 may be implemented by various types such as a resistive type, a capacitive type, an infrared ray, and a surface acoustic wave. The input unit 1230 may include other input devices 1232 in addition to the touch panel 1231. In particular, other input devices 1232 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.

The display unit 1240 may be used to display information input by the user or information provided to the user and various menus of the cellular phone. The Display unit 1240 may include a Display panel 1241, and optionally, the Display panel 1241 may be configured in the form of a Liquid Crystal Display (LCD), an Organic Light-Emitting Diode (OLED), or the like. Further, touch panel 1231 can overlay display panel 1241, and when touch panel 1231 detects a touch operation thereon or nearby, the touch panel 1231 can transmit the touch operation to processor 1280 to determine the type of the touch event, and then processor 1280 can provide a corresponding visual output on display panel 1241 according to the type of the touch event. Although in fig. 12, the touch panel 1231 and the display panel 1241 are implemented as two independent components to implement the input and output functions of the mobile phone, in some embodiments, the touch panel 1231 and the display panel 1241 may be integrated to implement the input and output functions of the mobile phone.

The cell phone may also include at least one sensor 1250, such as a light sensor, motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 1241 according to the brightness of ambient light, and the proximity sensor may turn off the display panel 1241 and/or the backlight when the mobile phone moves to the ear. As one of the motion sensors, the accelerometer sensor can detect the magnitude of acceleration in each direction (generally, three axes), can detect the magnitude and direction of gravity when stationary, and can be used for applications of recognizing the posture of a mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer and tapping), and the like; as for other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, and an infrared sensor, which can be configured on the mobile phone, further description is omitted here.

Audio circuitry 1260, speaker 1261, and microphone 1262 can provide an audio interface between a user and a cell phone. The audio circuit 1260 can transmit the received electrical signal converted from the audio data to the speaker 1261, and the audio signal is converted into a sound signal by the speaker 1261 and output; on the other hand, the microphone 1262 converts the collected sound signals into electrical signals, which are received by the audio circuit 1260 and converted into audio data, which are processed by the audio data output processor 1280, and then passed through the RF circuit 1210 to be transmitted to, for example, another cellular phone, or output to the memory 1220 for further processing.

WiFi belongs to short-distance wireless transmission technology, and the mobile phone can help a user to receive and send e-mails, browse webpages, access streaming media and the like through the WiFi module 1270, and provides wireless broadband internet access for the user. Although fig. 12 shows the WiFi module 1270, it is understood that it does not belong to the essential constitution of the handset, and may be omitted entirely as needed within the scope not changing the essence of the invention.

The processor 1280 is a control center of the mobile phone, connects various parts of the entire mobile phone by using various interfaces and lines, and performs various functions of the mobile phone and processes data by operating or executing software programs and/or modules stored in the memory 1220 and calling data stored in the memory 1220, thereby performing overall monitoring of the mobile phone. Optionally, processor 1280 may include one or more processing units; preferably, the processor 1280 may integrate an application processor, which mainly handles operating systems, user interfaces, application programs, etc., and a modem processor, which mainly handles wireless communications. It is to be appreciated that the modem processor described above may not be integrated into the processor 1280.

The handset also includes a power supply 1290 (e.g., a battery) for powering the various components, and preferably, the power supply may be logically connected to the processor 1280 via a power management system, so that the power management system may manage the charging, discharging, and power consumption.

Although not shown, the mobile phone may further include a camera, a bluetooth module, etc., which are not described herein.

In this embodiment, the processor 1280 included in the terminal further has the following functions:

Optionally, the processor 1280 is further configured to execute the steps of any implementation manner of the evaluation method representing the learning model provided in the embodiment of the present application.

The present application further provides a computer-readable storage medium for storing a computer program for implementing any one of the implementations of the evaluation method for representing a learning model described in the foregoing embodiments.

The present application further provides a computer program product including instructions, which when run on a computer, causes the computer to execute any one of the embodiments of the evaluation method representing a learning model described in the foregoing embodiments.

It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.

In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or contributed to by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.

The above embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions in the embodiments of the present application.

Claims

1. An evaluation method for representing a learning model, comprising:

2. The method of claim 1, wherein generating the first metric comprises:

obtaining a representation vector obtained by the representation learning model in the training process aiming at each sample of the first sample subset;

determining the inter-class distance and the intra-class distance of each type of sample according to the expression vector and the label of each sample of the first sample subset, and generating a composition ratio according to the ratio of the inter-class distance to the intra-class distance;

and taking the split-combination ratio as a first index.

3. The method of claim 2, wherein the performance evaluation index comprises a first index;

determining a training condition of the representation learning model according to the performance evaluation index, including:

4. The method of claim 1, wherein the generating a second index comprises:

obtaining a representation vector obtained by the representation learning model in a training process in multiple iteration rounds aiming at learning of each sample of a training sample set;

selecting a preset number of most similar samples for the samples in the second sample subset according to the representation vector of each sample in the training sample set learned by each iteration turn, and adding the preset number of selected samples into the similar sample sets corresponding to the samples in the second sample subset and the iteration turn;

and generating a Jacobian index corresponding to each sample in the second sample subset aiming at a plurality of similar sample sets corresponding to each sample in the second sample subset, wherein the Jacobian index is used as a second index.

5. The method of claim 4, wherein the performance evaluation index comprises a second index;

6. The method according to any one of claims 1 to 5, wherein the performance evaluation index includes a first index and a second index;

weighting the first index and the second index;

and determining the training condition of the representation learning model according to the weighting processing result.

7. The method according to any one of claims 1 to 5, further comprising:

and drawing and displaying a training effect curve of the representation learning model according to the performance evaluation indexes generated by different iteration turns of the representation learning model, wherein the training effect curve represents the change condition of the performance of the representation learning model along with the training process.

8. The method according to any one of claims 1 to 5, further comprising:

and drawing and displaying a comparison effect graph of the representation learning model, wherein the comparison effect graph is used for representing respective training effect curves of the representation learning model based on different hyper-parameters, and the training effect curves represent the variation situation of the performance of the representation learning model along with the training process.

9. The method of any one of claims 1 to 5, wherein the representation learning model is a word vector representation learning model.

10. An evaluation apparatus that represents a learning model, comprising:

11. The apparatus of claim 10, wherein the indicator generating module is specifically configured to:

and taking the split-combination ratio as a first index.

12. The apparatus of claim 10, wherein the indicator generating module is specifically configured to:

selecting a preset number of most similar samples for the samples in the second sample subset according to the representation vector of each sample in the training sample set learned by each iteration round, and adding the preset number of similar samples selected for the samples into the similar sample sets corresponding to the samples in the second sample subset and the iteration round; and generating a Jacobian index corresponding to each sample in the second sample subset aiming at a plurality of similar sample sets corresponding to each sample in the second sample subset, wherein the Jacobian index is used as a second index.

13. The method according to any one of claims 10 to 12, wherein the performance evaluation index includes a first index and a second index;

the evaluation module is specifically configured to:

weighting the first index and the second index;

14. A terminal device, comprising a processor and a memory:

the memory is used for storing a computer program;

the processor is configured to perform the method of any one of claims 1 to 9 in accordance with the computer program.

15. A computer-readable storage medium, characterized in that the computer-readable storage medium is used to store a computer program for performing the method of any of claims 1 to 9.