CN110263939A - A kind of appraisal procedure, device, equipment and medium indicating learning model - Google Patents

Abstract

The present application discloses a method for evaluating a representation learning model, including: generating a performance evaluation index for a representation learning model trained in an unsupervised manner, including at least one of a first index and a second index, wherein the first index It is based on the representation vector of each sample in the first sample subset learned by the representation learning model during the training process, and is a quantitative index generated to measure the similarity of similar samples and the distance between different types of samples. The second index is based on the representation learning model in The similarity vectors corresponding to the representation vectors of the samples in the second sample subset learned in the training process, the generated quantitative indicators for measuring the stability of the sample representations, and determining the training status of the representation learning model according to the performance evaluation indicators . Through the above quantitative indicators, it no longer depends on the subsequent machine learning tasks, and the training iteration process of the entire representation learning is greatly accelerated. The application also discloses the corresponding device, equipment and medium.

Description

A method, device, device and medium for evaluating a representation learning model

technical field

The present application relates to the field of computer technology, and in particular to an evaluation method, device, equipment and computer storage medium of a representation learning model.

Background technique

Representation learning refers to the task of converting raw data into a form that can be effectively developed by machine learning by learning the representation of data, making it easier to extract useful information when subsequently constructing classifiers or other prediction tasks. In layman's terms, it is to convert the data into a vector representation, and at the same time make the vector contain as much data information as possible that is useful for subsequent tasks. In recent years, representation learning has attracted a lot of attention in the fields of speech, image and so on.

Unsupervised representation learning refers to training representation learning models on unlabeled training data. It is difficult to evaluate unsupervised learning models because there are no known labels to compare the results of unsupervised learning with actual labels.

Usually, the evaluation of the representation learning model based on unsupervised training depends on the evaluation results of subsequent machine learning tasks, which leads to the extension of the training and optimization iteration cycle of the unsupervised representation learning model, increasing the time cost of model training , slowing down the iterative speed of the model, resulting in the loss of practical application.

Contents of the invention

This application provides an evaluation method for a representation learning model, which proposes two quantitative indicators for evaluating the training quality to measure the training status of an unsupervised representation learning model, so as to detect abnormalities in the training process in time and avoid prolonging the training cycle , slow down the training speed and increase the cost of training time, so as to avoid damage to the actual application. The present application also provides corresponding devices, devices, media and computer program products.

The first aspect of the present application provides a method for evaluating a representation learning model, the method comprising:

For a representation learning model trained in an unsupervised manner, generate a performance evaluation index of the representation learning model, where the performance evaluation index includes at least one of a first index and a second index;

Wherein, the first indicator is a quantitative indicator generated based on the representation vectors of each sample in the first sample subset learned by the representation learning model during the training process, and used to measure the similarity between samples of the same type and the distance between samples of different types; The first sample subset is generated by labeling a first subset of the training sample set of the representation learning model, and the first subset includes samples of different categories;

The second index is based on the similarity vectors corresponding to the representation vectors of the samples in the second sample subset learned by the representation learning model during the training process, and is a quantitative index for measuring the stability of sample representation generated; The two-sample subset is a second subset of the training sample set;

According to the performance evaluation index, the training situation of the representation learning model is determined.

The second aspect of the present application provides an evaluation device representing a learning model, the device comprising:

An index generation module, configured to generate a performance evaluation index of the representation learning model for the representation learning model trained in an unsupervised manner, where the performance evaluation index includes at least one of the first index and the second index;

Wherein, the first indicator is a quantitative indicator generated based on the representation vectors of each sample in the first sample subset learned by the representation learning model during the training process, and used to measure the similarity between samples of the same type and the distance between samples of different types; The first sample subset is generated by labeling a first subset of the training sample set of the representation learning model, and the first subset includes samples of different categories;

The second index is based on the similarity vectors corresponding to the representation vectors of the samples in the second sample subset learned by the representation learning model during the training process, and is a quantitative index for measuring the stability of sample representation generated; The two-sample subset is a second subset of the training sample set;

An evaluation module, configured to determine the training situation of the representation learning model according to the performance evaluation index.

Optionally, the indicator generation module includes:

The first acquisition sub-module is used to acquire the representation vector learned by the representation learning model for each sample of the first sample subset during the training process;

generating a submodule for determining inter-category distances and intra-category distances of various types of samples according to the representation vectors and labels of the samples in the first sample subset, and according to the ratio of the inter-category distances to the intra-category distances Generate split ratio;

The first determination sub-module is used to use the split ratio as the first index.

Optionally, the performance evaluation index includes a first index;

The assessment modules are specifically designed to:

When the plurality of dividing and combining ratios determined based on multiple iterative rounds are in a convergent state within a preset period of time and the convergence value is greater than a first reference threshold, it is determined that the training situation of the representation learning model tends to be stable.

Optionally, the indicator generation module includes:

The second acquisition sub-module is used to acquire the representation vector learned from each sample of the training sample set in multiple iteration rounds during the training process of the representation learning model;

Adding a submodule for selecting the most similar preset number of samples for each sample in the second sample subset according to the representation vectors of the samples in the training sample set learned in each iterative round, which will be The preset number of similar samples selected by the samples are added to the similar sample set corresponding to each sample in the second sample subset and the iteration round;

The second determining submodule is configured to generate a Jacquard index corresponding to each sample in the second sample subset for a plurality of similar sample sets corresponding to each sample in the second sample subset, and use the Jacquard index as the first Two indicators.

Optionally, the performance evaluation index includes a second index;

The evaluation module is then specifically used for:

When the proportion of samples greater than the Jacquard index threshold in the second sample subset exceeds a preset ratio, it is determined that the training situation of the representation learning model tends to be stable.

Optionally, the performance evaluation index includes a first index and a second index;

The assessment modules then include:

A weighting submodule, configured to perform weighting processing on the first index and the second index;

The evaluation submodule is used to determine the training situation of the representation learning model according to the weighted processing result.

Optionally, the device also includes:

A first display module, configured to draw and display a training effect curve of the representation learning model according to the performance evaluation indicators generated in different iteration rounds of the representation learning model, the training effect curve representing the performance of the representation learning model How performance changes as the training progresses.

Optionally, the indicator generation module is specifically used for:

For the representation learning model configured with different hyperparameters, generate the performance evaluation indicators of different iteration rounds;

The device also includes:

The second display module is used to draw and display the comparison effect graph of the representation learning model, the comparison effect graph is used to represent the respective training effect curves of the representation learning models based on different hyperparameters, and the training effect curve represents The performance of the representation learning model varies with the training process.

Optionally, the representation learning model is a word vector representation learning model.

The third aspect of the present application provides a terminal device, where the terminal device includes a processor and a memory:

The memory is used to store computer programs;

The processor is configured to execute the method for evaluating a representation learning model described in the first aspect of the present application according to the computer program.

A fourth aspect of the present application provides a computer-readable storage medium, where the computer-readable storage medium is used to store a computer program, and the computer program is used to execute the method for evaluating a representation learning model described in the first aspect above.

The fifth aspect of the present application provides a computer program product including instructions, which, when run on a computer, cause the computer to execute the method for evaluating a representation learning model described in the first aspect above.

It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

An evaluation method of a representation learning model is provided in an embodiment of the present application. For a representation learning model trained in an unsupervised manner, the method proposes two quantitative indicators, and at least one of the two quantitative indicators can be used to measure the The training status of the model, specifically, the first indicator is to label part of the data in the training sample set. During the training process, the intra-group distance and inter-group distance of various samples are calculated based on the labeled data. According to the above-mentioned intra-group distance and The quantitative index generated by the distance between groups is used to measure the similarity of the same type of samples and the distance of different types of samples. The first indicator determines the training situation of the learning model, and the second indicator is to determine a part of the samples from the training sample set. During the training process, periodically calculate the similarity between its representation vector and the sample representation in the entire training sample set, and determine the In the similar vector corresponding to the above representation vector, the quantitative index used to measure the stability of the sample representation is generated. The more stable the quantitative index is, the more stable the model is. In this way, the training of the representation learning model can be determined based on the second index. Happening.

Through the above quantitative indicators, users can grasp the model training status in a timely manner, that is, whether the model is getting better gradually, whether the training can be terminated, etc., and no longer depend on subsequent machine learning tasks, which greatly speeds up the training iteration process of the entire representation learning , which saves the time spent on adding a model to train, adjust and evaluate later. Moreover, this method provides quantitative evaluation results, and the subsequent adjustment method of the model can be determined through the performance of different parameter combinations. On the one hand, it avoids the problem of relying on historical experience, which is prone to errors due to strong subjectivity. On the other hand, it makes the automatic adjustment of hyperparameters a possible.

Description of drawings

FIG. 1 is a scene architecture diagram representing an evaluation method of a learning model in an embodiment of the present application;

Fig. 2 is the flow chart that represents the evaluation method of learning model in the embodiment of the present application;

FIG. 3 is a time-consuming comparison diagram of the training model representing the learning model in the embodiment of the present application;

FIG. 4 is a training effect diagram of different representation learning models in the embodiment of the present application;

FIG. 5A is a scene diagram representing an evaluation method of a learning model in an embodiment of the present application;

Fig. 5B is a flow chart representing the evaluation method of the learning model in the embodiment of the present application;

6 is a schematic structural diagram of an evaluation device representing a learning model in an embodiment of the present application;

FIG. 7 is a schematic structural diagram of an evaluation device representing a learning model in an embodiment of the present application;

FIG. 8 is a schematic structural diagram of an evaluation device representing a learning model in an embodiment of the present application;

FIG. 9 is a schematic structural diagram of an evaluation device representing a learning model in an embodiment of the present application;

FIG. 10 is a schematic structural diagram of an evaluation device representing a learning model in an embodiment of the present application;

FIG. 11 is a schematic structural diagram of an evaluation device representing a learning model in an embodiment of the present application;

FIG. 12 is a schematic structural diagram of a terminal in an embodiment of the present application.

Detailed ways

In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiment of the application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiment of the application. Obviously, the described embodiment is only It is a part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.

The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above drawings are used to distinguish similar objects, and not necessarily Used to describe a specific sequence or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances such that the embodiments of the application described herein, for example, can be practiced in sequences other than those illustrated or described herein. Furthermore, the terms "comprising" and "having", as well as any variations thereof, are intended to cover a non-exclusive inclusion, for example, a process, method, system, product or device comprising a sequence of steps or elements is not necessarily limited to the expressly listed instead, may include other steps or elements not explicitly listed or inherent to the process, method, product or apparatus.

Aiming at the current evaluation of the representation learning model trained in an unsupervised manner depends on the evaluation results of subsequent machine learning tasks, resulting in the prolongation of the training cycle of the unsupervised representation learning model, the slowdown of the training speed, and the increase in training time costs, this application provides A method that can realize quantitative evaluation during the training process. The method is realized through at least one of two quantitative indicators. One quantitative indicator is to pre-label a small amount of data and calculate a real-time value according to the model training progress. Quantitative evaluation of one dimension that characterizes the classification ability of the model, and another quantitative index is to determine a small number of samples tracked in the whole process, periodically calculate the similarity with the full amount of results, and evaluate the stability of the results, so as to realize the stability of the model Evaluate.

This method enables users to know the training status no longer depending on the subsequent machine learning tasks, and the training iteration process of the entire representation learning is greatly accelerated, saving the time spent on adding another model for training, adjustment and evaluation. Moreover, this method provides quantitative evaluation results, and the subsequent adjustment method of the model can be determined through the performance of different parameter combinations. On the one hand, it avoids the problem of relying on historical experience, which is prone to errors due to strong subjectivity. On the other hand, it makes the automatic adjustment of hyperparameters a possible.

It can be understood that the method for evaluating a representation learning model provided in this application can be applied to any processing device with data processing capabilities, and the processing device can be a terminal or a server. Wherein, the terminal may be a desktop computer, a portable mobile terminal device such as a tablet computer or a smart phone, or a mainframe computer or the like. A server refers to a device that provides computing services, which may be an independent computing device or a computing cluster composed of multiple computing devices.

The method for evaluating a representation learning model provided in the present application may be stored in a processing device in the form of a computer program, and the processing device implements the above method for evaluating a representation learning model by executing the computer program. Wherein, the computer program may be independent, or may be a functional module, plug-in or small program running on other programs.

In practical application, the evaluation method of the representation learning model provided in this application can be applied to but not limited to the application environment shown in FIG. 1 .

As shown in Figure 1, the terminal 102 is connected to the database 104, and the training sample set is stored in the database 104. The terminal 102 obtains samples from the training sample set in the database 104, and uses an unsupervised way to train the representation learning model. During the training process, the terminal 102 Based on the representation vectors of each sample in the first sample subset, generate a first indicator for measuring the similarity of samples of the same type and the distance between samples of different types, and generate a similar vector corresponding to the representation vectors of each sample in the second sample subset for measuring the distance between samples of the same type. The second indicator representing stability, the terminal 102 determines based on at least one of the first indicator and the second indicator indicates a training situation of the learning model.

Next, each step of the method for evaluating a representation learning model provided in the embodiment of the present application will be described in detail from the perspective of a terminal.

Referring to the flow chart representing the evaluation method of the learning model shown in Figure 2, the method includes:

S201: For a representation learning model trained in an unsupervised manner, generate a performance evaluation index of the representation learning model.

In this embodiment, the terminal uses the samples in the training sample set to train the representation learning model in an unsupervised manner. In order to evaluate the training status of the representation learning model, the terminal generates a performance evaluation indicator for the representation learning model, which is used to evaluate the representation learning model.

Wherein, the performance evaluation index includes at least one of the first index and the second index. The first indicator is a quantitative indicator calculated in real time based on pre-labeled samples. This quantitative indicator can represent the classification ability of the model, that is, the ability to represent similar samples as similar results and different types of samples as distant results. In other words, the Quantitative indicators can measure the degree of similarity between samples of the same type and the distance between samples of different types; the second indicator is a quantitative indicator calculated based on a small number of samples tracked in the whole process. The full amount indicates the similarity of the results, and quantitatively evaluates the stability of the model output results.

The first index is generated based on the representation vectors of the samples in the first sample subset learned by the representation learning model during the training process. Wherein, the first sample subset is generated based on the first subset in the training sample set. Specifically, a small number of samples are determined from the training sample set to form the first subset, which includes samples of different categories, so as to determine the classification ability of the model for samples of different categories, and labeling the samples in the first subset can be obtained first sample subset. That is, the first sample subset includes the first subset and labels corresponding to samples in the first subset.

In specific implementation, the terminal obtains the representation vectors learned by the representation learning model for each sample in the first sample subset during the training process, and then determines the representation vectors and labels of each sample in the first sample subset to determine the The inter-category distance and the intra-category distance generate a division ratio according to the ratio of the inter-category distance to the intra-category distance, and use the division ratio as a first index. The calculation process of the first indicator can be referred to the following formula:

Among them, Λ represents the split ratio, d _AB represents the distance between categories AB, d _A and d _B represent the sample distance in category A and the sample distance in category B respectively, and mean represents the average value. Based on this, mean(d _AB ) Indicates the average distance between categories AB, mean(d _A ) and mean(d _B ) respectively represent the average distance of samples in category A and the average distance of samples in category B.

Among them, the sample distance has various forms of expression. In some possible implementations, the cosine distance can be used for characterization. For details, please refer to the following formula:

Among them, d is the cosine distance, X and Y represent the representation vectors corresponding to different samples, respectively, and ‖X‖ and ‖Y‖ represent the lengths of X and Y respectively.

Of course, in practical applications, other methods may also be used to calculate the sample distance, such as Euclidean distance, Manhattan distance, Chebyshev distance, etc., which is not limited in this embodiment. In addition, in some cases, the terminal may also use the median of the distance instead of the average distance to calculate the splitting ratio. The above is only an example of the present application and does not constitute a limitation to the technical solution of the present application.

For the second index, it is a quantitative index for measuring the stability of sample representation generated based on the similarity vectors corresponding to the representation vectors of the samples in the second sample subset learned by the representation learning model during the training process. Wherein, the second sample subset is the second subset in the training sample set.

It can be understood that the representation learning model is iteratively trained in rounds. In order to calculate the second indicator for measuring the stability of the sample representation, the terminal can obtain the representation vector learned by the representation learning model for each sample in the training sample set in multiple iteration rounds. , according to the representation vector of each sample in the training sample set learned in each iterative round, select the most similar N samples for each sample in the second sample subset, and add the N similar samples selected for the sample to A similar sample set corresponding to each sample in the second sample subset and an iteration round, for a plurality of similar sample sets corresponding to each sample in the second sample subset, generate a sample corresponding to each sample in the second sample subset Carr index, using the Jacquard index as a second index. Wherein, N is a preset number, which can be set according to actual needs, for example, set as a positive integer greater than 1.

Jaccard Index (Jaccard Index), also known as Jaccard similarity coefficient, is a probability value for comparing the similarity or dispersion of sample sets, which is equal to the ratio of the intersection of sample sets and the union of sample sets, referred to as the intersection ratio. application, which can be calculated by the following formula:

in, Characterize the similar sample set of the i-th sample in the second sample subset at the t-th step of the model iteration value (that is, the t-th round), Characterize the similar sample set of the i-th sample in the second sample subset at the t+nth step of the model iteration value (that is, the t+nth round), i is a positive integer between 1 and k, and k is the second the number of elements in the sample subset, Characterize the number of intersection elements of the above two similar sample sets, The number of union elements representing the above two similar sample sets.

S202: Determine a training situation of the representation learning model according to the performance evaluation index.

In practical applications, the performance evaluation index index includes at least one of the division ratio and the Jacquard index. Based on this, the terminal can determine the training status of the representation learning model through the following implementation methods, which will be described in detail below.

The first implementation manner is to determine the training situation of the representation learning model based only on the division and combination ratio. Specifically, when a plurality of split ratios determined based on multiple iteration rounds are in a convergent state within a preset period of time and the convergence value is greater than a first reference threshold, it is determined that the training situation of the representation learning model tends to be stable .

It can be understood that the benchmark reference value of the split-combination ratio Λ is 1, which means that there is no difference between the average distance within the two categories and the average distance between the categories. Based on this, in an effective representation learning and training process, the split-combination ratio The value should be greater than 1, and gradually increase until it is stable. Based on this, the terminal may determine whether the training situation of the learning model tends to be stable based on the convergence status and the convergence value of the division/combination ratio determined in multiple iterations within a preset period of time.

For the convergence status and convergence value of the split-combination ratio, the terminal can be realized in the following manner. Specifically, for each round of iterative process within the preset time period, the inter-category distance and intra-category distance of various samples are calculated according to the representation vector and corresponding label of each sample in the first sample subset, based on the inter-category distance and The distance within the category is used to calculate the splitting ratio, where the splitting ratio before iteration is recorded as the first splitting ratio, and the splitting ratio after iteration is recorded as the second splitting ratio. When the first splitting ratio and the second splitting ratio are greater than The first reference threshold, and when the absolute value of the difference between the second split ratio and the first split ratio is smaller than the second reference threshold, it is determined that the split ratio is convergent, and the convergence value is greater than the first reference threshold.

It should be noted that the preset time period, the first reference threshold and the second reference threshold can be set according to actual needs. As an example of this application, the preset time period can be one day, the first reference threshold can be 2, the second The second reference threshold may be 0.01.

For the split ratio, the larger the value, the more obvious the difference between categories, and the more similar each category is, which reflects that the result of unsupervised representation learning meets the classification standards on small samples, and represents the representation vector output by the learning model Carrying valuable information, the effectiveness of the entire model training is guaranteed.

The second implementation is to determine the training condition of the representation learning model based only on the Jacquard index. Specifically, for the Jacquard index, when the proportion of samples larger than the Jacquard index threshold in the second sample subset exceeds a preset ratio, it is determined that the training situation of the representation learning model tends to be stable.

It can be understood that in an effective representation learning training process, with the deepening of the representation learning model training, the similar sample sets corresponding to each sample in the second sample subset should show a high correlation and have a gradually fixed trend, in other words, each The similar sample set corresponding to the sample no longer undergoes a large amount of variation. Based on this, the terminal may determine whether the training situation of the representation learning model tends to be stable based on the magnitude of the Jacquard index within the preset time period. For the samples in the second sample subset, if the Jacquard index is greater than the Jacquard index threshold, it indicates that the similar sample set of the sample is almost the same before and after iteration, if the Jacquard index of the samples exceeding the preset ratio is greater than the Jacquard index threshold, it indicates that the learning model tends to be stable.

It should be noted that the preset ratio and the Jacquard index threshold can be set according to actual needs. As an example of the present application, the Jacquard index threshold can be set to 70%, and the preset ratio can be set to 80%.

The third implementation method is to jointly determine the training situation of the representation learning model based on the split-combination ratio and the Jacquard index. When determining the training situation based on the split-combination ratio and the Jacquard index, it can be judged separately whether the split-combination ratio and the Jacquard index meet their corresponding standards, so as to determine the training situation of the learning model; it can also be the split-combination ratio and The Jacquard index performs weighting processing, and the training status of the representation learning model is determined according to the weighting processing result.

It should be noted that the split-combination ratio and the Jacquard index belong to different dimensions. When weighting the split-combination ratio and the Jacquard index, the split-combination ratio and the Jacquard index can also be normalized first, and then based on The normalized indicators are weighted. Wherein, the respective weights of the split ratio and the Jacquard index can be set according to actual needs.

It should also be noted that the above three implementations are illustrated with the first index as the split ratio and the second index as the Jacquard index. In other possible implementations of the embodiments of the present application, the first index and When the second indicator is other parameters, at least one of the other parameters may be referred to to determine the training status of the representation learning model.

It can be seen from the above that the embodiment of the present application provides an evaluation method for a representation learning model. For a representation learning model trained in an unsupervised manner, this method proposes two quantitative indicators, and at least one of these two quantitative indicators It can measure the training status of the model. Specifically, the first indicator is to label part of the data in the training sample set. During the training process, the intra-group distance and inter-group distance of various samples are calculated based on the labeled data. According to the above group The quantitative index generated by the intra-group distance and the inter-group distance is used to measure the similarity of similar samples and the distance between different types of samples. This quantitative index indicates that the closer the same type of samples are, the more distant the different types of samples are, indicating that the classification ability of the model is better. In this way, The training situation of the representation learning model can be determined based on the first index. The second index is to determine some samples from the training sample set. During the training process, periodically calculate the similarity between its representation vector and the sample representation in the entire training sample set, and determine The similar vectors corresponding to the above representation vectors in each cycle are then generated to measure the stability of the sample representation. The more stable the quantitative index is, the more stable the model is. In this way, the representation learning can be determined based on the second index The training status of the model.

Through the above quantitative indicators, users can grasp the model training status in a timely manner, that is, whether the model is getting better gradually, whether the training can be terminated, etc., and no longer depend on subsequent machine learning tasks, which greatly speeds up the training iteration process of the entire representation learning , which saves the time spent on adding a model to train, adjust and evaluate later.

This application provides a comparison chart of the time spent on unsupervised representation learning based on the evaluation method of the representation learning model of this application and the time spent on traditional unsupervised representation learning. As shown in Figure 3, the traditional machine learning takes 7 days, including the time spent on unsupervised representation learning (2 days) and the time it takes to determine the learning situation of unsupervised representation learning based on subsequent machine learning tasks (5 days), and this application can be based on performance in the process of unsupervised representation learning The evaluation index determines the learning status, and there is no need to determine the current learning status of unsupervised representation learning through subsequent machine learning, which directly saves the time-consuming follow-up machine learning tasks and speeds up the training progress.

Considering that the above performance evaluation index is variable, the terminal may also draw and display the training effect curve of the representation learning model according to the performance evaluation index generated in different iteration rounds of the representation learning model, the training effect curve represents the Indicates how the performance of the learning model changes with the training process, making the process and effect of model training visualized, allowing users to intuitively discover whether the training of the model is effective.

Further, the terminal may also generate the performance evaluation indicators of different iteration rounds for the representation learning model configured with different hyperparameters, draw and display a comparison effect diagram of the representation learning model, and the comparison effect diagram is used for Show the respective training effect curves of the representation learning models based on different hyperparameters, which can help users determine the direction of model selection and parameter adjustment. On the one hand, it avoids the problem of relying on historical experience and subjectivity that is prone to error. On the other hand, Aspects enable automatic tuning of hyperparameters.

For ease of understanding, the present application also provides a specific example of a comparison effect diagram. As shown in Figure 4, it shows the training effect curves of the representation learning models corresponding to 5 groups of different hyperparameter combinations, namely 41 to 45, wherein the split ratio of the representation learning models corresponding to curve 41 and curve 42 converges to a relatively Curve 43, Curve 44, and Curve 45 correspond to a high value, and curve 43, curve 44 and curve 45 indicate that the division ratio of the learning model converges to a higher value, and curve 43 indicates that the learning model reaches a stable state first.

The evaluation method of the representation learning model provided by this application can be applied to a variety of unsupervised representation learning tasks, such as t-distributed stochastic neighbor embedding learning (T-distributed Stochastic Neighbor Embedding, t-SNE), manifold learning, word vector Representation learning, and is applicable to various loss functions for representation learning, including but not limited to Noise-Contrastive Loss (NCE Loss).

In order to make the technical solution of this application clearer and easier to understand, the evaluation method of the representation learning model of this application will be introduced below in conjunction with the specific scenario of "vectorized representation of domain names visited by users within a month".

Referring to the application scenario diagram of the evaluation method of representation learning model shown in FIG. 5A and the flow chart of the evaluation method of representation learning shown in FIG. The domain name is extracted from the access record, and then the domain name is used as a sample to generate a training sample set. Based on the training sample set, an unsupervised learning method is used to train the representation learning model to realize domain name vectorization.

During the training process, the evaluation of the representation learning model is also realized through the following steps:

Step 1: Manually select some domain names from the training sample set to form the first subset, and mark the samples in the first subset to generate the first sample subset.

Step 2: Select part of the domain names from the training sample set to form a second subset, and use the second subset as the second sample subset.

Wherein, the first subset includes at least two types of domain names with large differences, for example, domain names of black-related websites and long-form video-on-demand websites. In practical application, about 50 samples are taken from each category for subsequent calculation of split ratio. For the second subset, the number of domain names can be set according to actual needs. In this embodiment, the second subset includes 9 domain names, and these 9 domain names are added to the watch list for subsequent calculation of the representation model based on the domain names in the watch list A second indicator of stability.

It should be noted that step 1 and step 2 may be executed in parallel, or may be executed successively according to the setting order, which is not limited in this embodiment.

Step 3: Train the domain name representation learning model based on the training sample set.

In this embodiment, the representation learning model for the domain name may be a word2vec model, which takes the domain name as input and outputs a representation vector corresponding to the domain name. Input the domain names in the training sample set into the word2vec model, and the model can extract corresponding features based on the learning algorithm and convert them into vectors to generate representation vectors for domain names.

Step 4: During the model training process, synchronously calculate the division ratio corresponding to each iteration round based on the identification vector and label of each sample in the first sample subset.

For each iteration round, obtain the representation vector learned by the word2vec model for each sample in the first sample subset, and then determine the inter-category distance and intra-category distance of various samples based on the representation vector and the labels of each sample, The split ratio can be obtained by calculating the ratio of the distance between categories and the distance within categories. In this way, the split ratio corresponding to each iteration round can be obtained.

Step 5: During the model training process, the domain names in the watch list and the full training samples (that is, all samples in the training sample set) are periodically calculated for cosine similarity, and similar domain names corresponding to the domain names in the watch list are obtained to form a similar sample set , generating the Jacquard index corresponding to each domain name in the second sample subset based on the above similar sample set.

During specific implementation, the number of selected similar domain names may be set according to requirements. As an example, in this embodiment, 8 domain names that are most similar to domain names in the attention list are selected to form a similar sample set.

Wherein, Step 4 and Step 5 can be executed in parallel, or can be executed successively according to the set order.

Step 6: Determine whether the split ratio and stability meet the preset conditions, if so, go to step 7, if not, go back to step 3.

Step 7: Determine the training situation of the representation learning model.

Specifically, for the separation and combination ratio, it can be judged whether it is converged, and whether the convergence value is greater than the first reference threshold, if so, it indicates that the model has a good representation effect on both samples of the same type and samples of different types, and the model tends to a stable state .

Regarding the Jacquard index, it may be determined whether the proportion of samples in the second sample subset greater than the threshold of the Jacquard index exceeds a preset ratio, and if so, the training situation of the representation model tends to be stable. In some cases, the above-mentioned preset ratio can be set to 100%, that is, to judge whether the Jacquard index of each sample in the first sample subset is greater than the Jacquard index threshold, if each sample satisfies Among them, p is the Jacquard index threshold, and it is considered that the training result of word2vec word vectorization representation learning tends to be stable.

When both the split ratio and the Jacquard index meet the conditions, it is determined that the word2vec model tends to a stable state. When at least one of the split ratio and the Jacquard index does not meet the conditions, it indicates that the model still has room for optimization, and the model can be optimized Adjustment.

The above are some specific implementations of the evaluation method of the representation learning model provided by the embodiment of the present application. Based on this, the embodiment of the present application also provides a corresponding device, which will be introduced from the perspective of functional modularization below.

Referring to the evaluation device representing a learning model shown in FIG. 6, the device 600 includes:

An index generation module 610, configured to generate a performance evaluation index of the representation learning model for the representation learning model trained in an unsupervised manner, where the performance evaluation index includes at least one of the first index and the second index;

Wherein, the first indicator is a quantitative indicator generated based on the representation vectors of each sample in the first sample subset learned by the representation learning model during the training process, and used to measure the similarity between samples of the same type and the distance between samples of different types; The first sample subset is generated by labeling a first subset of the training sample set of the representation learning model, and the first subset includes samples of different categories;

The second index is based on the similarity vectors corresponding to the representation vectors of the samples in the second sample subset learned by the representation learning model during the training process, and is a quantitative index for measuring the stability of sample representation generated; The two-sample subset is a second subset of the training sample set;

The evaluation module 620 is configured to determine the training situation of the representation learning model according to the performance evaluation index.

Optionally, refer to FIG. 7, which is a schematic structural diagram of an evaluation device representing a learning model provided in an embodiment of the present application. On the basis of the structure shown in FIG. 6, the index generation module 610 includes:

The first acquisition sub-module 611 is configured to acquire the representation vector learned by the representation learning model for each sample of the first sample subset during the training process;

The generation submodule 612 is used to determine the inter-category distance and the intra-category distance of each sample according to the representation vector and label of each sample in the first sample subset, and determine the inter-category distance and the intra-category distance according to the Ratio generates split ratio;

The first determining sub-module 613 is configured to use the split ratio as a first index.

Optionally, the performance evaluation index includes a first index;

The evaluation module 620 is specifically used for:

When the plurality of dividing and combining ratios determined based on multiple iterative rounds are in a convergent state within a preset period of time and the convergence value is greater than a first reference threshold, it is determined that the training situation of the representation learning model tends to be stable.

Optionally, refer to FIG. 8. FIG. 8 is a schematic structural diagram of an evaluation device representing a learning model provided in an embodiment of the present application. On the basis of the structure shown in FIG. 6, the index generation module 610 includes:

The second acquisition sub-module 614 is used to acquire the representation vector learned by the representation learning model for each sample of the training sample set during multiple iteration rounds during the training process;

Adding a submodule 615, configured to select the most similar preset number of samples for each sample in the second sample subset according to the representation vector of each sample in the training sample set learned in each iteration round, and adding a preset number of similar samples selected for the sample to a similar sample set corresponding to each sample in the second sample subset and the iteration round;

The second determining submodule 616 is configured to generate a Jacquard index corresponding to each sample in the second sample subset for multiple similar sample sets corresponding to each sample in the second sample subset, and use the Jacquard index as Second indicator.

Optionally, the performance evaluation index includes a second index;

Then the evaluation module 620 is specifically used for:

When the proportion of samples greater than the Jacquard index threshold in the second sample subset exceeds a preset ratio, it is determined that the training situation of the representation learning model tends to be stable.

Optionally, refer to FIG. 9. FIG. 9 is a schematic structural diagram of an evaluation device representing a learning model provided in an embodiment of the present application. On the basis of the structure shown in FIG. 6, the performance evaluation index includes a first index and a second index ;

The evaluation module 620 includes:

A weighting submodule 621, configured to perform weighting processing on the first index and the second index;

The evaluation sub-module 622 is configured to determine the training situation of the representation learning model according to the weighted processing result.

It should be noted that FIG. 9 may also be based on what is shown in FIG. 7 or 8 , including the above weighting sub-module and evaluation sub-module.

Optionally, refer to FIG. 10, which is a schematic structural diagram of an evaluation device representing a learning model provided in an embodiment of the present application. On the basis of the structure shown in FIG. 6, the performance evaluation index includes a first index and a second index ;

The device 600 also includes:

The first display module 630 is configured to draw and display a training effect curve of the representation learning model according to the performance evaluation indicators generated in different iteration rounds of the representation learning model, the training effect curve representing the representation learning model performance changes with the training process.

Certainly, Fig. 10 may also include a first display module on the basis of Fig. 6 to Fig. 9 .

Optionally, refer to FIG. 11. FIG. 11 is a schematic structural diagram of an evaluation device representing a learning model provided in an embodiment of the present application. On the basis of the structure shown in FIG. 6, the performance evaluation index includes a first index and a second index ;

The indicator generating module 610 is specifically used for:

For the representation learning model configured with different hyperparameters, generate the performance evaluation indicators of different iteration rounds;

The device 600 also includes:

The second display module 640 is configured to draw and display a comparison effect graph of the representation learning model, the comparison effect graph is used to represent the respective training effect curves of the representation learning models based on different hyperparameters, and the training effect curves Indicates how the performance of the representation learning model changes with the training process.

Wherein, Fig. 11 may also include a second display module on the basis of Fig. 6 to Fig. 9 .

Optionally, the representation learning model is a word vector representation learning model.

The embodiment of the present application also provides a device, which may specifically be a terminal, as shown in Figure 12, for the sake of illustration, only the parts related to the embodiment of the present application are shown, and the specific technical details are not disclosed, please refer to The method part of the embodiment of the present application. The terminal can be any terminal device including mobile phone, tablet computer, personal digital assistant (English full name: Personal Digital Assistant, English abbreviation: PDA), sales terminal (English full name: Point of Sales, English abbreviation: POS), vehicle-mounted computer, etc. The terminal is a mobile phone as an example:

FIG. 12 shows a block diagram of a partial structure of a mobile phone related to the terminal provided by the embodiment of the present application. Referring to Fig. 12, the mobile phone includes: radio frequency (English full name: Radio Frequency, English abbreviation: RF) circuit 1210, memory 1220, input unit 1230, display unit 1240, sensor 1250, audio circuit 1260, wireless fidelity (English full name: wirelessfidelity, English abbreviation: WiFi module 1270 , processor 1280 , power supply 1290 and other components. Those skilled in the art can understand that the structure of the mobile phone shown in FIG. 12 does not constitute a limitation to the mobile phone, and may include more or less components than shown in the figure, or combine some components, or arrange different components.

The following is a specific introduction to each component of the mobile phone in conjunction with Figure 12:

The RF circuit 1210 can be used for sending and receiving information or receiving and sending signals during a call. In particular, after receiving the downlink information from the base station, it is processed by the processor 1280; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit 1210 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (full English name: Low Noise Amplifier, English abbreviation: LNA), a duplexer, and the like. In addition, RF circuitry 1210 may also communicate with networks and other devices via wireless communications. The above-mentioned wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile Communication (English full name: Global System of Mobile communication, English abbreviation: GSM), General Packet Radio Service (English full name: General Packet Radio Service, GPRS ), Code Division Multiple Access (English full name: CodeDivision Multiple Access, English abbreviation: CDMA), Wideband Code Division Multiple Access (English full name: Wideband CodeDivision Multiple Access, English abbreviation: WCDMA), Long Term Evolution (English full name: Long TermEvolution, English Abbreviation: LTE), email, short message service (English full name: Short Messaging Service, SMS), etc.

The memory 1220 can be used to store software programs and modules, and the processor 1280 executes various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 1220 . The memory 1220 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required by a function (such as a sound playback function, an image playback function, etc.) and the like; Data created by the use of mobile phones (such as audio data, phonebook, etc.), etc. In addition, the memory 1220 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage devices.

The input unit 1230 can be used to receive input numbers or character information, and generate key signal input related to user settings and function control of the mobile phone. Specifically, the input unit 1230 may include a touch panel 1231 and other input devices 1232 . The touch panel 1231, also referred to as a touch screen, can collect touch operations of the user on or near it (for example, the user uses any suitable object or accessory such as a finger or a stylus on the touch panel 1231 or near the touch panel 1231). operation), and drive the corresponding connection device according to the preset program. Optionally, the touch panel 1231 may include two parts, a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, and detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and sends it to the to the processor 1280, and can receive and execute commands sent by the processor 1280. In addition, the touch panel 1231 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 1231 , the input unit 1230 may also include other input devices 1232 . Specifically, other input devices 1232 may include but not limited to one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), trackball, mouse, joystick, and the like.

The display unit 1240 may be used to display information input by or provided to the user and various menus of the mobile phone. The display unit 1240 may include a display panel 1241. Optionally, a liquid crystal display (English full name: Liquid Crystal Display, English abbreviation: LCD), an organic light-emitting diode (English full name: Organic Light-Emitting Diode, English abbreviation: OLED) etc. may be used. form to configure the display panel 1241 . Further, the touch panel 1231 can cover the display panel 1241, and when the touch panel 1231 detects a touch operation on or near it, it sends it to the processor 1280 to determine the type of the touch event, and then the processor 1280 determines the type of the touch event according to the The type provides a corresponding visual output on the display panel 1241 . Although in FIG. 12 , the touch panel 1231 and the display panel 1241 are used as two independent components to realize the input and input functions of the mobile phone, in some embodiments, the touch panel 1231 and the display panel 1241 can be integrated to form a mobile phone. Realize the input and output functions of the mobile phone.

The handset may also include at least one sensor 1250, such as a light sensor, motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 1241 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 1241 and/or when the mobile phone is moved to the ear. or backlight. As a kind of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes), and can detect the magnitude and direction of gravity when it is stationary, and can be used to identify the application of mobile phone posture (such as horizontal and vertical screen switching, related Games, magnetometer attitude calibration), vibration recognition related functions (such as pedometer, tap), etc.; as for other sensors such as gyroscope, barometer, hygrometer, thermometer, infrared sensor, etc. repeat.

The audio circuit 1260, the speaker 1261, and the microphone 1262 can provide an audio interface between the user and the mobile phone. The audio circuit 1260 can transmit the electrical signal converted from the received audio data to the speaker 1261, and the speaker 1261 converts it into an audio signal for output; After being received, it is converted into audio data, and then the audio data is processed by the output processor 1280, and then sent to another mobile phone through the RF circuit 1210, or the audio data is output to the memory 1220 for further processing.

WiFi is a short-distance wireless transmission technology. The mobile phone can help users send and receive emails, browse web pages, and access streaming media through the WiFi module 1270. It provides users with wireless broadband Internet access. Although FIG. 12 shows a WiFi module 1270, it can be understood that it is not an essential component of the mobile phone, and can be completely omitted as required without changing the essence of the invention.

The processor 1280 is the control center of the mobile phone. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and/or modules stored in the memory 1220, and calling data stored in the memory 1220, execution Various functions and processing data of the mobile phone, so as to monitor the mobile phone as a whole. Optionally, the processor 1280 may include one or more processing units; preferably, the processor 1280 may integrate an application processor and a modem processor, wherein the application processor mainly processes operating systems, user interfaces, and application programs, etc. , the modem processor mainly handles wireless communications. It can be understood that the foregoing modem processor may not be integrated into the processor 1280 .

The mobile phone also includes a power supply 1290 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 1280 through the power management system, so that functions such as charging, discharging, and power consumption management can be realized through the power management system.

Although not shown, the mobile phone may also include a camera, a Bluetooth module, etc., which will not be repeated here.

In this embodiment of the application, the processor 1280 included in the terminal also has the following functions:

For a representation learning model trained in an unsupervised manner, generate a performance evaluation index of the representation learning model, where the performance evaluation index includes at least one of a first index and a second index;

Wherein, the first indicator is a quantitative indicator generated based on the representation vectors of each sample in the first sample subset learned by the representation learning model during the training process, and used to measure the similarity between samples of the same type and the distance between samples of different types; The first sample subset is generated by labeling a first subset of the training sample set of the representation learning model, and the first subset includes samples of different categories;

The second index is based on the similarity vectors corresponding to the representation vectors of the samples in the second sample subset learned by the representation learning model during the training process, and is a quantitative index for measuring the stability of sample representation generated; The two-sample subset is a second subset of the training sample set;

According to the performance evaluation index, the training situation of the representation learning model is determined.

Optionally, the processor 1280 is further configured to execute steps in any implementation manner of the method for evaluating a representation learning model provided in the embodiments of the present application.

An embodiment of the present application further provides a computer-readable storage medium for storing a computer program, and the computer program is used to implement any one of the methods for evaluating a representation learning model described in the foregoing embodiments.

An embodiment of the present application further provides a computer program product including instructions, which, when run on a computer, cause the computer to execute any one of the methods for evaluating a representation learning model described in the foregoing embodiments.

Those skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or May be integrated into another system, or some features may be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

If the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , including several instructions to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (English full name: Read-OnlyMemory, English abbreviation: ROM), random access memory (English full name: Random Access Memory, English abbreviation: RAM), disk Or various media such as CDs that can store program codes.

As mentioned above, the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: it can still understand the foregoing The technical solutions described in each embodiment are modified, or some of the technical features are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the application.

Claims (15)

1. An evaluation method for representing a learning model, comprising:

generating a performance evaluation index of a representation learning model trained based on an unsupervised mode, wherein the performance evaluation index comprises at least one of a first index and a second index;

the first index is a quantization index which is generated for measuring similar samples and distant samples of different types based on the expression vector of each sample in the first sample subset which is learned by the expression learning model in the training process; the first subset of samples is generated by labeling a first subset of the training sample set representing the learning model, wherein the first subset comprises samples of different classes;

the second index is a quantitative index which is generated for measuring the representation stability of the samples based on the similar vectors corresponding to the representation vectors of the samples in the second sample subset which are learned by the representation learning model in the training process; the second subset of samples is a second subset of the set of training samples;

and determining the training condition of the representation learning model according to the performance evaluation index.

2. The method of claim 1, wherein generating the first metric comprises:

obtaining a representation vector obtained by the representation learning model in the training process aiming at each sample of the first sample subset;

determining the inter-class distance and the intra-class distance of each type of sample according to the expression vector and the label of each sample of the first sample subset, and generating a composition ratio according to the ratio of the inter-class distance to the intra-class distance;

and taking the split-combination ratio as a first index.

3. The method of claim 2, wherein the performance evaluation index comprises a first index;

determining a training condition of the representation learning model according to the performance evaluation index, including:

and when a plurality of the split-combination ratios determined based on a plurality of iteration turns within a preset time period are in a convergence state and the convergence value is greater than a first reference threshold value, determining that the training condition of the representation learning model tends to be stable.

4. The method of claim 1, wherein the generating a second index comprises:

obtaining a representation vector obtained by the representation learning model in a training process in multiple iteration rounds aiming at learning of each sample of a training sample set;

selecting a preset number of most similar samples for the samples in the second sample subset according to the representation vector of each sample in the training sample set learned by each iteration turn, and adding the preset number of selected samples into the similar sample sets corresponding to the samples in the second sample subset and the iteration turn;

and generating a Jacobian index corresponding to each sample in the second sample subset aiming at a plurality of similar sample sets corresponding to each sample in the second sample subset, wherein the Jacobian index is used as a second index.

5. The method of claim 4, wherein the performance evaluation index comprises a second index;

determining a training condition of the representation learning model according to the performance evaluation index, including:

and when the proportion of the samples in the second sample subset which are larger than the Jacard exponent threshold exceeds a preset proportion, determining that the training condition of the representation learning model tends to be stable.

6. The method according to any one of claims 1 to 5, wherein the performance evaluation index includes a first index and a second index;

determining a training condition of the representation learning model according to the performance evaluation index, including:

weighting the first index and the second index;

and determining the training condition of the representation learning model according to the weighting processing result.

7. The method according to any one of claims 1 to 5, further comprising:

and drawing and displaying a training effect curve of the representation learning model according to the performance evaluation indexes generated by different iteration turns of the representation learning model, wherein the training effect curve represents the change condition of the performance of the representation learning model along with the training process.

8. The method according to any one of claims 1 to 5, further comprising:

generating the performance evaluation indexes of different iteration rounds aiming at the representation learning model configured with different hyper-parameters;

and drawing and displaying a comparison effect graph of the representation learning model, wherein the comparison effect graph is used for representing respective training effect curves of the representation learning model based on different hyper-parameters, and the training effect curves represent the variation situation of the performance of the representation learning model along with the training process.

9. The method of any one of claims 1 to 5, wherein the representation learning model is a word vector representation learning model.

10. An evaluation apparatus that represents a learning model, comprising:

the index generation module is used for generating a performance evaluation index of the representation learning model aiming at the representation learning model trained on an unsupervised mode, wherein the performance evaluation index comprises at least one of a first index and a second index;

the first index is a quantization index which is generated for measuring similar samples and distant samples of different types based on the expression vector of each sample in the first sample subset which is learned by the expression learning model in the training process; the first subset of samples is generated by labeling a first subset of the training sample set representing the learning model, wherein the first subset comprises samples of different classes;

the second index is a quantitative index which is generated for measuring the representation stability of the samples based on the similar vectors corresponding to the representation vectors of the samples in the second sample subset which are learned by the representation learning model in the training process; the second subset of samples is a second subset of the set of training samples;

and the evaluation module is used for determining the training condition of the representation learning model according to the performance evaluation index.

11. The apparatus of claim 10, wherein the indicator generating module is specifically configured to:

obtaining a representation vector obtained by the representation learning model in the training process aiming at each sample of the first sample subset;

determining the inter-class distance and the intra-class distance of each type of sample according to the expression vector and the label of each sample of the first sample subset, and generating a composition ratio according to the ratio of the inter-class distance to the intra-class distance;

and taking the split-combination ratio as a first index.

12. The apparatus of claim 10, wherein the indicator generating module is specifically configured to:

obtaining a representation vector obtained by the representation learning model in a training process in multiple iteration rounds aiming at learning of each sample of a training sample set;

selecting a preset number of most similar samples for the samples in the second sample subset according to the representation vector of each sample in the training sample set learned by each iteration round, and adding the preset number of similar samples selected for the samples into the similar sample sets corresponding to the samples in the second sample subset and the iteration round; and generating a Jacobian index corresponding to each sample in the second sample subset aiming at a plurality of similar sample sets corresponding to each sample in the second sample subset, wherein the Jacobian index is used as a second index.

13. The method according to any one of claims 10 to 12, wherein the performance evaluation index includes a first index and a second index;

the evaluation module is specifically configured to:

weighting the first index and the second index;

and determining the training condition of the representation learning model according to the weighting processing result.

14. A terminal device, comprising a processor and a memory:

the memory is used for storing a computer program;

the processor is configured to perform the method of any one of claims 1 to 9 in accordance with the computer program.

15. A computer-readable storage medium, characterized in that the computer-readable storage medium is used to store a computer program for performing the method of any of claims 1 to 9.