CN110263939A - A kind of appraisal procedure, device, equipment and medium indicating learning model - Google Patents
A kind of appraisal procedure, device, equipment and medium indicating learning model Download PDFInfo
- Publication number
- CN110263939A CN110263939A CN201910549544.6A CN201910549544A CN110263939A CN 110263939 A CN110263939 A CN 110263939A CN 201910549544 A CN201910549544 A CN 201910549544A CN 110263939 A CN110263939 A CN 110263939A
- Authority
- CN
- China
- Prior art keywords
- index
- sample
- learning model
- representation
- training
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 104
- 238000012549 training Methods 0.000 claims abstract description 177
- 238000011156 evaluation Methods 0.000 claims abstract description 98
- 239000013598 vector Substances 0.000 claims abstract description 64
- 230000008569 process Effects 0.000 claims abstract description 57
- 230000000694 effects Effects 0.000 claims description 28
- 238000012545 processing Methods 0.000 claims description 18
- 238000004590 computer program Methods 0.000 claims description 15
- 238000002372 labelling Methods 0.000 claims description 7
- 239000013604 expression vector Substances 0.000 claims 4
- 238000013139 quantization Methods 0.000 claims 2
- 238000010801 machine learning Methods 0.000 abstract description 10
- 238000010586 diagram Methods 0.000 description 21
- 230000006870 function Effects 0.000 description 13
- 238000004891 communication Methods 0.000 description 6
- 238000011158 quantitative evaluation Methods 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000001514 detection method Methods 0.000 description 3
- 238000007726 management method Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 238000010295 mobile communication Methods 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 230000005856 abnormality Effects 0.000 description 1
- 230000001133 acceleration Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000012512 characterization method Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000007599 discharging Methods 0.000 description 1
- 239000006185 dispersion Substances 0.000 description 1
- 230000005484 gravity Effects 0.000 description 1
- 238000012905 input function Methods 0.000 description 1
- 238000012804 iterative process Methods 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 230000005236 sound signal Effects 0.000 description 1
- 238000010897 surface acoustic wave method Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Medical Informatics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Artificial Intelligence (AREA)
- Image Analysis (AREA)
Abstract
本申请公开了一种表示学习模型的评估方法,包括:针对基于无监督方式进行训练的表示学习模型生成性能评价指标,其包括第一指标和第二指标中的至少一个,其中,第一指标是基于表示学习模型在训练过程中学习到的第一样本子集中各样本的表示向量,生成的用于衡量同类样本相近且不同类样本相疏远的量化指标,第二指标是基于表示学习模型在训练过程中学习到的第二样本子集中各样本的表示向量对应的相似向量,生成的用于衡量样本表示稳定性的量化指标,根据所述性能评价指标,确定所述表示学习模型的训练情况。通过上述量化指标,使得不再依赖于后续的机器学习任务,整个表示学习的训练迭代过程大大加快。本申请还公开了对应的装置、设备及介质。
The present application discloses a method for evaluating a representation learning model, including: generating a performance evaluation index for a representation learning model trained in an unsupervised manner, including at least one of a first index and a second index, wherein the first index It is based on the representation vector of each sample in the first sample subset learned by the representation learning model during the training process, and is a quantitative index generated to measure the similarity of similar samples and the distance between different types of samples. The second index is based on the representation learning model in The similarity vectors corresponding to the representation vectors of the samples in the second sample subset learned in the training process, the generated quantitative indicators for measuring the stability of the sample representations, and determining the training status of the representation learning model according to the performance evaluation indicators . Through the above quantitative indicators, it no longer depends on the subsequent machine learning tasks, and the training iteration process of the entire representation learning is greatly accelerated. The application also discloses the corresponding device, equipment and medium.
Description
技术领域technical field
本申请涉及计算机技术领域,尤其涉及一种表示学习模型的评估方法、装置、设备及计算机存储介质。The present application relates to the field of computer technology, and in particular to an evaluation method, device, equipment and computer storage medium of a representation learning model.
背景技术Background technique
表示学习是指通过学习数据的表示,将原始数据转换成能够被机器学习来有效开发的形式,使得其后续构建分类器或者其他预测任务时更容易提取有用信息的任务。通俗来讲,就是将数据转换成向量表示,同时使得向量包含尽可能多的、对后续任务有用的数据信息。近年来,表示学习在语音、图像等领域广受关注。Representation learning refers to the task of converting raw data into a form that can be effectively developed by machine learning by learning the representation of data, making it easier to extract useful information when subsequently constructing classifiers or other prediction tasks. In layman's terms, it is to convert the data into a vector representation, and at the same time make the vector contain as much data information as possible that is useful for subsequent tasks. In recent years, representation learning has attracted a lot of attention in the fields of speech, image and so on.
无监督表示学习是指在无标签训练数据上训练表示学习模型。由于没有已知的标签,无法将无监督学习的结果与实际标签进行比较,所以很难评估无监督学习的模型。Unsupervised representation learning refers to training representation learning models on unlabeled training data. It is difficult to evaluate unsupervised learning models because there are no known labels to compare the results of unsupervised learning with actual labels.
通常,对基于无监督方式训练的表示学习模型的评估,是依赖于后续机器学习任务的评估结果,这就导致无监督表示学习模型的训练、优化迭代的周期延长,增加了模型训练的时间成本,拖慢模型的迭代速度,造成实际应用损失。Usually, the evaluation of the representation learning model based on unsupervised training depends on the evaluation results of subsequent machine learning tasks, which leads to the extension of the training and optimization iteration cycle of the unsupervised representation learning model, increasing the time cost of model training , slowing down the iterative speed of the model, resulting in the loss of practical application.
发明内容Contents of the invention
本申请提供了一种表示学习模型的评估方法,其提出了两种评估训练质量的量化指标,以衡量无监督表示学习模型的训练状况,从而及时发现训练过程中的异常情况,避免训练周期延长、训练速度放缓以及训练时间成本增加,进而避免对实际应用造成损伤。本申请还提供了对应的装置、设备、介质及计算机程序产品。This application provides an evaluation method for a representation learning model, which proposes two quantitative indicators for evaluating the training quality to measure the training status of an unsupervised representation learning model, so as to detect abnormalities in the training process in time and avoid prolonging the training cycle , slow down the training speed and increase the cost of training time, so as to avoid damage to the actual application. The present application also provides corresponding devices, devices, media and computer program products.
本申请第一方面提供了一种表示学习模型的评估方法,所述方法包括:The first aspect of the present application provides a method for evaluating a representation learning model, the method comprising:
针对基于无监督方式进行训练的表示学习模型,生成所述表示学习模型的性能评价指标,所述性能评价指标包括第一指标和第二指标中的至少一个;For a representation learning model trained in an unsupervised manner, generate a performance evaluation index of the representation learning model, where the performance evaluation index includes at least one of a first index and a second index;
其中,所述第一指标是基于所述表示学习模型在训练过程中学习到的第一样本子集中各样本的表示向量,生成的用于衡量同类样本相近且不同类样本相疏远的量化指标;所述第一样本子集是对所述表示学习模型的训练样本集中的第一子集进行标签标注生成的,所述第一子集包括不同类别的样本;Wherein, the first indicator is a quantitative indicator generated based on the representation vectors of each sample in the first sample subset learned by the representation learning model during the training process, and used to measure the similarity between samples of the same type and the distance between samples of different types; The first sample subset is generated by labeling a first subset of the training sample set of the representation learning model, and the first subset includes samples of different categories;
所述第二指标是基于所述表示学习模型在训练过程中学习到的第二样本子集中各样本的表示向量对应的相似向量,生成的用于衡量样本表示稳定性的量化指标;所述第二样本子集是所述训练样本集中的第二子集;The second index is based on the similarity vectors corresponding to the representation vectors of the samples in the second sample subset learned by the representation learning model during the training process, and is a quantitative index for measuring the stability of sample representation generated; The two-sample subset is a second subset of the training sample set;
根据所述性能评价指标,确定所述表示学习模型的训练情况。According to the performance evaluation index, the training situation of the representation learning model is determined.
本申请第二方面提供一种表示学习模型的评估装置,所述装置包括:The second aspect of the present application provides an evaluation device representing a learning model, the device comprising:
指标生成模块,用于针对基于无监督方式进行训练的表示学习模型,生成所述表示学习模型的性能评价指标,所述性能评价指标包括第一指标和第二指标中的至少一个;An index generation module, configured to generate a performance evaluation index of the representation learning model for the representation learning model trained in an unsupervised manner, where the performance evaluation index includes at least one of the first index and the second index;
其中,所述第一指标是基于所述表示学习模型在训练过程中学习到的第一样本子集中各样本的表示向量,生成的用于衡量同类样本相近且不同类样本相疏远的量化指标;所述第一样本子集是对所述表示学习模型的训练样本集中的第一子集进行标签标注生成的,所述第一子集包括不同类别的样本;Wherein, the first indicator is a quantitative indicator generated based on the representation vectors of each sample in the first sample subset learned by the representation learning model during the training process, and used to measure the similarity between samples of the same type and the distance between samples of different types; The first sample subset is generated by labeling a first subset of the training sample set of the representation learning model, and the first subset includes samples of different categories;
所述第二指标是基于所述表示学习模型在训练过程中学习到的第二样本子集中各样本的表示向量对应的相似向量,生成的用于衡量样本表示稳定性的量化指标;所述第二样本子集是所述训练样本集中的第二子集;The second index is based on the similarity vectors corresponding to the representation vectors of the samples in the second sample subset learned by the representation learning model during the training process, and is a quantitative index for measuring the stability of sample representation generated; The two-sample subset is a second subset of the training sample set;
评估模块,用于根据所述性能评价指标,确定所述表示学习模型的训练情况。An evaluation module, configured to determine the training situation of the representation learning model according to the performance evaluation index.
可选的,所述指标生成模块包括:Optionally, the indicator generation module includes:
第一获取子模块,用于获取所述表示学习模型在训练过程中针对第一样本子集各样本学习得到的表示向量;The first acquisition sub-module is used to acquire the representation vector learned by the representation learning model for each sample of the first sample subset during the training process;
生成子模块,用于根据所述第一样本子集各样本的表示向量和标签,确定各类样本的类别间距离和类别内距离,根据所述类别间距离和所述类别内距离的比值生成分合比;generating a submodule for determining inter-category distances and intra-category distances of various types of samples according to the representation vectors and labels of the samples in the first sample subset, and according to the ratio of the inter-category distances to the intra-category distances Generate split ratio;
第一确定子模块,用于将所述分合比作为第一指标。The first determination sub-module is used to use the split ratio as the first index.
可选的,所述性能评价指标包括第一指标;Optionally, the performance evaluation index includes a first index;
所述评估模块具体用于:The assessment modules are specifically designed to:
当预设时间段内基于多个迭代轮次所确定的多个所述分合比呈收敛状态且收敛值大于第一参考阈值时,确定所述表示学习模型的训练情况趋于稳定。When the plurality of dividing and combining ratios determined based on multiple iterative rounds are in a convergent state within a preset period of time and the convergence value is greater than a first reference threshold, it is determined that the training situation of the representation learning model tends to be stable.
可选的,所述指标生成模块包括:Optionally, the indicator generation module includes:
第二获取子模块,用于获取所述表示学习模型在训练过程中多个迭代轮次针对训练样本集各样本学习得到的表示向量;The second acquisition sub-module is used to acquire the representation vector learned from each sample of the training sample set in multiple iteration rounds during the training process of the representation learning model;
添加子模块,用于根据每个迭代轮次学习到的所述训练样本集中各样本的表示向量,为所述第二样本子集中的各样本分别选择最相似的预设数量个样本,将为样本所选择的预设数量个相似样本加入与所述第二样本子集中各样本以及迭代轮次对应的相似样本集;Adding a submodule for selecting the most similar preset number of samples for each sample in the second sample subset according to the representation vectors of the samples in the training sample set learned in each iterative round, which will be The preset number of similar samples selected by the samples are added to the similar sample set corresponding to each sample in the second sample subset and the iteration round;
第二确定子模块,用于针对所述第二样本子集中各样本对应的多个相似样本集,生成所述第二样本子集中各个样本对应的雅卡尔指数,将所述雅卡尔指数作为第二指标。The second determining submodule is configured to generate a Jacquard index corresponding to each sample in the second sample subset for a plurality of similar sample sets corresponding to each sample in the second sample subset, and use the Jacquard index as the first Two indicators.
可选的,所述性能评价指标包括第二指标;Optionally, the performance evaluation index includes a second index;
则所述评估模块具体用于:The evaluation module is then specifically used for:
当所述第二样本子集中大于雅卡尔指数阈值的样本占比超过预设比例时,确定所述表示学习模型的训练情况趋于稳定。When the proportion of samples greater than the Jacquard index threshold in the second sample subset exceeds a preset ratio, it is determined that the training situation of the representation learning model tends to be stable.
可选的,所述性能评价指标包括第一指标和第二指标;Optionally, the performance evaluation index includes a first index and a second index;
则所述评估模块包括:The assessment modules then include:
加权子模块,用于对所述第一指标和所述第二指标进行加权处理;A weighting submodule, configured to perform weighting processing on the first index and the second index;
评估子模块,用于根据加权处理结果确定所述表示学习模型的训练情况。The evaluation submodule is used to determine the training situation of the representation learning model according to the weighted processing result.
可选的,所述装置还包括:Optionally, the device also includes:
第一显示模块,用于根据所述表示学习模型不同迭代轮次生成的所述性能评价指标,绘制并显示所述表示学习模型的训练效果曲线,所述训练效果曲线表示所述表示学习模型的性能随着训练进程的变化情况。A first display module, configured to draw and display a training effect curve of the representation learning model according to the performance evaluation indicators generated in different iteration rounds of the representation learning model, the training effect curve representing the performance of the representation learning model How performance changes as the training progresses.
可选的,所述指标生成模块具体用于:Optionally, the indicator generation module is specifically used for:
针对配置有不同超参数的所述表示学习模型,生成不同迭代轮次的所述性能评价指标;For the representation learning model configured with different hyperparameters, generate the performance evaluation indicators of different iteration rounds;
所述装置还包括:The device also includes:
第二显示模块,用于绘制并显示所述表示学习模型的对比效果图,所述对比效果图用于表现基于不同超参数的所述表示学习模型各自的训练效果曲线,所述训练效果曲线表示所述表示学习模型的性能随着训练进程的变化情况。The second display module is used to draw and display the comparison effect graph of the representation learning model, the comparison effect graph is used to represent the respective training effect curves of the representation learning models based on different hyperparameters, and the training effect curve represents The performance of the representation learning model varies with the training process.
可选的,所述表示学习模型是词向量表示学习模型。Optionally, the representation learning model is a word vector representation learning model.
本申请第三方面提供一种终端设备,所述终端设备包括处理器以及存储器:The third aspect of the present application provides a terminal device, where the terminal device includes a processor and a memory:
所述存储器用于存储计算机程序;The memory is used to store computer programs;
所述处理器用于根据所述计算机程序执行本申请第一方面所述的表示学习模型的评估方法。The processor is configured to execute the method for evaluating a representation learning model described in the first aspect of the present application according to the computer program.
本申请第四方面提供一种计算机可读存储介质,所述计算机可读存储介质用于存储计算机程序,所述计算机程序用于执行上述第一方面所述的表示学习模型的评估方法。A fourth aspect of the present application provides a computer-readable storage medium, where the computer-readable storage medium is used to store a computer program, and the computer program is used to execute the method for evaluating a representation learning model described in the first aspect above.
本申请第五方面提供一种包括指令的计算机程序产品,当其在计算机上运行时,使得所述计算机执行上述第一方面所述的表示学习模型的评估方法。The fifth aspect of the present application provides a computer program product including instructions, which, when run on a computer, cause the computer to execute the method for evaluating a representation learning model described in the first aspect above.
从以上技术方案可以看出,本申请实施例具有以下优点:It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:
本申请实施例中提供了一种表示学习模型的评估方法,针对采用无监督方式训练的表示学习模型,该方法提出了两种量化指标,通过这两种量化指标中的至少一种可以衡量该模型的训练状况,具体地,第一指标是对训练样本集中的部分数据进行标注,在训练过程中,基于该标注数据计算各类样本的组内距离与组间距离,根据上述组内距离和组间距离生成的用于衡量同类样本相近且不同类样本相疏远的量化指标,该量化指标表征同类样本越相近,不同类样本越疏远,则表明模型的分类能力越好,如此,可以基于该第一指标确定表示学习模型的训练情况,第二指标是从训练样本集中确定部分样本,在训练过程中,周期性计算其表示向量与整个训练样本集中样本表示的相似程度,确定出每个周期中与上述表示向量对应的相似向量,进而生成的用于衡量样本表示稳定性的量化指标,该量化指标越稳定,则表明模型越稳定,如此,可以基于该第二指标确定表示学习模型的训练情况。An evaluation method of a representation learning model is provided in an embodiment of the present application. For a representation learning model trained in an unsupervised manner, the method proposes two quantitative indicators, and at least one of the two quantitative indicators can be used to measure the The training status of the model, specifically, the first indicator is to label part of the data in the training sample set. During the training process, the intra-group distance and inter-group distance of various samples are calculated based on the labeled data. According to the above-mentioned intra-group distance and The quantitative index generated by the distance between groups is used to measure the similarity of the same type of samples and the distance of different types of samples. The first indicator determines the training situation of the learning model, and the second indicator is to determine a part of the samples from the training sample set. During the training process, periodically calculate the similarity between its representation vector and the sample representation in the entire training sample set, and determine the In the similar vector corresponding to the above representation vector, the quantitative index used to measure the stability of the sample representation is generated. The more stable the quantitative index is, the more stable the model is. In this way, the training of the representation learning model can be determined based on the second index. Happening.
通过上述量化指标,使得用户可以及时掌握模型训练状况,即模型是否在逐渐变好,训练是否可以终止等等,而不再依赖于后续的机器学习任务,使得整个表示学习的训练迭代过程大大加快,省去了后续再加入一个模型训练、调整并评估所耗费的时间。而且,该方法提供了量化评估结果,可以通过不同参数组合的表现确定模型后续调整方法,一方面避免了依赖历史经验,主观性强导致容易出错的问题,另一方面使得超参数的自动调节成为可能。Through the above quantitative indicators, users can grasp the model training status in a timely manner, that is, whether the model is getting better gradually, whether the training can be terminated, etc., and no longer depend on subsequent machine learning tasks, which greatly speeds up the training iteration process of the entire representation learning , which saves the time spent on adding a model to train, adjust and evaluate later. Moreover, this method provides quantitative evaluation results, and the subsequent adjustment method of the model can be determined through the performance of different parameter combinations. On the one hand, it avoids the problem of relying on historical experience, which is prone to errors due to strong subjectivity. On the other hand, it makes the automatic adjustment of hyperparameters a possible.
附图说明Description of drawings
图1为本申请实施例中表示学习模型的评估方法的场景架构图;FIG. 1 is a scene architecture diagram representing an evaluation method of a learning model in an embodiment of the present application;
图2为本申请实施例中表示学习模型的评估方法的流程图;Fig. 2 is the flow chart that represents the evaluation method of learning model in the embodiment of the present application;
图3为本申请实施例中表示学习模型的训练耗时对比图;FIG. 3 is a time-consuming comparison diagram of the training model representing the learning model in the embodiment of the present application;
图4为本申请实施例中不同表示学习模型的训练效果图;FIG. 4 is a training effect diagram of different representation learning models in the embodiment of the present application;
图5A为本申请实施例中表示学习模型的评估方法的场景图;FIG. 5A is a scene diagram representing an evaluation method of a learning model in an embodiment of the present application;
图5B为本申请实施例中表示学习模型的评估方法的流程图;Fig. 5B is a flow chart representing the evaluation method of the learning model in the embodiment of the present application;
图6为本申请实施例中表示学习模型的评估装置的结构示意图;6 is a schematic structural diagram of an evaluation device representing a learning model in an embodiment of the present application;
图7为本申请实施例中表示学习模型的评估装置的结构示意图;FIG. 7 is a schematic structural diagram of an evaluation device representing a learning model in an embodiment of the present application;
图8为本申请实施例中表示学习模型的评估装置的结构示意图;FIG. 8 is a schematic structural diagram of an evaluation device representing a learning model in an embodiment of the present application;
图9为本申请实施例中表示学习模型的评估装置的结构示意图;FIG. 9 is a schematic structural diagram of an evaluation device representing a learning model in an embodiment of the present application;
图10为本申请实施例中表示学习模型的评估装置的结构示意图;FIG. 10 is a schematic structural diagram of an evaluation device representing a learning model in an embodiment of the present application;
图11为本申请实施例中表示学习模型的评估装置的结构示意图;FIG. 11 is a schematic structural diagram of an evaluation device representing a learning model in an embodiment of the present application;
图12为本申请实施例中终端的一个结构示意图。FIG. 12 is a schematic structural diagram of a terminal in an embodiment of the present application.
具体实施方式Detailed ways
为了使本技术领域的人员更好地理解本申请方案,下面将结合本申请实施例中的附图,对本申请实施例中的技术方案进行清楚、完整地描述,显然,所描述的实施例仅是本申请一部分实施例,而不是全部的实施例。基于本申请中的实施例,本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例,都属于本申请保护的范围。In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiment of the application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiment of the application. Obviously, the described embodiment is only It is a part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.
本申请的说明书和权利要求书及上述附图中的术语“第一”、“第二”、“第三”、“第四”等(如果存在)是用于区别类似的对象,而不必用于描述特定的顺序或先后次序。应该理解这样使用的数据在适当情况下可以互换,以便这里描述的本申请的实施例例如能够以除了在这里图示或描述的那些以外的顺序实施。此外,术语“包括”和“具有”以及他们的任何变形,意图在于覆盖不排他的包含,例如,包含了一系列步骤或单元的过程、方法、系统、产品或设备不必限于清楚地列出的那些步骤或单元,而是可包括没有清楚地列出的或对于这些过程、方法、产品或设备固有的其它步骤或单元。The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above drawings are used to distinguish similar objects, and not necessarily Used to describe a specific sequence or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances such that the embodiments of the application described herein, for example, can be practiced in sequences other than those illustrated or described herein. Furthermore, the terms "comprising" and "having", as well as any variations thereof, are intended to cover a non-exclusive inclusion, for example, a process, method, system, product or device comprising a sequence of steps or elements is not necessarily limited to the expressly listed instead, may include other steps or elements not explicitly listed or inherent to the process, method, product or apparatus.
针对目前对无监督方式训练的表示学习模型的评估依赖于后续机器学习任务的评估结果,导致无监督表示学习模型的训练周期延长、训练速度放缓、训练时间成本增加的问题,本申请提供了一种在训练过程中即可实现量化评估的方法,该方法具体是通过两种量化指标中的至少一种实现的,一种量化指标是对少量数据进行预标注,根据模型训练进度实时计算一种表征模型分类能力的量纲进行量化评估,另一种量化指标是确定少量全过程追踪的样本,周期性计算与全量表示结果的相似度,评估结果的稳定性,从而实现对模型稳定性的评估。Aiming at the current evaluation of the representation learning model trained in an unsupervised manner depends on the evaluation results of subsequent machine learning tasks, resulting in the prolongation of the training cycle of the unsupervised representation learning model, the slowdown of the training speed, and the increase in training time costs, this application provides A method that can realize quantitative evaluation during the training process. The method is realized through at least one of two quantitative indicators. One quantitative indicator is to pre-label a small amount of data and calculate a real-time value according to the model training progress. Quantitative evaluation of one dimension that characterizes the classification ability of the model, and another quantitative index is to determine a small number of samples tracked in the whole process, periodically calculate the similarity with the full amount of results, and evaluate the stability of the results, so as to realize the stability of the model Evaluate.
该方法使得用户不再依赖于后续的机器学习任务即可获知训练状况,整个表示学习的训练迭代过程大大加快,省去了后续再加入一个模型训练、调整并评估所耗费的时间。而且,该方法提供了量化评估结果,可以通过不同参数组合的表现确定模型后续调整方法,一方面避免了依赖历史经验,主观性强导致容易出错的问题,另一方面使得超参数的自动调节成为可能。This method enables users to know the training status no longer depending on the subsequent machine learning tasks, and the training iteration process of the entire representation learning is greatly accelerated, saving the time spent on adding another model for training, adjustment and evaluation. Moreover, this method provides quantitative evaluation results, and the subsequent adjustment method of the model can be determined through the performance of different parameter combinations. On the one hand, it avoids the problem of relying on historical experience, which is prone to errors due to strong subjectivity. On the other hand, it makes the automatic adjustment of hyperparameters a possible.
可以理解,本申请提供的表示学习模型的评估方法可以应用于任意具有数据处理能力的处理设备,该处理设备可以是终端,也可以是服务器。其中,终端可以是台式机desktop,也可以是平板电脑、智能手机等便携式移动终端设备,还可以是大型机等等。服务器是指提供计算服务的设备,其可以是独立的计算设备,也可以是多个计算设备组成的计算集群。It can be understood that the method for evaluating a representation learning model provided in this application can be applied to any processing device with data processing capabilities, and the processing device can be a terminal or a server. Wherein, the terminal may be a desktop computer, a portable mobile terminal device such as a tablet computer or a smart phone, or a mainframe computer or the like. A server refers to a device that provides computing services, which may be an independent computing device or a computing cluster composed of multiple computing devices.
本申请提供的表示学习模型的评估方法可以以计算机程序的形式存储于处理设备中,处理设备通过执行该计算机程序实现上述表示学习模型的评估方法。其中,计算机程序可以是独立的,也可以是运行于其他程序之上的功能模块、插件或者小程序等。The method for evaluating a representation learning model provided in the present application may be stored in a processing device in the form of a computer program, and the processing device implements the above method for evaluating a representation learning model by executing the computer program. Wherein, the computer program may be independent, or may be a functional module, plug-in or small program running on other programs.
在实际应用时,本申请提供的表示学习模型的评估方法可以但不限于应用于如图1所示的应用环境中。In practical application, the evaluation method of the representation learning model provided in this application can be applied to but not limited to the application environment shown in FIG. 1 .
如图1所示,终端102与数据库104连接,数据库104中存储有训练样本集,终端102从数据库104中的训练样本集获取样本,采用无监督方式训练表示学习模型,在训练过程中,终端102基于第一样本子集中各样本的表示向量生成用于衡量同类样本相近且不同类样本相疏远的第一指标,基于第二样本子集中各样本的表示向量对应的相似向量生成用于衡量样本表示稳定性的第二指标,终端102基于第一指标和第二指标中的至少一个确定表示学习模型的训练情况。As shown in Figure 1, the terminal 102 is connected to the database 104, and the training sample set is stored in the database 104. The terminal 102 obtains samples from the training sample set in the database 104, and uses an unsupervised way to train the representation learning model. During the training process, the terminal 102 Based on the representation vectors of each sample in the first sample subset, generate a first indicator for measuring the similarity of samples of the same type and the distance between samples of different types, and generate a similar vector corresponding to the representation vectors of each sample in the second sample subset for measuring the distance between samples of the same type. The second indicator representing stability, the terminal 102 determines based on at least one of the first indicator and the second indicator indicates a training situation of the learning model.
接下来,从终端的角度对本申请实施例提供的表示学习模型的评估方法的各个步骤进行详细说明。Next, each step of the method for evaluating a representation learning model provided in the embodiment of the present application will be described in detail from the perspective of a terminal.
参见图2所示的表示学习模型的评估方法的流程图,该方法包括:Referring to the flow chart representing the evaluation method of the learning model shown in Figure 2, the method includes:
S201:针对基于无监督方式进行训练的表示学习模型,生成所述表示学习模型的性能评价指标。S201: For a representation learning model trained in an unsupervised manner, generate a performance evaluation index of the representation learning model.
在本实施例中,终端采用训练样本集中的样本,利用无监督方式训练表示学习模型。为了评估表示学习模型的训练状况,终端生成针对该表示学习模型的性能评价指标,用于评价表示学习模型。In this embodiment, the terminal uses the samples in the training sample set to train the representation learning model in an unsupervised manner. In order to evaluate the training status of the representation learning model, the terminal generates a performance evaluation indicator for the representation learning model, which is used to evaluate the representation learning model.
其中,性能评价指标包括第一指标和第二指标中的至少一个。第一指标是基于预标注的样本实时计算得到的一种量化指标,该量化指标能够表征模型分类能力,即将同类样本表示为相近结果,将不同类样本表示为相疏远结果的能力,换言之,该量化指标能够衡量同类样本相近且不同类样本相疏远的程度;第二指标是基于少量全过程追踪的样本计算得到的一种量化指标,该量化指标基于周期性计算少量全过程追踪样本表示结果与全量表示结果的相似度,定量评估模型输出结果稳定性。Wherein, the performance evaluation index includes at least one of the first index and the second index. The first indicator is a quantitative indicator calculated in real time based on pre-labeled samples. This quantitative indicator can represent the classification ability of the model, that is, the ability to represent similar samples as similar results and different types of samples as distant results. In other words, the Quantitative indicators can measure the degree of similarity between samples of the same type and the distance between samples of different types; the second indicator is a quantitative indicator calculated based on a small number of samples tracked in the whole process. The full amount indicates the similarity of the results, and quantitatively evaluates the stability of the model output results.
第一指标是基于表示学习模型在训练过程中学习得到的第一样本子集中各样本的表示向量生成的。其中,第一样本子集是基于训练样本集中的第一子集生成的。具体地,从训练样本集确定少量样本形成第一子集,该第一子集包括不同类别的样本,以便确定模型对不同类别样本的分类能力,针对第一子集中的样本进行标签标注可以得到第一样本子集。也即,第一样本子集包括第一子集及所述第一子集中各样本对应的标签。The first index is generated based on the representation vectors of the samples in the first sample subset learned by the representation learning model during the training process. Wherein, the first sample subset is generated based on the first subset in the training sample set. Specifically, a small number of samples are determined from the training sample set to form the first subset, which includes samples of different categories, so as to determine the classification ability of the model for samples of different categories, and labeling the samples in the first subset can be obtained first sample subset. That is, the first sample subset includes the first subset and labels corresponding to samples in the first subset.
在具体实现时,终端获取表示学习模型在训练过程中针对第一样本子集各样本学习得到的表示向量,然后根据第一样本子集各样本的表示向量和标签,确定各类样本的类别间距离和类别内距离,根据所述类别间距离和所述类别内距离的比值生成分合比,将所述分合比作为第一指标。第一指标的计算过程可以参见如下公式:In specific implementation, the terminal obtains the representation vectors learned by the representation learning model for each sample in the first sample subset during the training process, and then determines the representation vectors and labels of each sample in the first sample subset to determine the The inter-category distance and the intra-category distance generate a division ratio according to the ratio of the inter-category distance to the intra-category distance, and use the division ratio as a first index. The calculation process of the first indicator can be referred to the following formula:
其中,Λ表示分合比,dAB表示类别AB之间的距离,dA、dB分别表示类别A内样本距离和类别B内样本距离,mean表示平均值,基于此,mean(dAB)表示类别AB之间的平均距离,mean(dA)、mean(dB)分别表示类别A内样本平均距离和类别B内样本平均距离。Among them, Λ represents the split ratio, d AB represents the distance between categories AB, d A and d B represent the sample distance in category A and the sample distance in category B respectively, and mean represents the average value. Based on this, mean(d AB ) Indicates the average distance between categories AB, mean(d A ) and mean(d B ) respectively represent the average distance of samples in category A and the average distance of samples in category B.
其中,样本距离具有多种表现形式,在一些可能的实现方式中,可以采用余弦距离进行表征,具体可以参见如下公式:Among them, the sample distance has various forms of expression. In some possible implementations, the cosine distance can be used for characterization. For details, please refer to the following formula:
其中,d为余弦距离,X和Y分别表示不同样本对应的表示向量,‖X‖和‖Y‖分别表示X和Y各自的长度。Among them, d is the cosine distance, X and Y represent the representation vectors corresponding to different samples, respectively, and ‖X‖ and ‖Y‖ represent the lengths of X and Y respectively.
当然,在实际应用时,也可以采用其他方式计算样本距离,如采用欧氏距离、曼哈顿距离、切比雪夫距离等等,本实施例对此不作限定。此外,在有些情况下,终端也可以采用距离的中位数代替距离均值计算分合比,以上仅为本申请的一个示例,不构成对本申请技术方案的限定。Of course, in practical applications, other methods may also be used to calculate the sample distance, such as Euclidean distance, Manhattan distance, Chebyshev distance, etc., which is not limited in this embodiment. In addition, in some cases, the terminal may also use the median of the distance instead of the average distance to calculate the splitting ratio. The above is only an example of the present application and does not constitute a limitation to the technical solution of the present application.
针对第二指标,其是基于所述表示学习模型在训练过程中学习到的第二样本子集中各样本的表示向量对应的相似向量,生成的用于衡量样本表示稳定性的量化指标。其中,第二样本子集是训练样本集中的第二子集。For the second index, it is a quantitative index for measuring the stability of sample representation generated based on the similarity vectors corresponding to the representation vectors of the samples in the second sample subset learned by the representation learning model during the training process. Wherein, the second sample subset is the second subset in the training sample set.
可以理解,表示学习模型是按照轮次进行迭代训练的,为了计算衡量样本表示稳定性的第二指标,终端可以获取表示学习模型在多个迭代轮次针对训练样本集各样本学习得到的表示向量,根据每个迭代轮次学习到的所述训练样本集中各样本的表示向量,为第二样本子集中的各样本分别选择最相似的N个样本,将为样本所选择的N个相似样本加入与该第二样本子集中各样本和迭代轮次对应的相似样本集,针对所述第二样本子集中各样本对应的多个相似样本集,生成所述第二样本子集中各个样本对应的雅卡尔指数,将所述雅卡尔指数作为第二指标。其中,N为预设数量,其可以根据实际需求而设置,例如设置为大于1的正整数。It can be understood that the representation learning model is iteratively trained in rounds. In order to calculate the second indicator for measuring the stability of the sample representation, the terminal can obtain the representation vector learned by the representation learning model for each sample in the training sample set in multiple iteration rounds. , according to the representation vector of each sample in the training sample set learned in each iterative round, select the most similar N samples for each sample in the second sample subset, and add the N similar samples selected for the sample to A similar sample set corresponding to each sample in the second sample subset and an iteration round, for a plurality of similar sample sets corresponding to each sample in the second sample subset, generate a sample corresponding to each sample in the second sample subset Carr index, using the Jacquard index as a second index. Wherein, N is a preset number, which can be set according to actual needs, for example, set as a positive integer greater than 1.
雅卡尔指数(Jaccard Index)也称Jaccard相似性系数,其为比较样本集中的相似性或分散性的一个概率值,等于样本集交集和样本集并集的比值,简称交并比,具体到本申请,其可以通过如下公式计算:Jaccard Index (Jaccard Index), also known as Jaccard similarity coefficient, is a probability value for comparing the similarity or dispersion of sample sets, which is equal to the ratio of the intersection of sample sets and the union of sample sets, referred to as the intersection ratio. application, which can be calculated by the following formula:
其中,表征第二样本子集中第i个样本在模型迭代值第t步(即第t轮次)的相似样本集,表征第二样本子集中第i个样本在模型迭代值第t+n步(即第t+n轮次)的相似样本集,i取值为1至k之间的正整数,k为第二样本子集中元素的个数,表征上述两个相似样本集的交集元素个数,表征上述两个相似样本集的并集元素的个数。in, Characterize the similar sample set of the i-th sample in the second sample subset at the t-th step of the model iteration value (that is, the t-th round), Characterize the similar sample set of the i-th sample in the second sample subset at the t+nth step of the model iteration value (that is, the t+nth round), i is a positive integer between 1 and k, and k is the second the number of elements in the sample subset, Characterize the number of intersection elements of the above two similar sample sets, The number of union elements representing the above two similar sample sets.
S202:根据所述性能评价指标,确定所述表示学习模型的训练情况。S202: Determine a training situation of the representation learning model according to the performance evaluation index.
在实际应用时,性能评价指标指数包括分合比和雅卡尔指数中的至少一个,基于此,终端可以通过以下几种实现方式确定表示学习模型的训练情况,下面进行详细说明。In practical applications, the performance evaluation index index includes at least one of the division ratio and the Jacquard index. Based on this, the terminal can determine the training status of the representation learning model through the following implementation methods, which will be described in detail below.
第一种实现方式为,仅基于分合比确定表示学习模型的训练情况。具体地,当预设时间段内基于多个迭代轮次所确定的多个所述分合比呈收敛状态且收敛值大于第一参考阈值时,确定所述表示学习模型的训练情况趋于稳定。The first implementation manner is to determine the training situation of the representation learning model based only on the division and combination ratio. Specifically, when a plurality of split ratios determined based on multiple iteration rounds are in a convergent state within a preset period of time and the convergence value is greater than a first reference threshold, it is determined that the training situation of the representation learning model tends to be stable .
可以理解,分合比Λ的基准参考值为1,其表示两个类别内部的平均距离与类别之间的平均距离无差别,基于此,在一个有效的表示学习训练过程中,分合比的取值应当大于1,并且逐渐增大直至稳定。基于此,终端可以基于预设时间段内多个迭代轮次确定的分合比的收敛状况以及收敛值确定表示学习模型的训练情况是否趋于稳定。It can be understood that the benchmark reference value of the split-combination ratio Λ is 1, which means that there is no difference between the average distance within the two categories and the average distance between the categories. Based on this, in an effective representation learning and training process, the split-combination ratio The value should be greater than 1, and gradually increase until it is stable. Based on this, the terminal may determine whether the training situation of the learning model tends to be stable based on the convergence status and the convergence value of the division/combination ratio determined in multiple iterations within a preset period of time.
针对分合比的收敛状况和收敛值,终端可以通过如下方式实现。具体地,针对预设时间段内的每一轮迭代过程,根据第一样本子集中各样本的表示向量和对应的标签计算各类样本的类别间距离和类别内距离,基于该类别间距离和类别内距离计算分合比,其中,迭代前分合比记作第一分合比,迭代后分合比记作第二分合比,当第一分合比和第二分合比均大于第一参考阈值,且第二分合比与第一分合比的差值绝对值小于第二参考阈值时,则确定分合比收敛,且收敛值大于第一参考阈值。For the convergence status and convergence value of the split-combination ratio, the terminal can be realized in the following manner. Specifically, for each round of iterative process within the preset time period, the inter-category distance and intra-category distance of various samples are calculated according to the representation vector and corresponding label of each sample in the first sample subset, based on the inter-category distance and The distance within the category is used to calculate the splitting ratio, where the splitting ratio before iteration is recorded as the first splitting ratio, and the splitting ratio after iteration is recorded as the second splitting ratio. When the first splitting ratio and the second splitting ratio are greater than The first reference threshold, and when the absolute value of the difference between the second split ratio and the first split ratio is smaller than the second reference threshold, it is determined that the split ratio is convergent, and the convergence value is greater than the first reference threshold.
需要说明的是,预设时间段、第一参考阈值和第二参考阈值可以根据实际需求而设置,作为本申请的一个示例,预设时间段可以是一天,第一参考阈值可以是2,第二参考阈值可以是0.01。It should be noted that the preset time period, the first reference threshold and the second reference threshold can be set according to actual needs. As an example of this application, the preset time period can be one day, the first reference threshold can be 2, the second The second reference threshold may be 0.01.
针对分合比,其数值越大,则表明类别之间的差别越明显,各类别内部越相似,其反映了无监督表示学习的结果符合小样本上的分类标准,表示学习模型输出的表示向量携带了有价值的信息,整个模型训练的有效性得到保障。For the split ratio, the larger the value, the more obvious the difference between categories, and the more similar each category is, which reflects that the result of unsupervised representation learning meets the classification standards on small samples, and represents the representation vector output by the learning model Carrying valuable information, the effectiveness of the entire model training is guaranteed.
第二种实现方式为,仅基于雅卡尔指数确定表示学习模型的训练情况。具体地,针对雅卡尔指数,当第二样本子集中大于雅卡尔指数阈值的样本占比超过预设比例时,确定所述表示学习模型的训练情况趋于稳定。The second implementation is to determine the training condition of the representation learning model based only on the Jacquard index. Specifically, for the Jacquard index, when the proportion of samples larger than the Jacquard index threshold in the second sample subset exceeds a preset ratio, it is determined that the training situation of the representation learning model tends to be stable.
可以理解,在一个有效的表示学习训练过程中,随着表示学习模型训练的深入,第二样本子集中各样本对应的相似样本集应当呈现高相关性,并且具有逐渐固定的趋势,换言之,各样本对应的相似样本集不再出现大量变动。基于此,终端可以基于预设时间段内雅卡尔指数的大小确定表示学习模型的训练情况是否趋于稳定。针对第二样本子集中的样本,若其雅卡尔指数大于雅卡尔指数阈值,则表明该样本的相似样本集在迭代前后几乎一致,若超过预设比例的样本的雅卡尔指数均大于雅卡尔指数阈值,则表明表示学习模型趋于稳定。It can be understood that in an effective representation learning training process, with the deepening of the representation learning model training, the similar sample sets corresponding to each sample in the second sample subset should show a high correlation and have a gradually fixed trend, in other words, each The similar sample set corresponding to the sample no longer undergoes a large amount of variation. Based on this, the terminal may determine whether the training situation of the representation learning model tends to be stable based on the magnitude of the Jacquard index within the preset time period. For the samples in the second sample subset, if the Jacquard index is greater than the Jacquard index threshold, it indicates that the similar sample set of the sample is almost the same before and after iteration, if the Jacquard index of the samples exceeding the preset ratio is greater than the Jacquard index threshold, it indicates that the learning model tends to be stable.
需要说明的是,预设比例和雅卡尔指数阈值可以根据实际需求而设置,作为本申请的一个示例,雅卡尔指数阈值可以设置为70%,预设比例可以设置为80%。It should be noted that the preset ratio and the Jacquard index threshold can be set according to actual needs. As an example of the present application, the Jacquard index threshold can be set to 70%, and the preset ratio can be set to 80%.
第三种实现方式为,基于分合比和雅卡尔指数共同确定表示学习模型的训练情况。在基于分合比和雅卡尔指数共同确定训练情况时,可以是分别判断分合比和雅卡尔指数是否满足各自对应的标准,从而确定表示学习模型的训练情况;也可以是对分合比和雅卡尔指数进行加权处理,根据加权处理结果确定所述表示学习模型的训练情况。The third implementation method is to jointly determine the training situation of the representation learning model based on the split-combination ratio and the Jacquard index. When determining the training situation based on the split-combination ratio and the Jacquard index, it can be judged separately whether the split-combination ratio and the Jacquard index meet their corresponding standards, so as to determine the training situation of the learning model; it can also be the split-combination ratio and The Jacquard index performs weighting processing, and the training status of the representation learning model is determined according to the weighting processing result.
需要说明的是,分合比和雅卡尔指数属于不同的量纲,在对分合比和雅卡尔指数进行加权处理时,还可以先对分合比和雅卡尔指数进行归一化,然后基于归一化后的指标进行加权处理。其中,分合比和雅卡尔指数各自的权值可以根据实际需要而设置。It should be noted that the split-combination ratio and the Jacquard index belong to different dimensions. When weighting the split-combination ratio and the Jacquard index, the split-combination ratio and the Jacquard index can also be normalized first, and then based on The normalized indicators are weighted. Wherein, the respective weights of the split ratio and the Jacquard index can be set according to actual needs.
还需要说明的是,上述三种实现方式是以第一指标为分合比,第二指标为雅卡尔指数进行示例性说明的,在本申请实施例其他可能的实现方式中,第一指标和第二指标为其他参数时,可以参考其他参数中的至少一种确定表示学习模型的训练情况。It should also be noted that the above three implementations are illustrated with the first index as the split ratio and the second index as the Jacquard index. In other possible implementations of the embodiments of the present application, the first index and When the second indicator is other parameters, at least one of the other parameters may be referred to to determine the training status of the representation learning model.
由上可知,本申请实施例提供了一种表示学习模型的评估方法,针对采用无监督方式训练的表示学习模型,该方法提出了两种量化指标,通过这两种量化指标中的至少一种可以衡量该模型的训练状况,具体地,第一指标是对训练样本集中的部分数据进行标注,在训练过程中,基于该标注数据计算各类样本的组内距离与组间距离,根据上述组内距离和组间距离生成的用于衡量同类样本相近且不同类样本相疏远的量化指标,该量化指标表征同类样本越相近,不同类样本越疏远,则表明模型的分类能力越好,如此,可以基于该第一指标确定表示学习模型的训练情况,第二指标是从训练样本集中确定部分样本,在训练过程中,周期性计算其表示向量与整个训练样本集中样本表示的相似程度,确定出每个周期中与上述表示向量对应的相似向量,进而生成的用于衡量样本表示稳定性的量化指标,该量化指标越稳定,则表明模型越稳定,如此,可以基于该第二指标确定表示学习模型的训练情况。It can be seen from the above that the embodiment of the present application provides an evaluation method for a representation learning model. For a representation learning model trained in an unsupervised manner, this method proposes two quantitative indicators, and at least one of these two quantitative indicators It can measure the training status of the model. Specifically, the first indicator is to label part of the data in the training sample set. During the training process, the intra-group distance and inter-group distance of various samples are calculated based on the labeled data. According to the above group The quantitative index generated by the intra-group distance and the inter-group distance is used to measure the similarity of similar samples and the distance between different types of samples. This quantitative index indicates that the closer the same type of samples are, the more distant the different types of samples are, indicating that the classification ability of the model is better. In this way, The training situation of the representation learning model can be determined based on the first index. The second index is to determine some samples from the training sample set. During the training process, periodically calculate the similarity between its representation vector and the sample representation in the entire training sample set, and determine The similar vectors corresponding to the above representation vectors in each cycle are then generated to measure the stability of the sample representation. The more stable the quantitative index is, the more stable the model is. In this way, the representation learning can be determined based on the second index The training status of the model.
通过上述量化指标,使得用户可以及时掌握模型训练状况,即模型是否在逐渐变好,训练是否可以终止等等,而不再依赖于后续的机器学习任务,使得整个表示学习的训练迭代过程大大加快,省去了后续再加入一个模型训练、调整并评估所耗费的时间。Through the above quantitative indicators, users can grasp the model training status in a timely manner, that is, whether the model is getting better gradually, whether the training can be terminated, etc., and no longer depend on subsequent machine learning tasks, which greatly speeds up the training iteration process of the entire representation learning , which saves the time spent on adding a model to train, adjust and evaluate later.
本申请提供了基于本申请的表示学习模型的评估方法进行无监督表示学习所耗费的时间和传统无监督表示学习所耗费的时间的对比图,如图3所示,传统机器学习耗时为7天,包括无监督表示学习耗费时间(2天)和基于后续机器学习任务确定无监督表示学习的学习情况所耗费时间(5天),而本申请在无监督表示学习过程中,即可基于性能评价指标确定学习状况,无需通过后续机器学习确定当前无监督表示学习的学习状况,直接节省后续机器学习任务耗时,加快了训练进度。This application provides a comparison chart of the time spent on unsupervised representation learning based on the evaluation method of the representation learning model of this application and the time spent on traditional unsupervised representation learning. As shown in Figure 3, the traditional machine learning takes 7 days, including the time spent on unsupervised representation learning (2 days) and the time it takes to determine the learning situation of unsupervised representation learning based on subsequent machine learning tasks (5 days), and this application can be based on performance in the process of unsupervised representation learning The evaluation index determines the learning status, and there is no need to determine the current learning status of unsupervised representation learning through subsequent machine learning, which directly saves the time-consuming follow-up machine learning tasks and speeds up the training progress.
考虑到上述性能评价指标是可变的,终端还可以根据表示学习模型不同迭代轮次生成的所述性能评价指标,绘制并显示所述表示学习模型的训练效果曲线,该训练效果曲线表示所述表示学习模型的性能随着训练进程的变化情况,使得模型训练的过程与效果可视化,直观地让用户发现模型的训练是否有效。Considering that the above performance evaluation index is variable, the terminal may also draw and display the training effect curve of the representation learning model according to the performance evaluation index generated in different iteration rounds of the representation learning model, the training effect curve represents the Indicates how the performance of the learning model changes with the training process, making the process and effect of model training visualized, allowing users to intuitively discover whether the training of the model is effective.
进一步地,终端还可以针对配置有不同超参数的所述表示学习模型,生成不同迭代轮次的所述性能评价指标,绘制并显示所述表示学习模型的对比效果图,该对比效果图用于表现基于不同超参数的所述表示学习模型各自的训练效果曲线,其能够帮助用户确定模型选型与调参的方向,一方面避免了依赖历史经验,主观性强导致容易出错的问题,另一方面使得超参数的自动调节成为可能。Further, the terminal may also generate the performance evaluation indicators of different iteration rounds for the representation learning model configured with different hyperparameters, draw and display a comparison effect diagram of the representation learning model, and the comparison effect diagram is used for Show the respective training effect curves of the representation learning models based on different hyperparameters, which can help users determine the direction of model selection and parameter adjustment. On the one hand, it avoids the problem of relying on historical experience and subjectivity that is prone to error. On the other hand, Aspects enable automatic tuning of hyperparameters.
为了便于理解,本申请还提供了对比效果图的一具体示例。如图4所示,其示出了5组不同超参数组合对应的表示学习模型的训练效果曲线,即41至45,其中,曲线41和曲线42对应的表示学习模型的分合比收敛于较高值,而曲线43、曲线44和曲线45对应的表示学习模型的分合比收敛于较高值,曲线43对应的表示学习模型最先达到稳定状态。For ease of understanding, the present application also provides a specific example of a comparison effect diagram. As shown in Figure 4, it shows the training effect curves of the representation learning models corresponding to 5 groups of different hyperparameter combinations, namely 41 to 45, wherein the split ratio of the representation learning models corresponding to curve 41 and curve 42 converges to a relatively Curve 43, Curve 44, and Curve 45 correspond to a high value, and curve 43, curve 44 and curve 45 indicate that the division ratio of the learning model converges to a higher value, and curve 43 indicates that the learning model reaches a stable state first.
本申请提供的表示学习模型的评估方法可以应用于多种无监督表示学习任务中,如t-分布随机邻域嵌入学习(T-distributed Stochastic Neighbor Embedding,t-SNE)、流形学习、词向量表示学习,并适用于各种用于表示学习的损失函数,包括但不限于噪声对比估计损失函数(Noise-Contrastive Loss,NCE Loss)。The evaluation method of the representation learning model provided by this application can be applied to a variety of unsupervised representation learning tasks, such as t-distributed stochastic neighbor embedding learning (T-distributed Stochastic Neighbor Embedding, t-SNE), manifold learning, word vector Representation learning, and is applicable to various loss functions for representation learning, including but not limited to Noise-Contrastive Loss (NCE Loss).
为了使得本申请的技术方案更加清楚、易于理解,下面结合“用户在一个月内访问过的域名的向量化表示”这一具体场景,对本申请的表示学习模型的评估方法进行介绍。In order to make the technical solution of this application clearer and easier to understand, the evaluation method of the representation learning model of this application will be introduced below in conjunction with the specific scenario of "vectorized representation of domain names visited by users within a month".
参见图5A所示的表示学习模型的评估方法的应用场景图和图5B所示的表示学习的评估方法的流程图,该应用场景中包括终端102,终端102从本地缓存中获取其在一个月内访问记录,从访问记录中提取出域名,然后将该域名作为样本生成训练样本集,基于该训练样本集利用无监督学习方式训练表示学习模型,以实现域名向量化。Referring to the application scenario diagram of the evaluation method of representation learning model shown in FIG. 5A and the flow chart of the evaluation method of representation learning shown in FIG. The domain name is extracted from the access record, and then the domain name is used as a sample to generate a training sample set. Based on the training sample set, an unsupervised learning method is used to train the representation learning model to realize domain name vectorization.
在训练过程中,还通过如下步骤实现对表示学习模型的评估:During the training process, the evaluation of the representation learning model is also realized through the following steps:
步骤一:从训练样本集中人工选出部分域名形成第一子集,对第一子集中的样本进行标注生成第一样本子集。Step 1: Manually select some domain names from the training sample set to form the first subset, and mark the samples in the first subset to generate the first sample subset.
步骤二:从训练样本集中选取部分域名形成第二子集,将该第二子集作为第二样本子集。Step 2: Select part of the domain names from the training sample set to form a second subset, and use the second subset as the second sample subset.
其中,第一子集包括至少两类差别较大的域名,例如可以包括涉黑网站域名与长视频点播网站域名。在实际应用时,每类各取约50个样本,用于后续计算分合比。针对第二子集,其域名数量可以根据实际需求而设置,在本实施例中,第二子集包括9个域名,这9个域名被加入关注列表,以便后续基于关注列表中域名计算表征模型稳定性的第二指标。Wherein, the first subset includes at least two types of domain names with large differences, for example, domain names of black-related websites and long-form video-on-demand websites. In practical application, about 50 samples are taken from each category for subsequent calculation of split ratio. For the second subset, the number of domain names can be set according to actual needs. In this embodiment, the second subset includes 9 domain names, and these 9 domain names are added to the watch list for subsequent calculation of the representation model based on the domain names in the watch list A second indicator of stability.
需要说明的是,步骤一、步骤二可以并行执行,也可以按照设置顺序先后执行,本实施例对此不作限定。It should be noted that step 1 and step 2 may be executed in parallel, or may be executed successively according to the setting order, which is not limited in this embodiment.
步骤三:基于训练样本集训练针对域名的表示学习模型。Step 3: Train the domain name representation learning model based on the training sample set.
在本实施例中,针对域名的表示学习模型可以是word2vec模型,该模型以域名为输入,以域名对应的表示向量为输出。将训练样本集中的域名输入到word2vec模型中,模型能够基于学习算法从中提取相应的特征,将其转换为向量,从而生成针对域名的表示向量。In this embodiment, the representation learning model for the domain name may be a word2vec model, which takes the domain name as input and outputs a representation vector corresponding to the domain name. Input the domain names in the training sample set into the word2vec model, and the model can extract corresponding features based on the learning algorithm and convert them into vectors to generate representation vectors for domain names.
步骤四:在模型训练过程中,基于第一样本子集各样本的标识向量和标签同步计算各迭代轮次对应的分合比。Step 4: During the model training process, synchronously calculate the division ratio corresponding to each iteration round based on the identification vector and label of each sample in the first sample subset.
针对每一迭代轮次,获取该word2vec模型针对第一样本子集各样本学习得到的表示向量,然后基于该表示向量以及各样本的标签可以确定各类样本的类别间距离和类别内距离,计算类别间距离和类别内距离的比值即可获得分合比。如此,可以获得各迭代轮次对应的分合比。For each iteration round, obtain the representation vector learned by the word2vec model for each sample in the first sample subset, and then determine the inter-category distance and intra-category distance of various samples based on the representation vector and the labels of each sample, The split ratio can be obtained by calculating the ratio of the distance between categories and the distance within categories. In this way, the split ratio corresponding to each iteration round can be obtained.
步骤五:在模型训练过程中,将关注列表中域名与全量训练样本(即训练样本集中的所有样本)周期性进行余弦相似度计算,得到关注列表中域名各自对应的相似域名,形成相似样本集,针对上述相似样本集生成第二样本子集中各个域名对应的雅卡尔指数。Step 5: During the model training process, the domain names in the watch list and the full training samples (that is, all samples in the training sample set) are periodically calculated for cosine similarity, and similar domain names corresponding to the domain names in the watch list are obtained to form a similar sample set , generating the Jacquard index corresponding to each domain name in the second sample subset based on the above similar sample set.
在具体实现时,可以根据需求设置选择的相似域名的数量。作为一个示例,本实施例选择关注列表中域名最相近的8个域名形成相似样本集。During specific implementation, the number of selected similar domain names may be set according to requirements. As an example, in this embodiment, 8 domain names that are most similar to domain names in the attention list are selected to form a similar sample set.
其中,步骤四和步骤五可以并行执行,也可以按照设定顺序先后执行。Wherein, Step 4 and Step 5 can be executed in parallel, or can be executed successively according to the set order.
步骤六:判断分合比和稳定是否均满足预设条件,若是则执行步骤七,若否,则返回步骤三。Step 6: Determine whether the split ratio and stability meet the preset conditions, if so, go to step 7, if not, go back to step 3.
步骤七:确定表示学习模型的训练情况。Step 7: Determine the training situation of the representation learning model.
具体地,针对分合比,可以判断其是否收敛,且收敛值是否大于第一参考阈值,若是,则表明模型对同类样本和不同类样本均具有较好的表示效果,并且模型趋于稳定状态。Specifically, for the separation and combination ratio, it can be judged whether it is converged, and whether the convergence value is greater than the first reference threshold, if so, it indicates that the model has a good representation effect on both samples of the same type and samples of different types, and the model tends to a stable state .
针对雅卡尔指数,可以判断当所述第二样本子集中大于雅卡尔指数阈值的样本占比是否超过预设比例,若是,则表征模型的训练情况趋于稳定。在有些情况下,上述预设比例可以设置为100%,即判断第一样本子集中个样本的雅卡尔指数是否均大于雅卡尔指数阈值,若每个样本均满足其中,p为雅卡尔指数阈值,则认为word2vec词向量化表示学习的训练结果趋于稳定。Regarding the Jacquard index, it may be determined whether the proportion of samples in the second sample subset greater than the threshold of the Jacquard index exceeds a preset ratio, and if so, the training situation of the representation model tends to be stable. In some cases, the above-mentioned preset ratio can be set to 100%, that is, to judge whether the Jacquard index of each sample in the first sample subset is greater than the Jacquard index threshold, if each sample satisfies Among them, p is the Jacquard index threshold, and it is considered that the training result of word2vec word vectorization representation learning tends to be stable.
当分合比和雅卡尔指数均满足条件时,则确定word2vec模型趋于稳定状态,当分合比和雅卡尔指数中至少有一个不满足条件时,则表明模型还具有优化空间,可以对模型进行优化调整。When both the split ratio and the Jacquard index meet the conditions, it is determined that the word2vec model tends to a stable state. When at least one of the split ratio and the Jacquard index does not meet the conditions, it indicates that the model still has room for optimization, and the model can be optimized Adjustment.
以上为本申请实施例提供的表示学习模型的评估方法的一些具体实现方式,基于此,本申请实施例还提供了对应的装置,下面从功能模块化的角度对其进行介绍。The above are some specific implementations of the evaluation method of the representation learning model provided by the embodiment of the present application. Based on this, the embodiment of the present application also provides a corresponding device, which will be introduced from the perspective of functional modularization below.
参见图6所示的表示学习模型的评估装置,该装置600包括:Referring to the evaluation device representing a learning model shown in FIG. 6, the device 600 includes:
指标生成模块610,用于针对基于无监督方式进行训练的表示学习模型,生成所述表示学习模型的性能评价指标,所述性能评价指标包括第一指标和第二指标中的至少一个;An index generation module 610, configured to generate a performance evaluation index of the representation learning model for the representation learning model trained in an unsupervised manner, where the performance evaluation index includes at least one of the first index and the second index;
其中,所述第一指标是基于所述表示学习模型在训练过程中学习到的第一样本子集中各样本的表示向量,生成的用于衡量同类样本相近且不同类样本相疏远的量化指标;所述第一样本子集是对所述表示学习模型的训练样本集中的第一子集进行标签标注生成的,所述第一子集包括不同类别的样本;Wherein, the first indicator is a quantitative indicator generated based on the representation vectors of each sample in the first sample subset learned by the representation learning model during the training process, and used to measure the similarity between samples of the same type and the distance between samples of different types; The first sample subset is generated by labeling a first subset of the training sample set of the representation learning model, and the first subset includes samples of different categories;
所述第二指标是基于所述表示学习模型在训练过程中学习到的第二样本子集中各样本的表示向量对应的相似向量,生成的用于衡量样本表示稳定性的量化指标;所述第二样本子集是所述训练样本集中的第二子集;The second index is based on the similarity vectors corresponding to the representation vectors of the samples in the second sample subset learned by the representation learning model during the training process, and is a quantitative index for measuring the stability of sample representation generated; The two-sample subset is a second subset of the training sample set;
评估模块620,用于根据所述性能评价指标,确定所述表示学习模型的训练情况。The evaluation module 620 is configured to determine the training situation of the representation learning model according to the performance evaluation index.
可选的,参见图7,图7为本申请实施例提供的表示学习模型的评估装置的结构示意图,在图6所示结构的基础上,所述指标生成模块610包括:Optionally, refer to FIG. 7, which is a schematic structural diagram of an evaluation device representing a learning model provided in an embodiment of the present application. On the basis of the structure shown in FIG. 6, the index generation module 610 includes:
第一获取子模块611,用于获取所述表示学习模型在训练过程中针对第一样本子集各样本学习得到的表示向量;The first acquisition sub-module 611 is configured to acquire the representation vector learned by the representation learning model for each sample of the first sample subset during the training process;
生成子模块612,用于根据所述第一样本子集各样本的表示向量和标签,确定各类样本的类别间距离和类别内距离,根据所述类别间距离和所述类别内距离的比值生成分合比;The generation submodule 612 is used to determine the inter-category distance and the intra-category distance of each sample according to the representation vector and label of each sample in the first sample subset, and determine the inter-category distance and the intra-category distance according to the Ratio generates split ratio;
第一确定子模块613,用于将所述分合比作为第一指标。The first determining sub-module 613 is configured to use the split ratio as a first index.
可选的,所述性能评价指标包括第一指标;Optionally, the performance evaluation index includes a first index;
所述评估模块620具体用于:The evaluation module 620 is specifically used for:
当预设时间段内基于多个迭代轮次所确定的多个所述分合比呈收敛状态且收敛值大于第一参考阈值时,确定所述表示学习模型的训练情况趋于稳定。When the plurality of dividing and combining ratios determined based on multiple iterative rounds are in a convergent state within a preset period of time and the convergence value is greater than a first reference threshold, it is determined that the training situation of the representation learning model tends to be stable.
可选的,参见图8,图8为本申请实施例提供的表示学习模型的评估装置的结构示意图,在图6所示结构的基础上,所述指标生成模块610包括:Optionally, refer to FIG. 8. FIG. 8 is a schematic structural diagram of an evaluation device representing a learning model provided in an embodiment of the present application. On the basis of the structure shown in FIG. 6, the index generation module 610 includes:
第二获取子模块614,用于获取所述表示学习模型在训练过程中多个迭代轮次针对训练样本集各样本学习得到的表示向量;The second acquisition sub-module 614 is used to acquire the representation vector learned by the representation learning model for each sample of the training sample set during multiple iteration rounds during the training process;
添加子模块615,用于根据每个迭代轮次学习到的所述训练样本集中各样本的表示向量,为所述第二样本子集中的各样本分别选择最相似的预设数量个样本,将为样本所选择的预设数量个相似样本加入与所述第二样本子集中各样本以及迭代轮次对应的相似样本集;Adding a submodule 615, configured to select the most similar preset number of samples for each sample in the second sample subset according to the representation vector of each sample in the training sample set learned in each iteration round, and adding a preset number of similar samples selected for the sample to a similar sample set corresponding to each sample in the second sample subset and the iteration round;
第二确定子模块616,用于针对所述第二样本子集中各样本对应的多个相似样本集,生成所述第二样本子集中各个样本对应的雅卡尔指数,将所述雅卡尔指数作为第二指标。The second determining submodule 616 is configured to generate a Jacquard index corresponding to each sample in the second sample subset for multiple similar sample sets corresponding to each sample in the second sample subset, and use the Jacquard index as Second indicator.
可选的,所述性能评价指标包括第二指标;Optionally, the performance evaluation index includes a second index;
则所述评估模块620具体用于:Then the evaluation module 620 is specifically used for:
当所述第二样本子集中大于雅卡尔指数阈值的样本占比超过预设比例时,确定所述表示学习模型的训练情况趋于稳定。When the proportion of samples greater than the Jacquard index threshold in the second sample subset exceeds a preset ratio, it is determined that the training situation of the representation learning model tends to be stable.
可选的,参见图9,图9为本申请实施例提供的表示学习模型的评估装置的结构示意图,在图6所示结构的基础上,所述性能评价指标包括第一指标和第二指标;Optionally, refer to FIG. 9. FIG. 9 is a schematic structural diagram of an evaluation device representing a learning model provided in an embodiment of the present application. On the basis of the structure shown in FIG. 6, the performance evaluation index includes a first index and a second index ;
所述评估模块620包括:The evaluation module 620 includes:
加权子模块621,用于对所述第一指标和所述第二指标进行加权处理;A weighting submodule 621, configured to perform weighting processing on the first index and the second index;
评估子模块622,用于根据加权处理结果确定所述表示学习模型的训练情况。The evaluation sub-module 622 is configured to determine the training situation of the representation learning model according to the weighted processing result.
需要说明的是,图9也可以是在图7或图8所示基础上,包括上述加权子模块和评估子模块。It should be noted that FIG. 9 may also be based on what is shown in FIG. 7 or 8 , including the above weighting sub-module and evaluation sub-module.
可选的,参见图10,图10为本申请实施例提供的表示学习模型的评估装置的结构示意图,在图6所示结构的基础上,所述性能评价指标包括第一指标和第二指标;Optionally, refer to FIG. 10, which is a schematic structural diagram of an evaluation device representing a learning model provided in an embodiment of the present application. On the basis of the structure shown in FIG. 6, the performance evaluation index includes a first index and a second index ;
所述装置600还包括:The device 600 also includes:
第一显示模块630,用于根据所述表示学习模型不同迭代轮次生成的所述性能评价指标,绘制并显示所述表示学习模型的训练效果曲线,所述训练效果曲线表示所述表示学习模型的性能随着训练进程的变化情况。The first display module 630 is configured to draw and display a training effect curve of the representation learning model according to the performance evaluation indicators generated in different iteration rounds of the representation learning model, the training effect curve representing the representation learning model performance changes with the training process.
当然,图10也可以是在图6至图9的基础上还包括第一显示模块。Certainly, Fig. 10 may also include a first display module on the basis of Fig. 6 to Fig. 9 .
可选的,参见图11,图11为本申请实施例提供的表示学习模型的评估装置的结构示意图,在图6所示结构的基础上,所述性能评价指标包括第一指标和第二指标;Optionally, refer to FIG. 11. FIG. 11 is a schematic structural diagram of an evaluation device representing a learning model provided in an embodiment of the present application. On the basis of the structure shown in FIG. 6, the performance evaluation index includes a first index and a second index ;
所述指标生成模块610具体用于:The indicator generating module 610 is specifically used for:
针对配置有不同超参数的所述表示学习模型,生成不同迭代轮次的所述性能评价指标;For the representation learning model configured with different hyperparameters, generate the performance evaluation indicators of different iteration rounds;
所述装置600还包括:The device 600 also includes:
第二显示模块640,用于绘制并显示所述表示学习模型的对比效果图,所述对比效果图用于表现基于不同超参数的所述表示学习模型各自的训练效果曲线,所述训练效果曲线表示所述表示学习模型的性能随着训练进程的变化情况。The second display module 640 is configured to draw and display a comparison effect graph of the representation learning model, the comparison effect graph is used to represent the respective training effect curves of the representation learning models based on different hyperparameters, and the training effect curves Indicates how the performance of the representation learning model changes with the training process.
其中,图11也可以是在图6至图9的基础上还包括第二显示模块。Wherein, Fig. 11 may also include a second display module on the basis of Fig. 6 to Fig. 9 .
可选的,所述表示学习模型是词向量表示学习模型。Optionally, the representation learning model is a word vector representation learning model.
本申请实施例还提供了一种设备,该设备具体可以是终端,如图12所示,为了便于说明,仅示出了与本申请实施例相关的部分,具体技术细节未揭示的,请参照本申请实施例方法部分。该终端可以为包括手机、平板电脑、个人数字助理(英文全称:Personal DigitalAssistant,英文缩写:PDA)、销售终端(英文全称:Point of Sales,英文缩写:POS)、车载电脑等任意终端设备,以终端为手机为例:The embodiment of the present application also provides a device, which may specifically be a terminal, as shown in Figure 12, for the sake of illustration, only the parts related to the embodiment of the present application are shown, and the specific technical details are not disclosed, please refer to The method part of the embodiment of the present application. The terminal can be any terminal device including mobile phone, tablet computer, personal digital assistant (English full name: Personal Digital Assistant, English abbreviation: PDA), sales terminal (English full name: Point of Sales, English abbreviation: POS), vehicle-mounted computer, etc. The terminal is a mobile phone as an example:
图12示出的是与本申请实施例提供的终端相关的手机的部分结构的框图。参考图12,手机包括:射频(英文全称:Radio Frequency,英文缩写:RF)电路1210、存储器1220、输入单元1230、显示单元1240、传感器1250、音频电路1260、无线保真(英文全称:wirelessfidelity,英文缩写:WiFi)模块1270、处理器1280、以及电源1290等部件。本领域技术人员可以理解,图12中示出的手机结构并不构成对手机的限定,可以包括比图示更多或更少的部件,或者组合某些部件,或者不同的部件布置。FIG. 12 shows a block diagram of a partial structure of a mobile phone related to the terminal provided by the embodiment of the present application. Referring to Fig. 12, the mobile phone includes: radio frequency (English full name: Radio Frequency, English abbreviation: RF) circuit 1210, memory 1220, input unit 1230, display unit 1240, sensor 1250, audio circuit 1260, wireless fidelity (English full name: wirelessfidelity, English abbreviation: WiFi module 1270 , processor 1280 , power supply 1290 and other components. Those skilled in the art can understand that the structure of the mobile phone shown in FIG. 12 does not constitute a limitation to the mobile phone, and may include more or less components than shown in the figure, or combine some components, or arrange different components.
下面结合图12对手机的各个构成部件进行具体的介绍:The following is a specific introduction to each component of the mobile phone in conjunction with Figure 12:
RF电路1210可用于收发信息或通话过程中,信号的接收和发送,特别地,将基站的下行信息接收后,给处理器1280处理;另外,将设计上行的数据发送给基站。通常,RF电路1210包括但不限于天线、至少一个放大器、收发信机、耦合器、低噪声放大器(英文全称:LowNoise Amplifier,英文缩写:LNA)、双工器等。此外,RF电路1210还可以通过无线通信与网络和其他设备通信。上述无线通信可以使用任一通信标准或协议,包括但不限于全球移动通讯系统(英文全称:Global System of Mobile communication,英文缩写:GSM)、通用分组无线服务(英文全称:General Packet Radio Service,GPRS)、码分多址(英文全称:CodeDivision Multiple Access,英文缩写:CDMA)、宽带码分多址(英文全称:Wideband CodeDivision Multiple Access,英文缩写:WCDMA)、长期演进(英文全称:Long TermEvolution,英文缩写:LTE)、电子邮件、短消息服务(英文全称:Short Messaging Service,SMS)等。The RF circuit 1210 can be used for sending and receiving information or receiving and sending signals during a call. In particular, after receiving the downlink information from the base station, it is processed by the processor 1280; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit 1210 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (full English name: Low Noise Amplifier, English abbreviation: LNA), a duplexer, and the like. In addition, RF circuitry 1210 may also communicate with networks and other devices via wireless communications. The above-mentioned wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile Communication (English full name: Global System of Mobile communication, English abbreviation: GSM), General Packet Radio Service (English full name: General Packet Radio Service, GPRS ), Code Division Multiple Access (English full name: CodeDivision Multiple Access, English abbreviation: CDMA), Wideband Code Division Multiple Access (English full name: Wideband CodeDivision Multiple Access, English abbreviation: WCDMA), Long Term Evolution (English full name: Long TermEvolution, English Abbreviation: LTE), email, short message service (English full name: Short Messaging Service, SMS), etc.
存储器1220可用于存储软件程序以及模块,处理器1280通过运行存储在存储器1220的软件程序以及模块,从而执行手机的各种功能应用以及数据处理。存储器1220可主要包括存储程序区和存储数据区,其中,存储程序区可存储操作系统、至少一个功能所需的应用程序(比如声音播放功能、图像播放功能等)等;存储数据区可存储根据手机的使用所创建的数据(比如音频数据、电话本等)等。此外,存储器1220可以包括高速随机存取存储器,还可以包括非易失性存储器,例如至少一个磁盘存储器件、闪存器件、或其他易失性固态存储器件。The memory 1220 can be used to store software programs and modules, and the processor 1280 executes various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 1220 . The memory 1220 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required by a function (such as a sound playback function, an image playback function, etc.) and the like; Data created by the use of mobile phones (such as audio data, phonebook, etc.), etc. In addition, the memory 1220 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage devices.
输入单元1230可用于接收输入的数字或字符信息,以及产生与手机的用户设置以及功能控制有关的键信号输入。具体地,输入单元1230可包括触控面板1231以及其他输入设备1232。触控面板1231,也称为触摸屏,可收集用户在其上或附近的触摸操作(比如用户使用手指、触笔等任何适合的物体或附件在触控面板1231上或在触控面板1231附近的操作),并根据预先设定的程式驱动相应的连接装置。可选的,触控面板1231可包括触摸检测装置和触摸控制器两个部分。其中,触摸检测装置检测用户的触摸方位,并检测触摸操作带来的信号,将信号传送给触摸控制器;触摸控制器从触摸检测装置上接收触摸信息,并将它转换成触点坐标,再送给处理器1280,并能接收处理器1280发来的命令并加以执行。此外,可以采用电阻式、电容式、红外线以及表面声波等多种类型实现触控面板1231。除了触控面板1231,输入单元1230还可以包括其他输入设备1232。具体地,其他输入设备1232可以包括但不限于物理键盘、功能键(比如音量控制按键、开关按键等)、轨迹球、鼠标、操作杆等中的一种或多种。The input unit 1230 can be used to receive input numbers or character information, and generate key signal input related to user settings and function control of the mobile phone. Specifically, the input unit 1230 may include a touch panel 1231 and other input devices 1232 . The touch panel 1231, also referred to as a touch screen, can collect touch operations of the user on or near it (for example, the user uses any suitable object or accessory such as a finger or a stylus on the touch panel 1231 or near the touch panel 1231). operation), and drive the corresponding connection device according to the preset program. Optionally, the touch panel 1231 may include two parts, a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, and detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and sends it to the to the processor 1280, and can receive and execute commands sent by the processor 1280. In addition, the touch panel 1231 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 1231 , the input unit 1230 may also include other input devices 1232 . Specifically, other input devices 1232 may include but not limited to one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), trackball, mouse, joystick, and the like.
显示单元1240可用于显示由用户输入的信息或提供给用户的信息以及手机的各种菜单。显示单元1240可包括显示面板1241,可选的,可以采用液晶显示器(英文全称:Liquid Crystal Display,英文缩写:LCD)、有机发光二极管(英文全称:Organic Light-Emitting Diode,英文缩写:OLED)等形式来配置显示面板1241。进一步的,触控面板1231可覆盖显示面板1241,当触控面板1231检测到在其上或附近的触摸操作后,传送给处理器1280以确定触摸事件的类型,随后处理器1280根据触摸事件的类型在显示面板1241上提供相应的视觉输出。虽然在图12中,触控面板1231与显示面板1241是作为两个独立的部件来实现手机的输入和输入功能,但是在某些实施例中,可以将触控面板1231与显示面板1241集成而实现手机的输入和输出功能。The display unit 1240 may be used to display information input by or provided to the user and various menus of the mobile phone. The display unit 1240 may include a display panel 1241. Optionally, a liquid crystal display (English full name: Liquid Crystal Display, English abbreviation: LCD), an organic light-emitting diode (English full name: Organic Light-Emitting Diode, English abbreviation: OLED) etc. may be used. form to configure the display panel 1241 . Further, the touch panel 1231 can cover the display panel 1241, and when the touch panel 1231 detects a touch operation on or near it, it sends it to the processor 1280 to determine the type of the touch event, and then the processor 1280 determines the type of the touch event according to the The type provides a corresponding visual output on the display panel 1241 . Although in FIG. 12 , the touch panel 1231 and the display panel 1241 are used as two independent components to realize the input and input functions of the mobile phone, in some embodiments, the touch panel 1231 and the display panel 1241 can be integrated to form a mobile phone. Realize the input and output functions of the mobile phone.
手机还可包括至少一种传感器1250,比如光传感器、运动传感器以及其他传感器。具体地,光传感器可包括环境光传感器及接近传感器,其中,环境光传感器可根据环境光线的明暗来调节显示面板1241的亮度,接近传感器可在手机移动到耳边时,关闭显示面板1241和/或背光。作为运动传感器的一种,加速计传感器可检测各个方向上(一般为三轴)加速度的大小,静止时可检测出重力的大小及方向,可用于识别手机姿态的应用(比如横竖屏切换、相关游戏、磁力计姿态校准)、振动识别相关功能(比如计步器、敲击)等;至于手机还可配置的陀螺仪、气压计、湿度计、温度计、红外线传感器等其他传感器,在此不再赘述。The handset may also include at least one sensor 1250, such as a light sensor, motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 1241 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 1241 and/or when the mobile phone is moved to the ear. or backlight. As a kind of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes), and can detect the magnitude and direction of gravity when it is stationary, and can be used to identify the application of mobile phone posture (such as horizontal and vertical screen switching, related Games, magnetometer attitude calibration), vibration recognition related functions (such as pedometer, tap), etc.; as for other sensors such as gyroscope, barometer, hygrometer, thermometer, infrared sensor, etc. repeat.
音频电路1260、扬声器1261,传声器1262可提供用户与手机之间的音频接口。音频电路1260可将接收到的音频数据转换后的电信号,传输到扬声器1261,由扬声器1261转换为声音信号输出;另一方面,传声器1262将收集的声音信号转换为电信号,由音频电路1260接收后转换为音频数据,再将音频数据输出处理器1280处理后,经RF电路1210以发送给比如另一手机,或者将音频数据输出至存储器1220以便进一步处理。The audio circuit 1260, the speaker 1261, and the microphone 1262 can provide an audio interface between the user and the mobile phone. The audio circuit 1260 can transmit the electrical signal converted from the received audio data to the speaker 1261, and the speaker 1261 converts it into an audio signal for output; After being received, it is converted into audio data, and then the audio data is processed by the output processor 1280, and then sent to another mobile phone through the RF circuit 1210, or the audio data is output to the memory 1220 for further processing.
WiFi属于短距离无线传输技术,手机通过WiFi模块1270可以帮助用户收发电子邮件、浏览网页和访问流式媒体等,它为用户提供了无线的宽带互联网访问。虽然图12示出了WiFi模块1270,但是可以理解的是,其并不属于手机的必须构成,完全可以根据需要在不改变发明的本质的范围内而省略。WiFi is a short-distance wireless transmission technology. The mobile phone can help users send and receive emails, browse web pages, and access streaming media through the WiFi module 1270. It provides users with wireless broadband Internet access. Although FIG. 12 shows a WiFi module 1270, it can be understood that it is not an essential component of the mobile phone, and can be completely omitted as required without changing the essence of the invention.
处理器1280是手机的控制中心,利用各种接口和线路连接整个手机的各个部分,通过运行或执行存储在存储器1220内的软件程序和/或模块,以及调用存储在存储器1220内的数据,执行手机的各种功能和处理数据,从而对手机进行整体监控。可选的,处理器1280可包括一个或多个处理单元;优选的,处理器1280可集成应用处理器和调制解调处理器,其中,应用处理器主要处理操作系统、用户界面和应用程序等,调制解调处理器主要处理无线通信。可以理解的是,上述调制解调处理器也可以不集成到处理器1280中。The processor 1280 is the control center of the mobile phone. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and/or modules stored in the memory 1220, and calling data stored in the memory 1220, execution Various functions and processing data of the mobile phone, so as to monitor the mobile phone as a whole. Optionally, the processor 1280 may include one or more processing units; preferably, the processor 1280 may integrate an application processor and a modem processor, wherein the application processor mainly processes operating systems, user interfaces, and application programs, etc. , the modem processor mainly handles wireless communications. It can be understood that the foregoing modem processor may not be integrated into the processor 1280 .
手机还包括给各个部件供电的电源1290(比如电池),优选的,电源可以通过电源管理系统与处理器1280逻辑相连,从而通过电源管理系统实现管理充电、放电、以及功耗管理等功能。The mobile phone also includes a power supply 1290 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 1280 through the power management system, so that functions such as charging, discharging, and power consumption management can be realized through the power management system.
尽管未示出,手机还可以包括摄像头、蓝牙模块等,在此不再赘述。Although not shown, the mobile phone may also include a camera, a Bluetooth module, etc., which will not be repeated here.
在本申请实施例中,该终端所包括的处理器1280还具有以下功能:In this embodiment of the application, the processor 1280 included in the terminal also has the following functions:
针对基于无监督方式进行训练的表示学习模型,生成所述表示学习模型的性能评价指标,所述性能评价指标包括第一指标和第二指标中的至少一个;For a representation learning model trained in an unsupervised manner, generate a performance evaluation index of the representation learning model, where the performance evaluation index includes at least one of a first index and a second index;
其中,所述第一指标是基于所述表示学习模型在训练过程中学习到的第一样本子集中各样本的表示向量,生成的用于衡量同类样本相近且不同类样本相疏远的量化指标;所述第一样本子集是对所述表示学习模型的训练样本集中的第一子集进行标签标注生成的,所述第一子集包括不同类别的样本;Wherein, the first indicator is a quantitative indicator generated based on the representation vectors of each sample in the first sample subset learned by the representation learning model during the training process, and used to measure the similarity between samples of the same type and the distance between samples of different types; The first sample subset is generated by labeling a first subset of the training sample set of the representation learning model, and the first subset includes samples of different categories;
所述第二指标是基于所述表示学习模型在训练过程中学习到的第二样本子集中各样本的表示向量对应的相似向量,生成的用于衡量样本表示稳定性的量化指标;所述第二样本子集是所述训练样本集中的第二子集;The second index is based on the similarity vectors corresponding to the representation vectors of the samples in the second sample subset learned by the representation learning model during the training process, and is a quantitative index for measuring the stability of sample representation generated; The two-sample subset is a second subset of the training sample set;
根据所述性能评价指标,确定所述表示学习模型的训练情况。According to the performance evaluation index, the training situation of the representation learning model is determined.
可选的,处理器1280还用于执行本申请实施例提供的表示学习模型的评估方法的任意一种实现方式的步骤。Optionally, the processor 1280 is further configured to execute steps in any implementation manner of the method for evaluating a representation learning model provided in the embodiments of the present application.
本申请实施例还提供一种计算机可读存储介质,用于存储计算机程序,该计算机程序用于执行前述各个实施例所述的一种表示学习模型的评估方法中的任意一种实施方式。An embodiment of the present application further provides a computer-readable storage medium for storing a computer program, and the computer program is used to implement any one of the methods for evaluating a representation learning model described in the foregoing embodiments.
本申请实施例还提供一种包括指令的计算机程序产品,当其在计算机上运行时,使得计算机执行前述各个实施例所述的一种表示学习模型的评估方法中的任意一种实施方式。An embodiment of the present application further provides a computer program product including instructions, which, when run on a computer, cause the computer to execute any one of the methods for evaluating a representation learning model described in the foregoing embodiments.
所属领域的技术人员可以清楚地了解到,为描述的方便和简洁,上述描述的系统,装置和单元的具体工作过程,可以参考前述方法实施例中的对应过程,在此不再赘述。Those skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.
在本申请所提供的几个实施例中,应该理解到,所揭露的系统,装置和方法,可以通过其它的方式实现。例如,以上所描述的装置实施例仅仅是示意性的,例如,所述单元的划分,仅仅为一种逻辑功能划分,实际实现时可以有另外的划分方式,例如多个单元或组件可以结合或者可以集成到另一个系统,或一些特征可以忽略,或不执行。另一点,所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口,装置或单元的间接耦合或通信连接,可以是电性,机械或其它的形式。In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or May be integrated into another system, or some features may be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的,作为单元显示的部件可以是或者也可以不是物理单元,即可以位于一个地方,或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.
另外,在本申请各个实施例中的各功能单元可以集成在一个处理单元中,也可以是各个单元单独物理存在,也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现,也可以采用软件功能单元的形式实现。In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.
所述集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时,可以存储在一个计算机可读取存储介质中。基于这样的理解,本申请的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来,该计算机软件产品存储在一个存储介质中,包括若干指令用以使得一台计算机设备(可以是个人计算机,服务器,或者网络设备等)执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括:U盘、移动硬盘、只读存储器(英文全称:Read-OnlyMemory,英文缩写:ROM)、随机存取存储器(英文全称:Random Access Memory,英文缩写:RAM)、磁碟或者光盘等各种可以存储程序代码的介质。If the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , including several instructions to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (English full name: Read-OnlyMemory, English abbreviation: ROM), random access memory (English full name: Random Access Memory, English abbreviation: RAM), disk Or various media such as CDs that can store program codes.
以上所述,以上实施例仅用以说明本申请的技术方案,而非对其限制;尽管参照前述实施例对本申请进行了详细的说明,本领域的普通技术人员应当理解:其依然可以对前述各实施例所记载的技术方案进行修改,或者对其中部分技术特征进行等同替换;而这些修改或者替换,并不使相应技术方案的本质脱离本申请各实施例技术方案的精神和范围。As mentioned above, the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: it can still understand the foregoing The technical solutions described in each embodiment are modified, or some of the technical features are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the application.
Claims (15)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201910549544.6A CN110263939B (en) | 2019-06-24 | 2019-06-24 | Assessment method, device, equipment and medium for representing learning model |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201910549544.6A CN110263939B (en) | 2019-06-24 | 2019-06-24 | Assessment method, device, equipment and medium for representing learning model |
Publications (2)
Publication Number | Publication Date |
---|---|
CN110263939A true CN110263939A (en) | 2019-09-20 |
CN110263939B CN110263939B (en) | 2024-11-29 |
Family
ID=67920900
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201910549544.6A Active CN110263939B (en) | 2019-06-24 | 2019-06-24 | Assessment method, device, equipment and medium for representing learning model |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN110263939B (en) |
Cited By (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111047049A (en) * | 2019-12-05 | 2020-04-21 | 北京小米移动软件有限公司 | Method, apparatus and medium for processing multimedia data based on machine learning model |
CN111459820A (en) * | 2020-03-31 | 2020-07-28 | 北京九章云极科技有限公司 | Model application method and device and data analysis processing system |
CN111679829A (en) * | 2020-06-11 | 2020-09-18 | 北京百度网讯科技有限公司 | Method and device for determining user interface design |
CN111797993A (en) * | 2020-06-16 | 2020-10-20 | 东软睿驰汽车技术(沈阳)有限公司 | Evaluation method and device for deep learning model, electronic equipment and storage medium |
CN112256537A (en) * | 2020-11-12 | 2021-01-22 | 腾讯科技(深圳)有限公司 | Model running state display method and device, computer equipment and storage medium |
CN112529167A (en) * | 2020-12-25 | 2021-03-19 | 东云睿连(武汉)计算技术有限公司 | Interactive automatic training system and method for neural network |
CN114880305A (en) * | 2022-05-25 | 2022-08-09 | 北京明易达科技股份有限公司 | AI algorithm modeling method and system based on data middlebox |
CN115358392A (en) * | 2022-10-21 | 2022-11-18 | 北京百度网讯科技有限公司 | Deep learning network training method, text detection method and text detection device |
CN115545570A (en) * | 2022-11-28 | 2022-12-30 | 四川大学华西医院 | Method and system for checking and accepting achievements of nursing education training |
WO2024131395A1 (en) * | 2022-12-20 | 2024-06-27 | 中国电信股份有限公司 | Service performance measurement method and apparatus, and device, storage medium and program product |
WO2024255039A1 (en) * | 2023-06-13 | 2024-12-19 | Huawei Technologies Co., Ltd. | Communication method and communication apparatus |
Citations (20)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN107729908A (en) * | 2016-08-10 | 2018-02-23 | 阿里巴巴集团控股有限公司 | A kind of method for building up, the apparatus and system of machine learning classification model |
CN107908928A (en) * | 2017-12-21 | 2018-04-13 | 天津科技大学 | A kind of hemoglobin Dynamic Spectrum Analysis Forecasting Methodology based on depth learning technology |
CN107945175A (en) * | 2017-12-12 | 2018-04-20 | 百度在线网络技术(北京)有限公司 | Evaluation method, device, server and the storage medium of image |
CN108021931A (en) * | 2017-11-20 | 2018-05-11 | 阿里巴巴集团控股有限公司 | A kind of data sample label processing method and device |
CN108229592A (en) * | 2018-03-27 | 2018-06-29 | 四川大学 | Outlier detection method and device based on GMDH neuroids |
CN109086791A (en) * | 2018-06-25 | 2018-12-25 | 阿里巴巴集团控股有限公司 | A kind of training method, device and the computer equipment of two classifiers |
CN109215368A (en) * | 2018-08-23 | 2019-01-15 | 百度在线网络技术(北京)有限公司 | A kind of method, apparatus, equipment and computer storage medium that auxiliary drives |
CN109299161A (en) * | 2018-10-31 | 2019-02-01 | 阿里巴巴集团控股有限公司 | A kind of data selecting method and device |
CN109344201A (en) * | 2018-10-17 | 2019-02-15 | 国网江苏省电力有限公司信息通信分公司 | A system and method for evaluating database performance load based on machine learning |
CN109409528A (en) * | 2018-09-10 | 2019-03-01 | 平安科技(深圳)有限公司 | Model generating method, device, computer equipment and storage medium |
CN109446520A (en) * | 2018-10-17 | 2019-03-08 | 北京神州泰岳软件股份有限公司 | For constructing the data clustering method and device of knowledge base |
CN109492772A (en) * | 2018-11-28 | 2019-03-19 | 北京百度网讯科技有限公司 | The method and apparatus for generating information |
CN109670940A (en) * | 2018-11-12 | 2019-04-23 | 深圳壹账通智能科技有限公司 | Credit Risk Assessment Model generation method and relevant device based on machine learning |
CN109684478A (en) * | 2018-12-18 | 2019-04-26 | 腾讯科技(深圳)有限公司 | Disaggregated model training method, classification method and device, equipment and medium |
CN109740738A (en) * | 2018-12-29 | 2019-05-10 | 腾讯科技(深圳)有限公司 | A kind of neural network model training method, device, equipment and medium |
CN109784578A (en) * | 2019-01-24 | 2019-05-21 | 中国科学院软件研究所 | An online learning stagnation prediction system combined with business rules |
CN109783617A (en) * | 2018-12-11 | 2019-05-21 | 平安科技(深圳)有限公司 | For replying model training method, device, equipment and the storage medium of problem |
CN109840904A (en) * | 2019-01-24 | 2019-06-04 | 西南交通大学 | A kind of high iron catenary large scale difference parts testing method |
CN109918684A (en) * | 2019-03-05 | 2019-06-21 | 腾讯科技(深圳)有限公司 | Model training method, interpretation method, relevant apparatus, equipment and storage medium |
CN111860850A (en) * | 2019-04-28 | 2020-10-30 | 第四范式(北京)技术有限公司 | Method for training model, information processing method, device and electronic device |
-
2019
- 2019-06-24 CN CN201910549544.6A patent/CN110263939B/en active Active
Patent Citations (20)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN107729908A (en) * | 2016-08-10 | 2018-02-23 | 阿里巴巴集团控股有限公司 | A kind of method for building up, the apparatus and system of machine learning classification model |
CN108021931A (en) * | 2017-11-20 | 2018-05-11 | 阿里巴巴集团控股有限公司 | A kind of data sample label processing method and device |
CN107945175A (en) * | 2017-12-12 | 2018-04-20 | 百度在线网络技术(北京)有限公司 | Evaluation method, device, server and the storage medium of image |
CN107908928A (en) * | 2017-12-21 | 2018-04-13 | 天津科技大学 | A kind of hemoglobin Dynamic Spectrum Analysis Forecasting Methodology based on depth learning technology |
CN108229592A (en) * | 2018-03-27 | 2018-06-29 | 四川大学 | Outlier detection method and device based on GMDH neuroids |
CN109086791A (en) * | 2018-06-25 | 2018-12-25 | 阿里巴巴集团控股有限公司 | A kind of training method, device and the computer equipment of two classifiers |
CN109215368A (en) * | 2018-08-23 | 2019-01-15 | 百度在线网络技术(北京)有限公司 | A kind of method, apparatus, equipment and computer storage medium that auxiliary drives |
CN109409528A (en) * | 2018-09-10 | 2019-03-01 | 平安科技(深圳)有限公司 | Model generating method, device, computer equipment and storage medium |
CN109446520A (en) * | 2018-10-17 | 2019-03-08 | 北京神州泰岳软件股份有限公司 | For constructing the data clustering method and device of knowledge base |
CN109344201A (en) * | 2018-10-17 | 2019-02-15 | 国网江苏省电力有限公司信息通信分公司 | A system and method for evaluating database performance load based on machine learning |
CN109299161A (en) * | 2018-10-31 | 2019-02-01 | 阿里巴巴集团控股有限公司 | A kind of data selecting method and device |
CN109670940A (en) * | 2018-11-12 | 2019-04-23 | 深圳壹账通智能科技有限公司 | Credit Risk Assessment Model generation method and relevant device based on machine learning |
CN109492772A (en) * | 2018-11-28 | 2019-03-19 | 北京百度网讯科技有限公司 | The method and apparatus for generating information |
CN109783617A (en) * | 2018-12-11 | 2019-05-21 | 平安科技(深圳)有限公司 | For replying model training method, device, equipment and the storage medium of problem |
CN109684478A (en) * | 2018-12-18 | 2019-04-26 | 腾讯科技(深圳)有限公司 | Disaggregated model training method, classification method and device, equipment and medium |
CN109740738A (en) * | 2018-12-29 | 2019-05-10 | 腾讯科技(深圳)有限公司 | A kind of neural network model training method, device, equipment and medium |
CN109784578A (en) * | 2019-01-24 | 2019-05-21 | 中国科学院软件研究所 | An online learning stagnation prediction system combined with business rules |
CN109840904A (en) * | 2019-01-24 | 2019-06-04 | 西南交通大学 | A kind of high iron catenary large scale difference parts testing method |
CN109918684A (en) * | 2019-03-05 | 2019-06-21 | 腾讯科技(深圳)有限公司 | Model training method, interpretation method, relevant apparatus, equipment and storage medium |
CN111860850A (en) * | 2019-04-28 | 2020-10-30 | 第四范式(北京)技术有限公司 | Method for training model, information processing method, device and electronic device |
Cited By (21)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111047049A (en) * | 2019-12-05 | 2020-04-21 | 北京小米移动软件有限公司 | Method, apparatus and medium for processing multimedia data based on machine learning model |
CN111047049B (en) * | 2019-12-05 | 2023-08-11 | 北京小米移动软件有限公司 | Method, device and medium for processing multimedia data based on machine learning model |
CN111459820A (en) * | 2020-03-31 | 2020-07-28 | 北京九章云极科技有限公司 | Model application method and device and data analysis processing system |
CN111459820B (en) * | 2020-03-31 | 2021-01-05 | 北京九章云极科技有限公司 | Model application method and device and data analysis processing system |
CN111679829B (en) * | 2020-06-11 | 2023-03-21 | 北京百度网讯科技有限公司 | Method and device for determining user interface design |
CN111679829A (en) * | 2020-06-11 | 2020-09-18 | 北京百度网讯科技有限公司 | Method and device for determining user interface design |
CN111797993A (en) * | 2020-06-16 | 2020-10-20 | 东软睿驰汽车技术(沈阳)有限公司 | Evaluation method and device for deep learning model, electronic equipment and storage medium |
CN111797993B (en) * | 2020-06-16 | 2024-02-27 | 东软睿驰汽车技术(沈阳)有限公司 | Evaluation method and device of deep learning model, electronic equipment and storage medium |
CN112256537A (en) * | 2020-11-12 | 2021-01-22 | 腾讯科技(深圳)有限公司 | Model running state display method and device, computer equipment and storage medium |
CN112256537B (en) * | 2020-11-12 | 2024-03-29 | 腾讯科技(深圳)有限公司 | Model running state display method and device, computer equipment and storage medium |
WO2022134600A1 (en) * | 2020-12-25 | 2022-06-30 | 东云睿连(武汉)计算技术有限公司 | Interactive automatic training system and method for neural network |
CN112529167A (en) * | 2020-12-25 | 2021-03-19 | 东云睿连(武汉)计算技术有限公司 | Interactive automatic training system and method for neural network |
CN112529167B (en) * | 2020-12-25 | 2024-05-14 | 东云睿连(武汉)计算技术有限公司 | Neural network interactive automatic training system and method |
CN114880305A (en) * | 2022-05-25 | 2022-08-09 | 北京明易达科技股份有限公司 | AI algorithm modeling method and system based on data middlebox |
CN114880305B (en) * | 2022-05-25 | 2023-06-20 | 北京明易达科技股份有限公司 | AI algorithm modeling method and system based on data center |
CN115358392A (en) * | 2022-10-21 | 2022-11-18 | 北京百度网讯科技有限公司 | Deep learning network training method, text detection method and text detection device |
CN115358392B (en) * | 2022-10-21 | 2023-05-05 | 北京百度网讯科技有限公司 | Training method of deep learning network, text detection method and device |
CN115545570A (en) * | 2022-11-28 | 2022-12-30 | 四川大学华西医院 | Method and system for checking and accepting achievements of nursing education training |
CN115545570B (en) * | 2022-11-28 | 2023-03-24 | 四川大学华西医院 | Achievement acceptance method and system for nursing education training |
WO2024131395A1 (en) * | 2022-12-20 | 2024-06-27 | 中国电信股份有限公司 | Service performance measurement method and apparatus, and device, storage medium and program product |
WO2024255039A1 (en) * | 2023-06-13 | 2024-12-19 | Huawei Technologies Co., Ltd. | Communication method and communication apparatus |
Also Published As
Publication number | Publication date |
---|---|
CN110263939B (en) | 2024-11-29 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN110263939A (en) | A kind of appraisal procedure, device, equipment and medium indicating learning model | |
CN110176226B (en) | Speech recognition and speech recognition model training method and device | |
CN109241431B (en) | Resource recommendation method and device | |
CN110069715B (en) | Information recommendation model training method, information recommendation method and device | |
CN109918684B (en) | Model training method, translation method, related device, equipment and storage medium | |
CN112820299B (en) | Voiceprint recognition model training method and device and related equipment | |
CN111125523B (en) | Searching method, searching device, terminal equipment and storage medium | |
CN108984731A (en) | Sing single recommended method, device and storage medium | |
CN108319657B (en) | Method for detecting strong rhythm point, storage medium and terminal | |
CN110245293B (en) | Network content recall method and device | |
CN113284142B (en) | Image detection method, image detection device, computer-readable storage medium and computer equipment | |
CN110209810B (en) | Similar text recognition method and device | |
CN110995810B (en) | Object identification method based on artificial intelligence and related device | |
CN111027854A (en) | Comprehensive portrait index generation method based on enterprise big data and related equipment | |
CN111090877B (en) | Data generation and acquisition methods, corresponding devices and storage medium | |
CN111913848A (en) | Data monitoring and analyzing method and related equipment | |
CN110597957B (en) | Text information retrieval method and related device | |
CN110929882A (en) | A kind of artificial intelligence-based feature vector calculation method and related device | |
CN117153186A (en) | Sound signal processing method, device, electronic equipment and storage medium | |
CN109544241A (en) | A kind of construction method of clicking rate prediction model, clicking rate predictor method and device | |
CN112948763B (en) | Piece quantity prediction method and device, electronic equipment and storage medium | |
CN110390102B (en) | Emotion analysis method and related device | |
CN111104823B (en) | Face recognition method and device, storage medium and terminal equipment | |
CN117332844A (en) | Challenge sample generation method, related device and storage medium | |
CN113918822B (en) | Content search method, device, electronic device and storage medium |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |