CN112200273B

CN112200273B - Data annotation method, device, equipment and computer storage medium

Info

Publication number: CN112200273B
Application number: CN202011413612.5A
Authority: CN
Inventors: 闾凡兵; 曾海文
Original assignee: Changsha Hisense Intelligent System Research Institute Co ltd
Current assignee: Changsha Hisense Intelligent System Research Institute Co ltd
Priority date: 2020-12-07
Filing date: 2020-12-07
Publication date: 2021-05-07
Anticipated expiration: 2040-12-07
Also published as: CN112200273A

Abstract

The application discloses a data annotation method, a device, equipment and a computer storage medium; the data labeling method comprises the following steps: inputting a sample data set to be labeled into a teacher model for labeling to obtain a first labeled sample set and the reliability of each labeled sample in the first labeled sample set; in a training period, determining a reliability threshold, determining a second labeled sample set from the first labeled sample set, training the student model by using the second labeled sample set to obtain a trained student model, and acquiring a target evaluation index of the trained student model; and determining a target labeling sample set from N second labeling sample sets determined in N training periods according to N target evaluation indexes obtained in N training periods, wherein N is an integer larger than 1. According to the embodiment of the application, the quality of the target labeling sample set can be verified, and the quality of the target labeling sample set is effectively guaranteed.

Description

Data annotation method, device, equipment and computer storage medium

Technical Field

The present application belongs to the technical field of data processing, and in particular, to a data annotation method, apparatus, device, and computer storage medium.

Background

Generally speaking, labeling sample data belongs to a common link before deep learning models are trained. At present, in some application occasions, in order to increase the efficiency of labeling sample data, an automatic labeling device may be adopted to realize automatic labeling of the sample data.

However, when the sample data is automatically labeled in the prior art, the quality of the labeled sample is difficult to ensure.

Disclosure of Invention

Embodiments of the present application provide a data labeling method, apparatus, device, and computer storage medium, which have solved the problem in the prior art that it is difficult to ensure the quality of an obtained labeled sample when automatically labeling sample data.

In one aspect, an embodiment of the present application provides a data annotation method, where the method includes:

inputting a sample data set to be labeled into a teacher model for labeling to obtain a first labeled sample set and the reliability of each labeled sample in the first labeled sample set;

in a training period, determining a reliability threshold, determining a second labeled sample set from the first labeled sample set, training the student model by using the second labeled sample set to obtain a trained student model, and acquiring a target evaluation index of the trained student model; the reliability of each labeled sample in the second labeled sample set is greater than or equal to a reliability threshold value;

and determining a target labeling sample set from N second labeling sample sets determined in N training periods according to N target evaluation indexes obtained in N training periods, wherein N is an integer larger than 1.

On the other hand, an embodiment of the present application provides a data annotation device, including:

the marking module is used for inputting the sample data set to be marked into the teacher model for marking to obtain a first marked sample set and the reliability of each marked sample in the first marked sample set;

the training module is used for determining a reliability threshold value in a training period, determining a second labeling sample set from the first labeling sample set, training the student model by using the second labeling sample set to obtain a trained student model, and acquiring a target evaluation index of the trained student model; the reliability of each labeled sample in the second labeled sample set is greater than or equal to a reliability threshold value;

and the determining module is used for determining a target labeling sample set from N second labeling sample sets determined in N training periods according to N target evaluation indexes obtained in N training periods, wherein N is an integer larger than 1.

In another aspect, an embodiment of the present application provides an electronic device, where the electronic device includes: a processor and a memory storing computer program instructions;

the processor, when executing the computer program instructions, implements the data annotation methods described above.

In another aspect, an embodiment of the present application provides a computer storage medium, where computer program instructions are stored on the computer storage medium, and the computer program instructions, when executed by a processor, implement the data annotation method described above.

According to the data labeling method, the data labeling device, the data labeling equipment and the computer storage medium, a teacher model is used for labeling the sample data sets to be labeled to obtain a first labeled sample set and the reliability of each labeled sample in the first labeled sample set; in a plurality of training periods, respectively determining a reliability threshold value, determining a second labeling sample set from the first labeling sample set, training the student model by using the second labeling sample set, and acquiring a target evaluation index of the trained student model; and determining a target labeling sample set from the second labeling sample set determined in each training period based on the target evaluation index obtained in each training period. In the embodiment of the application, a teacher model is used for labeling a sample data set to be labeled to obtain a first labeled sample set, the first labeled sample set is screened by using a determined credibility threshold value, and a target labeled sample set is determined from second labeled sample sets obtained through multiple screening based on the performances of the second labeled sample sets in student model training respectively, so that the quality of the target labeled sample set can be verified, and the quality of the target labeled sample set is effectively guaranteed.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments of the present application will be briefly described below, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.

FIG. 1 is a schematic flow chart diagram illustrating a data annotation method according to an embodiment of the present application;

FIG. 2 is a schematic flow chart of a data annotation method according to another embodiment of the present application;

FIG. 3 is a schematic flow chart diagram of a specific application example of the data annotation method provided in the present application;

FIG. 4 is a schematic structural diagram of a data annotation device according to an embodiment of the present application;

fig. 5 is a schematic structural diagram of an electronic device provided in an embodiment of the present application.

Detailed Description

Features and exemplary embodiments of various aspects of the present application will be described in detail below, and in order to make objects, technical solutions and advantages of the present application more apparent, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are intended to be illustrative only and are not intended to be limiting. It will be apparent to one skilled in the art that the present application may be practiced without some of these specific details. The following description of the embodiments is merely intended to provide a better understanding of the present application by illustrating examples thereof.

It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.

In order to solve the prior art problems, embodiments of the present application provide a data annotation method, apparatus, device, and computer storage medium. First, a data annotation method provided in the embodiment of the present application is described below.

Fig. 1 shows a schematic flow chart of a data annotation method according to an embodiment of the present application. As shown in fig. 1, the data annotation method includes:

step 101, inputting a sample data set to be labeled into a teacher model for labeling to obtain a first labeled sample set and the reliability of each labeled sample in the first labeled sample set;

102, in a training period, determining a reliability threshold, determining a second labeled sample set from the first labeled sample set, training a student model by using the second labeled sample set to obtain a trained student model, and acquiring a target evaluation index of the trained student model; the reliability of each labeled sample in the second labeled sample set is greater than or equal to a reliability threshold value;

and 103, determining a target labeling sample set from N second labeling sample sets determined in N training periods according to N target evaluation indexes obtained in N training periods, wherein N is an integer larger than 1.

It is easy to understand that, the teacher model and the student model can be one of transfer learning, and the function of the transfer learning can be simply described as transferring the performance of the teacher model to the student model; generally speaking, a teacher model can be considered as a large model, and evaluation indexes of the model, such as accuracy, false detection rate, accuracy, recall rate or generalization ability, are relatively superior; the student model can be considered as a model to be trained or a model to be verified, and the evaluation index of the student model has a certain promotion space under a specific scene.

In this embodiment, the sample data set to be annotated may be a picture, a video, and the like, which is not specifically limited herein; the teacher model may be used to label the sample data set to be labeled. Specifically, the teacher model may be some existing labeling tools, such as Labelme, easyll, or a labeling tool developed based on the recognition function of the deep learning model, and the like, and is not limited in detail here. The teacher model labels the sample data set to be labeled, and can assign corresponding classification labels or labels to the sample data set to be labeled; the first labeled sample set obtained by labeling can be considered to correspond to the sample data set to be labeled. Meanwhile, each labeled sample in the first labeled sample set may carry an evaluation index such as reliability.

The reliability may be embodied in a probability or score manner, for example, a plurality of pictures may exist in the sample data set to be labeled, and after one picture is labeled, an obtained labeling result may be characterized as: the probability of being a picture of an animal cat is 90%; here, 90% can be regarded as the above-described reliability.

It is easy to understand that, although the teacher model labels the to-be-labeled sample data set to obtain the first labeled sample set, the labeling quality of the first labeled sample set is difficult to be effectively ensured, and therefore, in this embodiment, the first labeled sample set can be subjected to quality evaluation or screening by obtaining the influence of the first labeled sample set on some evaluation indexes of the student model when the first labeled sample set is used for training the student model.

In particular, the student model may be trained based on the first set of annotated samples from a plurality of training cycles. In each training period, a confidence threshold may be determined, where the confidence threshold is used to determine a second labeled sample set from the first labeled sample set, for example, labeled samples in the first labeled sample set whose confidence level is greater than or equal to the confidence threshold may be extracted to form the second labeled sample set; and the second labeled sample set is used for training the student model in the training period. It is easy to understand that, for the student model, there are corresponding evaluation indexes, such as accuracy, false detection rate, accuracy rate or recall rate, and one of the evaluation indexes may be used as the target evaluation index. When the student model is trained, the quality of the second labeled sample set influences the training effect of the student model, and the training effect can be reflected through the target evaluation index.

Specifically, when the quality of the second labeled sample set is low, the target evaluation index of the trained student model may be poor, for example, the accuracy is low, or the accuracy is reduced relative to the accuracy of the learning model before training; on the contrary, when the quality of the second labeled sample set is higher, the target evaluation index of the trained student model may be optimized. In other words, based on the target evaluation index of the trained student model, the quality of the second labeled sample set can be actually verified or evaluated. In other words, for the second set of annotated samples, the set of sample data to be verified or evaluated may be considered.

As shown above, in this embodiment, a plurality of training periods are used to train the student model, and the target evaluation index of the student model trained in each training period is obtained. Generally speaking, a suitable confidence threshold is selected to screen the first annotation sample set, so that the obtained second annotation sample set performs better in the training of the student model. On one hand, the appropriate credibility threshold value is used for avoiding that the quality of the second labeled sample set is poor due to undersize, so that the student model is difficult to fit during training; on the other hand, the situation that the number of the second labeled sample sets is small due to overlarge size, so that overfitting of the trained student model is caused, and generalization performance is poor is avoided. However, the determination of the appropriate confidence threshold may be influenced by various factors, for example, when the teacher model labels different sample data sets to be labeled, the performance may be different; as another example, the student model may have been trained before it was trained using the second set of annotated samples, but the degree of training here may not be deterministic. Therefore, in the present embodiment, a mode of determining the confidence threshold in each of the plurality of training periods is adopted to correspondingly obtain the plurality of target evaluation indexes, and the target annotation sample set is determined based on the plurality of target evaluation indexes.

For example, in the first training period, the confidence threshold may be determined to be 85%, then the labeled samples with the confidence degrees greater than or equal to 85% in the first labeled sample set may be extracted to form a second labeled sample set, so as to train the student model, and obtain a target evaluation index of the trained student model, which is denoted as Ed 1; in the second training period, the confidence threshold may be determined to be 86%, and similarly, the labeled samples with the confidence level greater than or equal to 86% in the first labeled sample set may be extracted to form a second labeled sample set, so as to train the student model, and obtain a target evaluation index of the trained student model, which is denoted as Ed 2. Of course, in practical applications, there may be more training cycles, and accordingly, more target evaluation indexes may also be obtained, which may be sequentially denoted as Ed3, Ed4, and … … EdN.

In this embodiment, a target labeled sample set is determined from a plurality of second labeled sample sets according to a plurality of target evaluation indexes obtained in a plurality of training periods. In brief, according to the description of the training process for each training period in step 102, it can be considered that each second labeled sample set can be associated with a target evaluation index through the training period. For simplicity, it can be described that each second labeled sample set is associated with a target evaluation index. It is easily understood that when the target evaluation index performs better, the quality of the associated second labeled sample set is generally higher.

In practical application, the target labeling sample set can be a second labeling sample set with the optimal associated target evaluation index; for example, in the example combining Ed1 and Ed2, if Ed2 is better than Ed1, the second labeled sample set determined in the second training cycle may be used as the target labeled sample set. Of course, the target annotation sample set may also be determined based on other rules, which are not specifically limited herein. In addition, when the evaluation target evaluation index is better, the evaluation target may be evaluated by using an evaluation index such as accuracy or recall, which is not limited herein.

In the embodiment of the application, a teacher model is used for marking the sample data set to be marked to obtain a first marked sample set and the reliability of each marked sample in the first marked sample set; in a plurality of training periods, respectively determining a reliability threshold value, determining a second labeling sample set from the first labeling sample set, training the student model by using the second labeling sample set, and acquiring a target evaluation index of the trained student model; and determining a target labeling sample set from the second labeling sample set determined in each training period based on the target evaluation index obtained in each training period. In the embodiment of the application, a teacher model is used for labeling a sample data set to be labeled to obtain a first labeled sample set, the first labeled sample set is screened by using a determined credibility threshold value, and a target labeled sample set is determined from second labeled sample sets obtained through multiple screening based on the performances of the second labeled sample sets in student model training respectively, so that the quality of the target labeled sample set can be verified, and the quality of the target labeled sample set is effectively guaranteed.

In one example, the student model trained in the ith training period is the student model trained in the (i-1) th training period; i is an integer greater than 1 and less than or equal to N.

In other words, in this example, it may be considered a continuously trained process for the student model. The initial student model may not have received training before the first training period begins; or training is carried out to a certain degree, and the evaluation index has a certain promotion space. Compared with the prior art, the overfitting condition is easy to occur when the former student model is trained based on the second labeling sample set, and the verification capability of the student model on the second labeling sample set is difficult to ensure.

In other words, when the student model is trained to a certain extent and the promotion space of the evaluation index exists, the verification capability of the second labeled sample set can be better exerted. Based on the consideration, in the example, the student model obtained by training in one training period is used as the trained student model in the next training period, so that the student model can be continuously trained, and the quality verification effect on the second labeling sample set is better achieved.

Meanwhile, after a sufficient number of second labeling sample sets are trained based on the training mode for the student model, each evaluation index of the student model may reach a better state, and at the moment, the evaluation index can be directly used for verifying whether the labeling result of the teacher model is accurate or not; alternatively, the student model can also be used for labeling the sample data set to be labeled.

In one example, the determined reliability threshold value in the ith training period is larger than the determined reliability threshold value in the (i-1) th training period and is smaller than or equal to the preset reliability threshold value upper limit value; i is an integer greater than 1 and less than or equal to N.

In other words, in this example, the determined confidence thresholds may be raised sequentially over N training cycles that are performed sequentially.

For example, assuming that the upper limit of the preset confidence threshold is 95%, and the confidence threshold is determined to be 85% in the first training cycle, the confidence thresholds may be determined to be 86%, 87%, and … … 95% in the subsequent training cycles, respectively.

Of course, the above is only an example of the determination manner of the confidence threshold of each training cycle, and the gradient of the change of the confidence threshold in two adjacent training cycles, or the setting of the upper limit value of the confidence threshold, may be adjusted according to actual needs.

As can be seen, in this example, by adjusting the confidence threshold, the adopted second labeling sample sets are different in each training period, so that quality verification can be performed on different second labeling sample sets, and a target labeling sample set with higher quality can be obtained therefrom.

Of course, in addition, when the student model trained in the ith training period is the student model trained in the (i-1) th training period, the following effects can be obtained by continuously increasing the confidence threshold value:

it is easy to understand that the lower the confidence threshold value is, the more the number of the corresponding second labeled sample set labeled samples is, and under the condition that the training degree of the student model is relatively low, the second labeled sample set labeled with the larger number of samples is adopted to train the student model, so that the overfitting condition of the student model can be effectively avoided, and the verification capability of the student model is ensured; on the other hand, after the learning model is trained in a plurality of training periods, the evaluation index data is improved, and the method can be suitable for quality verification of a second labeled sample set with a smaller number of labeled samples.

Optionally, in step 102, training the student model by using the second labeled sample set to obtain a trained student model, and acquiring a target evaluation index of the trained student model, where the method includes:

dividing the second labeled sample set into a training set and a first verification set;

training the student model by using a training set to obtain a trained student model;

and acquiring a target evaluation index of the trained student model based on the first verification set.

In this embodiment, the second labeled sample set may be divided into a training set and a first verification set, where the training set is used to train the student model, and generally speaking, network parameters of the trained student model may change correspondingly; and the first validation set is used for validating the trained student model.

Taking the student model as the classification model as an example, the first verification set may have verification samples and corresponding classification labels; the trained student model can be classified according to the verification samples to obtain classification results, and whether the trained student model is correctly classified according to the verification samples is determined by comparing the classification results with the classification marks. Similarly, according to the performance of classifying the first verification set by the trained student model, the evaluation indexes such as accuracy, false detection rate or accuracy of the trained student model can be obtained, and the target evaluation index can be one or more of the evaluation indexes.

Of course, the above is only an example for the student model, and in practical application, the student model may be other types of recognition models; the target evaluation index may be another type of evaluation index.

In this embodiment, the target evaluation index of the trained student model is obtained based on the first validation set, and the target evaluation index is relatively easy to obtain.

In the process of using the student model to perform quality verification on different second labeling sample sets to obtain a target labeling sample set, the situation that the difference of each labeling sample in the second labeling sample set is small may exist, so that the student model trained through the training set may have strong recognition performance on the first verification set, but actually has the defect of low generalization degree; in other words, the quality verification result of the student model on the second labeled sample set may be biased due to the above situation.

To overcome the above drawback, optionally, the obtaining a target evaluation index of the trained student model based on the first validation set includes:

acquiring a preset second verification set;

and acquiring a target evaluation index of the trained student model based on the first verification set and the second verification set.

Specifically, with reference to fig. 2, the data annotation method provided in this embodiment may include:

step 201, inputting a sample data set to be labeled into a teacher model for labeling to obtain a first labeled sample set and the reliability of each labeled sample in the first labeled sample set;

step 202, determining a reliability threshold value, and determining a second labeled sample set from the first labeled samples, wherein the reliability of each labeled sample in the second labeled sample set is greater than or equal to the reliability threshold value;

step 203, dividing the second labeled sample into a training set and a first verification set;

step 204, acquiring a preset second verification set, and merging the first verification set and the second verification set to be used as a verification set;

step 205, training the student model by using the training set, and acquiring a target evaluation index of the trained student model based on the verification set;

step 206, judging whether the acquisition frequency of the target evaluation index meets a preset condition, if so, executing step 207, otherwise, returning to execute step 202;

the preset condition may be the preset number of times, or may be a number of times determined based on a change condition of the confidence threshold, for example, the confidence threshold has an initial value, a preset value may be added to the initial value before each time the step 202 is executed, and when the confidence threshold is greater than a preset upper limit value of the confidence threshold, it is determined that the number of times of acquiring the target evaluation index satisfies the preset condition.

And step 207, determining a target labeling sample set from the associated second labeling sample sets according to the target evaluation indexes obtained for multiple times.

The second verification set may be preset, and may be obtained by manual marking, or labeled sample data separately prepared in another manner, for example. The second validation set may not participate in the training process of the student model, and may be used for validation after each training of the student model based on the training set.

That is to say, the second verification set may not be affected by the sample data set to be labeled or the second labeling sample set, so that when the trained student model is subjected to target evaluation index acquisition, the situation that the training set and the verification set are all derived from the same sample data set to be labeled can be effectively avoided. Accordingly, the second verification set is used, the situation that some second labeling sample sets which possibly cause overfitting of the trained student model are verified to be high-quality labeling sample sets and finally serve as target labeling sample sets can be effectively avoided, and the reliability of the quality verification result of the student model on the second labeling sample sets is guaranteed.

Optionally, in step 103, determining a target labeled sample set from N second labeled sample sets determined in N training periods according to N target evaluation indexes obtained in N training periods, including:

determining a second labeled sample set determined in the nth training period as a target labeled sample set;

the target evaluation index obtained in the nth training period is the optimal target evaluation index of the N target evaluation indexes, the target evaluation index obtained in the nth training period is superior to the target evaluation index of the initial student model, the initial student model is the student model before the first training period starts, and N is an integer less than or equal to N.

As described in the above embodiments, it can be considered that each second labeled sample set is associated with a target evaluation index through the training period, that is, each second labeled sample set is associated with a target evaluation index. In this embodiment, the selection of the target labeled sample set is performed based on the target evaluation index associated with each second labeled sample set.

For example, the target annotation sample set may be a second annotation sample set determined in each training period, and the associated target evaluation index is the best second annotation sample set, and the association relationship is implemented based on the training period. In other words, after comparing the target evaluation indexes, the second labeled sample set determined in the nth training period may be obtained as the target labeled sample set by obtaining the training period number n (that is, the number corresponding to the nth training period) of the optimal target evaluation index.

The optimal target evaluation index can be screened out through the numerical value of the target evaluation index or a scoring mode and the like. For example, when the target evaluation index is accuracy, the highest accuracy value may be used as the optimal target evaluation index information; alternatively, when the target evaluation index includes a plurality of evaluation indexes, it may be obtained by comprehensively scoring the evaluation indexes, and the highest comprehensive score is taken as the optimum target evaluation index information.

Of course, in this embodiment, the target labeling sample set needs to satisfy the condition that the associated target evaluation index is better than the target evaluation index of the initial student model, in addition to the condition that the associated target evaluation index is optimal.

It will be readily appreciated that the initial learning model may be considered to be the student model prior to the start of the first training period, which may be a model that has been trained to some extent in advance, as described above. In this embodiment, after N training periods, if the trained student model obtained in each training period is inferior to the initial learning model, it may be considered that each second labeling sample set is difficult to bring sufficient positive influence to the learning model, or it may be further considered that the labeling quality of the teaching model on a certain batch of sample data sets to be labeled is poor, and a target labeling sample set with higher quality cannot be selected from the corresponding first labeling sample set.

Therefore, the quality of the determined target labeling sample set can be effectively ensured by limiting the determination condition of the target labeling sample set.

In one example, when the optimal target evaluation index of the N target evaluation indexes is still inferior to the target evaluation index of the initial student model, the second labeled sample set associated with the optimal target evaluation index may be added to the first labeled sample set again for quality verification; or the corresponding first set of annotated samples may be discarded.

Optionally, in step 103, after determining the target labeled sample set from the N second labeled sample sets determined in the N training periods according to the N target evaluation indexes obtained in the N training periods, the data labeling method further includes:

and acquiring a target reliability threshold, wherein the target reliability threshold is a reliability threshold determined in a training period corresponding to the target labeling sample set, and the target reliability threshold is used for guiding the selection of the labeling sample set.

In this embodiment, the reliability threshold determined in the training period corresponding to the target labeling sample set is obtained, for example, when the target labeling sample set is obtained in the nth training period, and the reliability threshold determined in the training period is 90%, the reliability threshold may be obtained, and meanwhile, the reliability threshold corresponds to the target reliability threshold.

The target reliability threshold is used to guide selection of the labeled sample set, for example, in combination with an actual application scenario, a lot of samples to be labeled may be counted up, and may be classified into a plurality of sample data sets to be labeled, and after one sample data set to be labeled completes the labeling described in the above embodiments, the obtaining in a plurality of training cycles, and the determining process of the target labeled sample set, the target reliability threshold may be obtained. The target reliability threshold may be used in a data annotation process of other sample data sets to be annotated to which the batch of samples to be annotated belongs, for example, in a first target annotation sample set determination process, a process of sequentially determining the reliability thresholds as 85% and 86% … … 95% in a plurality of training cycles may be performed; after that, the target confidence threshold is determined to be 90%; in the second time of determining the target labeling sample set, the confidence threshold values of the training periods are respectively determined to be 90% and 91% … … 95%.

In the embodiment, the target credibility threshold value is obtained, guidance can be provided for selecting the labeled sample set, so that the labeling time of the whole batch of samples to be labeled is shortened on the basis of ensuring the labeling quality, and the labeling efficiency is improved.

As shown in fig. 3, the following describes a data annotation method provided in an embodiment of the present application with reference to a specific application example, where the specific application example specifically includes:

1) inputting original sample data to be labeled into a teacher model (hereinafter referred to as TModel), labeling the samples, such as classification labeling and labeling, and outputting labeled data sets (corresponding to the first labeled sample set in the above, hereinafter referred to as IADataset), wherein each data in the IADataset has a credibility evaluation index;

the original sample data to be labeled can be video streams or pictures and the like, and can be input into the teacher model for labeling after being stored in the cache; the specific way of labeling may be to classify or label the raw data of these samples.

2) An initial threshold (hereinafter referred to as IFValue) and an upper threshold adjustment limit (corresponding to the above-mentioned upper limit of the predetermined reliability threshold, hereinafter referred to as UPValue) of the evaluation index, i.e., reliability, are set, for example: IFValue can be 0.85, UPValue can be 0.95, and data with the reliability evaluation index larger than that of the IADataset data set is screened out to form a sample data set (hereinafter referred to as WVDataset) to be subjected to quality verification. It should be noted that IFValue can be considered as an initially set value, and will change according to the change of the training period in the subsequent training.

The WVDataset can be scaled into a training set (hereinafter TDataset) and a validation set (hereinafter VDataset) according to model training requirements, for example, according to TDataset: VDataset ═ 10: 1;

furthermore, an initial verification data set (hereinafter referred to as IVDataset) may be prepared from the model in advance, and the IVDataset and VDataset may be combined to form a final model verification set (LVDataset). Wherein, the data ratio of IVDataset to VDataset in LVDataset can be 1: 1.

in one example, the number of ivdatasets may be initially set to 5000, and the specific values may be adjusted according to the needs of the student model.

3) Inputting TDataset and LVDataset into SModel for training to obtain information (hereinafter referred to as EIndex) of a target evaluation index of the SModel after training, and a threshold value (hereinafter referred to as FValue) and WVDataset of reliability of the evaluation index set correspondingly;

4) and (3) repeating the steps 2 and 3 after the IFValue +0.01, until the IFValue +0.01 > UPValue is finished, or finishing the process of the loop when the IFValue is judged to be larger than the UPValue for the first time, wherein each loop process can be regarded as a training period.

5) And comparing the EIndex obtained in each training period to obtain an optimal EIndex value and corresponding Fvalue and WVDataset, and if the optimal EIndex value is superior to the EIndex before SModel training, considering the WVDataset as an effective sample set and putting the effective sample set into a formal sample library.

In one example, in any training period (i.e., the ith training period), it may be determined whether the obtained EIndex is improved relative to the EIndex obtained in the previous training period (i.e., the (i-1) th training period); if the EIndex value is increased, recording the EIndex value obtained in the ith training period, and the corresponding Fvalue and the corresponding WVDataset for a subsequent process of obtaining the optimal EIndex value.

According to the embodiment of the application, the original data of the sample can be automatically labeled through the teacher model, the sample data set after high-quality labeling can be obtained based on the training performance of the labeled sample data set on the student model, manual labeling is compared, and efficiency and cost are greatly improved. In some application scenarios, the possibility of forming a high-quality annotation sample database of tens of millions or even higher orders of magnitude in a short period can be provided.

Fig. 4 shows a schematic structural diagram of a data annotation device provided in an embodiment of the present application.

As shown in fig. 4, the data annotation device includes:

the marking module 401 is configured to input the sample data set to be marked into the teacher model for marking, so as to obtain a first marked sample set and the reliability of each marked sample in the first marked sample set;

a training module 402, configured to determine a reliability threshold in a training period, determine a second labeled sample set from the first labeled sample set, train the student model using the second labeled sample set, obtain a trained student model, and obtain a target evaluation index of the trained student model; the reliability of each labeled sample in the second labeled sample set is greater than or equal to a reliability threshold value;

the determining module 403 is configured to determine, according to N target evaluation indexes obtained in N training periods, a target labeling sample set from N second labeling sample sets determined in N training periods, where N is an integer greater than 1.

Optionally, the student model trained in the ith training period is a student model trained in an (i-1) th training period; i is an integer greater than 1 and less than or equal to N.

Optionally, the reliability threshold determined in the ith training cycle is greater than the reliability threshold determined in the (i-1) th training cycle and is less than or equal to a preset upper limit value of the reliability threshold; i is an integer greater than 1 and less than or equal to N.

Optionally, the training module 402 includes:

the dividing unit is used for dividing the second labeled sample set into a training set and a first verification set;

the training unit is used for training the student models by using a training set to obtain trained student models;

and the acquisition unit is used for acquiring the target evaluation index of the trained student model based on the first verification set.

Optionally, based on the first verification set, the obtaining unit includes:

the first obtaining subunit is used for obtaining a preset second verification set;

and the second obtaining subunit is used for obtaining the target evaluation index of the trained student model based on the first verification set and the second verification set.

Optionally, the determining module 403 includes:

the determining unit is used for determining the second labeled sample set determined in the nth training period as a target labeled sample set;

Optionally, the data annotation device may further include:

and the acquisition module is used for acquiring a target reliability threshold, wherein the target reliability threshold is a reliability threshold determined in a training period corresponding to the target labeling sample set, and the target reliability threshold is used for guiding the selection of the labeling sample set.

Fig. 5 shows a hardware structure diagram of an electronic device provided in an embodiment of the present application.

The electronic device may comprise a processor 501 and a memory 502 in which computer program instructions are stored.

Specifically, the processor 501 may include a Central Processing Unit (CPU), or an Application Specific Integrated Circuit (ASIC), or may be configured to implement one or more Integrated circuits of the embodiments of the present Application.

Memory 502 may include mass storage for data or instructions. By way of example, and not limitation, memory 502 may include a Hard Disk Drive (HDD), a floppy Disk Drive, flash memory, an optical Disk, a magneto-optical Disk, tape, or a Universal Serial Bus (USB) Drive or a combination of two or more of these. Memory 502 may include removable or non-removable (or fixed) media, where appropriate. The memory 502 may be internal or external to the integrated gateway disaster recovery device, where appropriate. In a particular embodiment, the memory 502 is non-volatile solid-state memory.

The memory may include Read Only Memory (ROM), Random Access Memory (RAM), magnetic disk storage media devices, optical storage media devices, flash memory devices, electrical, optical, or other physical/tangible memory storage devices. Thus, in general, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software comprising computer-executable instructions and when the software is executed (e.g., by one or more processors), it is operable to perform operations described with reference to the methods according to an aspect of the present disclosure.

The processor 501 reads and executes the computer program instructions stored in the memory 502 to implement any one of the data annotation methods in the above embodiments.

In one example, the electronic device can also include a communication interface 503 and a bus 504. As shown in fig. 5, the processor 501, the memory 502, and the communication interface 503 are connected via a bus 504 to complete communication therebetween.

The communication interface 503 is mainly used for implementing communication between modules, apparatuses, units and/or devices in the embodiments of the present application.

Bus 504 comprises hardware, software, or both to couple the components of the online data traffic billing device to each other. By way of example, and not limitation, a bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hypertransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an infiniband interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a video electronics standards association local (VLB) bus, or other suitable bus or a combination of two or more of these. Bus 504 may include one or more buses, where appropriate. Although specific buses are described and shown in the embodiments of the application, any suitable buses or interconnects are contemplated by the application.

In addition, in combination with the data annotation method in the foregoing embodiments, the embodiments of the present application may provide a computer storage medium to implement. The computer storage medium having computer program instructions stored thereon; the computer program instructions, when executed by a processor, implement any of the data annotation methods of the embodiments described above.

It is to be understood that the present application is not limited to the particular arrangements and instrumentality described above and shown in the attached drawings. A detailed description of known methods is omitted herein for the sake of brevity. In the above embodiments, several specific steps are described and shown as examples. However, the method processes of the present application are not limited to the specific steps described and illustrated, and those skilled in the art can make various changes, modifications, and additions or change the order between the steps after comprehending the spirit of the present application.

The functional blocks shown in the above structural block diagrams may be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, it may be, for example, an electronic circuit, an Application Specific Integrated Circuit (ASIC), suitable firmware, plug-in, function card, or the like. When implemented in software, the elements of the present application are the programs or code segments used to perform the required tasks. The program or code segments may be stored in a machine-readable medium or transmitted by a data signal carried in a carrier wave over a transmission medium or a communication link. A "machine-readable medium" may include any medium that can store or transfer information. Examples of a machine-readable medium include electronic circuits, semiconductor memory devices, ROM, flash memory, Erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, Radio Frequency (RF) links, and so forth. The code segments may be downloaded via computer networks such as the internet, intranet, etc.

It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, the present application is not limited to the order of the above-described steps, that is, the steps may be performed in the order mentioned in the embodiments, may be performed in an order different from the order in the embodiments, or may be performed simultaneously.

Aspects of the present disclosure are described above with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions/acts specified in the flowchart and/or block diagram block or blocks. Such a processor may be, but is not limited to, a general purpose processor, a special purpose processor, an application specific processor, or a field programmable logic circuit. It will also be understood that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware for performing the specified functions or acts, or combinations of special purpose hardware and computer instructions.

As will be apparent to those skilled in the art, for convenience and brevity of description, the specific working processes of the systems, modules and units described above may refer to corresponding processes in the foregoing method embodiments, and are not described herein again. It should be understood that the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive various equivalent modifications or substitutions within the technical scope of the present application, and these modifications or substitutions should be covered within the scope of the present application.

Claims

1. A method for annotating data, comprising:

in a training period, determining a reliability threshold value, determining a second labeling sample set from the first labeling sample set, training a student model by using the second labeling sample set to obtain a trained student model, and acquiring a target evaluation index of the trained student model; wherein the reliability of each labeled sample in the second labeled sample set is greater than or equal to the reliability threshold;

determining a target labeling sample set from N second labeling sample sets determined in N training periods according to N target evaluation indexes obtained in N training periods, wherein N is an integer larger than 1;

the sample data set to be marked is a picture or a video;

the target evaluation index comprises at least one of accuracy, false detection rate, precision rate, recall rate and generalization ability;

and the target evaluation index of the trained student model is obtained by verifying the trained student model through a verification sample.

2. The method according to claim 1, wherein the student model trained in the ith training period is the student model trained in the (i-1) th training period; i is an integer greater than 1 and less than or equal to N.

3. The method according to claim 1 or 2, wherein the confidence threshold determined in the ith training cycle is greater than the confidence threshold determined in the (i-1) th training cycle and is less than or equal to a preset confidence threshold upper limit value; i is an integer greater than 1 and less than or equal to N.

4. The method according to claim 1, wherein the training a student model by using the second labeled sample set to obtain a trained student model, and obtaining a target evaluation index of the trained student model comprises:

training the student model by using the training set to obtain a trained student model;

5. The method according to claim 4, wherein the obtaining of the target evaluation index of the trained student model based on the first validation set comprises:

acquiring a preset second verification set;

6. The method of claim 1, wherein the determining a target labeled sample set from N second labeled sample sets determined in N training periods according to N target evaluation indexes obtained in N training periods comprises:

the target evaluation index obtained in the nth training period is the optimal target evaluation index of the N target evaluation indexes, the target evaluation index obtained in the nth training period is superior to the target evaluation index of an initial student model, the initial student model is a student model before the first training period starts, and N is an integer less than or equal to N.

7. The method according to claim 1, wherein after determining a target labeled sample set from N second labeled sample sets determined in N training periods according to N target evaluation indexes obtained in N training periods, the method further comprises:

8. A data annotation device, said device comprising:

the system comprises a marking module, a teacher model and a plurality of marking modules, wherein the marking module is used for inputting a sample data set to be marked into the teacher model for marking to obtain a first marked sample set and the reliability of each marked sample in the first marked sample set;

the training module is used for determining a reliability threshold value in a training period, determining a second labeling sample set from the first labeling sample set, training a student model by using the second labeling sample set to obtain a trained student model, and acquiring a target evaluation index of the trained student model; wherein the reliability of each labeled sample in the second labeled sample set is greater than or equal to the reliability threshold;

the determining module is used for determining a target labeling sample set from N second labeling sample sets determined in N training periods according to N target evaluation indexes obtained in N training periods, wherein N is an integer larger than 1;

the sample data set to be marked is a picture or a video;

9. An electronic device, characterized in that the device comprises: a processor and a memory storing computer program instructions;

the processor, when executing the computer program instructions, implements the data annotation method of any one of claims 1-7.

10. A computer storage medium having computer program instructions stored thereon which, when executed by a processor, implement the data annotation method of any one of claims 1-7.