CN115100484A - Data labeling method, computer program product, storage medium and electronic device - Google Patents

Abstract

The embodiment of the application provides a data annotation method, a computer program product, a storage medium and an electronic device. The data labeling method comprises the following steps: acquiring first data to be labeled and a reasoning result thereof, wherein the reasoning result of the first data to be labeled is information obtained by predicting the first data to be labeled by using a labeling model; acquiring a label of second data to be labeled marked by a user, wherein the second data to be marked is part of data to be marked in the first data to be marked; estimating whether the labeling accuracy of third data to be labeled which is not labeled in the first data to be labeled reaches the standard or not according to the labeling label of the second data to be labeled, wherein the labeling accuracy of the third data to be labeled is the accuracy of a reasoning label determined according to the reasoning result; and if so, determining the inference label of the third data to be labeled as the labeling label of the third data to be labeled. The method obviously improves the efficiency of data annotation, and the annotation accuracy is effectively guaranteed.

Description

Data labeling method, computer program product, storage medium and electronic device

technical field

The present application relates to the field of artificial intelligence technology, and in particular, to a data labeling method, a computer program product, a storage medium and an electronic device.

Background technique

Data labeling is an important task in the field of artificial intelligence. The so-called data labeling is to add reliable label information to the data. For example, in the image classification task, data annotation can refer to marking the real category of the training image, so that the real category can be used as a supervision signal to perform supervised training on the image classification model, so that the model can learn the content and content of the training image. Correspondence between its ground-truth categories, which in turn can effectively classify other images.

However, the current data labeling is basically completely done manually, and its efficiency is very low.

SUMMARY OF THE INVENTION

The purpose of the embodiments of the present application is to provide a data labeling method, a computer program product, a storage medium and an electronic device to improve the above technical problems.

To achieve the above purpose, the application provides the following technical solutions:

In a first aspect, an embodiment of the present application provides a data labeling method, including: acquiring first data to be labelled and an inference result thereof; wherein the reasoning result of the first data to be labelled is to use a labeling model for the first data to be labelled. information obtained by predicting the labeled data; obtaining the labeling label of the second data to be labeled marked by the user; wherein, the second data to be labeled is part of the data to be labeled in the first data to be labeled; according to the second data to be labeled The labeling label of the data to be labeled estimates whether the labeling accuracy of the third data to be labeled that has not been labeled in the first data to be labeled has reached the standard; wherein, the labeling accuracy of the third data to be labeled is based on the third The accuracy of the inference label determined by the inference result of the data to be labeled; if the labeling accuracy of the third data to be labeled has reached the standard, the inference label of the third data to be labeled is determined as the third data to be labeled. Annotation labels for the data.

The above method only needs to manually label the second to-be-labeled data in the first to-be-labeled data, and based on the labeling result, the labeling accuracy evaluation can be performed on the unlabeled third to-be-labeled data in the first to-be-labeled data, And when the accuracy rate reaches the standard, the third data to be labeled is automatically labeled without manual labeling of all the data to be labeled. Therefore, the method significantly improves the efficiency of data labeling, and the labeling accuracy is also effectively guaranteed. Among them, the labeling accuracy rate of the third to-be-labeled data reaches the expected standard (that is, up to the standard), while the second to-be-labeled data is manually labeled, and its labeling accuracy can be considered to be close to or equal to 100 when the user is in a normal state %.

In a second aspect, an embodiment of the present application provides a computer program product, including computer program instructions that, when read and run by a processor, execute the first aspect or any possible implementation manner of the first aspect provided method.

In a third aspect, embodiments of the present application provide a computer-readable storage medium, where computer program instructions are stored on the computer-readable storage medium, and when the computer program instructions are read and run by a processor, the first aspect or A method provided by any possible implementation manner of the first aspect.

In a fourth aspect, an embodiment of the present application provides an electronic device, including: a memory and a processor, where computer program instructions are stored in the memory, and when the computer program instructions are read and run by the processor, execute a first A method provided by the aspect or any one of the possible implementations of the first aspect.

Description of drawings

In order to explain the technical solutions of the embodiments of the present application more clearly, the following briefly introduces the accompanying drawings that need to be used in the embodiments of the present application. It should be understood that the following drawings only show some embodiments of the present application, therefore It should not be regarded as a limitation of the scope. For those of ordinary skill in the art, other related drawings can also be obtained from these drawings without any creative effort.

FIG. 1 shows the flow of the data labeling method provided by the embodiment of the present application;

Figure 2 (A) and Figure 2 (B) show an implementation of the user annotation interface;

Figure 3 (A) and Figure 3 (B) show another implementation of the user annotation interface;

Fig. 4 shows the flow of an iterative implementation of the method in Fig. 1;

Fig. 5 shows the reduction principle of the fourth to-be-labeled data;

FIG. 6 shows the architecture and working principle of the data labeling system provided by the embodiment of the present application;

Fig. 7 shows an implementation manner of model training and inference performed by the system in Fig. 6;

FIG. 8 shows the functional modules included in the data labeling apparatus provided by the embodiment of the present application;

FIG. 9 shows a structure of an electronic device provided by an embodiment of the present application.

Detailed ways

In recent years, important progress has been made in the research of artificial intelligence-based computer vision, deep learning, machine learning, image processing, image recognition and other technologies. Artificial Intelligence (AI) is an emerging science and technology that studies and develops theories, methods, technologies and application systems for simulating and extending human intelligence. Artificial intelligence is a comprehensive discipline, involving chips, big data, cloud computing, Internet of Things, distributed storage, deep learning, machine learning, neural networks and many other types of technologies. As an important branch of artificial intelligence, computer vision is to let machines recognize the world. Computer vision technology usually includes face recognition, living detection, fingerprint recognition and anti-counterfeiting verification, biometric recognition, face detection, pedestrian detection, target detection, pedestrian detection Recognition, image processing, image recognition, image semantic understanding, image retrieval, text recognition, video processing, video content recognition, 3D reconstruction, virtual reality, augmented reality, simultaneous positioning and map construction, computational photography, robot navigation and positioning and other technologies. With the research and progress of artificial intelligence technology, this technology has been applied in many fields, such as safety prevention and control, urban management, traffic management, building management, park management, face access, face attendance, logistics management, warehouse management , robotics, intelligent marketing, computational photography, mobile imaging, cloud services, smart home, wearable devices, driverless, autonomous driving, smart medical care, face payment, face unlock, fingerprint unlock, personal verification, smart screen, smart TV, camera, mobile Internet, webcast, beauty, beauty, medical beauty, intelligent temperature measurement and other fields. The data labeling method in the embodiment of the present application also belongs to the category of artificial intelligence technology.

Before introducing the technical solutions in the embodiments of the present application, the meaning of data labeling and the meaning of performing data labeling are briefly explained.

The so-called data labeling is the process of labeling the data by some means (for example, manual confirmation), and the label can be understood as the value of a certain (or certain) attributes of the data. If the labeling effect is ideal, The label should then be the true value of the property.

In the field of artificial intelligence, data labeling is often performed on the training samples of a certain model (for example, a neural network model). The labeled training samples can be used for supervised training of the model, and the trained model can be used to perform specific tasks. For the convenience of description later, it may be called the task model. These tasks can be summarized as: input the task data into the task model, after the operation of the task model, predict the value of a certain (or certain) attributes of the task data, and the predicted value may have actual physical meaning in the specific problem .

For example, a possible training process is: input the training sample into the task model, obtain the prediction result output by the model for an attribute of the training sample, and substitute the prediction result and the label of the training sample into a preset loss function to calculate the prediction loss, which characterizes the difference between the predicted result and the labels of the training samples, iteratively optimizes the parameters of the task model until the loss converges.

If the label of the training sample represents the true value of an attribute of the training sample, then the training process can also be regarded as adjusting the parameters of the task model so that the output prediction result of the attribute for the training sample is closer to its true value. approaching process. Therefore, after the task model is trained, the task data of the same type as the training sample can be processed, and a reasonable result can be predicted for this attribute of the task data, that is, a specific task can be better completed.

For example, the task model can be an image classification model, and the image classification model is used to predict the category of a specific object in the image, for example, to predict which number from 0 to 9 the handwritten number in the image is, the training samples at this time can be pre-collected An image of handwritten digits, the labels of the training samples are the real digits in the pre-labeled images.

For another example, the task model may be a target detection model, and the target detection model is used to detect the position of a specific object in an image, for example, to detect the position of a vehicle in an image, and the training sample at this time may be a pre-collected road image, the label of the training sample. is the true position of the vehicle in the pre-annotated image.

For another example, the task model may be a part-of-speech analysis model, which is used to predict the part-of-speech of each word in a sentence, such as a noun, a verb, a preposition, etc. The training samples at this time may be pre-collected sentences and labels of the training samples. is the true part-of-speech of the words in the pre-annotated sentences.

In a word, the solution of this application does not limit the specific form of the data to be labeled, such as images, text, voice, etc.; nor does it limit the specific form of the label, which is related to the specific task to be performed by the task model, for example, it can be The real class, real location, etc. of the object in the image. In the following, for the sake of simplicity, the data labeling method proposed in the present application is mainly introduced by taking the case where the data to be labelled is an image as an example, but this should not be regarded as a limitation on the protection scope of the present application.

Further, because in the field of artificial intelligence, the human cognitive result is usually regarded as the correct result, and the task model is only a simulation of the cognitive process of a person on a specific task, so the current data annotation is usually done manually. , take the human-annotated labels as the true values of the attributes in the data. However, the training of the task model is likely to require a large number of labeled training samples, resulting in a huge workload for labelers and very low labeling efficiency.

The data labeling method proposed in this application aims to combine manual labeling and machine automatic labeling, thereby significantly improving the efficiency of data labeling and ensuring that the accuracy of labeling results can meet the task requirements.

The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings in the embodiments of the present application. It should be noted that like numerals and letters refer to like items in the following figures, so once an item is defined in one figure, it does not require further definition and explanation in subsequent figures.

The terms "comprising", "comprising" or any other variation thereof are intended to encompass non-exclusive inclusion such that a process, method, article or device comprising a list of elements includes not only those elements, but also other not expressly listed elements, or also include elements inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.

The terms "first", "second", etc. are only used to distinguish one entity or operation from another, and should not be construed to indicate or imply relative importance, nor to require or imply such entities or operations. There is no such actual relationship or sequence between operations.

FIG. 1 shows the flow of the data labeling method in the embodiment of the present application. The method can be, but is not limited to, executed on the electronic device shown in FIG. 9 . For the specific structure of the electronic device, refer to the description of FIG. 9 below. . Referring to Figure 1, the data labeling method includes:

Step S110: Acquire the first to-be-labeled data and its inference result.

The first data to be labeled is a batch of data to be labeled. For example, the first data to be labeled can be a training sample that has not been labeled, and can be used as a formal training sample of the task model after being labeled. The reasoning result of the first data to be labeled is the information obtained by using the labeling model to predict the first data to be labeled, or in other words, after the first data to be labeled is input into the labeling model, the obtained model output is the reasoning result of the first data to be labeled. . For example, if the labeling model is a classification model, the reasoning result of the first data to be labelled may be the probability that the first data to be labelled belongs to each category predicted by the model; if the labeling model is a regression model, the reasoning result of the first data to be labelled The result can be the regression value predicted by the model.

The annotation model is the model used in the data annotation phase. The annotation model can have the same form of input and output as the task model mentioned above. For example, the task model is an image classification model, the input is an image, and the output is the probability that the image belongs to each category (for example, for handwritten digit recognition, it is the probability that the image belongs to the numbers 0-9, and there is only one in the default image. handwritten digits), then the labeling model can also be an image classification model, whose input is an image, and the output is the probability that the image belongs to each category. For another example, if the task model is a target detection model, the input is an image, and the output is the position of each target in the image (for example, for vehicle detection, it is the position of each vehicle in the image), then the annotation model can also be the target. A detection model whose input is an image, output is the location of each object in that image, and so on.

However, the annotation model and the task model may, but are not necessarily the same model, that is, the two may have the same structure but different parameters, or even different structures. There are many factors that cause the two to be designed as different models:

For example, in a certain scenario, it is desirable to assign accurate labels to the data to be labeled as much as possible, that is to say, there is no need to consider resource consumption too much during data labeling. In this case, a more complex model can be used as the labeling model; but for The task model is likely to be deployed in the actual production environment (for example, mobile phones, cameras), the resources that can be used are strictly limited, and only a simpler model can be used.

For another example, according to the following content, in some implementations of the data labeling method, the labeling model will only use the manually labelled data for training, and will not use the data automatically labelled by the machine for training, so it is directly used for training. Executing the task will not achieve very good results, because the training data it has "seen" is very limited, and the task model may use human-labeled data and machine-labeled data during training, so the effect of its execution of the task. Better because it has "seen" more training data. How to train the labeling model will not be elaborated here.

For the sake of simplicity, in the following examples, the annotation model and the task model can be considered to be the same model (for example, both are image classification models), and have the same form of input and output, but this should not be regarded as a Restrictions on the scope of application for protection.

"Acquiring" in step S110 generally refers to obtaining through various means: for example, it may be that the first data to be labeled and its inference result have been obtained and stored somewhere before step S110 is executed, and directly read when step S110 is executed The first to-be-labeled data and its inference result; for another example, the first to-be-labeled data and its inference result may not be obtained before step S110 is performed, and the first data to be labeled and its inference result may not be obtained before step S110 is performed, and the inference result is obtained from somewhere (for example, the reasoning mentioned later) result pool) to obtain the first data to be labeled, and use the labeling model to calculate the inference result of the first data to be labeled, and so on.

Step S120: Obtain the labeling label of the second data to be labelled marked by the user.

The second data to be labeled is part of the data to be labeled in the first data to be labeled. The second data to be labeled may be obtained by sampling from the first data to be labeled, and the sampling method is not limited, for example, it may be random sampling, or sampling according to a fixed pattern, and so on. Second, there may be no duplicate data in the data to be labeled, so as to avoid repeated labeling.

After the second data to be labeled is obtained, it can be provided to the user for manual labeling, and the user here is the index annotator. There are many ways to provide the second data to be labeled to the user: for example, if the device (hereinafter referred to as the user terminal) used by the user for data labeling and the device (hereinafter referred to as the current device) that performs step S120 are not the same device, Then the second data to be marked can be sent from the current device to the user terminal, and displayed on the user marking interface (for example, a web page or client interface) of the user terminal for the user to mark; If the devices are the same device, the second to-be-labeled data can be directly displayed on the user labeling interface of the current device for the user to label, and so on.

It should be understood that although the user must know the content of the second to-be-labeled data in some way to label it, the way of knowing is not limited to displaying it on the user's labeling interface. For example, the second to-be-labeled data can also be played by voice. (if it can be played by voice), or print out the second data to be marked for the user to watch and so on.

When the user labels the second data to be labeled, it is manually labeled, and this label is called a label. If the default manual labeling is accurate, the label is the final label of the second data to be labeled. For example, if the first data to be labeled is used for training a classification model (for example, the task model and the labeling model are both classification models), the user can use the category of the first data to be labeled as his label as his label; another example , if the first data to be labeled is used for training a regression model (for example, the task model and the labeled model are both regression models), the user can use the regression value of the first data to be labeled as his label as his label. When introducing FIG. 2(A), FIG. 2(B), FIG. 3(A), and FIG. 3(B), the user's labeling process will be further elaborated, which will not be expanded here for the time being.

After the user's annotation is completed, the user can trigger the confirmation operation representing the completion of the annotation (for example, click "Submit" on the user annotation interface), and return the annotation result, that is, the annotation label of the second data to be annotated to the current device (if the user originally The labeling on the current device can be considered as being returned to the main program that executes the data labeling process), so that the current device obtains the labeling label of the second data to be labelled. After step S120 is performed, the second to-be-labeled data is finished.

Step S130: Estimate whether the labeling accuracy of the third data to be labelled has reached the standard according to the labeling label of the second data to be labelled.

The third data to be labeled is the data that has not been labeled in the first data to be labeled. Since the second data to be labeled is only part of the data to be labeled in the first data to be labeled, after the second data to be labeled is labeled, the first data to be labeled There must be unlabeled data in the labeled data, so third data to be labeled may be generated. However, it should be noted that all the unlabeled data in the first data to be labeled may, but not necessarily, belong to the third data to be labeled, and there are different implementations here. For example, it can be seen from the following that in some implementations, some unlabeled data that is evaluated as having a low labeling accuracy rate will be excluded from the third data to be labeled.

The labeling accuracy of the third to-be-labeled data is the accuracy of the inference label determined according to the inference result of the third to-be-labeled data. Wherein, since the third to-be-labeled data belongs to the first to-be-labeled data, and the inference result of the first to-be-labeled data has been obtained in step S110, the inference result of the third to-be-labeled data is known when step S130 is executed.

The inference label of the third data to be labeled is a label obtained based on the inference result of the third data to be labeled and combined with specific rules. The label is not manually labeled, so its accuracy is usually not 100% (the default manually labeled label is correct). The labeling accuracy can be estimated according to the label of the second to-be-labeled data. The principle Yes: the second to-be-labeled data also has an inference label, and the rules for obtaining the inference label are the same as the third to-be-labeled data, so the difference between the inference label of the second to-be-labeled data and the labeling label (correct labeling result) is It can reflect the accuracy of the inference label calculated according to this rule. There are many rules for obtaining inference labels. Examples will be given later, which will not be expanded here.

It should be pointed out that the concept of the inference label introduced above is only to explain the meaning of the labeling accuracy of the third to-be-labeled data, and does not mean that the inference label of the third to-be-labeled data must be calculated when estimating the labeling accuracy. It does not mean that the third to-be-labeled data must be labeled with an inference label when estimating the labeling accuracy, but the definition of the labeling accuracy can be understood as follows: if the third to-be-labeled data is labeled according to its inference label, the labeling result is accurate. The rate is in line with the estimated accuracy. In conclusion, it is not contradictory that the third to-be-labeled data has an inference label and that it is in an unlabeled state.

Third, the accuracy of the data to be labeled meets the standard may mean that the estimated accuracy of the labeled data is greater than the target accuracy. The target accuracy rate may be an accuracy rate set according to the training requirements of the task model before starting the data labeling, for example, it may be 99%, 98%, and the like. If it is estimated that the accuracy of the third to-be-labeled data meets the standard, step S140 is executed. If it is estimated that the accuracy of the third to-be-labeled data does not meet the standard, different processing methods may be adopted: for example, sampling more second to-be-labeled data Label the data, and perform steps S120 to S130 to re-estimate the labeling accuracy of the third data to be labelled. Of course, the third data to be labelled may change at this time; for another example, retrain the labeling model and perform steps S110 to S130 , re-estimating the labeling accuracy of the third data to be labeled; for another example, directly manually labeling the third data to be labeled, and so on.

Step S140: Determine the inference label of the third data to be labeled as the labeling label of the third data to be labeled.

Since the labeling accuracy of the third to-be-labeled data can meet the requirements, it is sufficient to label the third to-be-labeled data directly according to the inference label, and the label of the third to-be-labeled data is also its final label. If the inference label of the third data to be labeled has not been calculated before, the inference label of the third data to be labeled needs to be calculated first in step S140. Since the third data to be labeled is automatically labeled, its labeling efficiency is much higher than that of manual labeling, and its labeling accuracy is also guaranteed, for example, it is sufficient to meet the training needs of the task model.

After step S140 is performed, both the second data to be labeled and the third data to be labeled in the first data to be labeled have been labeled, but it does not rule out that there are data remaining in the first data to be labeled that have not been labeled. For these data, Different processing methods can be adopted: for example, labeling directly by humans; another example, retraining the labeling model and then labeling (at this time, the inference results may change, thereby changing the labeling results), and so on.

To briefly summarize the method in Figure 1, this method only needs to manually label the second to-be-labeled data in the first to-be-labeled data. The labeling data is evaluated for labeling accuracy, and the third to-be-labeled data is automatically labeled when its accuracy reaches the standard, without the need to manually label all the to-be-labeled data. Therefore, the method significantly improves the efficiency of data labeling, and the labeling accuracy is also effectively guaranteed, for example, it can meet the training requirements of the task model. Among them, the labeling accuracy of the third to-be-labeled data has reached the expected standard, while the second to-be-labeled data is manually labeled, and its labeling accuracy can be considered to be close to or equal to 100% when the user is in a normal state.

Note that the inventors have found that the data to be labeled is often huge in quantity, and the first data to be labeled is likely to be only one batch of data. Even if not every batch of the first data to be labeled can trigger the execution of step S140, there are still many batches of data to be labeled. The first data to be labeled can trigger the execution of step S140, that is, automatic labeling is almost always executed, and the above beneficial effects are not only achieved in a few special cases.

In addition, it should be pointed out that although the data labeling and the training of the task model are always linked in the above description, in fact, the two do not have a binding relationship, that is, the labeled data (the second to-be-labeled data and the first (3) Data to be labeled) is not necessarily used for training the task model, for example, it may only be used for display and storage purposes, that is, its use is not limited.

As mentioned above when describing step S120, in order to obtain the labeling label of the second data to be labelled, the second data to be labelled may be displayed on the user labeling interface, and the user labels. Since the second data to be labeled has not been labeled before, in an implementation manner, the second data to be labeled with an unknown label can be displayed on the user labeling interface, and the labeling label is completely given by the user. This implementation method is relatively simple , but the workload of the user is relatively large.

Therefore, in another implementation manner, the prediction label of the second to-be-labeled data may be determined first according to the inference result of the second to-be-labeled data, and not only the second to-be-labeled data but also the second to-be-labeled data should be displayed on the user labeling interface. its predicted label.

Among them, the predicted label is a label obtained based on the inference result of the second data to be labeled and combined with specific rules. The predicted label is no different in form from the previously described label, except that the predicted label is not a manually labeled label. There are many specific rules for obtaining predicted labels. Examples will be given later, which will not be expanded here, but it should be noted that the specific rules for calculating predicted labels and the specific rules for calculating inference labels may be the same or different. Second, the prediction label and the inference label of the data to be labeled may be the same or different.

The predicted label can represent the labeling result predicted according to the inference result of the labeling model. If the inference result of the labeling model is accurate enough, the predicted label may be very close to the labeled label, so the predicted label can be used as an important reference for users to manually label. , so that the user can directly confirm the prediction label displayed on the user labeling interface as the labeling label without modifying it, or only need to slightly modify the prediction label displayed on the user labeling interface (modify it to the labeling label that the user deems appropriate), which will greatly Some predicted labels are confirmed as label labels, which significantly improves the labeling efficiency of users.

Optionally, if the labeling model is a classification model, the prediction label of the second data to be labelled is the prediction class of the second data to be labelled. At this time, it can be classified and displayed on the user labeling interface according to the prediction class of the second data to be labelled. The second data to be labeled.

2 (A), the labeling model is an emotion classification model, the second data to be labelled is a face image, and the reasoning result of the second data to be labelled may be that the face in the image belongs to each emotion category (for example, happy, sad) , surprise, etc.), the predicted category of the second to-be-labeled data may be the emotional category to which the image belongs, such as happy, sad, surprised, etc.

In Figure 2(A), the face images whose predicted category is happy are displayed in one row, the face images whose predicted category is sad are displayed in another row, and so on. After the face images are classified and displayed, it is easy for users to confirm the emotional category to which each face image belongs, and it is also easier to pick out images with wrong predicted categories and change their categories, because in a large number of similar data with commonality, a small amount of The heterogeneous data will be more obvious. For example, for the second face image in the happy category, the bending direction of the corners of the mouth is significantly different from other face images, and the user can quickly confirm that the predicted category is wrong.

Further, the second data to be labeled and its prediction category can be displayed in the first area of the user labeling interface, and the user can perform a data selection operation in the first area. If a second data to be labeled is selected by the user, it indicates that the The predicted category of the second to-be-labeled data is not recognized by the user.

For example, referring to FIG. 2(B), the upper part of the user annotation interface is the first area. If the user uses a PC to make annotations, the data selection operation can be a click operation of the left mouse button. If there is an error in the predicted category of the second face image, you can move the cursor on the screen to the face image and click the left mouse button. At this time, the word unknown category will appear below the face image, indicating the category of the image. Pending.

In response to the data selection operation triggered in the first area, the user annotation interface may display the selected second data to be annotated in the second area in the interface, and display the selected second data to be annotated in the second area The candidate category for the user to select when manually labeling.

For example, referring to Figure 2(B), the lower part of the user annotation interface is the second area, the second face image in the happy category is displayed here, and all possible emotion categories are displayed on the right side for The user selects a label for the callout. In particular, the unknown category can also be used as a candidate category. If the user selects the unknown category, it means that the labeling label of the face image is unknown, or it can also be considered that it has not been labeled.

It should be understood that the layout of the first area and the second area in the user annotation interface is not limited to the manner shown in FIG. 2(A) and FIG. 2(B), and even the first area and the second area may overlap.

The beneficial effect of labeling according to the above method is that in the first area, the user only needs to pay attention to whether the prediction category of a second piece of data to be labelled is correct, that is, only need to make a judgment (judging "yes" or "no"), and There is no need to make a selection (choose the correct callout label), and in the second area, the user simply makes a selection without judgment. Therefore, the user can first select all the second to-be-labeled data whose predicted category is wrong in the first area, and then uniformly modify the category in the second area (change to the category in the label), that is, in each area, Users only need to operate according to a single labeling logic, without switching back and forth between the two thinking logics. Experiments show that this method can significantly improve the user's labeling speed.

It should be understood that it is not necessary to divide the first area and the second area on the labeling interface. For example, only the first area may be set. After the user selects a piece of second data to be labelled in the first area, it will be displayed directly in the first area. The category is changed at the location, but at this time, the user needs to switch back and forth between the two thinking logics of judgment and selection, and the labeling efficiency may be reduced.

In addition, even if the labeling model is a classification model, on the user labeling interface, it is not necessary to display the second data to be labeled according to the predicted category. For example, the second data to be labeled of all categories can be mixed and displayed. It suffices to display the predicted category at the display position of the labeled data.

Further, the inventor's research found that when a user is annotating, it is likely to judge whether the predicted label is correct from the part of the second to-be-labeled data, or to determine what the label is. For example, in Figure 2(B), the user may judge the emotion category mainly from the bending direction of the mouth corners of the face image (bending up is happy, bending down is sad), but not so much about the rest of the face image. focus on. Therefore, if the user annotation interface can focus the displayed content on the key part of the user's attention (that is, the position where the user's attention is focused) according to the user's needs when displaying the second to-be-annotated data, it is more beneficial for the user to quickly and accurately Gives the callout label.

For the case where the second data to be labeled is an image, the possible methods of displaying data based on the user's attention are as follows:

The second to-be-labeled data and its predicted label are displayed in the first area of the user annotation interface, and the reference image is displayed in the third area of the user annotation interface. The reference image is an image selected from all the second data to be marked currently to be marked, which can be selected by the user or automatically selected by the program (for example, the first image is automatically selected).

The user may perform an image transformation operation on the reference image in the third area, the purpose of which may be to display the user's attention-focused area closely related to the annotation result in the reference image in a more prominent manner. Wherein, the image transformation operation includes at least one operation of image translation, image rotation, image scaling, and frame selection of a local area of the image.

In response to the image transformation operation triggered in the third area, the user annotation interface (or the background) may record the image transformation operation. Since each operation corresponds to specific operation parameters, logging an operation is actually recording these operation parameters. For example, the translation amount can be recorded for the image translation, the rotation angle can be recorded for the image rotation, the zoom factor can be recorded for the image zoom, the position of the local area can be recorded for the frame selection of the partial area of the image, and so on.

Finally, the recorded image transformation operation may be applied to each image in the second to-be-annotated data, and the second to-be-annotated data after the image transformation operation is refreshed and displayed in the first area. Therefore, each image in the second data to be annotated is also the same as the reference image, and the area in which the user's attention is concentrated is displayed in a more prominent manner, which is convenient for the user to observe and annotate.

For example, referring to FIG. 3(A) , the left part of the user annotation interface is the third area, the upper half of the right part is the first area, and the lower half of the right part is the second area. There is a fixed-size window (grey) in the third area, where the first face image in the happy category is selected as the reference image and displayed in this window.

After that, the user can perform at least one operation of image translation, rotation, zoom, and frame selection of a local area in the fixed window, and the operation parameters will be recorded, and the part beyond the fixed window size will not be displayed during the transformation process. Referring to Figure 3 (B), the user has performed a translation operation to move the face in the image to the center of the window, and the offset of the center of the image relative to the center of the window can be recorded as a translation parameter at this time; the user also performed a zoom-in operation, The face in the image fills the entire window, and the magnification of the image can be recorded as a zoom parameter; the user also selects the mouth area that needs to be focused on when annotating the image, as shown by the thick dotted box, at this time The position of the mouth area can be recorded as a parameter for the local area selection.

Finally, a transformation matrix can be generated according to the recorded translation parameters and zoom parameters, and the transformation matrix can be applied to each face image in the second to-be-annotated data, so that the face can be moved to the center of the screen and enlarged. Then, according to the recorded position of the mouth area, frame the same position from each face, and further enlarge the frame selection until it fills the original display area of the second to-be-annotated image (it can also be a fixed-size window). ), the final refresh display effect is shown in Figure 3(B). It can be seen that in the first area, the face image has been replaced by a partial enlarged image of its mouth. According to these enlarged images, the user can more clearly see the bending direction of the corners of the mouth, so that the wrong prediction label can be quickly selected. , or give the correct callout label.

Note that, first, when the image transformation operation is applied to each image in the second to-be-labeled data, the effect is not necessarily exactly the same as that applied to the reference image. For example, in Fig. 3(B), when the operation of selecting a local area of the image by a frame is applied to the reference image, only the position of the mouth area is given, and when it is applied to each image in the second to-be-annotated data When the image is displayed, not only the position of the mouth area should be determined, but also the mouth area should be enlarged and displayed.

Second, the layout of the first area and the third area in the user annotation interface is not limited to the method shown in FIG. 3(A) and FIG. 3(B), and even the first area and the third area may overlap.

Third, although the first area, the second area and the third area are shown in FIG. 3(A) and FIG. 3(B), in the above scheme of data display based on user attention, whether to set the second area Regions are optional.

Fourth, the above solution for displaying data based on user attention has nothing to do with whether the labeling model is a classification model. For example, if the labeling model is a target detection model, this solution can also be applied to facilitate the user to more accurately label the location of the target.

On the basis of the above embodiment, the method for estimating whether the labeling accuracy rate of the third data to be labelled in step S130 is up to the standard will be described below. In an implementation manner, step S130 may be performed according to the following process:

According to the label of the second to-be-labeled data, the fourth to-be-labeled data is determined from the first to-be-labeled data in combination with specific rules; if the fourth to-be-labeled data includes the third to-be-labeled data that has not The number of correctly labeled second data to be labeled in the labeled data, and it is estimated whether the labeling accuracy of the third data to be labeled meets the standard.

Among them, the fourth data to be labeled may include the correctly labeled second data to be labeled, or the third data to be labeled that has not been labeled, or both types of data to be labeled. Design related. The correct labeling of the second data to be labeled means that the labeling label of the second data to be labeled is the same as the reasoning label of the second data to be labeled determined according to the inference result of the second data to be labeled. The meanings of the reasoning label and the labeling label are in The previous article has been explained and will not be repeated.

Consider first the case where the fourth data to be labeled includes the correctly labeled second data to be labeled and the third data to be labeled that has not been labeled. At this time, the fourth data to be labeled is equivalent to the first data to be labeled. A range is defined, and the inference labels of some data to be labeled within this range have been manually confirmed as correct. Since the rules for determining the inference labels of each data to be labeled are unified, the correct data is labeled according to the range. number, the accuracy of the inference labels of the remaining data to be labeled can be estimated.

If the fourth data to be labeled only contains the third data to be labeled that has not been labeled, it is equivalent to the situation where the number of correctly labeled second data to be labeled is 0 in the fourth data to be labeled. 2. If the number of data to be labeled is not 0, it will be processed uniformly. If the fourth data to be labeled only includes the correctly labeled second data to be labeled, it means that the fourth batch of data to be labeled has been labeled at this time, and there is no data that needs to be automatically labeled. It should be understood that the rule for determining the fourth to-be-labeled data should not make the fourth to-be-labeled data always include only the third to-be-labeled data that has not been labeled, because the labeling accuracy is bound to be difficult to achieve, nor should it always only contain Label the correct second data to be labeled, because this is equivalent to always not automatically labeling.

Further, when the above implementation manner is adopted in step S130, the method in FIG. 1 can also be implemented as the iterative process in FIG. 4:

Step S210: Obtain the first to-be-labeled data and its inference result.

Step S210 is similar to step S110 and will not be repeated. Note that although FIG. 4 is an iterative process as a whole, step S210 does not participate in the iteration. Steps S220-S260 are an iterative implementation of steps S120-S140, or each round of iteration logically includes steps S220-S260 (of course, not all of steps S220-S260 are necessarily executed in each round of iteration).

Step S220: Sampling the second to-be-labeled data provided to the user for labeling in the current iteration from the fourth to-be-labeled data.

Wherein, the fourth data to be labeled in step S220 may be considered to be the fourth data to be labeled at the end of the previous iteration, or the fourth data to be labeled at the beginning of the current iteration, and in each iteration , the fourth to-be-labeled data may, but not necessarily, be updated (see step S230 ). For the first round of iteration, since there is no previous round, the fourth to-be-labeled data at the beginning of the first-round iteration is the first to-be-labeled data.

The method of sampling the second to-be-labeled data from the fourth to-be-labeled data is not limited. The second to-be-labeled data must be re-sampled in each iteration, and optionally, for those samples that have been sampled in previous rounds The second to-be-labeled data, since it has been labeled by the user in the previous iteration round, can no longer be sampled repeatedly, that is, it can be limited that the second to-be-labeled data for each sampling is from the first iteration after the previous iteration. The fourth data to be labeled is sampled from the third data to be labeled that has not been labeled, so that repeated labeling of the data can be avoided.

Step S230: Obtain the labeling label of the second data to be labelled marked by the user, and filter the fourth data to be labelled according to the labeling label of the second data to be labelled.

The sampled second data to be labeled can be sent to the user for labeling, and the labeling label thereof can be obtained. For the specific implementation method, refer to the foregoing description, which will not be repeated.

If the fourth data to be labeled only includes the correctly labeled second data to be labeled and/or the third data to be labeled that has not been labeled, then after the second data to be labeled in this round has been labeled, the current fourth data to be labeled The labeled data may no longer meet this requirement, so the fourth to-be-labeled data needs to be screened according to specific rules to obtain new fourth to-be-labeled data, and the new fourth to-be-labeled data only includes the correctly labeled second to-be-labeled data and/or the third data to be labeled that has not been labeled, so that the labeling accuracy of the third data to be labeled can continue to be effectively evaluated.

As for the filtered data to be labeled, it includes the wrongly labeled second data to be labeled, and may also include some unlabeled data to be labeled. The processing method is not limited. For example, the wrongly labeled second data to be labeled is only It is said that there is an error in its inference label, but its labeling label is not wrong (the default manual labeling is accurate), so it can be considered that it has been labelled, and for the data to be labelled that has not been labelled, you can re-inference after the labeling model is updated. , relabel, etc.

Note that, in the steps after step S230, any reference to the fourth data to be labeled refers to the new fourth data to be labeled after screening, and for simplicity, it is still simply referred to as the fourth data to be labeled.

Step S240: Determine whether the fourth data to be labeled includes the third data to be labeled that has not been labeled.

If the fourth data to be labeled includes the third data to be labeled that has not been labeled, step S250 is continued. If the fourth data to be labeled does not contain the third data to be labeled that has not been labeled, that is, the fourth data to be labeled only contains the correctly labeled second data to be labeled, automatic labeling cannot be performed at this time. Batch the labeling of the fourth data to be labelled. And since the part of the first to-be-labeled data except the fourth to-be-labeled data is generated by the screening operation in step S230, if this part of the data is not to be labeled temporarily, at this time, the current batch of A label for the data to be labelled.

Step S250: According to the number of correctly labeled second data to be labeled in the fourth data to be labeled, it is estimated whether the labeling accuracy of the third data to be labeled meets the standard.

If the labeling accuracy of the third to-be-labeled data has reached the standard, proceed to step S260; if the labeling accuracy of the third to-be-labeled data has not yet met the standard, the next round of iteration can be started, that is, jump to step S220 to continue execution. Note that the correct second data to be labeled in step S250 is not limited to the second data to be labeled labeled in this iteration (that is, not limited to the second data to be labeled sampled in step S220), and the data labeled in previous iterations The second data to be labeled should be taken into consideration.

Step S260: Determine the inference label of the third data to be labeled as the labeling label of the third data to be labeled.

Step S260 is similar to step S140 and will not be repeated. After the execution of step S260, the labeling of the fourth batch of data to be labelled also ends, and if the part of the first data to be labelled except for the fourth data to be labelled will not be labelled for the time being, it can also end at this time. Annotation of the first batch of data to be labeled at present.

Note that although step S260 is nominally an iterative step (for example, this step can be included in the code of the iterative part), in fact, in the entire iterative process, this step is executed at most once when the conditions are met, and after the execution is completed, the will jump out of the iteration.

The key to the above iterative process is to continuously screen the fourth to-be-labeled data, so that the estimated accuracy of the third to-be-labeled data in the fourth to-be-labeled data becomes higher and higher until it reaches the standard (of course, it may eventually fail to meet the standard) . Since the flow in FIG. 4 is only an implementation manner of the method in FIG. 1 , when describing the steps in FIG. 4 , reference may be made to the description of the relevant steps in FIG. 1 for the parts that are not mentioned.

In the following, two implementation manners of the iterative steps S220 to S260 are illustrated by taking the case where the labeling model is a classification model as an example. At this time, the reasoning result of the first data to be labeled in step S210 includes the probability that the first data to be labeled belongs to each category.

way 1

Before executing the first round of iterations, a label candidate pool corresponding to each category is created according to the inference result of the first data to be labelled. A label candidate pool can be understood as a dataset or a storage space. For example, if the classification results of the annotation model have nc categories, you need to create nc annotation candidate pools. The annotation candidate pool i corresponds to category i, where i is the category number. For example, i can take any integer from 0 to nc-1.

Wherein, each labeling candidate pool includes the first to-be-labeled data and its inference result, the probability that the first to-be-labeled number belongs to the category corresponding to the labeling candidate pool. For example, the labeling candidate pool i includes the first to-be-labeled data and the probability that the first to-be-labeled number belongs to the category i among the first to-be-labeled data and its inference result.

Note that the so-called labeling candidate pool contains the first data to be labelled, it does not mean that a copy of the first data to be labelled must be copied for each labelling candidate pool, or just the identifier of the first data to be labelled (for example, each The number of the first to-be-labeled data) is saved in the labeling candidate pool. Since the corresponding first to-be-labeled data can be found according to the identifier of the first to-be-labeled data, it is equivalent to saving the first to-be-labeled data in the labeling candidate pool. A copy of the annotation data. For other data pools mentioned later, if there are different data pools storing the same data, it can be understood similarly.

After creating an annotation candidate pool, each iteration can include the following steps:

Step A1: Sampling the second to-be-labeled data provided to the user for labeling in this round of iterations from the fourth to-be-labeled data in all the labeling candidate pools.

In step A1, the fourth to-be-labeled data in each labeling candidate pool can be regarded as either the fourth to-be-labeled data in the labeling candidate pool at the end of the previous iteration, or the labeling at the beginning of the current iteration The fourth data to be labeled in the candidate pool, in each round of iteration, the fourth data to be labeled in each labeled candidate pool may, but not necessarily, be updated (see step C1). For the first round of iteration, since there is no previous round, at the beginning of the first round of iteration, the fourth to-be-labeled data in each labeling candidate pool is the first to-be-labeled data in the labeling candidate pool.

The method of sampling the second to-be-labeled data from the fourth to-be-labeled data in all the labeling candidate pools is not limited, and the second to-be-labeled data needs to be resampled in each iteration. For example, a union of the fourth to-be-labeled data in all the labeling candidate pools may be taken to remove duplicate to-be-labeled data, and then the second to-be-labeled data to be labeled in this round is generated by sampling from the union. Further, it is also possible to take a union of the third unlabeled data to be labeled in the fourth to-be-labeled data in all the labeling candidate pools, and then sample the second to-be-labeled data to be labeled in this round from the union, so as to avoid Data that has already been labeled is resampled.

Step B1: Obtain the labeling category of the second data to be labelled marked by the user.

Since the classification problem is aimed at at this time, the labeling category of the second data to be labelled is the labeling label of the second data to be labelled mentioned in step S230.

For each annotation candidate pool, perform the following steps, taking an arbitrary annotation candidate pool i as an example:

Step C1: For each piece of second data to be labeled obtained in step A1, taking any second data to be labeled j as an example, if the labeling category of the second data to be labeled j and the category corresponding to the labeling candidate pool i (that is, the category i ) is different, and the second to-be-labeled data j is included in the fourth to-be-labeled data in the labeling candidate pool i, then the fourth to-be-labeled data in the labeling candidate pool i is reduced to only include the data to be labeled that meets the following conditions: The probability of belonging to category i in the inference result is greater than the probability of belonging to category i in the inference result of the second to-be-labeled data j.

The reduction in step C1 is an implementation of the screening in step 230 . Category i is the inference label of all the first to-be-labeled data in the label candidate pool i, so if the labeling class of the second to-be-labeled data j is different from the class i, it means that the inference label of the second to-be-labeled data j is wrong, According to the above definition of the fourth data to be labeled, it may only include the correctly labeled second data to be labeled and/or the third data to be labeled that has not been labeled, so the second data to be labeled j should be changed from the current fourth data to be labeled. To be removed from the labeled data.

In addition, for each piece of fourth to-be-labeled data (the second to-be-labeled data itself is also the fourth to-be-labeled data) in the labeling candidate pool i except for the wrongly labeled second data to be labeled j, if the reasoning result contains The probability of belonging to category i is not greater than the probability of belonging to category i in the inference result of the second to-be-labeled data j, indicating that the inference label of the fourth to-be-labeled data can only be equal, or even worse than that of the second to-be-labeled data j The reasoning label (already an error label) is reliable. If it is kept in the fourth to-be-labeled data, it will make it difficult to further improve the labeling accuracy of the third to-be-labeled data, so such fourth to-be-labeled data can also be obtained from the current is removed from the fourth to-be-labeled data.

It is not difficult to see that, except for the removed data to be labeled, the remaining data in the fourth data to be labeled satisfies the conditions mentioned in step C1.

FIG. 5 shows the reduction principle of the fourth to-be-labeled data. The thin line arrows in Figure 5 represent the correctly labeled second data to be labeled (not limited to the second data to be labeled in this round of labeling), the thick line arrows represent the wrongly labeled second data to be labeled, and the unlabeled third data to be labeled Data is not shown.

Referring to FIG. 5, it is assumed that the second data to be labeled j is the second data to be labeled that is wrongly labeled first from the left, and the fourth data to be labeled is located on the left side of the second data to be labeled j, including a number of correctly labeled second data. The data to be labeled and the third data to be labeled not shown, because the probability of belonging to category i in the inference result is greater than the probability of belonging to category i in the inference result of the second to-be-labeled data j (the bottom of Figure 5 belongs to category i). The change trend of the probability), so it is retained after the reduction, the second to-be-labeled data j itself, and the fourth to-be-labeled data located on the right side of the second to-be-labeled data j, including a wrongly labeled second to-be-labeled data, Several correctly labeled second data to be labeled and some unlabeled data to be labeled that are not shown, since the probability of belonging to category i in the inference result is not greater than the probability of belonging to category i in the inference result of the second data to be labeled j, Therefore, it is removed after reduction, so that the second half of the rectangular box representing the fourth to-be-labeled data becomes a dotted line.

Note that according to the reduction method described above, when removing the wrongly labeled second data j to be labeled, the second data to be labeled k may also be removed "by the way", so in step C1, it is also required to perform Before reducing, first confirm whether the second to-be-labeled data j is still included in the fourth to-be-labeled data in the labeling candidate pool i to avoid repeated removal.

Referring to FIG. 5 , it is assumed that the second to-be-labeled data k to be processed is the second to-be-labeled data from the left that is wrongly labeled, because the second to-be-labeled data k has been removed, so the second to-be-labeled data k is not currently included in the fourth to-be-labeled data, and it will not cause the fourth to-be-labeled data to be reduced.

In addition, the fourth to-be-labeled data that is removed during the reduction process does not necessarily have to be deleted from the labeled data pool i, and it can also continue to remain in the labeled data pool i.

In this round of iterative steps after step C1, any reference to the fourth data to be labeled refers to the reduced fourth data to be labeled, which is still simply referred to as the fourth data to be labeled for simplicity.

Step D1: Determine whether the fourth data to be labeled in the labeled data pool i includes the third data to be labeled that has not been labeled.

If the fourth to-be-labeled data includes the unlabeled third to-be-labeled data, step E1 is continued. If the fourth data to be labeled does not contain the third data to be labeled that has not been labeled, that is, the fourth data to be labeled only contains the correctly labeled second data to be labeled, automatic labeling cannot be performed at this time, so that the iterative process can be jumped out. End the labeling of labeling data pool i.

Step E1: According to the number of correctly labeled second data to be labeled in the fourth data to be labeled in the labeled data pool i, it is estimated whether the labeling accuracy of the third data to be labeled meets the standard.

Here is a judgment formula for estimating whether the labeling accuracy meets the standard: 1/(1+w ² /Ci)>acc.

Wherein, the left side of the greater than sign is the estimation formula of the labeling accuracy, and the acc on the right is the target accuracy rate mentioned in the description of step S130. If the left side is greater than the right side, it means that the labeling accuracy rate meets the standard. In the expression on the left, Ci is the number of correctly labeled second to-be-labeled data in the fourth to-be-labeled data of the labeled candidate pool i (note that the Ci second to-be-labeled data is not necessarily in this round, If the second to-be-labeled data with correct standards in previous iteration rounds still belongs to the fourth to-be-labeled data in this round, it will also contribute to Ci), w is a preset value related to confidence, and the higher the value of w The higher the confidence of the estimated accuracy, for example, when w=2, the confidence is 95%.

It should be understood that other formulas may also be used to estimate the labeling accuracy, which is not limited in this application. In addition, judging whether the labeling accuracy meets the standard does not mean that it is necessary to explicitly calculate the labeling accuracy estimate. For example, transform the above formula to get Ci>w ² /(1/acc-1), that is, use Comparison of Ci and a certain threshold value can also be used for judgment.

If the labeling accuracy of the third data to be labeled has reached the standard, continue to step F1; if the labeling accuracy of the third data to be labeled has not met the standard, the next round of iteration can be started, that is, jump to step A1 to continue execution.

Step F1: Determine the category i as the labeling category of the third data to be labelled.

Wherein, according to the description in step C1, category i is the reasoning label of the third data to be labeled, and the labeling category of the third data to be labeled is the labeling label of the third data to be labeled mentioned in step S260. After the execution of step F1, the iterative process can be jumped out, and the labeling of the labeling data pool i can be ended.

Optionally, when creating an annotation candidate pool in method 1, the first data to be annotated in each annotation candidate pool may also be sorted in descending order according to the probability of belonging to the category corresponding to the annotation candidate pool in the inference result. At this time, step C1 can be implemented as follows:

For each piece of second data to be labeled obtained in step A1, if the labeling category of the second data to be labeled j is different from the class i, and the sorting index of the second data to be labeled j is not greater than the fourth data to be labeled in the labeling candidate pool i The largest sorting index of the data, the fourth to-be-labeled data in the labeling candidate pool i is reduced to only include the to-be-labeled data that satisfies the following conditions: its sorting index is smaller than the sorting index of the second to-be-labeled data j.

Among them, the sorting index of the data to be labeled may refer to its sorting sequence number in the labeling candidate pool i. Since the first data to be labeled in the labeling candidate pool i is sorted in descending order according to the probability corresponding to the category i in the inference result, so If the sorting index of a fourth piece of data to be labeled is smaller than the sorting index of the second data to be labeled j, it indicates that the probability of belonging to category i in the inference result is greater than the probability of belonging to category i in the inference result of the second data to be labeled j, that is, It is the same as the condition in step C1.

It is not difficult to see that, by reducing in the above manner, the distribution range of the sorting index of the fourth to-be-labeled data is reduced in one direction. For example, if the sorting index starts counting up from 0, at the beginning, the distribution range of the sorting index of the fourth data to be labeled is [0, m-1], where m is the number of the first data to be labeled, and after one iteration , the distribution range of the sorting index of the fourth data to be labeled is [0, m1-1], m1≤m, after two rounds of iterations, the distribution range of the sorting index of the fourth data to be labeled is [0, m2-1], m2≤m1, and so on.

Further, in order to avoid repeated removal of data, step C1 also requires that the condition "the second to-be-labeled data j is included in the fourth to-be-labeled data in the labeling candidate pool i" should be satisfied before the reduction, and the labeling candidate pool is based on the probability In the case of descending sorting, this condition can be equivalent to that the sorting index of the second to-be-labeled data j is not greater than the maximum sorting index of the fourth to-be-labeled data in the labeling candidate pool i.

It can be seen that if the data in the labeling candidate pool is sorted first, when performing the fourth reduction of the data to be labelled, it is only necessary to perform a simple comparison of the sorting index instead of the comparison of the probability values, thus improving the processing efficiency.

Assuming that the sorting index of the data to be labeled starts counting up from 0, and Li represents the number of the fourth data to be labeled in the labeling candidate pool i (the maximum sorting index of the fourth data to be labeled is Li-1 at this time), then the above steps The implementation of C1 can be further simplified and described as:

If the labeling category of the second to-be-labeled data j is different from the category i, the quantity Li of the fourth to-be-labeled data in the labeling candidate pool i is reduced to min(index(j,i),Li).

Among them, index(j,i) represents the sorting index of the second data to be labeled j in the labeling candidate pool i. The meaning of this formula is briefly explained below:

As can be seen from the foregoing, since the distribution range of the sorting index of the fourth data to be labeled is reduced in one direction during the reduction process, the number Li of the fourth data to be labeled is reduced to min(index(j,i),Li), The distribution range of the sorting index equivalent to the fourth to-be-labeled data is reduced to [0, min(index(j,i),Li)-1]. Obviously, any number in the interval [0, min(index(j,i),Li)-1] is smaller than index(j,i), that is, the sorted index of the reduced fourth data to be labeled is smaller than the second index to be labeled Sort index of label data j.

Moreover, Li will change only when index(j,i)<Li or index(j,i)≤Li-1, otherwise Li still maintains the original value, that is, only when the second to-be-labeled data j is The reduction is possible only when the sorting index is not greater than the largest sorting index Li-1 of the fourth to-be-labeled data in the labeling candidate pool i.

Optionally, before step B1 is performed in Mode 1, the predicted category (ie, predicted label) of the second data to be labeled may also be determined according to the inference result of the second data to be labeled, and the second data to be labeled may be displayed on the user labeling interface. data and its predicted categories to improve the efficiency of user annotation.

If the first to-be-labeled data in the labeling candidate pool is sorted in descending order according to the probability of belonging to the category corresponding to the labeling candidate pool in its inference result, then in each round of iteration, the predicted class of the second to-be-labeled data can be It is determined in the following manner, taking any second piece of data j to be labeled as an example:

First, obtain the ranking index of the second to-be-labeled data j in each labeling candidate pool, for example, index(j, i), and i is taken from 0 to nc-1, that is, a total of nc ranking indexes are obtained.

Then, each sorting index is normalized by the number of the fourth data to be labeled in the corresponding labeling candidate pool, to obtain a normalized sorting index. For example, index(j,i) can be index(j,i)/Li after normalization. Note that since the step of giving the predicted label is performed before step B1, the reduction of the fourth to-be-labeled data in this round of iteration has not been performed, so Li should be the fourth to-be-labeled data at the beginning of this iteration. quantity.

Finally, the category corresponding to the smallest sorting index among all the normalized sorting indexes is determined as the predicted category of the second to-be-labeled data j. For example, take argmin_i(index(j,i)/Li) as the predicted category of the second data j to be labeled. Among them, argmin_i represents the i value when the value in the following parentheses is calculated to take the minimum value.

The prediction category determined in this way, because the sorting index is normalized by the number of the fourth to-be-labeled data in the labeling candidate pool, which is equivalent to weakening the difference in the distribution of data of different categories, which is conducive to improving the prediction. The accuracy of the category, thereby improving the efficiency of manual annotation.

way 2

There is no need to create an annotation candidate pool in method 2, or it can be considered that there is only one annotation candidate pool in method 2, and all the first to-be-labeled data are in this annotation candidate pool. Each iteration can include the following steps:

Step A2: Sampling the second to-be-labeled data provided to the user for labeling in this round of iterations from the fourth to-be-labeled data.

The fourth data to be labeled at the beginning of the first iteration is the first data to be labeled. Step A2 is similar to step S220 and step A1 and will not be repeated.

Step B2: Obtain the labeling category of the second data to be labelled marked by the user.

Since the classification problem is aimed at at this time, the labeling category of the second data to be labelled is the labeling label of the second data to be labelled mentioned in step S230.

Step C2: For each piece of second data to be labeled obtained in step A2, taking any second data to be labeled j as an example, if the labeling category of the second data to be labeled j is different from the category corresponding to the maximum probability in its inference result , and the second to-be-labeled data j is included in the fourth to-be-labeled data, then the fourth to-be-labeled data is reduced to only include the to-be-labeled data that satisfies the following conditions: the maximum probability in the inference result is greater than the second to-be-labeled data j The maximum probability in the inference result of .

The reduction in step C2 is an implementation of the screening in step 230 . The reasoning label of all the first data to be labeled is the category corresponding to the maximum value of the probabilities of the first data to be labeled belonging to each category (referred to as the maximum probability in the reasoning result) in the reasoning result. Therefore, if the labeling category of the second to-be-labeled data j is different from the class corresponding to the maximum probability in the inference result, it indicates that the inference label of the second to-be-labeled data j is wrong, and according to the previous definition of the fourth to-be-labeled data , which may only include the correctly labeled second data to be labeled and/or the third data to be labeled that has not been labeled, so the second data to be labeled j should be removed from the current fourth data to be labeled.

In addition, for each piece of fourth to-be-labeled data except the wrongly labeled second to-be-labeled data j (the second to-be-labeled data itself is also the fourth to-be-labeled data), if the maximum probability in the inference result is not greater than the second The maximum probability in the inference result of the data j to be labeled indicates that the inference label of the fourth data to be labeled can only be equal, or even less reliable than the inference label of the second data j to be labeled (already an error label). It remains in the fourth to-be-labeled data, which will make it difficult to further improve the estimation of the labeling accuracy of the third to-be-labeled data, so that such fourth to-be-labeled data can also be removed from the current fourth to-be-labeled data.

It is not difficult to see that, except for the removed data to be labeled, the remaining data in the fourth data to be labeled satisfies the conditions mentioned in step C2.

In this round of iterative steps after step C2, any reference to the fourth data to be labeled refers to the reduced fourth data to be labeled, which is still simply referred to as the fourth data to be labeled for simplicity.

Step D2: Determine whether the fourth data to be labeled includes the third data to be labeled that has not been labeled.

If the fourth to-be-labeled data includes the unlabeled third to-be-labeled data, step E2 is continued. If the fourth data to be labeled does not contain the third data to be labeled that has not been labeled, that is, the fourth data to be labeled only contains the correctly labeled second data to be labeled, automatic labeling cannot be performed at this time, so that the iterative process can be jumped out. End the labeling of the first data to be labelled.

Step E2: According to the number of correctly labeled second data to be labeled in the fourth data to be labeled, it is estimated whether the labeling accuracy of the third data to be labeled meets the standard.

The judgment formula for estimating whether the labeling accuracy meets the standard can be: 1/(1+w ² /C)>acc. Among them, C is the number of correctly labeled second data to be labeled in the fourth data to be labeled (note that the C second data to be labeled are not necessarily in this round, and the second data to be labeled with correct standards in previous iteration rounds) If the data to be labeled still belongs to the fourth data to be labeled in this round, it will also contribute to C), w is a preset value related to confidence, and acc is the target accuracy rate mentioned in step S130. It is not difficult to see that this formula is similar to the previous formula 1/(1+w ² /Ci)>acc, except that Ci is replaced by C, because method 2 can be regarded as only one annotation candidate pool.

It should be understood that other formulas may also be used to determine whether the labeling accuracy rate meets the standard, which is not limited in this application.

If the labeling accuracy of the third to-be-labeled data has reached the standard, proceed to step F2; if the labeling accuracy of the third to-be-labeled data has not met the standard, the next round of iteration can be started, that is, jump to step A2 to continue execution.

Step F2: Determine the category corresponding to the maximum probability in the inference result of the third to-be-labeled data as the labeling class of the third to-be-labeled data.

Wherein, according to the description in step C2, the category corresponding to the maximum probability in the inference result of the third data to be labeled is the inference label of the third data to be labeled, and the labeling class of the third data to be labeled is the one mentioned in step S260 The third label of the data to be labelled. After the execution of step F2, the iterative process can be jumped out, and the labeling of the first data to be labelled is ended.

Optionally, before starting the iteration in Manner 2, the first data to be labeled may be sorted in descending order according to the maximum probability in the inference result that the first data to be labeled belong to each category. At this time, step C2 can be implemented as follows:

For each piece of second data to be labeled obtained in step A2, if the labeling category of the second data to be labeled j is different from the class corresponding to the maximum probability in the inference result, and the sorting index of the second data to be labeled j is not greater than the first 4. The largest sorting index in the data to be labeled, the fourth data to be labeled is reduced to only include data to be labeled that meets the following conditions: its sorting index is smaller than the sorting index of the second data to be labeled j.

The sorting index of the data to be marked may refer to its sorting sequence number in the first data to be marked. Since the first data to be marked is sorted in descending order according to the maximum probability in the inference result, if a fourth The sorting index of j is smaller than the sorting index of the second to-be-labeled data j, indicating that the maximum probability in the inference result is greater than the maximum probability in the inference result of the second to-be-labeled data j, that is, it is consistent with the condition in step C2. It is not difficult to see that, by reducing in the above manner, the distribution range of the sorting index of the fourth to-be-labeled data is reduced in one direction.

Further, in order to avoid repeated removal of data, step C2 also requires that the condition "the second data to be labeled j is included in the fourth data to be labeled" should be satisfied before the reduction, and the first data to be labeled is sorted in descending order by probability. In the case of , this condition can be equivalent to that the sorting index of the second to-be-labeled data j is not greater than the maximum sorting index of the fourth to-be-labeled data.

It can be seen that if the first to-be-labeled data is sorted first, when performing the reduction of the fourth to-be-labeled data, it is only necessary to perform a simple comparison of sorting indexes instead of probability values, thus improving processing efficiency.

Assuming that the sorting index of the data to be marked starts counting up from 0, and L represents the number of the fourth data to be marked (at this time, the maximum sorting index of the fourth data to be marked is L-1), then the above implementation of step C2 can also be A further simplified description is:

If the labeling category of the second to-be-labeled data j is different from the category corresponding to the maximum probability in the inference result, the quantity L of the fourth to-be-labeled data is reduced to min(index(j), L). Among them, index(j) represents the sorting index of the second data to be labeled j in the first data to be labeled, and the formula can be understood by referring to the formula min(index(j,i),Li) in Mode 1, and it is not Repeat the elaboration.

Optionally, before step B2 is performed in Mode 2, the predicted category (ie, the predicted label) of the second data to be labeled can also be determined according to the inference result of the second data to be labeled, and the second data to be labeled can be displayed on the user labeling interface. data and its predicted categories to improve the efficiency of user annotation.

If the first to-be-labeled data is sorted in descending order according to the highest probability in its inference result, the prediction category of the second to-be-labeled data can be determined in the following manner (regardless of the iteration round), using any second to-be-labeled data j for example:

The category corresponding to the maximum probability in the inference result of the second data j to be labeled is determined as the predicted category of the second data j to be labeled. Determining the prediction category in this way is simple and fast.

A simple comparison between method 1 and method 2, method 2 is relatively simple in implementation, and there is no need to create multiple annotation candidate pools. The advantage of method 1 is that there is an annotation candidate pool for each category of data, and the number of fourth to-be-annotated data in each annotation candidate pool is different, which is equivalent to adaptively setting each category. A discriminant threshold to decide which data can be labeled as this category, and not every category is set with the same discrimination threshold (eg, probability value 1/nc), which can overcome the bias in the data to be labeled to a certain extent. Setting (that is, the data distribution of each category is different), to improve the accuracy of labeling.

It should be understood that even in the case where the labeling model is a classification model, the iterative steps S220 to S260 are not limited to the two implementations of Mode 1 and Mode 2. The probability sorts the first data to be labeled in descending order, and the maximum probability can also be replaced with other indicators.

When describing the steps of Mode 1 and Mode 2, for the parts not mentioned, reference may be made to the description of the relevant steps in FIG. 1 and FIG. 4 , or the two modes may refer to each other.

According to the previous description, the data labeling method in Fig. 4 only gives the labeling accuracy estimate for the third data to be labelled that has not been labelled in the current fourth data to be labelled. For other data to be labelled, the labeling accuracy is It is unknown, and its label is given only when the labeling accuracy of the third to-be-labeled data reaches the standard. screened out in step S230), its label is also unknown.

That is, it is not guaranteed that each piece of data to be labeled has a current reasonable labeling accuracy rate and a current reasonable label at any time, which brings difficulties to some application scenarios: for example, after the user labels for a period of time, Therefore, we have to stop labeling, and hope to obtain as much data and labels with labeling accuracy greater than 96% (target accuracy rate 99%) as possible for training the task model according to the current labeling progress, but according to the previous introduction method, at most it can only give an estimate of the labeling accuracy of the current third data to be labelled, which is difficult to meet user needs.

In response to the above problems, in an improved scheme, two attributes are added for each piece of data to be labeled: the labeling accuracy attribute and the labeling label attribute. The labeling accuracy attribute stores the current labeling accuracy estimate of the piece of data to be labeled. value, the label attribute of the label stores the current label estimate of the piece of data to be labelled. In this way, the user can end the labeling at any time, and according to the value of the labeling accuracy attribute of each piece of data to be labeled, whether it can be used as a training sample of the task model, and label the training sample according to the value of its labeling attribute.

In the following, combined with the steps in the previous method 1, an example is given to illustrate how to calculate the annotation accuracy attribute and the annotation label attribute of each piece of data to be labeled. Since the method 1 is aimed at the classification problem, the annotation label attribute can also be renamed as the annotation category attribute.

First, perform the initialization step, and you can set initial values for these two attributes of the first data to be labeled before executing the iterative step. For example, for the labeling accuracy attribute, you can set its initial value to 0, 1/nc, etc., for Annotation label attribute, its initial value can be set to the label unknown (can be regarded as a special label), or it can also be set to its inference label, etc.

Note that if the two items of the first piece of data to be labeled already have values and are not initial values, there is no need to initialize them, because these data to be labeled may have been labeled before (such as a previous batch of data) The first data to be labeled), the values of these two attributes were calculated during the previous labeling, but were reduced during the previous labeling, and were not marked with the final labeling label, so this time they were added to the current In the first batch of data to be labeled, re-label (because the labeling model may have been optimized at this time, its inference result for the same data to be labeled may also be different from the previous one, and re-labeling may result in new results).

After step B1 is performed, the value of the labeling accuracy attribute of each second piece of data to be labeled may be set to the first numerical value, and the value of its labeling category attribute may be set to the labeling category given by the user. The first value may be a preset value. For example, if the default labeling category given by the user is correct, the first value may be 1. Of course, it is not excluded that the first value takes other values.

When step C1 is performed, part of the data to be labeled may be removed, and the fourth data to be labeled may be removed. The removed data to be labeled includes three components: one is the second data to be labeled, and its labeling accuracy attribute is The value is the first value, and the value of the label attribute is the label given by the user; the second is the data that has been reduced in the first iteration and has not been labeled. Due to initialization, its labeling accuracy attribute and labeling label attribute are their respective initial values; the third is the data that has been reduced in the non-first round of iterations and has not yet been labeled. According to the following content, its labeling accuracy attribute and labeling label attribute may be assigned specific values.

After step D1 is performed, if the fourth data to be labeled includes the third data to be labeled that has not been labeled, the labeling accuracy rate of the third data to be labeled is obtained. Wherein, if the labeling accuracy of the third data to be labelled is calculated in the judgment in step E1, the calculation result in E1 can be directly obtained here, if the labeling of the third data to be labelled is not calculated when the judgment is made in step E1 Accuracy (as mentioned above, it is not necessary to explicitly calculate the estimate of the labeling accuracy to estimate whether the labeling accuracy meets the standard), then the labeling accuracy of the third to-be-labeled data needs to be calculated.

For each piece of third data to be labeled, taking any third data to be labeled s as an example, if the value of the labeling accuracy attribute of the third data to be labeled s is less than the obtained labeling accuracy, the third data to be labeled The value of the labeling accuracy attribute of s is set to the obtained labeling accuracy, and the value of its labeling category attribute is set to the category i (taking the case in the labeling candidate pool i as an example).

For example, the labeling accuracy rate of the third to-be-labeled data s in the labeling candidate pool i can be estimated according to the formula 1/(1+w ² /Ci), and the value of the labeling accuracy attribute of the third to-be-labeled data s is denoted as p (s), the value of the label category attribute is recorded as c(s), if p(s)<1/(1+w ² /Ci), then p(s) is set to 1/(1+w ² /Ci ), set c(s) to i, otherwise the values of p(s) and c(s) remain unchanged. In particular, if p(s)>acc, the value of c(s) is the final label of the third data to be labelled.

Note that since the same piece of data s to be labeled will retain a copy in each labeling candidate pool, the labeling accuracy may be estimated in multiple labeling candidate pools, but according to the above formula, the maximum value in the calculation result will be automatically selected p(s).

It should be understood that even if the steps in FIG. 4 are not implemented by way 1, the values of the labeling accuracy attribute and labeling category attribute of each first piece of data to be labelled can be calculated in a similar manner, which will not be described in detail.

The embodiment of the present application also provides a data labeling system, which can be, but is not limited to, deployed on the electronic device shown in FIG. 9 to perform the steps of the data labeling method and any implementation manner thereof introduced above, FIG. 6 shows the architecture and working principle of the data labeling system 300 . The data labeling method provided by the embodiment of the present application will be described below in conjunction with the data labeling system 300 .

6, the data annotation system 300 includes:

Two functional modules: model training and reasoning module 310 and labeling strategy module 330; wherein, the functional module can be understood as the program module of the data labeling program;

Four data pools: annotation result pool 322 , inference data candidate pool 324 , training data pool 326 and inference result pool 328 ; wherein, a data pool can be understood as a data set, or can also be understood as a storage space.

And, one interface: the user annotation interface 340 .

In conjunction with Figure 6, the working engineering of the system is outlined as follows:

Step 1: The model training and inference module 310 samples at least zero batches of training data from the training data pool 326 to train the labeling model, and uses the labeling model obtained after training to perform training on at least zero batches of training data sampled from the inference data candidate pool 324. Inference is performed on a batch of data to be labeled, and an inference result of at least one batch of data to be labeled is obtained.

The sampling method in step 1 is not limited. The training data pool 326 is the data source of the training data. The training data in the training data pool 326 includes training samples and their labels (eg, images and their label categories). How to obtain the training data will be described later. A batch of data includes at least one piece of data. If the sampled training data is zero batch, it actually means that no data sampling will be performed at this time, so the training of the labeling model will not be performed, and the original labeling model will continue to be used. Just for simplicity of presentation, the case of extracting zero batches of training data is combined with the case of extracting at least one batch of training data. It is possible to extract zero batches of training data. For example, the performance of the current annotation model is considered to be good enough, and there is no need to adjust its parameters to avoid wasting computing resources. The sampled training data need not be removed from the training data pool 326 . Regarding the training process of the labeling model, reference may be made to the prior art, which will not be explained in detail here.

The inference data candidate pool 326 is the data source of the data to be labeled, and the data to be labeled is unlabeled data. The sampled data to be labeled does not need to be removed from the inference data candidate pool 326 temporarily. Regarding the reasoning process of the labeling model, reference may be made to the prior art, which will not be explained in detail here.

Optionally, when the data labeling system 300 is initialized, the model training and inference module 310 will initialize the labeling model, including constructing the structure of the model, setting the initial parameters of the model, setting the loss function used by the model, and the like.

Then, the model training and inference module 310 can write all the data to be labeled into the inference data candidate pool 326, so that the original data in the inference data candidate pool 326 is available, so that the data sampling mentioned in step 1 can be supported. Then, the labeling strategy module 330 can read a preset number (eg, 30 pieces) of data to be labelled from the inference data candidate pool 326 , and directly hand over the data to be labelled to the user on the user labeling interface 340 for labeling and labeling. The labeling label returned by the user labeling interface 340 is obtained. Note that this part of the data to be labelled does not contain a prediction label, because the labeling model has not been trained at this time, and the prediction label cannot be given. After that, the labeling strategy module 330 writes this part of the data to be labelled and its labeling labels into the training data pool 326, so that the original data in the training data pool 326 is available, so that the data sampling mentioned in step 1 can be supported.

FIG. 7 shows an implementation of model training and inference in step 1. Referring to FIG. 7, step 1 can be implemented as at least one model training/inference cycle, and each model training/inference cycle includes at least zero model training (each batch of training data sampled from the training data pool 326 corresponds to one model training) , and at least zero-time model reasoning performed using the labeling model obtained after training (each batch of data to be labelled is sampled from the reasoning data candidate pool 326 corresponds to one model reasoning).

However, it should be noted that the number of training and inference in each model training/inference cycle cannot be zero, and at least one model inference should be included in all model training/inference cycles, otherwise the inference results of the data to be labeled cannot be obtained. . For example, performing two model training/inference cycles can be implemented as 1 model training, 1 model inference, 1 model training, and 1 model inference in sequence, or 0 model training and 1 model inference. , 1 model training, 0 model inference, can also be implemented as 0 model training, 1 model inference, 0 model training, 2 model inference, and so on.

Step 2: The model training and inference module 310 writes at least one batch of the data to be labeled in Step 1 and its inference results into the inference result pool 328 .

Optionally, when the data annotation system 300 is initialized, the model training and inference module 310 may initialize the inference result pool 328 to be empty.

Step 3: The labeling strategy module 330 reads the first data to be labelled and its reasoning result from the reasoning result pool 328 .

The number of the first data to be labeled does not exceed the total number of data to be labeled currently in the inference result pool 328 , and the read first data to be labeled and its inference result can be considered to be removed from the inference result pool 328 .

There are many possible implementations of step 3: in terms of reading timing, the labeling strategy module 330 can read the first data to be labelled and its reasoning result from the reasoning result pool 328 at regular intervals, or it can wait for the last data to be labelled. After the labeling of the first batch of data to be labelled ends (the end condition has been described above), a new batch of first data to be labelled and its reasoning results are read from the inference result pool 328, and so on. In terms of the amount of data read, the labeling strategy module 330 can read all the current to-be-labeled data and its inference results in the inference result pool 328 as the inference results of the first to-be-labeled data, or it can always read the inference results from the inference result pool 328. The inference result of the first to-be-labeled data of a fixed number is read in the result pool 328 (unless the amount of data in the inference result pool 328 is insufficient), and so on.

Steps 1 to 3 may also be considered as an implementation manner of step S110.

Step 4: The labeling strategy module 330 executes the labeling process of the first data to be labelled.

Step 4 corresponds to steps S120 to S140. During the labeling process, it is also necessary to display the second data to be labelled and its predicted label (optional) on the user labeling interface 340, and obtain the second data to be labelled through the user labeling interface 340. the callout label.

For the second to-be-labeled data and its labeling label, the labeling strategy module 330 may write it into the training data pool 326 to increase the amount of training data therein, thereby improving the performance of the labeling model when step 1 is subsequently performed again. For the automatically marked third data to be marked and its marking label (obtained in step S140), it can also be written into the training data pool 326, but this is not necessary, because after all, the automatic marking result has not been manually confirmed, and its marking accuracy is only is estimated by the algorithm and may negatively affect the performance of the annotation model.

For the second to-be-labeled data and the third to-be-labeled data for which the labeling label has been determined, the labeling strategy module 330 may remove them from the inference data candidate pool 324 to avoid repeated reasoning and repeated labeling. However, for those to-be-labeled data for which the labeling label is not determined, it can continue to be kept in the inference data candidate pool 324, allowing it to reason and label again. With the optimization of the labeling model, it is possible to infer these data. Different results were successfully annotated.

In addition, when the data labeling system 300 is initialized, the model training and reasoning module 310 can write all the data to be labelled and a special label (representing an unknown label) into the labeling result pool 322. The second data to be labeled and the third data to be labeled, the labeling strategy module 330 can use the labeling labels to modify the corresponding special labels in the labeling result pool 322 . Optionally, the labeling accuracy attribute and labeling label attribute of the data to be labelled mentioned above may also be stored in the labeling result pool 322 .

In an alternative solution, the labeling result pool 322 can also be initialized to be empty, and the labeling strategy module 330 is responsible for writing the second to-be-labeled data and the third to-be-labeled data for which the labeling label has been determined, and the labeling labels of the two into the labeling result pool 322.

Steps 1 to 4 may be repeatedly performed. In an implementation manner (referred to as a synchronous mode), when steps 3 to 4 are performed, steps 1 to 2 are not allowed to be performed. In another implementation (abbreviated as asynchronous mode), when steps 3 to 4 are executed, steps 1 to 2 are allowed to be executed at the same time, that is, after the labeling strategy module 330 reads the first data to be labelled from the inference result pool 328, After the labeling of the first to-be-labeled data ends, the model training and inference module 310 may write the new to-be-labeled data and its inference results into the inference result pool 328 again.

In the asynchronous mode, since the model training and reasoning module 310 and the labeling strategy module 330 can work independently according to their respective logics, the labeling efficiency is higher than that in the synchronous mode. For the second to-be-labeled data and the third to-be-labeled data for which the labeling label has been determined, the labeling strategy module 330 will remove them from the inference data candidate pool 324, but before the removal, these data may be trained and matched by the model again. The inference module 310 samples from the inference data candidate pool 324 and performs inference, causing it to re-enter the inference result pool 328, thereby causing duplicate annotations.

In order to improve this problem, for the second to-be-labeled data and the third to-be-labeled data for which the labeling label has been determined, the labeling strategy module 330 may also remove it from the inference data candidate pool 324, if it exists in the inference data candidate pool. 324 words.

Further, in the asynchronous mode, since the labeling strategy module 330 may write new training data to the training data pool 326 at any time, if step 1 is implemented in a training/inference loop, in different rounds of training/inference loops , the training data in the training data pool 326 may be different.

It should be understood that performing steps 1-4 through the model training and reasoning module 310 and the labeling strategy module 330 is only an optional way, and the division of program modules is relatively free, for example, steps 1-4 can also be performed by the same module. 4.

In the following, on the basis of the above embodiments, the two labeling modes provided by the embodiments of the present application will be continuously introduced:

Optionally, the data labeling process can be performed in two labeling modes, one is a semi-automatic labeling mode and the other is a manual labeling mode. Among them, the semi-automatic labeling mode refers to the mode in which user labeling is combined with automatic labeling. For example, the data labeling method described above is the semi-automatic labeling mode, in which the second data to be labelled is labelled by the user, and the third data to be labelled whose labeling accuracy meets the standard is labelled by the machine. Automatic labeling, manual labeling mode refers to the mode that is completely labelled by the user, that is, all the data to be labelled are handed over to the user for labeling, and an example of the manual labeling mode will be given later.

When the switching conditions of the labeling mode are met, the two labeling modes can be switched between each other, specifically: when the switching conditions of the manual labeling mode are met, the semi-automatic labeling mode is switched to the manual labeling mode; when the switching conditions of the semi-automatic labeling mode are met, the Switch from manual labeling mode to semi-automatic labeling mode.

For example, if the third to-be-labeled data in a labeling candidate pool is automatically labelled as executing a skip strategy, if there are M consecutive labeling candidate pools (M greater than a certain threshold), the skip strategy is not executed, It is because the fourth to-be-labeled data in the labeling candidate pool only contains the correctly labeled second to-be-labeled data and ends the labeling, which means that the data at this time is more difficult for the machine to label (or the performance of the labeling model at this time). does not meet the requirements of automatic labeling), so that the semi-automatic labeling mode can be switched to the manual labeling mode.

However, it should be noted that even if you continue to use the semi-automatic labeling mode for labeling, the labeling can still be completed, but the labeling method in the manual labeling mode can usually be designed to be simpler than the semi-automatic labeling mode (for example, without creating several labeling candidate pools), The labeling efficiency can be improved after mode switching.

For another example, if the labeling strategy module 330 reads all the data to be labelled and its reasoning results from the reasoning result pool 328 each time as the first data to be labelled and its reasoning result, then when the labeling strategy module 330 detects the reasoning result pool When the amount of data to be labeled is less than a certain amount threshold, you can switch from the semi-automatic labeling mode to the manual labeling mode.

Because if the model training and inference module 310 performs inference at a roughly constant speed, the number of data to be labeled in the inference result pool 328 is small, indicating that the labeling has entered the final stage, and the remaining data to be labeled in the final stage is likely to be the previous The data that has not been successfully labeled (for example, reduced) in the previous labeling process is difficult data, so it can be reasonably inferred that these data cannot trigger the skip strategy, or can only trigger a small amount of skipping strategy, it is better to label it directly by the user .

Among them, for the case where the labeling model is a classification model, and the classification results are nc categories, and the method 1 is used for semi-automatic labeling, the number threshold can be, but is not limited to, set to (w ² /(1/acc-1))× nc×h, where h is a preset multiple, for example, 3, 5, etc. can be taken.

Next, take the case where the labeling model is a classification model as an example to introduce an implementation method of the manual labeling mode:

For a batch of data to be labeled, it is assumed to be called the fifth data to be labeled. The reasoning result of the fifth data to be labeled includes the probability that the fifth data to be labeled belongs to each category. The labeling method of the fifth data to be labeled in the manual labeling mode is: :

First, obtain the fifth data to be labeled and its inference result; wherein, the inference result of the fifth data to be labeled is obtained by inferring the fifth data to be labeled by using a labeling model. For example, the fifth to-be-labeled data and its inference result can be read from the inference result pool 328 .

Then, sort the fifth data to be labeled in descending order according to the maximum probability in the inference result, and provide the fifth data to be labeled and its prediction category to the user batch by batch (for example, 100 pieces of data per batch) in the order of increasing sorting index. Annotation; wherein, the predicted category of each fifth piece of data to be annotated is the category corresponding to the maximum probability in its inference result.

Obtain the labeling category of each batch of the fifth data to be labelled marked by the user until all the fifth data to be labelled is labelled, or, after each batch of the fifth batch of data to be labelled, recheck whether the current semi-automatic data is satisfied Labeling mode switching conditions, if the switching conditions are met, switch from manual labeling mode to semi-automatic labeling mode, and change the remaining fifth data to be labelled that has not been labelled to semi-automatic labeling.

The key to the above labeling method is to label in batches according to the sorting index, and the labeling difficulty of each batch of the fifth data to be labelled is from low to high (the lower the probability in the inference result, the more uncertain the labeling model is about the data category, which makes labeling difficult. The user will mark the low-difficulty data with higher efficiency first. It is possible that after the low-difficulty data is marked, the remaining data does not need to be marked by the user because the semi-automatic marking mode switching conditions are met. Avoid users completely relying on themselves to label difficult data.

It should be understood that it is not necessary to use two labeling modes, for example, the semi-automatic labeling mode may always be used for labeling.

In some application scenarios, it is desirable to distinguish between simple datasets and difficult datasets in the labeled data, so that the performance of the task model can be evaluated on these two datasets respectively, and then its optimization direction during training can be determined.

With the help of the data labeling method provided by the embodiments of the present application, the division of simple data sets and difficult data sets can be realized:

For example, the data labeled in the semi-automatic labeling mode are considered to belong to the simple data set, and the data labeled in the manual labeling mode are considered to belong to the difficult data set. For another example, after all the data to be labeled is labeled, it is judged according to the value of its labeling accuracy attribute. If it is greater than a certain threshold, the data to be labeled is considered to belong to a simple data set, otherwise it is considered to belong to a difficult data set, etc. Wait.

FIG. 8 shows a possible structure of the data labeling apparatus 400 provided by the embodiment of the present application. 8, the data labeling apparatus 400 includes:

The data obtaining unit 410 is configured to obtain the first data to be labeled and an inference result thereof; wherein, the inference result of the first data to be labeled is information obtained by using a labeling model to predict the first data to be labeled;

The manual labeling unit 420 is configured to obtain the labeling label of the second data to be labelled marked by the user; wherein, the second data to be labelled is part of the data to be labelled in the first data to be labelled;

An accuracy evaluation unit 430, configured to estimate whether the labeling accuracy of the third data to be labelled that has not been labelled in the first data to be labelled has reached the standard according to the labelling label of the second data to be labelled; wherein, the third data to be labelled The labeling accuracy rate of the data to be labeled is the accuracy rate of the inference label determined according to the inference result of the third data to be labeled;

The automatic labeling unit 440 is configured to determine the inference label of the third data to be labelled as the labeling label of the third data to be labelled when the labeling accuracy of the third data to be labelled has reached the standard.

The data labeling apparatus 400 provided by the embodiments of the present application, the implementation principle and the technical effects produced have been introduced in the foregoing method embodiments, and it can implement all the contents in the foregoing method embodiments. For a brief description, the apparatus includes various units For the specific implementation process of the corresponding function, reference may be made to the corresponding content in the foregoing method embodiments, which will not be repeated here.

FIG. 9 shows a structure of an electronic device 500 provided by an embodiment of the present application. 9, an electronic device 500 includes a processor 510, a memory 520, and a communication interface 530, and these components are interconnected and communicate with each other through a communication bus 540 and/or other forms of connection mechanisms (not shown).

Among them, the processor 510 includes one or more (only one is shown in the figure), which may be an integrated circuit chip, and has signal processing capability. The above-mentioned processor 510 may be a general-purpose processor, including a central processing unit (Central Processing Unit, referred to as CPU), a micro control unit (Micro Controller Unit, referred to as MCU), a network processor (Network Processor, referred to as NP) or other conventional processors ; It can also be a special-purpose processor, including a graphics processor (Graphics Processing Unit, GPU), a neural network processor (Neural-network Processing Unit, referred to as NPU), a digital signal processor (Digital Signal Processor, referred to as DSP), application-specific integrated circuits (Application Specific Integrated Circuits, ASIC for short), Field Programmable Gate Array (Field Programmable Gate Array, FPGA for short) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components. Also, when there are multiple processors 510, some of them may be general-purpose processors, and the other may be dedicated processors.

The memory 520 includes one or more (only one is shown in the figure), which may be, but not limited to, a random access memory (Random Access Memory, RAM for short), a read only memory (Read Only Memory, ROM for short), or Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrical Erasable Programmable Read-Only Memory (EPROM) Only Memory, referred to as EEPROM) and so on.

The processor 510 and possibly other components may access the memory 520, read and/or write data therein. In particular, one or more computer program instructions may be stored in the memory 520, and the processor 510 may read and execute these computer program instructions, so as to implement the data labeling method provided by the embodiments of the present application.

The communication interface 530 includes one or more (only one is shown in the figure), which can be used to communicate directly or indirectly with other devices for data exchange. Communication interface 530 may include an interface for wired and/or wireless communication.

It can be understood that the structure shown in FIG. 9 is only for illustration, and the electronic device 500 may further include more or less components than those shown in FIG. 9 , or have different configurations from those shown in FIG. 9 . For example, when the electronic device 500 does not need to communicate with other devices, the communication interface 530 may not be included.

Each component shown in FIG. 9 may be implemented in hardware, software, or a combination thereof. The electronic device 500 may be a physical device, such as a server, a PC, a notebook computer, a tablet computer, a mobile phone, etc., or a virtual device, such as a virtual machine, a container, and the like. In addition, the electronic device 500 is not limited to a single device, and may be a combination of a plurality of devices or a cluster composed of a large number of devices.

Embodiments of the present application further provide a computer-readable storage medium, where computer program instructions are stored on the computer-readable storage medium. When these computer program instructions are read and run by a processor, the data labeling method provided by the embodiments of the present application is executed. . For example, the computer-readable storage medium may be implemented as the memory 520 in the electronic device 500 in FIG. 9 .

Embodiments of the present application further provide a computer program product, the computer program product includes computer program instructions, and when the computer program instructions are read and run by a processor, the data labeling method provided by the embodiments of the present application is executed.

The above descriptions are merely examples of the present application, and are not intended to limit the protection scope of the present application. For those skilled in the art, various modifications and changes may be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall be included within the protection scope of this application.

Claims (18)

1. A method for annotating data, comprising:

acquiring first data to be marked and an inference result thereof; the inference result of the first data to be labeled is information obtained by predicting the first data to be labeled by using a labeling model;

acquiring a labeling label of second data to be labeled marked by a user; the second data to be marked is part of data to be marked in the first data to be marked;

estimating whether the labeling accuracy of third data to be labeled which is not labeled in the first data to be labeled reaches the standard or not according to the labeling label of the second data to be labeled; the labeling accuracy of the third data to be labeled is the accuracy of the inference label determined according to the inference result of the third data to be labeled;

and if the labeling accuracy of the third data to be labeled reaches the standard, determining the inference label of the third data to be labeled as the labeling label of the third data to be labeled.

2. The data annotation method of claim 1, wherein the estimating whether the annotation accuracy of the third data to be annotated in the first data to be annotated reaches the standard according to the annotation tag of the second data to be annotated comprises:

determining fourth data to be marked in the first data to be marked according to the label of the second data to be marked; the fourth data to be labeled comprises second data to be labeled with correct labeling and/or third data to be labeled which is not labeled, wherein the second data to be labeled with correct labeling refers to: the labeling label of the second data to be labeled is the same as the inference label of the second data to be labeled determined according to the inference result of the second data to be labeled;

and if the fourth data to be labeled contains third data to be labeled which is not labeled yet, estimating whether the labeling accuracy of the third data to be labeled reaches the standard or not according to the quantity of the second data to be labeled which is labeled correctly in the fourth data to be labeled.

3. The data annotation method of claim 2, wherein the steps from the obtaining of the annotation tag of the second data to be annotated by the user to the determining of the inference tag of the third data to be annotated as the annotation tag of the third data to be annotated are performed in an iterative manner, and each iteration includes the following steps:

sampling the second data to be labeled, which is provided for the user to label in the iteration, from the fourth data to be labeled; for the first iteration, the fourth data to be labeled is the first data to be labeled;

acquiring a label of the second data to be labeled marked by the user, and screening the fourth data to be labeled according to the label of the second data to be labeled;

after screening, if the fourth data to be labeled contains third data to be labeled which is not labeled yet, estimating whether the labeling accuracy of the third data to be labeled reaches the standard or not according to the quantity of second data to be labeled which is labeled correctly in the fourth data to be labeled;

and if the labeling accuracy of the third data to be labeled reaches the standard, determining the inference label of the third data to be labeled as the labeling label of the third data to be labeled.

4. The data labeling method of claim 3, wherein the labeling model is a classification model, and the inference result of the first data to be labeled comprises a probability that the first data to be labeled belongs to each class;

prior to performing the first iteration, the method further comprises: creating a labeling candidate pool corresponding to each category according to the inference result of the first data to be labeled; each marking candidate pool comprises the first data to be marked and the probability of the first data to be marked belonging to the category corresponding to the marking candidate pool in the inference result;

each iteration comprises the following steps:

sampling the second data to be labeled, which is provided for the user to label in the iteration, from the fourth data to be labeled in all the labeling candidate pools; for the first iteration, the fourth data to be labeled in each labeling candidate pool is the first data to be labeled in the labeling candidate pool;

acquiring the labeling category of the second data to be labeled marked by the user; the labeling type of the second data to be labeled is a labeling label of the second data to be labeled;

for each annotation candidate pool, performing the following steps:

for each piece of second data to be annotated, if the annotation category of the piece of second data to be annotated is different from the category corresponding to the annotation candidate pool, and the piece of second data to be annotated is included in the fourth data to be annotated in the annotation candidate pool, reducing the fourth data to be annotated in the annotation candidate pool to only include the data to be annotated satisfying the following conditions: the probability of the class corresponding to the labeling candidate pool in the inference result is greater than the probability of the class corresponding to the labeling candidate pool in the inference result of the second piece of data to be labeled;

after the fourth data to be labeled in the labeling candidate pool is reduced, if the fourth data to be labeled contains third data to be labeled which is not labeled yet, estimating whether the labeling accuracy of the third data to be labeled reaches the standard or not according to the quantity of second data to be labeled which is labeled correctly in the fourth data to be labeled;

if the marking accuracy rate of the third data to be marked already reaches the standard, determining the category corresponding to the marking candidate pool as the marking category of the third data to be marked; the category corresponding to the labeling candidate pool is the inference label of the third data to be labeled, and the labeling category of the third data to be labeled is the labeling label of the third data to be labeled.

5. The data annotation method of claim 4, wherein in each annotation candidate pool, the first data to be annotated is sorted in descending order of probability of the class corresponding to the annotation candidate pool in the inference result;

if the category of the second to-be-labeled data is different from the category corresponding to the labeling candidate pool and the second to-be-labeled data is contained in the fourth to-be-labeled data in the labeling candidate pool, reducing the fourth to-be-labeled data in the labeling candidate pool to only contain the to-be-labeled data meeting the following conditions: the probability of the inference result belonging to the category corresponding to the labeling candidate pool is greater than the probability of the inference result belonging to the category corresponding to the labeling candidate pool of the second piece of data to be labeled, and the method comprises the following steps:

if the labeling category of the piece of second data to be labeled is different from the category corresponding to the labeling candidate pool, and the sorting index of the piece of second data to be labeled is not greater than the maximum sorting index of the fourth data to be labeled in the labeling candidate pool, reducing the fourth data to be labeled in the labeling candidate pool to only contain the data to be labeled which meets the following conditions: the sorting index of the second to-be-labeled data is smaller than that of the second to-be-labeled data.

6. The data annotation method of claim 4 or 5, wherein each piece of first data to be annotated comprises an annotation accuracy attribute and an annotation category attribute, and after the obtaining of the annotation category of the second data to be annotated labeled by the user, the method further comprises:

setting the value of the labeling accuracy rate attribute of each piece of second data to be labeled as a first numerical value, and setting the value of the labeling category attribute as the labeling category;

for each annotation candidate pool, the following steps are also performed:

after the fourth data to be labeled in the labeling candidate pool is reduced, if the fourth data to be labeled contains third data to be labeled which is not labeled yet, acquiring the labeling accuracy of the third data to be labeled; the labeling accuracy is estimated according to the quantity of second data to be labeled which are correctly labeled in the fourth data to be labeled;

and for each piece of third data to be labeled, if the value of the labeling accuracy attribute of the piece of third data to be labeled is smaller than the labeling accuracy, setting the value of the labeling accuracy attribute of the piece of third data to be labeled as the labeling accuracy, and setting the value of the labeling category attribute as the category corresponding to the candidate labeling pool.

7. The data labeling method of claim 3, wherein the labeling model is a classification model, and the inference result of the first data to be labeled comprises a probability that the first data to be labeled belongs to each class;

each iteration comprises the following steps:

sampling the second data to be labeled provided for the user to label in the current iteration from the fourth data to be labeled; for the first iteration, the fourth data to be labeled is the first data to be labeled;

acquiring the labeling category of the second data to be labeled marked by the user; the labeling category of the second data to be labeled is the labeling label of the second data to be labeled;

for each piece of second data to be labeled, if the labeling category of the piece of second data to be labeled is different from the category corresponding to the maximum probability in the inference result, and the piece of second data to be labeled is included in the fourth data to be labeled, reducing the fourth data to be labeled to only include the data to be labeled which meets the following conditions: the maximum probability in the inference result is greater than the maximum probability in the inference result of the second data to be labeled;

after the fourth data to be labeled is reduced, if the fourth data to be labeled contains third data to be labeled which is not labeled, estimating whether the labeling accuracy of the third data to be labeled reaches the standard or not according to the quantity of second data to be labeled which is labeled correctly in the fourth data to be labeled;

if the labeling accuracy of the third data to be labeled reaches the standard, determining the category corresponding to the maximum probability in the inference result of the third data to be labeled as the labeling category of the third data to be labeled; the class corresponding to the maximum probability in the inference result of the third data to be labeled is the inference label of the third data to be labeled, and the labeling class of the third data to be labeled is the labeling label of the third data to be labeled.

8. The data annotation method of claim 7, wherein the first data to be annotated is sorted in descending order according to the maximum probability among the probabilities of the inference results that the first data to be annotated belongs to each category;

if the labeling category of the second piece of data to be labeled is different from the category corresponding to the maximum probability in the inference result, and the second piece of data to be labeled is contained in the fourth piece of data to be labeled, reducing the fourth piece of data to be labeled to only contain the data to be labeled meeting the following conditions: the maximum probability in the inference result is greater than the maximum probability in the inference result of the second piece of data to be labeled, and the method comprises the following steps:

if the labeling category of the second data to be labeled is different from the category corresponding to the maximum probability in the inference result, and the sorting index of the second data to be labeled is not greater than the maximum sorting index in the fourth data to be labeled, reducing the fourth data to be labeled to only contain the data to be labeled which meets the following conditions: the sorting index of the second piece of data to be labeled is smaller than that of the second piece of data to be labeled.

9. The data annotation method of any one of claims 1 to 8, wherein before the obtaining of the annotation tag of the second data to be annotated by the user, the method further comprises:

determining a prediction label of the second data to be labeled according to the inference result of the second data to be labeled;

and displaying the second data to be labeled and the prediction label thereof on a user labeling interface.

10. The data annotation method of claim 9, wherein the annotation model is a classification model, the inference result of the first data to be annotated includes a probability that the first data to be annotated belongs to each class, the first data to be annotated is annotated by performing at least one iteration, before performing the first iteration, an annotation candidate pool corresponding to each class is created according to the inference result of the first data to be annotated, and in each annotation candidate pool, the first data to be annotated is sorted in a descending order according to the probability that the first data to be annotated belongs to the class corresponding to the annotation candidate pool in the inference result;

when each iteration is executed, determining the prediction category of any one piece of second data to be labeled according to the following mode, wherein the prediction category of the second data to be labeled is a prediction label of the second data to be labeled:

obtaining a sorting index of the piece of second data to be labeled in each labeling candidate pool;

normalizing the quantity of the fourth data to be labeled in the labeling candidate pool corresponding to each sequencing index reference to obtain a normalized sequencing index;

and determining the category corresponding to the minimum sorting index in all the normalized sorting indexes as the prediction category of the second data to be labeled.

11. The data labeling method of claim 9, wherein the labeling model is a classification model, the inference result of the first data to be labeled includes a probability that the first data to be labeled belongs to each class, and the first data to be labeled is labeled by performing at least one iteration;

determining the prediction category of any piece of second data to be labeled according to the following mode, wherein the prediction category of the second data to be labeled is the prediction label of the second data to be labeled:

and determining the class corresponding to the maximum probability in the inference result of the second piece of data to be labeled as the prediction class of the second piece of data to be labeled.

12. The data annotation method of any one of claims 9 to 11, wherein the annotation model is a classification model, the prediction label of the second data to be annotated is a prediction category of the second data to be annotated, and the displaying the second data to be annotated and the prediction label thereof on the user annotation interface comprises:

and displaying the second data to be labeled on the user labeling interface in a classified manner according to the prediction category of the second data to be labeled.

13. The data annotation method of claim 12, wherein the second data to be annotated and its prediction category are displayed in a first area of the user annotation interface;

the method further comprises the following steps:

responding to data selection operation which is triggered in the first area and aims at the second data to be annotated, displaying the selected second data to be annotated in a second area of the user annotation interface, and displaying the selected candidate category of the second data to be annotated in the second area;

if one piece of second data to be labeled is selected by the user, the prediction type of the piece of second data to be labeled is not approved by the user.

14. The data annotation method of any one of claims 9 to 13, wherein the second data to be annotated is an image, and the second data to be annotated and a prediction tag thereof are displayed in a first area of the user annotation interface;

the method further comprises the following steps:

displaying a reference image in a third area of the user labeling interface, wherein the reference image is an image selected from the second data to be labeled;

recording an image transformation operation triggered in the third area and aiming at the reference image, applying the image transformation operation to each image in the second data to be labeled, and refreshing and displaying the second data to be labeled after the image transformation operation is executed in the first area;

wherein the image transformation operation comprises at least one operation of image translation, image rotation, image scaling and image local area frame selection.

15. The data annotation method according to any one of claims 1 to 14, wherein the obtaining of the first data to be annotated and the inference result thereof includes:

sampling at least zero batches of training data from a training data pool to train the labeling model, and reasoning at least one batch of data to be labeled sampled from a reasoning data candidate pool by using the labeling model to obtain a reasoning result of the at least one batch of data to be labeled;

writing the at least one batch of data to be marked and the reasoning result thereof into a reasoning result pool;

and reading the first data to be labeled and the inference result thereof from the inference result pool.

16. A computer program product comprising computer program instructions which, when read and executed by a processor, perform the method of any one of claims 1 to 15.

17. A computer-readable storage medium having computer program instructions stored thereon, which when read and executed by a processor, perform the method of any one of claims 1-15.

18. An electronic device, comprising: a memory having stored therein computer program instructions which, when read and executed by the processor, perform the method of any of claims 1-15.

Images