WO2023029436A1 - Data labeling method and apparatus, and device and storage medium - Google Patents

Abstract

The present application belongs to the technical field of AI. Provided are a data labeling method and apparatus, and a computing device and a storage medium. The method comprises: a data labeling platform sending, to at least two labeling users, data to be labeled; acquiring at least two labeling results of the at least two labeling users for said data; determining whether to execute arbitration on said data according to the at least two labeling results; if it is determined to execute arbitration on said data, sending said data to an auditing user for labeling; and if it is determined not to execute arbitration on said data, fusing the at least two labeling results according to a fusion rule, so as to obtain a tag of said data. By means of the present application, the same data is labeled by means of multi-person coordination, such that the problem of low labeling accuracy of complex data is solved.

Description

Method, device, equipment and storage medium for data labeling

This application claims the priority of the Chinese patent application with the application number 202111003760.4 and the title of the invention "Data labeling method, device, equipment and storage medium" submitted on August 30, 2021, the entire content of which is incorporated by reference in this application middle.

technical field

The present application relates to the technical field of artificial intelligence (AI), and in particular to a data labeling method, device, device and storage medium.

Background technique

With the widespread application of AI technology, a large amount of labeled data is required for algorithm training in AI technology, so efficient and accurate labeling of data has become a top priority.

In related technologies, an intelligent labeling algorithm is pre-trained, and each data to be labeled is input into the intelligent labeling algorithm to obtain a labeling result of each data to be labeled. The labeling result of each data to be labeled is determined as a label. In this way, the intelligent labeling algorithm can label simple labels, but for more complex labels, the labeling accuracy of the intelligent labeling algorithm is often relatively low, so it is necessary to provide a method with a higher labeling accuracy.

Contents of the invention

The present application provides a data labeling method, device, device, and storage medium, which can improve the accuracy of data labeling.

In a first aspect, the present application provides a method for labeling data, the method comprising: sending the data to be labeled to at least two labeling users, and obtaining at least two labels of the at least two labeling users on the data to be labelled. result. According to the at least two tagging results, it is determined whether to perform arbitration on the data to be tagged. If it is determined to perform arbitration on the data to be marked, send the data to be marked to the review user for marking. If it is determined not to perform arbitration on the data to be labeled, the at least two labeling results are fused according to a fusion rule to obtain the label of the data to be labeled.

In the scheme shown in this application, the data to be marked is distributed to at least two marking users for marking, and at least two marking results are obtained. Determine whether to perform arbitration on the data to be labeled. When it is determined that the data to be marked is to be arbitrated, the data to be marked is sent to the review user for marking, and when it is determined that the data to be marked is not to be arbitrated, at least two marking results are fused according to the fusion rules to obtain the data to be marked The label of the data. In this way, multiple people collaborate to mark the same data, which solves the problem of low labeling accuracy of complex data, and also provides an arbitration function to arbitrate the data to be marked that needs to be arbitrated, further improving the labeling accuracy.

In a possible implementation manner, if it is determined to perform arbitration on the data to be labeled, the method further includes: obtaining a labeling result of the data to be labeled by the review user, and using the labeling result of the review user as a label of the data to be labeled. In this way, more accurate labels can be obtained for the data to be labeled that require arbitration.

In a possible implementation manner, the method further includes: receiving configuration information for marking users, reviewing users and fusion rules input by the management user. In this way, the labeling users, reviewing users and fusion rules can be flexibly configured, which improves the application flexibility of the method.

In a possible implementation manner, determining whether to perform arbitration on the data to be labeled according to at least two labeling results includes: if the labeling type of the data to be labeled is classification, if at least two labeling results are different, Then it is determined to arbitrate the data to be labeled. If at least two labeling results are the same, it is determined not to arbitrate the data to be labeled; If the difference between the label boxes for the same object does not meet the preset conditions, it is determined that the data to be labeled is arbitrated, and if the difference between the label boxes for the same object in at least two labeling results meets the preset conditions, it is determined not to be labeled data for arbitration. In this way, for the data to be labeled of different labeling types, there are different standards for judging whether to perform arbitration, so it is possible to accurately determine whether to perform arbitration.

In a possible implementation, the at least two labeling results are fused according to the fusion rule to obtain the label of the data to be labeled, including: when the labeling type of the data to be labeled is classification, combining at least two labeling results One of the labeling results in the results is determined as the label of the data to be labeled; when the labeling type of the data to be labeled is detection or segmentation, the union of the labeling frames for the same object is determined as the label of the data to be labeled Label.

In the solution shown in this application, there are different ways of determining labels for different labeling types of data to be labeled. In this way, when there are multiple labeling results, the label can be accurately determined based on the labeling results.

In a possible implementation, the method further includes: when it is detected that the labeling user triggers the labeling interface, providing a viewing tool and a labeling tool corresponding to the labeling type of the data to be labeled through the labeling interface, and the viewing tool is used for the labeling The user views the data to be labeled, and the labeling tool is used for the labeling user to add labeling results corresponding to the labeling type for the data to be labelled.

In the solution shown in this application, when it is detected that the labeling user triggers the labeling interface, the labeling user is provided with the viewing tool and labeling tool corresponding to the labeling type through the labeling interface, so that the labeling user can use the viewing tool to view the data to be labeled , and enable the annotation user to use the annotation tool to add labels to the data to be annotated. In this way, a more intelligent labeling method is provided for labeling users, which can not only unify the labeling format of labeling results, but also improve labeling efficiency.

In a possible implementation manner, after obtaining the label of the data to be labeled, the method further includes: acquiring label version information of the data to be labeled input by the management user; combining the label of the data to be labeled with the label version Information corresponds to storage.

In the scheme shown in this application, the data labeling platform provides the function of the management user to input label version information, and the management user can input the label version information for the data to be labeled, and the label version information is used to indicate the label version of the data to be labelled. The data labeling platform stores the label of the data to be labeled in correspondence with the label version information, so that the label of the data to be labeled can be distinguished based on the label version information.

In the second aspect, the present application provides a data labeling device, which is applied to a data labeling platform, and the device includes:

A data labeling module, configured to send the data to be labeled to at least two labeling users;

An annotation result management module, configured to: acquire at least two annotation results of the at least two annotation users on the data to be annotated;

Determine whether to perform arbitration on the data to be marked according to the at least two marking results;

The data labeling module is also used to send the data to be marked to the review user for labeling if it is determined to perform arbitration on the data to be marked;

The labeling result management module is further configured to fuse the at least two labeling results according to fusion rules to obtain the label of the data to be labeled if it is determined not to perform arbitration on the data to be labeled.

In a possible implementation, the labeling result management module is also used to: if it is determined to perform arbitration on the data to be labeled, obtain the labeling result of the review user for the data to be label, and the labeling result of the review user The result serves as the label of the data to be labeled.

In a possible implementation manner, the data labeling module is further configured to receive configuration information of the labeling user, the auditing user and the fusion rule input by the management user.

In a possible implementation manner, the labeling result management module is configured to: if the labeling type of the data to be labeled is classification, if the at least two labeling results are different, determine The data is arbitrated, and if the at least two labeling results are the same, it is determined not to arbitrate the data to be labeled;

In the case where the labeling type of the data to be labeled is detection or segmentation, if the difference between the labeling frames of the same object in the at least two labeling results does not meet the preset condition, then it is determined to arbitrate the data to be labeled , if the difference between the labeling frames of the same object in the at least two labeling results satisfies the preset condition, it is determined not to arbitrate the data to be labeled.

In a possible implementation manner, the labeling result management module is configured to: when the labeling type of the data to be labeled is classified, determine one of the at least two labeling results as the labeling result to be labeled the label of the annotated data;

In the case where the labeling type of the data to be labeled is detection or segmentation, the union of the labeling frames for the same object is determined as the label of the data to be labeled.

In a possible implementation manner, the data labeling module is further configured to: when it is detected that the labeling user triggers the labeling interface, provide a viewing tool and a labeling tool corresponding to the labeling type of the data to be labeled through the labeling interface, The viewing tool is used for the labeling user to view the data to be labeled, and the labeling tool is used for the labeling user to add labeling results corresponding to the labeling type to the data to be labelled.

In a possible implementation, the tagging result management module is further configured to: after obtaining the tag of the data to be tagged, obtain the tag version information of the data to be tagged input by the management user;

The label of the data to be labeled is stored in correspondence with the label version information. .

In a third aspect, the present application provides a computing device for data labeling, the computing device includes a processor and a memory, wherein:

computer instructions are stored in the memory;

The processor executes the computer instructions to implement the data labeling method described in the first aspect.

In a fourth aspect, the present application provides a computer-readable storage medium, the computer-readable storage medium stores computer instructions, and when the computer-readable storage medium is executed by a computing device, the computing device executes the first In one aspect, the data labeling method.

In a fifth aspect, the present application provides a computer program product containing instructions, which, when run on a computing device, causes the computing device to execute the data labeling method described in the first aspect above.

Description of drawings

Fig. 1 is a logical schematic diagram of an AI platform provided by an exemplary embodiment of the present application;

FIG. 2 is a logical schematic diagram of a data labeling platform provided by an exemplary embodiment of the present application;

Fig. 3 is a schematic diagram of interaction between a data labeling module and a management user provided by an exemplary embodiment of the present application;

FIG. 4 is a logical schematic diagram of a data labeling platform provided by an exemplary embodiment of the present application;

Fig. 5 is a logical schematic diagram of a data labeling platform provided by an exemplary embodiment of the present application;

Fig. 6 is a schematic structural diagram of a computing device provided by an exemplary embodiment of the present application;

Fig. 7 is a schematic flow chart of a data labeling method provided by an exemplary embodiment of the present application;

Fig. 8 is a schematic flowchart of a data labeling method provided by an exemplary embodiment of the present application;

Fig. 9 is a schematic diagram of an annotation interface provided by an exemplary embodiment of the present application;

Fig. 10 is a schematic diagram of an annotation interface provided by an exemplary embodiment of the present application;

Fig. 11 is a schematic diagram of an annotation interface provided by an exemplary embodiment of the present application;

Fig. 12 is a schematic flowchart of a data labeling method provided by an exemplary embodiment of the present application;

Fig. 13 is a schematic structural diagram of a data tagging device provided by an exemplary embodiment of the present application.

Detailed ways

In order to make the purpose, technical solution and advantages of the present application clearer, the implementation manners of the present application will be further described in detail below in conjunction with the accompanying drawings.

With the widespread application of AI technology, a large amount of labeled data is required for algorithm training in AI technology, so efficient and accurate labeling of data has become a top priority. In a related solution, the data is labeled using the intelligent labeling algorithm, but for complex data, the accuracy of the intelligent labeling algorithm is often relatively low, so a method with a relatively high labeling accuracy is needed.

The embodiment of the present application provides a data labeling method, which can be applied to various data labeling scenarios, and can have good labeling accuracy in various scenarios. For example, image data annotation scenarios, semantic analysis annotation scenarios, etc. Exemplarily, the image data may be medical images.

The embodiment of the present application uses medical image labeling as an example to describe the solution in detail. In the medical field, the professionalism of medical imaging is relatively high, and it is required that the labeling user has more professional knowledge of medical imaging. In this way, different annotation users will have relatively large differences in annotation results due to inconsistent professional knowledge levels. Moreover, there are many medical image data formats and different modalities, and the method of data annotation is also required to be able to adapt to various medical images. For example, in medical images of different lungs, the size and shape of pulmonary nodules are not the same. Moreover, in order to make the developed medical AI model more accurate, more medical images are required, so the labeling of medical images also requires a lot of manpower and time.

Based on the above situation, in the embodiment of this application, a data labeling function that supports multi-person labeling and fusion and arbitration of labeling results is set in the AI platform, so that each data to be labeled is allocated to at least two labeling users for labeling. After the at least two annotation users complete the annotation, the annotation results of the at least two annotation users are fused. When the annotation results of at least two annotation users are quite different, the data to be annotated is arbitrated. In this way, the data labeling function provided by the AI platform can not only make the labeling results accurate, but also make the labeling results consistent in format through multiple people’s collaborative labeling and arbitration when the labeling results differ greatly, which is convenient for subsequent AI model training , the AI model can be a medical aided diagnosis model, etc.

Exemplarily, in the embodiment of the present application, after a label is added to the data to be labeled, the data to be labeled and the label can be used as samples for training an AI model.

In some embodiments, the execution body of the data labeling method may be an AI platform. Exemplarily, the execution body may specifically be a data labeling platform included in the AI platform. The AI platform is a platform that provides a convenient AI development environment and convenient development tools for AI developers and users. The AI platform provides data annotation functions and AI model training functions.

FIG. 1 is a schematic structural diagram of the AI platform 100 provided in the embodiment of the present application. It should be understood that FIG. division of modules. As shown in Figure 1, the AI platform 100 includes an AI interaction module 101, a cloud base platform 102, a platform as a service (platform-as-a-service, PaaS) 103, an infrastructure as a service (infrastructure as a service, IaaS) 104, etc. .

The functions of each module in the AI platform 100 shown in FIG. 1 are briefly described below:

Exemplarily, the AI interaction module 101 is used to provide data management functions, AI algorithm development functions, auxiliary diagnosis and treatment service functions, and high-level AI image assistance functions.

The data management function is used for the import of received data, desensitization of data, image reading function and sketching function. The data may be data to be labeled or the like. Desensitization refers to the anonymization of sensitive information of user data. The image reading function is used to display the data to be marked to the user. The sketching function is used for users to add labeling results for the data to be labeled.

AI algorithm development functions include dataset management functions, data labeling functions, model training functions, model evaluation functions, deployment and online functions, algorithm management functions, etc. Data set management functions include data set management functions and data archiving functions. The management functions of datasets include the functions of creating, modifying and deleting datasets. The data archiving function refers to the function of archiving various information of the same data. The data labeling function includes the management function of the labeling task, the management function of the labeling team, the management function of the labeling tool, etc. The labeling task is the task of labeling some data. An annotation team refers to a team of multiple users who annotate a dataset. Labeling tools refer to tools used to label data, such as electronic brushes. The model training function is used to manage the training of AI models, including but not limited to resource pool management (i.e. management of training data), scheduling of training jobs, etc. The model evaluation function is used to evaluate the trained AI model, including but not limited to model verification, image packaging, and model visualization. The deployment function is used to deploy the AI model that has passed the evaluation to the line, such as service deployment, model sharing, etc. The algorithm management function is used for the management of various algorithms, such as: medical segmentation algorithm management, 3D pre-training (pre-segmentation of medical images, etc.), 3D image content retrieval, 3D cross-modal registration, self-supervised small sample learning , 3D network neural structure search (neural architecture search, NAS) optimization, etc.

The auxiliary diagnosis function is used to provide auxiliary diagnosis of various types of diseases.

High-level AI image assistance functions are used to provide AI image processing functions, including but not limited to image content retrieval functions, image registration functions, etc.

The cloud base platform 102 may also be called an AI development platform. The cloud infrastructure platform 102 provides various cloud infrastructure services. The cloud base platform 102 includes, but is not limited to, a development environment management platform, a data processing platform, a data labeling platform, a training operation platform, a model management function, a service management function, and the like. The development environment management platform is used to provide development environment management services. The data processing platform is used to provide data processing services. The data annotation platform is used to provide data annotation services. For example, the data labeling platform is used to manage the labeling results of each data to be labeled. The training job platform is used to provide AI model training job management services. The model management function is used to provide model management services, such as updating models, deleting models, and so on. The service management function is used to manage the provided services.

PaaS103, including resource pools. Resource pools include but are not limited to shared resource pools and tenant-specific resource pools. The shared resource pool includes resources available to each tenant, such as model training resources. Tenant-specific resource pools include resources available to each tenant, such as model training resources and model service resources. Exemplarily, when applied to the field of medical imaging, the resource pool further includes an AI-aided diagnosis resource pool, for example, the AI-aided diagnosis resource pool includes an AI model for medical-aided diagnosis.

IaaS104, including server resources. Server resources include but are not limited to elastic cloud server (elastic cloud server, ECS), ECS+image processing unit (graphics processing unit, GPU) server, bare metal server (bare metal server, BMS)+GPU server, etc. A cloud server can also be called a computing unit. BMS is a dedicated physical server provided for tenants.

It should be understood that the various modules included in the AI platform shown in FIG. 1 are only an example, and in some embodiments, the AI platform may only include the functions of some of the modules, or, in other embodiments, the AI platform may also include Functions of other modules may be included, which is not limited in this application.

FIG. 2 is a schematic structural diagram of the data labeling platform (subsequently represented as the data labeling platform 200) in the AI platform 100 provided in the embodiment of the present application. It should be understood that FIG. 2 is only an example showing a structure of the data labeling platform 200 As a schematic diagram, the embodiment of the present application does not limit the division of modules in the data labeling platform 200. As shown in FIG. 2 , the data labeling platform 200 includes a data labeling module 201 , a data storage module 202 and a labeling result management module 203 .

The functions of each module in the data labeling platform 200 shown in FIG. 2 are briefly described below:

The data labeling module 201 provides labeling project management services, labeling team management services, labeling task management services, data labeling services and label version management services.

Exemplarily, the labeling project management service is used to manage labeling projects. Each labeling project targets a data set, which is used to train an AI model. For example, this data set is a lung medical image set, which is used to train an AI model for assisting in the diagnosis of the lungs. For another example, the data set is a brain medical image set, which is used to train an AI model that assists in diagnosing the brain. The labeling project management service includes the functions of creating a labeling project, modifying a labeling project, deleting a labeling project, viewing the list of labeling projects, viewing the overview of labeling projects, etc. Management users can manage the marked items based on these functions.

In this embodiment of the application, the management user is a developer, or a third-party user of the AI platform (such as an independent software vendor (ISV)).

Exemplarily, the labeling team management service is used to manage the labeling team. Annotation teams are teams of multiple users. The annotation team includes annotation users and review users, for example, usually the professional knowledge level of review users is higher than that of annotation users. Both labeling users and review users can label data, and review users can also arbitrate the results of data labeling. The labeling team management service includes the functions of creating labeling teams, deleting labeling teams, modifying labeling team members, adding labeling team members, deleting labeling team members, viewing labeling team lists, etc. Administrative users can manage the labeling team based on these functions.

Exemplarily, the labeling task management service is used to manage labeling tasks. A labeling task is a task of labeling at least one data to be labeled, for example, a labeling task is a task of labeling part of data in a labeling project, or a task of labeling all data in a labeling project. Each data to be labeled corresponds to a label type, which can include any one of classification, detection or segmentation. A labeling task can be assigned to at least one labeling team, and each data to be labeled in the labeling task is assigned to at least two labeling users for labeling. The labeling task management service includes the functions of creating labeling tasks, deleting labeling tasks, viewing the progress of labeling tasks, etc. Management users can manage labeling tasks based on these functions.

Exemplarily, the labeling task management service also includes the function of accepting the labeling results and viewing the acceptance report. Through the function of accepting the marked results, the management user can submit the marked results to the review user for arbitration. The management user can view the acceptance result through the function of viewing the acceptance report.

Exemplarily, the data labeling service includes the functions of adding, modifying, deleting and arbitrating the result of labeling. And the data annotation service can also provide annotation tools for annotation users or review users. Annotation users or review users can use annotation tools to annotate data.

Exemplarily, the tag version management service is used to manage tag version information of the data to be tagged. Label version information is used to distinguish different batches of labels for a data to be labeled. The tag version management service includes the functions of querying tag version information, deleting tag version information, setting the current tag version information, publishing tag version information, etc. Management users can query tag version information through the function of querying tag version information. Administrative users can delete tag version information through the function of deleting tag version information. The management user can select different label version information of the data to be labeled through the function of setting the label version information, so as to view the labels of the data to be labeled under different label version information. The function of publishing label version information is used to instruct to store the label of the data to be labeled in correspondence with the label version information indicated by the management user.

Exemplarily, the data labeling module platform 200 also provides a label viewing service. Administrative users can view labels of labeled data through this service.

It should be noted that the management user, labeling user and auditing user can interact with the data labeling platform 200 through a graphical user interface (graphical user interface, GUI) or by calling an application program interface (application program interface, API).

The data storage module 202 may be a data storage resource corresponding to an object storage service (object storage service, OBS) provided by a cloud service provider. The data storage module 202 is used to store the data to be marked uploaded by the user and the marking results, for example, the data set to be marked uploaded by the user is stored, and the data set to be marked includes the data to be marked. Exemplarily, the data labeling module 201 reads the data to be labeled from the OBS, and writes the labeling result into the OBS after the labeling is completed.

Exemplarily, the data storage module 202 is also used to store the information of the labeling team and the like.

The labeling result management module 203 is configured to determine whether to perform arbitration on the data to be labeled. When the data to be marked is not arbitrated, at least two labeling results of each data to be marked are fused to obtain the label of the data to be marked; when the data to be marked is arbitrated, the data to be marked is sent to the reviewing user. Exemplarily, the tagging result management module 203 reads at least two tagging results of each data to be tagged from the data storage module 202, performs fusion processing on the at least two tagging results, and obtains a tag of each data to be tagged. Exemplarily, Spark can be used to perform fusion processing on the tagging results. After the labeling result management module 203 acquires the label of the data to be labeled, it can store the label. Exemplarily, the label can be stored in a database, such as storing the label in HBase.

Exemplarily, the data labeling platform 200 further includes an AI reasoning module 204 . The AI reasoning module 204 performs intelligent labeling on the data to be labeled to obtain intelligent labeling results, and provides the intelligent labeling results to labeling users and/or review users through user input/output (I/O) functions for reference.

Exemplarily, the data labeling platform 200 may also include an AI model training module and a data preprocessing module. The AI model training module is used to obtain the labeled data set after obtaining the label of each data to be labeled in the data set, and the labeled data set is all labeled data. Based on the labeled data set, train the AI model. The data preprocessing module is used to mark the completed data set for preprocessing operation. Exemplarily, preprocessing the marked data in the marked data set can make the marked data consistent in size, and can also remove inappropriate data in the marked data and perform Desensitization treatment, etc. The data preprocessing module can store the preprocessed labeled data in the data storage module 202 .

It should be noted that the data labeling platform 200 in this application can be a system that can interact with users. This system can be a software system or a hardware system, or a system combining software and hardware, which is not limited in this application. .

In order to better illustrate the interaction between the data labeling module 201 in the data labeling platform 200 in FIG. 2 and the management user, the interaction diagram shown in FIG. 3 is provided. In FIG. 3 , management users can use the labeling project management service, labeling team management service, label version management service, labeling task management service, data labeling service and label viewing service provided by the data labeling module 201. Among them, the management user creates a marked item, deletes a marked item, modifies a marked item, checks a list of marked items, etc. through the marked item management service. Management users can create labeling teams, delete labeling teams, modify labeling team members, add labeling team members, delete labeling team members, view labeling team lists, etc. through the labeling team management service. Management users create tag version information, delete tag version information, set current tag version information, and publish tag version information through the tag version management service. Management users create labeling tasks, delete labeling tasks, modify labeling tasks, view labeling task progress, and accept labeling results through the labeling task management service. Administrative users can view the labels of labeled data through the label viewing service. The management user can view the labeling results of the data to be labeled through the data labeling service.

In addition, in Figure 3, the labeling user and the reviewing user can use the data labeling service to perform data labeling. Exemplarily, the labeling user uses the data labeling service to label the data to be labelled, and saves the labeling result. The review user reviews the labeling results of the data to be labeled and saves the labeling results through the data labeling service.

FIG. 4 is a schematic diagram of an application scenario of an AI platform 100 provided by an exemplary embodiment of the present application. As shown in FIG. 4 , the AI platform 100 can all be deployed in a cloud environment. A cloud environment is an entity that uses basic resources to provide cloud services to users under the cloud computing model. The cloud environment includes a large number of basic resources (including computing resources, storage resources and network resources) owned by the cloud service provider, and the computing resources may be a large number of computing devices (such as servers). The AI platform 100 can be independently deployed on servers or virtual machines in the cloud environment, and the AI platform 100 can also be deployed on multiple servers in the cloud environment in a distributed manner, or distributed on multiple virtual machines in the cloud environment. On-machine, or distributed on servers and virtual machines in cloud environments. As shown in FIG. 4 , the AI platform 100 is abstracted into a cloud service by the cloud service provider in the cloud environment and provided to the user, and the cloud environment uses the AI platform 100 deployed in the cloud environment to provide the cloud service to the user. When using cloud services, management users can upload the data to be marked to the cloud environment through API or GUI. The AI platform 100 (such as the data labeling platform 200 ) in the cloud environment receives the data to be labeled, and provides data labeling service functions to users (labeling users, review users and management users).

The deployment of the data labeling platform 200 provided by this application is relatively flexible, as shown in FIG. 5 , in another embodiment, the data labeling platform 200 provided by this application can also be distributed and deployed in different environments. The data labeling platform 200 provided in this application can be logically divided into multiple parts, and each part has different functions. For example, in one embodiment, the data labeling platform 200 includes a data labeling module 201 , a data storage module 202 and a labeling result management module 203 . Each part of the data labeling platform 200 can be deployed in any two or three environments of the terminal computing device, the edge environment and the cloud environment. Terminal computing devices include: terminal servers, smart phones, notebook computers, tablet computers, personal desktop computers, smart cameras, etc. The edge environment is an environment that includes a collection of edge computing devices that are relatively close to the terminal computing device, and the edge computing devices include: edge servers, edge small stations with computing capabilities, and the like. Various parts of the data labeling platform 200 deployed in different environments or devices cooperate to provide data labeling functions for users. For example, in one scenario, the data labeling module 201 and the data storage module 202 of the data labeling platform 200 are deployed on the terminal computing device, and the labeling result management module 203 of the data labeling platform 200 is deployed on the edge computing device of the edge environment. It should be understood that this application does not restrictively divide which parts of the data labeling platform 200 are deployed in which environment. In actual application, it can be based on the computing power of the terminal computing device, the resource occupancy of the edge environment and the cloud environment, or specific application requirements. Make adaptive deployments.

The data labeling platform 200 can also be independently deployed on a computing device in any environment (for example, independently deployed on an edge server in an edge environment). FIG. 6 is a schematic diagram of a hardware structure of a computing device 600 deployed with the data labeling platform 200 . The computing device 600 shown in FIG. 6 includes a memory 601 , a processor 602 , a communication interface 603 and a bus 604 . Wherein, the memory 601 , the processor 602 , and the communication interface 603 are connected to each other through a bus 604 .

The memory 601 may be a read only memory (read only memory, ROM), a random access memory (random access memory, RAM), a hard disk, a flash memory or any combination thereof. The memory 601 may store programs, and when the programs stored in the memory 601 are executed by the processor 602, the processor 602 and the communication interface 603 are used to execute the data labeling method performed by the data labeling platform 200 . The memory can also store data sets. For example, a part of storage resources in the memory 601 is divided into storing data sets and labels of data in the data sets, and a part of storage resources in the memory 601 is divided into storing labeling results of data to be labeled.

The processor 602 may be a central processing unit (central processing unit, CPU), an application specific integrated circuit (application specific integrated circuit, ASIC), a GPU or any combination thereof. Processor 602 may include one or more chips. The processor 602 may include an AI accelerator, for example, a neural network processor (neural processing unit, NPU).

The communication interface 603 uses a transceiver module such as a transceiver to implement communication between the computing device 600 and other devices or communication networks. For example, data can be acquired through the communication interface 603 .

Bus 604 may include pathways for transferring information between various components of computing device 600 (eg, memory 601 , processor 602 , communication interface 603 ).

The flow of the data labeling method will be described below, and FIG. 7 shows a schematic diagram of the process of the data labeling method. The subject of execution of the data labeling method may be the aforementioned data labeling platform 200, which will be briefly described as the data labeling platform hereinafter.

Step 701, sending the data to be tagged to at least two tagging users.

Among them, the data to be marked is the unlabeled data submitted by the management user to the data labeling platform. One of the data to be labeled, the multiple data to be labeled constitute a data set to be labeled. The type of data to be labeled may be an image, the image may be a two-dimensional image, a three-dimensional image, etc., and the three-dimensional image may be a complex medical image.

In this embodiment, the management user instructs the data labeling platform to label the data to be labeled. The data labeling platform provides the data to be labeled to at least two labeling users. Exemplarily, the data annotation platform sends the access path of the annotation interface of the data to be annotated to at least two annotation users. The access path may be an address link. Alternatively, the data annotation platform sends annotation notification messages to at least two annotation users. Exemplarily, in order to save human resources for labeling, the foregoing at least two labeling users may be only two labeling users.

The annotation user can trigger the access path of the annotation interface, trigger the annotation user's terminal, and display the login window of the annotation interface. Mark the user to log in with their own account and password. Alternatively, the labeling user can directly log in to the data labeling platform with his account and password, and enter the labeling interface in the data labeling platform. In the annotation interface, the data to be annotated by the annotating user will be displayed, and the data that does not need to be annotated by the annotating user will not be displayed. For example, the data labeling platform binds the labeling user’s account with the data to be labeled by the labeling user in advance, and the data labeling platform detects that the labeling user has logged in, and provides the labeling user with the data to be labeled corresponding to the labeling user’s account . Annotation users can add annotation results to the data to be annotated in the annotation interface. After the labeling user completes the labeling of the data to be labeled, the labeling result of the data is submitted to the data labeling platform. The data labeling platform stores the labeling results of each data to be labeled.

It should be noted that, when the data labeling platform stores the labeling results of each data to be labeled, the labeling results may not be directly added to the data to be labeled for storage, but the labeling results may be stored separately. In this way, the annotation user will not be affected by the existing annotation results when annotating the data to be annotated. And the labeling result is stored corresponding to the identifier of the data to be labeled, so as to correspond the labeling result to the data to be labeled. Exemplarily, when the data labeling platform stores the labeling result, it can also correspond the labeling result with the identifier of the labeler who performed the labeling.

It should also be noted that, for the data to be labeled, the forms of labeling results of different labeling types are different. For example, when the labeling type is classification, the labeling result is the category to which the data to be labeled belongs; when the labeling type is detection or segmentation, the labeling result is the labeling box in the data to be labeling and the description information of the labeling box, and the description information is used to indicate The content in the label box, if the data to be labeled is a medical image, the content in the label box is the right lung, and the description information is the right lung. It is worth noting here that when the label type is segmentation, the boundary of the region obtained by segmentation can also be considered as a label box.

Step 702, acquiring at least two tagging results of data to be tagged by at least two tagging users.

In this embodiment, after the data labeling platform detects that the labeling of the data to be labeled is completed, at least two labeling results of the data to be labelled are obtained from the storage location of the labeling results.

Alternatively, the management user submits a dataset to be labeled for labeling, and the data to be labeled belongs to the dataset to be labeled. After the data labeling platform detects that the labeling of each data to be labeled in the data set to be labeled is completed, at least two labeling results of each data to be labeled are obtained from the storage location of the labeling results.

Step 703, according to at least two tagging results, determine whether to perform arbitration on the data to be tagged.

In this embodiment, the data labeling platform uses at least two labeling results, determines a difference between the at least two labeling results, and determines whether to perform arbitration on the data to be labeled based on the difference. When the difference is large, it is determined to perform arbitration on the data to be marked, and when the difference is small, it is determined not to perform arbitration on the data to be marked.

Exemplarily, for labeling results of different labeling types, there are different rules for judging whether to perform arbitration on the data to be labeled. When the labeling type of the data to be labeled is classification, the data labeling platform judges whether at least two labeling results are the same, if not, then decides to perform arbitration on the data to be labeled, and if they are the same, determines not to perform arbitration on the data to be labeled arbitration.

When the labeling type of the data to be labeled is detection or segmentation, the data labeling platform determines the labeling boxes of the same object in at least two labeling results. The data labeling platform determines the difference between the label boxes for the same object in at least two labeling results, and judges whether the difference meets the preset condition. The preset condition can be preset. For example, the preset condition is that the difference is less than or equal to the target value, etc. . For different objects, there may be different preset conditions. If the difference between the label frames of the same object in at least two labeling results does not meet the preset condition, it is determined to perform arbitration on the data to be labeled, and if the difference between the label frames of the same object in at least two label results meets the preset condition Preset conditions, it is determined not to perform arbitration on the data to be labeled.

Here, the difference value of the label box can be one or more of the difference between the area of the label box, the coincidence degree of the label box, the distance between the center of the label box, and the difference between the longest distance and the shortest distance of the boundary points of the label box kind.

It should be noted that the object is the content marked by the user in the data to be labeled. For example, the data to be labeled is a medical image of the lung, and the objects are right lung, left lung, etc. For example, the data to be labeled is a medical image of the thyroid gland. Imaging, the object is thyroid nodules, etc.

The process for the data annotation platform to determine the annotation frames of the same object in at least two annotation results is as follows: the data annotation platform obtains the description information of each annotation frame in at least two annotation results, and determines the annotation frames with the same description information as the annotation of the same object frame. Here, there may be multiple labeling boxes with the same description information in each labeling result, and the labeling boxes with the same description information in at least two labeling results and the closest distance are determined to be the labeling boxes of the same object.

Step 704, if it is determined to perform arbitration on the data to be marked, send the data to be marked to the review user for marking.

Among them, the professional knowledge of review users is usually richer than that of labeling users, and it can also be considered that the labeling accuracy of review users is higher than that of labeling users. For example, in the medical field, review users as experts and mark users as general doctors. In some cases, moderator users can also serve as callout users. If the data to be marked that needs to be arbitrated is marked by the review user as the markup user, the data to be marked will not be submitted to the review user, but to other review users.

In this embodiment, when it is determined that arbitration is performed on the data to be marked, the data labeling platform sends the data to be marked to the review user. Exemplarily, the data annotation platform sends an arbitration notification message to the review user, and the arbitration notification message may be provided to the review user by email, short message, or the like. The arbitration notification message may include the access path of the arbitration interface, or the arbitration notification message may not include the access path of the arbitration interface, and only reminds the auditing user that the data to be marked needs to be arbitrated.

After the audit user receives the arbitration notification message, the audit user can trigger the terminal to display the login window of the arbitration interface. Audit users use their own accounts and passwords to log in to the arbitration interface. Or, after receiving the arbitration notification message, the review user can directly log in to the data labeling platform with his account and password, and enter the arbitration interface in the data labeling platform.

After the audit user enters the arbitration interface, the data to be marked to be arbitrated by the audit user is displayed in the arbitration interface, and the audit user selects each data to be marked in turn for arbitration. When arbitrating each data to be labeled, at least two options of labeling results for the data to be labeled may be displayed. The review user can judge whether there are correct labeling results in at least two labeling results. If there is a correct labeling result, the reviewing user will submit the correct labeling result; if there is no correct labeling result, the reviewing user will submit a correct labeling result. The results are modified, and the revised labeling results are submitted; or, the review user re-labels the data to be labeled, and submits the re-labeled labeling results.

Exemplarily, after the data labeling platform obtains the labeling result of the data to be labeled by the auditing user, the labeling result is used as a label of the data to be labeled.

Step 705, if it is determined that the data to be labeled is not to be arbitrated, at least two labeling results are fused according to a fusion rule to obtain a label of the data to be labeled.

In this embodiment, the data labeling platform acquires a fusion rule of the data to be labeled, and the fusion rule refers to a rule for obtaining a label when there are at least two labeling results for one piece of data to be labeled. Based on the fusion rule, the data labeling platform performs fusion processing on at least two labeling results of the data to be labeled to obtain the label of the data to be labeled. The data labeling platform stores the labels of the data to be labeled.

Exemplarily, the fusion rules of different application scenarios are different, and the fusion rules of different annotation types are different. The fusion processing process of image annotation is provided as follows:

When the labeling type of the data to be labeled is classification, one labeling result among at least two labeling results is determined as the label of the data to be labelled. When the labeling type of the data to be labeled is detection or segmentation, the union of the labeling frames for the same object is determined as the label of the data to be labeled. Here is just an example of a method, and the fusion processing process can also be other processes, for example, when the label type of the data to be labeled is detection or segmentation, the labeling frame for the same object is averaged , determined as the label of the data to be labeled.

It should be noted that there is no sequence between step 704 and step 705, and the execution order of the two steps can be interchanged, or the two steps can be executed in parallel.

Exemplarily, a piece of data to be labeled may be labeled multiple times, and the data labeling platform may add different label version information to each label to distinguish each label. After obtaining the label of the data to be labeled, the data labeling platform obtains the label version information of the data to be labeled input by the management user, and stores the label version information corresponding to the label of the data to be labeled. The management user can input the label version information of the data to be labeled when submitting the data to be labeled to the data labeling platform, or input the label version information of the data to be labeled after the data labeling platform obtains the label of the data to be labeled .

In the above solution, the data to be labeled is submitted to at least two labeling users for labeling, so that the data to be labeled obtains at least two labeling results. And judge whether to arbitrate the data to be marked, when not arbitrating the data to be marked, perform fusion processing of at least two labeling results of the data to be marked, determine the label of the data to be marked; arbitrate the data to be marked , the moderation is performed by the auditing user. It can be seen that the problem of low labeling efficiency and accuracy of complex data (such as three-dimensional images) is solved through the collaborative labeling of the same data to be labeled by multiple people. Moreover, the entire labeling process is based on the data labeling platform, which can be adapted to various cloud computing scenarios and is more flexible.

Fig. 7 is an overview of the flow of the data labeling method. FIG. 8 provides a specific description of the flow of the data labeling method shown in FIG. 7 . In FIG. 8 , labeling a data set to be labeled of a labeling item is taken as an example for illustration.

In step 801, the data labeling platform creates labeling items.

In this embodiment, the task of labeling a data set required for training an AI model may be referred to as a labeling item. For example, the AI model is an assisted diagnosis model for pulmonary nodules, the data in the dataset is medical images of the lungs, and the labeling task of the dataset of medical images of the lungs is a labeling item. The data labeling platform provides management users with a labeling project creation interface. The management user enters the name of the labeling project in the labeling project creation interface, and imports the dataset to be labeled into the data labeling platform. The data labeling platform records the labeling item. The dataset to be labeled includes multiple pieces of data to be labeled.

In addition, administrative users can also manage the created labeling projects through the data labeling platform. For example, modify marked items, delete marked items, view marked items, etc.

Exemplarily, when creating a labeling item, the management user may also input label version information, where the label version information indicates the label version of the dataset to be labeled of the labeling item.

Step 802, the data labeling platform creates a labeling team.

Among them, the labeling team is used to label the data to be labeled in the dataset to be labeled.

In this embodiment, the data labeling platform provides a team management interface for the management user, and the management user can input the information of the labeling team in the team management interface, and the information includes each user in the labeling team and the role information of each user. The role information of each user includes labeling users and reviewing users. Exemplarily, each member of the labeling team is a labeling user or an auditing user; or some members of the labeling team are only labeling users or reviewing users, and some members are both labeling users and reviewing users. Then the management user submits the information of the labeling team to the data labeling platform. The data annotation platform stores the information of the annotation team.

Exemplarily, after the management user enters the created labeling project, the data labeling platform provides the management user with a labeling team interface, and the management user can create a labeling team through the labeling team interface. In this way, after the data labeling platform creates the labeling team, it stores the labeling team as the labeling team of the labeling project.

Exemplarily, the management user can also directly create an annotation team through the team management interface. When creating a labeling team, enter a name for the labeling project. In this way, after the data labeling platform creates the labeling team, it stores the labeling team as the labeling team of the labeling project.

Exemplarily, one labeling project may correspond to one or more labeling teams. For the method of creating each labeling team, refer to step 802 . An annotation team can also correspond to one or more annotation projects.

Step 803, the data labeling platform creates a labeling task.

Among them, the labeling task is a task of labeling at least one data to be labeled. For example, a labeling task is used to partially label the dataset of the labeling project, and the part labeling indicates two situations. The first situation is: the The data set to be labeled of the labeling project is divided into multiple sub-datasets to be labeled. The data in each sub-dataset to be labeled is different. Each labeling task corresponds to the data to be labeled in a sub-dataset to be labeled. The labeling types of the sub-datasets to be labeled are the same; the second case is: each labeling task is to label the entire data set to be labeled of the labeling project, but the labeling types are not the same, the entire data set to be labeled The data to be labeled is the data to be labeled corresponding to each labeling task. For example, three labeling tasks are performed on classification labeling, detection labeling and segmentation labeling respectively. In the second case here, the data set to be labeled of the labeling item is used to train an AI model that integrates classification, detection and segmentation.

In this embodiment, after the management user creates a labeling project on the data labeling platform, the data labeling platform provides the management user with a labeling task interface. The management user triggers the labeling task interface, and the data labeling platform provides the management user with a labeling task creation interface. The management user enters the name of the labeling task, the data to be labeled corresponding to the labeling task, the labeling type, and the labeling team in the labeling task creation interface. In addition, the management user can also enter the label saving path in the labeling task creation interface. Then the management user submits the created labeling task to the data labeling platform. After receiving the labeling task, the data labeling platform stores the name of the labeling task, labeling type, data to be labeled, labeling team and labeling project.

Exemplarily, when creating a labeling project, the management user does not indicate label version information, and the management user may also input label version information corresponding to the labeling task when creating a labeling task.

It should be noted that when the management user inputs the data to be labeled for each labeling task, he can select the data to be labeled in the dataset of the labeling project without re-importing the data to be labeled. The data labeling platform stores the labeling task and the identifier of the data to be labeled corresponding to the labeling task.

Exemplarily, the fusion rules corresponding to different application scenarios are different, and the management user can also input the fusion rules corresponding to the labeling task when creating the labeling task on the data labeling platform. Exemplarily, the data annotation platform provides the management user with a fusion rule configuration interface; the data annotation platform receives the fusion rule input by the management user or selected in the configuration interface.

In this embodiment, the data labeling platform provides management users with a configuration interface for fusion rules. Exemplarily, the data labeling platform provides an input interface for fusion rules in the interface for creating labeling tasks, the management user triggers the input interface, and the data labeling platform provides the management user with a configuration interface for fusion rules.

In the configuration interface, the management user can input fusion rules in a prescribed format. Alternatively, the configuration interface provides a variety of fusion rules that can be selected by the management user, and the management user can choose one of the various fusion rules. After the management user enters or selects the fusion rules, he submits the fusion rules to the data labeling platform. After receiving the fusion rules, the data labeling platform stores the fusion rules and labeling tasks correspondingly. In this way, fusion rules can be customized for different scenarios or different fusion rules can be customized for an application scenario, which has higher flexibility.

In addition, if the management user inputs a fusion rule in a specified format, the data labeling platform can store the fusion rule, and the subsequent data labeling platform can provide the fusion rule to other management users. In this way, multiple fusion rules input by management users form a fusion rule set for selection, which provides convenience for management users to select fusion rules. In addition, when the data labeling platform stores the fusion rules, it can also store the brief content of the fusion rules, which is convenient for management users to understand and select the fusion rules.

In addition, after the labeling task is created, the data labeling platform also provides management users with functions such as viewing the labeling task, modifying the labeling task, deleting the labeling task, and viewing the progress of the labeling task. For example, the management user modifies the function of the labeling task, modifies the fusion rules, and so on.

It should be noted that, when the management user does not select a fusion rule, the data labeling platform automatically selects a fusion rule for the labeling task from the stored fusion rules based on the labeling type of the labeling task.

Step 804, the data labeling platform provides labeling tasks to the labeling team, so that each data to be labeled corresponding to the labeling task is assigned to at least two labeling users in the labeling team.

In this embodiment, the data labeling platform provides labeling tasks to the labeling team based on preset data distribution rules. The preset data allocation rule is a pre-set rule for labeling users for data allocation.

Exemplarily, the data labeling platform determines the number of labeling users in the labeling team, equally distributes the data to be labeled corresponding to the labeling task to the labeling users, and allocates each data to be labelled to at least two labeling users. Alternatively, the data labeling platform provides the management user with a division function of the data to be labeled corresponding to the labeling task, and the management user divides each data to be labeled corresponding to the labeling task to at least two labeling users. The data labeling platform determines the labeling user assigned to each data to be labeled, and the data labeling platform can store the corresponding relationship between the labeling user's identification (the identification can be the labeling user's account) and the labeling data's identification.

The data labeling platform provides labeling tasks to the labeling team. Exemplarily, the data annotation platform sends an annotation notification message to each annotation user in the annotation team, and the annotation notification message can be sent by email, short message, etc. The annotation notification message may include an access path of the annotation interface.

Step 801 to step 804 are an implementation manner of step 701 . Alternatively, steps 802 to 804 are an implementation of step 701 . Alternatively, steps 803 to 804 are an implementation manner of step 701 .

Step 805, when it is detected that the labeling user triggers the labeling interface, the data labeling platform provides the viewing tool and labeling tool corresponding to the labeling type of the data to be labeled through the labeling interface. For labeling, the user adds labeling results corresponding to the labeling type for the data to be labeled.

In this embodiment, after the tagging user receives the tagging notification message through the terminal, the tagging user triggers the access path of the tagging interface, triggers the tagging user's terminal to log in to the data tagging platform, and the terminal displays the login window of the tagging interface. Note that users can log in with their own account and password. After the data labeling platform detects that the labeling user has logged in, the data labeling platform provides the labeling user with the data to be labelled, based on the stored correspondence between the labeling user's account and the labeling user's corresponding data to be labelled.

Based on the labeling type indicated by the labeling task, the data labeling platform provides viewing tools and labeling tools through the labeling interface. When annotating a certain data to be annotated, the annotation user uses the viewing tool to display the data to be annotated to better view the data to be annotated. The functions of the viewing tool include but are not limited to zoom in, zoom out, change color, Flip, move layers down, move layers up, and more. When adding labeling results to the data to be labeled, the labeling user uses the labeling tool to add labeling results to the data to be labelled. In this way, suitable viewing tools and labeling tools are provided for labeling users, so that labeling users can perform labeling efficiently.

Exemplarily, when the labeling type is classification, the labeling tool provides a selection function, and the labeling user can select a labeling result from the provided categories, or the labeling user can input a category. For example, FIG. 9 shows an annotation interface when the annotation type is classification. In Figure 9, the data to be labeled is the medical image of the lung, the left column shows the list of data to be labeled and the total number of data to be labeled (4), the middle column shows the currently labeled data (data 1), and the right column Show a list of categories (categories are 1, 2, 3). In the uppermost part of the middle, viewing tools, etc. are displayed. In Fig. 9, there are three categories, and the labeling user can select one category as the labeling result.

When the annotation type is detection, the annotation tool provides the selection function and the function of adding a annotation frame, etc. The annotation user can use the function of adding a annotation frame to add a annotation frame on the data to be annotated, and use the selection function to select the corresponding description information or enter the description information . When the labeling type is detection, the labeling frame and the corresponding description information are the labeling results, where the shape of the labeling frame can be arbitrary, and the color of the labeling frame can also be any color. For example, FIG. 10 shows an annotation interface when the annotation type is detection. In Figure 10, the data to be labeled is the medical image of the lung, the left column shows the list of the data to be labeled and the total number of data to be labeled (8), the middle column shows the currently labeled data, and the right column shows the descriptive information list. In the uppermost part of the middle, viewing tools, etc. are displayed. In FIG. 10 , a labeling frame is shown, which is represented by a dotted rectangular box in FIG. 10 . There are circles at the upper left corner and lower right corner of the labeling frame. By dragging the circles, the size of the labeling frame can be changed. In addition, in Figure 10, two description information are displayed in the description information list on the right side, description information 1 and description information 2. When the description information 2 is selected, the position coordinates corresponding to the label box are displayed correspondingly, and the position coordinates can be Coordinates of the upper left corner and lower right corner of the label box.

When the annotation type is segmentation, the annotation tool provides the selection function and the segmentation function, etc. The annotation user can use the segmentation function to mark the segmentation area on the data to be annotated, and use the selection function to select the corresponding description information. When the labeling type is segmentation, the segmented area and the corresponding description information are the labeling results. Here, the shape of the segmented area can be any polygon, and the color of the segmented area can also be any color. For example, FIG. 11 shows the labeling interface when the labeling type is segmentation. In Figure 11, the data to be labeled is the medical image of the lung, the left column shows the list of data to be labeled and the total number of data to be labeled (10), the middle column shows the data to be labeled, and the right column shows the description information list . Viewing tools, etc. are displayed at the top of the middle, and labeling tools are displayed at the bottom of the middle, such as segmentation labeling tools, segmentation area erasing tools, etc. In Fig. 11, a segmented region of the right lung is shown.

After the labeling user completes all labeling of the data to be labeled, the labeling result is submitted to the data labeling platform. Or the labeling user submits to the data labeling platform every time the labeling user completes a piece of data to be labelled. The data labeling platform stores the labeling results of each data to be labeled.

Exemplarily, when the annotation user is currently annotating the data to be annotated, the annotation interface also provides the option of importing historical labels, which is used to trigger the display of historical labels. Label the results. Annotation users can refer to historical labels when annotating the data to be annotated to improve annotation efficiency.

Exemplarily, the data labeling platform can also provide intelligent labeling results for reference by labeling users. For example, the data labeling platform includes an AI reasoning model, which labels the data to be marked and obtains the intelligent labeling results of the data to be marked. In the labeling interface, provide users with the option to import smart labeling results. Annotation users can use this option to trigger the display of smart annotation results in the annotation interface. In this way, the labeling efficiency can be improved by importing the intelligent labeling results for the reference of labeling users.

In step 806, the data labeling platform obtains at least two labeling results of each data to be labeled corresponding to the labeling task.

For the process of step 806, see the description of step 702.

In step 807, the data labeling platform determines whether to perform arbitration on each data to be marked according to at least two labeling results of each data to be marked.

For the process of step 807, see the description of step 703.

Step 808, if it is determined that arbitration is performed on the data to be marked, send the data to be marked to the review user for marking. If it is determined not to perform arbitration on the data to be labeled, at least two labeling results are fused according to fusion rules to obtain the label of the data to be labeled.

In step 808, the data labeling platform determines to perform arbitration on the data to be marked, and determines an audit user in the labeling team corresponding to the labeling task. The data labeling platform sends the data to be marked to the review user. Refer to the description in step 704 for the process of the auditing user performing arbitration on the data to be marked. The data labeling platform determines not to perform arbitration on the data to be labeled, and refer to the description of step 705 for the process of determining the label of the data to be labeled.

Exemplarily, in this embodiment, the data labeling platform provides the data to be marked to review users within the labeling team for arbitration, and in some cases, may also submit the data to be marked to review users outside the labeling team for arbitration. After the data labeling platform obtains the labeling result of the data to be labeled by the review user, it determines the labeling result as the label of the data to be labelled.

In addition, in the process of reviewing the data to be labeled, the review user determines that the data to be labeled is a hard sample, and can also mark it for subsequent training of the AI model. Difficult examples are data that are prone to errors in the labeling process during the process of labeling data.

Step 809, the data labeling platform obtains the label version information of the data to be labeled input by the management user, and stores the label of each data to be labeled and the label version information correspondingly.

In this embodiment, the data labeling platform obtains label version information (input when creating a labeling project or creating a labeling task), and stores the label of each data to be labeled corresponding to the labeling task and the label version information.

Alternatively, after the data labeling platform determines that a labeling task is completed, it feeds back a feedback message indicating that the labeling task is completed to the management user. The management user's terminal receives the feedback message, and the feedback message also prompts the management user whether to add label version information, and the management user inputs the label version information and submits it to the data labeling platform. The data labeling platform stores the label and label version information of each data to be labeled corresponding to the labeling task.

It should be noted that, in the process shown in Figure 8, the data labeling platform automatically provides the data to be marked to the review user for arbitration. In some implementation manners, after the data labeling platform determines to arbitrate the data to be marked, it feeds back the data to be marked to the management user. After receiving the data to be labeled, the management user submits the review task of the data to be labeled to the data labeling platform, and the data labeling platform executes step 808 . In addition, management users can also submit labels obtained through fusion rules to review users for arbitration.

It should also be noted that after the data labeling platform determines the label of the data to be labeled corresponding to the labeling task, the management user can control the data to be labeled to be reviewed multiple times, and after multiple reviews are completed, it is determined that the labeling task is completed. Multiple audits here can also be referred to as multiple acceptances.

In addition, after the labeling of the dataset to be labeled corresponding to a labeling project is completed, the data labeling platform trains the AI model based on the data set.

It should also be noted that the management user is a type of management personnel, which is to be distinguished from the labeling user and the auditing user, and the management user does not mean only one person.

In addition, in order to better understand the process of data labeling, the embodiment of this application also provides an easy-to-understand process, see Figure 12, Figure 12 is a schematic diagram of interaction between management users, labeling users, data labeling platforms (computing devices) and auditing users . In Fig. 12, it is the management user who provides the data to be labeled to the data labeling platform for arbitration.

Step S1, the management user creates a labeling task. Step S2, the management user selects at least two labeling users for each data to be labeled corresponding to the labeling task, and notifies the at least two labeling users. Step S3, the marking user marks the data to be marked. Step S4, the labeling user submits the labeling result to the computing device. Step S5, for any data to be marked, determine whether to perform arbitration on the data to be marked. Step S6, the computing device stores the tags of the data to be marked that do not require arbitration, and feeds back the data to be marked to the management user. Step S7, the management user notifies the review user to review the data to be marked. Step S8, the auditing user conducts arbitration on the data to be marked. Step S9 , the review user feeds back the labeling result of the data to be labeled by the review user to the management user. Step S10 , the management user controls the computing device to store the arbitrated labeling result as the label of the data to be labeled. After determining that each data to be labeled corresponding to the labeling task has been labeled, the management user determines that the labeling task is completed. Only the execution process of one labeling task is described here, and the execution process of each labeling task is the same, and will not be repeated here.

In the embodiment of this application, through a unified human-computer interaction platform, it is possible to support multiple people to mark the same data to be marked, and to obtain the marking results in the same format, and to support the arbitration and review of the marking results by review users. Effectively improve the overall labeling quality and increase the error tolerance rate. Moreover, it is also possible to configure different fusion rules for different labeling scenarios, effectively improving the quality of manual labeling. Moreover, viewing tools and annotation tools are provided in the annotation interface, which is more convenient for annotating users to annotate difficult 3D images.

FIG. 12 is a structural diagram of a data labeling device provided by an embodiment of the present application. The device may be part or all of the data labeling platform 200 . The device can be implemented as a part or all of the device through software, hardware or a combination of the two. The device provided in the embodiment of the present application can implement the processes described in Figure 7 and Figure 8 in the embodiment of the present application, and the device includes: a data labeling module 201 and a labeling result management module 203, wherein:

The data tagging module 201 is configured to send the data to be tagged to at least two tagging users, specifically, it can be used to perform the data tagging function in step 701 and the implicit steps it contains;

An annotation result management module 203, configured to: obtain at least two annotation results of the at least two annotation users on the data to be annotated;

According to the at least two tagging results, determine whether to perform arbitration on the data to be tagged, which can be specifically used to execute the tagging result management function of step 702 and step 703 and the implicit steps contained therein;

The data labeling module 201 is further configured to send the data to be marked to the auditing user for labeling if it is determined to perform arbitration on the data to be marked, specifically, it can be used to perform the data labeling function of step 704 and its included implicit steps;

The labeling result management module 203 is further configured to fuse the at least two labeling results according to the fusion rules to obtain the label of the data to be labeled if it is determined not to perform arbitration on the data to be labeled. The labeling result management function of step 705 and the hidden steps contained therein are executed.

In a possible implementation manner, the tagging result management module 203 is further configured to:

If it is determined to perform arbitration on the data to be marked, obtain a marking result of the data to be marked by the review user, and use the mark result of the review user as a label of the data to be marked.

In a possible implementation manner, the data labeling module 201 is further configured to receive configuration information on the labeling user, the auditing user and the fusion rule input by the management user.

In a possible implementation manner, the labeling result management module 203 is configured to:

In the case where the labeling type of the data to be labeled is classified, if the at least two labeling results are not the same, it is determined to arbitrate the data to be labeled, and if the at least two labeling results are the same, then Determine not to arbitrate the data to be labeled;

In the case where the labeling type of the data to be labeled is detection or segmentation, if the difference between the labeling frames of the same object in the at least two labeling results does not meet the preset condition, it is determined to The data is arbitrated, and if the difference between the tagging frames of the same object in the at least two tagging results satisfies the preset condition, it is determined not to arbitrate the data to be tagged.

In a possible implementation manner, the labeling result management module 203 is configured to:

In the case where the labeling type of the data to be labeled is classification, one labeling result of the at least two labeling results is determined as the label of the data to be labeled;

When the labeling type of the data to be labeled is detection or segmentation, a union of labeling frames for the same object is determined as the label of the data to be labeled.

In a possible implementation manner, the data labeling module 201 is further configured to:

When it is detected that the labeling user triggers the labeling interface, a viewing tool and a labeling tool corresponding to the labeling type of the data to be marked are provided through the labeling interface, and the viewing tool is used for the labeling user to view the labeling to be marked data, the labeling tool is used for the labeling user to add labeling results corresponding to the labeling type for the data to be labelled.

In a possible implementation manner, the tagging result management module 203 is further configured to:

After obtaining the label of the data to be marked, obtain the label version information of the data to be marked input by the management user;

The label of the data to be labeled is stored in correspondence with the label version information.

The division of the modules in the embodiment of the present application is schematic, and it is only a logical function division. In actual implementation, there may be other division methods. In addition, each functional module in each embodiment of the present application can be integrated into one In the processor, it may exist separately physically, or two or more modules may be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software function modules.

If the integrated module is realized in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , including several instructions for enabling a terminal device (which may be a personal computer, a mobile phone, or a network device, etc.) or a processor (processor) to execute all or part of the steps of the method in each embodiment of the present application. The aforementioned storage medium includes: various media capable of storing program codes such as U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk.

In the embodiment of the present application, a computer program product containing instructions is also provided, and when it is run on a computing device, it causes the computing device to execute the data labeling method provided above, or enables the computing device to implement the above provided method. The function of the device labeled by the data.

In the above-mentioned embodiments, all or part may be implemented by software, hardware, firmware or any combination thereof, and when software is used, all or part may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions, and when the computer program instructions are loaded and executed on the server or terminal, all or part of the processes or functions according to the embodiments of the present application will be generated. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website, computer, server or data center Transmission to another website site, computer, server, or data center by wired (eg, coaxial cable, optical fiber, DSL) or wireless (eg, infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that can be accessed by a server or a terminal, or a data storage device such as a server or a data center integrated with one or more available media. The available medium may be a magnetic medium (such as a floppy disk, a hard disk, and a magnetic tape, etc.), an optical medium (such as a digital video disk (DVD), etc.), or a semiconductor medium (such as a solid-state hard disk, etc.).

Claims (17)

1. A method for data labeling, characterized in that the method is applied to a data labeling platform, comprising:

   sending the data to be labeled to at least two labeling users;

   Acquiring at least two tagging results of the at least two tagging users on the data to be tagged;

   Determine whether to perform arbitration on the data to be marked according to the at least two marking results;

   If it is determined to perform arbitration on the data to be marked, send the data to be marked to the review user for marking;

   If it is determined not to perform arbitration on the data to be labeled, the at least two labeling results are fused according to a fusion rule to obtain a label of the data to be labeled.
2. The method according to claim 1, wherein if it is determined to perform arbitration on the data to be marked, the method further comprises:

   An annotation result of the audit user on the data to be annotated is obtained, and the annotation result of the audit user is used as a label of the data to be annotated.
3. The method according to claim 1 or 2, further comprising: receiving configuration information for the marking user, the reviewing user, and the fusion rule input by a management user.
4. The method according to any one of claims 1-3, wherein the determining whether to perform arbitration on the data to be marked according to the at least two marking results includes:

   In the case where the labeling type of the data to be labeled is classified, if the at least two labeling results are not the same, it is determined to arbitrate the data to be labeled, and if the at least two labeling results are the same, then Determine not to arbitrate the data to be labeled;

   In the case where the labeling type of the data to be labeled is detection or segmentation, if the difference between the labeling frames of the same object in the at least two labeling results does not meet the preset condition, it is determined to The data is arbitrated, and if the difference between the tagging frames of the same object in the at least two tagging results satisfies the preset condition, it is determined not to arbitrate the data to be tagged.
5. The method according to any one of claims 1-4, wherein the merging of the at least two labeling results according to a fusion rule to obtain the label of the data to be labeled comprises:

   In the case where the labeling type of the data to be labeled is classification, one labeling result of the at least two labeling results is determined as the label of the data to be labeled;

   When the labeling type of the data to be labeled is detection or segmentation, a union of labeling frames for the same object is determined as the label of the data to be labeled.
6. The method according to any one of claims 1-5, wherein the method further comprises:

   When it is detected that the labeling user triggers the labeling interface, a viewing tool and a labeling tool corresponding to the labeling type of the data to be marked are provided through the labeling interface, and the viewing tool is used for the labeling user to view the labeling to be marked data, the labeling tool is used for the labeling user to add labeling results corresponding to the labeling type for the data to be labelled.
7. The method according to any one of claims 1-6, wherein after obtaining the label of the data to be labeled, further comprising:

   Acquiring label version information of the data to be labeled input by the management user;

   The label of the data to be labeled is stored in correspondence with the label version information.
8. A device for data labeling, characterized in that it includes:

   A data labeling module, configured to send the data to be labeled to at least two labeling users;

   An annotation result management module, configured to: obtain at least two annotation results of the at least two annotation users on the data to be annotated;

   Determine whether to perform arbitration on the data to be marked according to the at least two marking results;

   The data tagging module is further configured to send the data to be tagged to an auditing user for tagging if it is determined to perform arbitration on the data to be tagged;

   The tagging result management module is further configured to, if it is determined not to perform arbitration on the data to be tagged, fuse the at least two tagging results according to a fusion rule to obtain a tag of the data to be tagged.
9. The device according to claim 8, wherein the labeling result management module is also used for:

   If it is determined to perform arbitration on the data to be marked, obtain a marking result of the data to be marked by the review user, and use the mark result of the review user as a label of the data to be marked.
10. The device according to claim 8 or 9, wherein the data labeling module is further configured to receive configuration information on the labeling user, the auditing user and the fusion rule input by the management user.
11. The device according to any one of claims 8-10, wherein the labeling result management module is configured to:

    In the case where the labeling type of the data to be labeled is classified, if the at least two labeling results are not the same, it is determined to arbitrate the data to be labeled, and if the at least two labeling results are the same, then Determine not to arbitrate the data to be labeled;

    In the case where the labeling type of the data to be labeled is detection or segmentation, if the difference between the labeling frames of the same object in the at least two labeling results does not meet the preset condition, it is determined to The data is arbitrated, and if the difference between the tagging frames of the same object in the at least two tagging results satisfies the preset condition, it is determined not to arbitrate the data to be tagged.
12. The device according to any one of claims 8-11, wherein the labeling result management module is configured to:

    In the case where the labeling type of the data to be labeled is classification, one labeling result of the at least two labeling results is determined as the label of the data to be labeled;

    When the labeling type of the data to be labeled is detection or segmentation, a union of labeling frames for the same object is determined as the label of the data to be labeled.
13. The device according to any one of claims 8-12, wherein the data labeling module is also used for:

    When it is detected that the labeling user triggers the labeling interface, a viewing tool and a labeling tool corresponding to the labeling type of the data to be marked are provided through the labeling interface, and the viewing tool is used for the labeling user to view the labeling to be marked data, the labeling tool is used for the labeling user to add labeling results corresponding to the labeling type for the data to be labelled.
14. The device according to any one of claims 8-13, wherein the labeling result management module is also used for:

    After obtaining the label of the data to be marked, obtain the label version information of the data to be marked input by the management user;

    The label of the data to be labeled is stored in correspondence with the label version information.
15. A computing device for data labeling, characterized in that the computing device includes a processor and a memory, wherein:

    computer instructions are stored in the memory;

    The processor executes the computer instructions to implement the method of any one of claims 1-7.
16. A computer-readable storage medium, characterized in that the computer-readable storage medium stores computer instructions, and when the computer-readable storage medium is executed by a computing device, the computing device executes the The method according to any one of claims 1-7.
17. A computer program product, characterized by comprising computer instructions, when the computer instructions are executed by a computing device, causing the computing device to execute the method according to any one of claims 1-7.

Images