CN111178845A

CN111178845A - Data annotation system and method based on network service platform

Info

Publication number: CN111178845A
Application number: CN201911402745.XA
Authority: CN
Inventors: 孙辉; 潘陶嘉诚; 陈小琴
Original assignee: Suzhou Automotive Research Institute of Tsinghua University
Current assignee: Suzhou Automotive Research Institute of Tsinghua University
Priority date: 2019-12-31
Filing date: 2019-12-31
Publication date: 2020-05-19

Abstract

The invention discloses a data labeling system and method based on a network service platform, wherein the system comprises: a software client and a network service platform; the software client comprises an uploading and downloading module, a manual marking module and an automatic marking module; the network service platform comprises a task publisher module, a task manager module and a task annotator module; the uploading and downloading module downloads or uploads the marked data; the manual marking module integrates at least one of point marking, line marking and two-dimensional frame marking; the automatic labeling module realizes semantic segmentation and labeling of drivable area analysis and assists in labeling through an artificial intelligence algorithm; the task publisher module publishes the annotation task; the task manager module manages and marks the progress of the task; and the task annotator module annotates data and uploads an annotation result. The automatic labeling and the manual labeling are combined, so that the defect of efficiency of the existing labeling software on large-scale labeling is overcome.

Description

Data annotation system and method based on network service platform

Technical Field

The invention belongs to the field of artificial intelligence, and relates to a data annotation system and method based on a network service platform.

Background

With the development of deep learning in recent years, some great breakthroughs appear in the aspects of computer vision, natural speech processing and the like, and with the rapid development of deep learning methods, the demand of data annotation is greatly increased. The marked mass data is grain of an artificial intelligence related algorithm, the marked training data is a necessary and expensive business, most marked data are manually annotated at present, and the process is very slow and inefficient, and people are required to sit in front of a computer screen to operate an annotator, click on the data and mark the data one by one. Also, as the amount of data increases, this process becomes more expensive and error prone. Taking ImageNet as an example, ImageNet is the largest database of image recognition in the world at present, namely a data set of 1,500 million labeled pictures, which is obtained by cleaning, classifying and marking nearly one billion pictures collected through the Internet, wherein 48,940 workers from 167 countries spend 2 years. A large amount of marked data can train a more excellent model to solve the requirements of a real scene, and the scene application of each AI technology needs the support of mass data. The more accurate and the more the data are labeled, the better the effect of the algorithm model is, and the high-quality data label determines the industry competitiveness of an AI company. At present, a large-scale data annotation system based on a network service platform is still lacking in the market to solve the requirement, and the types which can be annotated by the existing annotation software are still too single.

Disclosure of Invention

The invention aims to: the data annotation system based on the network service platform for efficiently annotating the data on a large scale is provided.

The technical scheme of the invention is as follows:

in a first aspect, a data annotation system based on a network service platform includes: a software client and a network service platform;

the software client comprises an uploading and downloading module, a manual marking module and an automatic marking module; the network service platform comprises a task publisher module, a task manager module and a task annotator module;

the uploading and downloading module downloads the marked data and the marked types required by the tasks to the local software client, and uploads the marked results to the network service platform after marking is finished;

the manual marking module integrates at least one of point marking, line marking and two-dimensional frame marking;

the automatic labeling module realizes semantic segmentation labeling of drivable areas and assists labeling through an artificial intelligence algorithm;

the task publisher module publishes a labeling task;

the task manager module manages the progress of the annotated task;

and the task annotator module annotates data and uploads an annotation result.

The further technical scheme is as follows: the software client further comprises an acceptance module, a decryption module, a price calculation module, a remote updating module and a format conversion module;

the acceptance module accepts the marked data according to the set acceptance mode;

the decryption module is used for carrying out advanced encryption on the marked data;

the price calculation module counts the number and the type of the marked data, and calculates the marking cost according to the number and the type of the marked data;

the remote updating module remotely updates the software;

and the format conversion module is used for converting the format of the marked data.

The further technical scheme is as follows: the automatic labeling module integrates at least one of basic labeling, sensor fusion labeling and three-dimensional point cloud labeling.

In a second aspect, a data annotation method based on a network service platform is applied to the data annotation system based on the network service platform in the first aspect, and the data annotation method includes:

the method comprises the steps that a task publisher module publishes a labeling task;

the assignment of the labeling tasks is carried out through a task manager module;

receiving a labeling task through a task annotator module;

the method comprises the steps that annotation data are downloaded to a local software client through an uploading and downloading module;

data marking is completed through an automatic marking module and a manual marking module;

uploading the labeling result to a network service platform through an uploading and downloading module;

checking the marking result on line through a task manager module;

if the inspection is qualified, the task manager module uploads a final labeling result;

if the data is not qualified, the data needing to be re-marked is sent to the task marker module, and the step of downloading the marked data to the local software client through the uploading and downloading module is continuously executed.

The further technical scheme is as follows: if the inspection is qualified, after the task manager module uploads the final labeling result, the method further comprises the following steps:

counting the number of labels and the label type of the labeled data through a price calculation module, and calculating the labeling cost according to the number of labels and the label type;

and paying the cost through the task manager module according to the calculated marking cost.

The further technical scheme is as follows: accomplish data annotation through automatic marking module and manual marking module, include:

analyzing the data by adopting a deep learning algorithm, and training to obtain classification detection characteristics;

performing large sample training through a convolutional learning network to obtain a classifier;

extracting a target object in the data through a classifier, and automatically marking the extracted object;

the automatically labeled results are verified manually.

The invention has the advantages that:

1. the automatic labeling and the manual labeling are combined, so that the defect of the efficiency of the current labeling software on large-scale labeling is overcome, the effective management of large-scale data labeling tasks is realized based on the integration, statistics, evaluation and management functions of a network service platform, and personnel, the labeling progress and the labeling quality can be managed simultaneously;

2. an acceptance module, a price calculation module and the like are added to assist acceptance personnel in evaluating submitted marking data, and meanwhile, the workload, the marking quality, the marking time and the like can be counted;

3. by integrating basic labeling, sensor fusion labeling, three-dimensional point cloud labeling and the like, the defects of the conventional labeling system in management and labeling types are overcome, a 3D labeling function is added, and the joint labeling of the three-dimensional laser radar point cloud and the cloud image is realized; automatic labeling is realized through a deep learning algorithm, and manpower is greatly reduced.

Drawings

The invention is further described with reference to the following figures and examples:

FIG. 1 is a schematic structural diagram of a data annotation system based on a web service platform provided in the present application;

FIG. 2 is a schematic structural diagram of another data annotation system based on a network service platform provided in the present application;

FIG. 3 is a flowchart of a data annotation method based on a web service platform provided in the present application;

fig. 4 is a flowchart of another data annotation method based on a network service platform provided in the present application.

Detailed Description

Example (b): the application provides a data annotation system based on a network service platform, as shown in fig. 1, the data annotation system may include: software client and network service platform.

The software client comprises an uploading and downloading module, a manual labeling module and an automatic labeling module.

And the uploading and downloading module downloads the marking data and the marking types required by the tasks to the local software client, and uploads the marking results to the network service platform after marking is finished.

The manual marking module integrates at least one of point marking, line marking and two-dimensional frame marking.

For example, point labeling can be used for face key point positioning and face recognition; line marking can be used for lane line detection and identification; the two-dimensional frame can be a vehicle, a pedestrian, a traffic sign and other target identification, such as a rectangular frame marked out. In practical application, the method can also comprise three-dimensional point cloud labeling for detecting a three-dimensional target.

The automatic labeling module realizes semantic segmentation and labeling of drivable areas, and labels are assisted through an artificial intelligence algorithm. Compared with manual labeling, the method reduces time and cost, is fused with sensor data, and can realize more accurate labeling.

The network service platform can be used for dispatching and receiving the labeling tasks, the functions comprise task issuing, task inquiring, progress inquiring, labor cost inquiring, account summarizing and the like, complex manual management is integrated on background service, the burden of a labeling manager is reduced, and the labeling efficiency is improved. The network service platform comprises a task publisher module, a task manager module and a task annotator module, and all the modules are endowed with different authorities.

The task publisher module publishes the annotation task, can publish the annotation task on the network service platform and check the task progress.

And the task manager module manages the progress of the labeling task.

The specific method can specifically comprise the steps of feeding back the labeling task, performing labor cost statistics, summarizing accounts and the like.

The task annotator module annotates data and uploads annotation results, and is mainly used for managing own annotation tasks.

And the annotation task publisher publishes the annotation task, manages the progress of the annotation task, a task manager distributes the annotation task, manages the progress of annotation personnel, the task annotator annotates data and uploads an annotation result, and finally the project manager calculates the price.

Optionally, with reference to fig. 2, the software client further includes an acceptance module, a decryption module, a price calculation module, a remote update module, and a format conversion module.

And the acceptance module accepts the marked data according to the set acceptance mode.

Optionally, the acceptance module provides functions of one-key drawing, one-key rework, rapid warehousing and the like.

The decryption module performs advanced encryption on the marked data to ensure data security.

And the price calculation module counts the number of labels and the label types of the labeled data and calculates the labeling cost according to the number of labels and the label types.

The remote updating module remotely updates the software.

The software bug can be solved in time through remote updating, and new functions can be added according to the feedback of the user.

The format conversion module is used for carrying out format conversion on the annotation data and carrying out format conversion on the annotation data needing to be converted.

The software client can be divided into a server side and a user side, the server side realizes an automatic labeling function, and the user side performs manual modification, acceptance check and other work and has a two-dimensional frame, point, line, contour, three-dimensional frame and sensor fusion labeling function.

In practical application, a labeling task publisher can publish a labeling task through a network service platform, and a task manager distributes a labeling task after querying the task from the service platform. And the annotator downloads the annotation data from the service platform to the local software client for annotation operation, uploads the annotation data to the network service platform after completing the annotation task, and the task manager verifies the annotation result and submits the final annotation data.

Optionally, the automatic labeling module integrates at least one of basic labeling, sensor fusion labeling, and three-dimensional point cloud labeling.

The basic annotation comprises the annotation functions of points, lines, circles, polygons and the like, the image content to be annotated by the basic annotation comprises figures, buildings, plants, roads, traffic signs, vehicles and the like, and different annotation tools can be used for annotation under each item of content.

The sensor fusion marking is mainly to fuse the image and the laser radar.

And the three-dimensional point cloud labeling is to project the frame on the picture into a three-dimensional space, modify the point cloud in the three-dimensional space and finally generate qualified three-dimensional labeling data.

The present application further provides a data annotation method based on a network service platform, which is applied to a data annotation system based on a network service platform as shown in fig. 1 or fig. 2, and as shown in fig. 3, the method may include:

and step 10, issuing the annotation task through a task issuer module.

And the task publisher creates a labeling task on the network service platform, uploads data, uploads a labeling document and sets the deadline.

And 20, allocating the labeling tasks through the task manager module.

And (4) configuring a project by a task manager, and assigning task annotating personnel on the network service platform.

And step 30, receiving the labeling task through the task labeling module.

And the task annotator receives the annotation task, downloads the annotation data and the annotation rules from the network service platform and starts to perform annotation operation.

And step 40, downloading the annotation data to the local software client through the uploading and downloading module.

And 50, completing data annotation through the automatic annotation module and the manual annotation module.

And 60, uploading the labeling result to a network service platform through an uploading and downloading module.

And after the annotating personnel finishes the annotating task, uploading the annotated task to the network service platform by using the uploading and downloading module.

And 70, checking the labeling result on line through the task manager module.

And the task manager checks the labeling quality, picks out data with unqualified labels and prompts the task annotator to label again.

In addition, in practical application, if the task progress is delayed, the task progress can be reminded through the network service platform.

And 80, if the inspection is qualified, uploading a final labeling result by the task manager module.

And step 90, if the inspection is not qualified, sending the data needing to be re-labeled to the task labeling module, and continuing to execute the step 40.

Optionally, for the data annotation system based on the network service platform in fig. 2, after step 80, the method may further include: counting the number of labels and the label type of the labeled data through a price calculation module, and calculating the labeling cost according to the number of labels and the label type; and paying the cost through the task manager module according to the calculated marking cost.

The network service platform can calculate the number of the labels and calculate the price after the task is completed. Correspondingly, after the labeling task is completed, the labeling task publisher checks the labeling condition on the network service platform, downloads the labeling result and pays the labeling cost.

Referring collectively to FIG. 4, a flow diagram of a web services platform based data annotation is shown.

Adopt the deep learning algorithm automatic scene in with the picture to directly cut apart and automatic mark, compensatied artifical mark and wasted time and energy the problem, combine with the labeling software through artificial intelligence algorithm, realize the semi-automatic and automatic marking function of big data, for artificial intelligence algorithm provides extensive marking data, to step 50, can include following step:

firstly, analyzing data by adopting a deep learning algorithm, and training to obtain classification detection characteristics.

Optionally, the pictures can be analyzed by adopting a deep learning fast-RCNN, and image classification detection features are obtained through training.

And secondly, performing large sample training through a convolution learning network to obtain a classifier.

Alternatively, a CNN convolutional learning network may be employed.

And thirdly, extracting the target object in the data through a classifier, and automatically marking the extracted object.

Various scenes or objects can be extracted from the picture through the classifier, and various separated objects are automatically marked.

And fourthly, checking the automatically marked result manually.

The manual reinspection can improve the accuracy of the labeling.

Aiming at the problems that the existing labeling software can only label data with a few types and cannot provide rapid interaction of large-scale labeling tasks, the application provides that the effective management of the large-scale labeling tasks is realized by establishing a network service platform. Meanwhile, aiming at the increasing three-dimensional data labeling requirement, the labeling system provides the labeling support of the 3D point cloud data. The automatic labeling is realized by utilizing a deep learning algorithm, so that the labor is greatly reduced, the manual labeling level is exceeded, and more precise labeling service is provided.

The effective management of large-scale data annotation tasks is realized by integrating a statistic evaluation function through a network service platform, the statistic evaluation is carried out on the workload, the annotation quality, the annotation time and the like, and meanwhile, the management of annotation personnel, the management of annotation progress and the management of annotation quality are carried out, so that an information background management system for realizing remote update of training suggestions of the annotation personnel and modification suggestions of software is formed.

Aiming at the defects of the labeling type of the current labeling software, a 3D labeling function is added, and 3D point cloud data obtained by collecting data through a laser radar are labeled.

The automatic labeling is realized through a deep learning algorithm, the manpower is greatly reduced, the labeling level exceeding the human labeling level is realized, more precise labeling service is provided, core algorithms such as object detection and regression, interactive image segmentation, multi-sensor fusion and the like are researched and developed and continuously optimized, and a new technical scheme is provided for the research of products of the type.

And (3) performing multi-sensor data fusion labeling, aligning data obtained by the laser radar and the camera in time, and realizing joint labeling of the point cloud of the three-dimensional laser radar and the image.

In summary, the data annotation system and method based on the network service platform provided by the application make up the deficiency of efficiency of current annotation software on large-scale annotation by combining automatic annotation and manual annotation, assist the acceptance personnel to carry out statistical evaluation on the result submitted by the annotation personnel, can carry out statistical evaluation on workload, annotation quality, annotation time and the like, realize effective management of large-scale data annotation tasks based on the integrated statistical evaluation management function of the network service platform, and can manage personnel, annotation progress and annotation quality simultaneously.

In addition, an acceptance module, a price calculation module and the like are added to assist acceptance staff in evaluating submitted annotation data, and meanwhile, the workload, the annotation quality, the annotation time and the like can be counted.

In addition, by integrating basic labeling, sensor fusion labeling, three-dimensional point cloud labeling and the like, the defects of the conventional labeling system in management and labeling types are overcome, a 3D labeling function is added, and the joint labeling of the three-dimensional laser radar point cloud and the cloud image is realized; automatic labeling is realized through a deep learning algorithm, and manpower is greatly reduced.

The terms "first" and "second" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implying a number of the indicated technical features. Thus, a defined feature of "first", "second", may explicitly or implicitly include one or more of that feature. In the description of the present application, "a plurality" means two or more unless otherwise specified.

The above-mentioned serial numbers of the embodiments of the present application are merely for description and do not represent the merits of the embodiments.

It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, where the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk, an optical disk, or the like.

The above description is only exemplary of the present application and should not be taken as limiting the present application, as any modification, equivalent replacement, or improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A data annotation system based on a network service platform is characterized by comprising: a software client and a network service platform;

the task publisher module publishes a labeling task;

the task manager module manages the progress of the annotated task;

and the task annotator module annotates data and uploads an annotation result.

2. The data annotation system based on the network service platform as claimed in claim 1, wherein the software client further comprises an acceptance module, a decryption module, a price calculation module, a remote update module, and a format conversion module;

the remote updating module remotely updates the software;

3. The network service platform based data annotation system of claim 2, wherein said automatic annotation module integrates at least one of a base annotation, a sensor fusion annotation, and a three-dimensional point cloud annotation.

4. A data annotation method based on a network service platform, which is applied to the data annotation system based on the network service platform according to any one of claims 1 to 3, the data annotation method comprising:

receiving a labeling task through a task annotator module;

checking the marking result on line through a task manager module;

5. The data annotation method based on network service platform as claimed in claim 4, applied to the data annotation system based on network service platform as claimed in claim 2, wherein if the check is qualified, after the task manager module uploads the final annotation result, the method further comprises:

6. The data annotation method based on the network service platform as claimed in claim 4 or 5, wherein the data annotation is performed by an automatic annotation module and a manual annotation module, and comprises:

the automatically labeled results are verified manually.