CN110674355B

CN110674355B - DSL application system for describing data labeling task and method thereof

Info

Publication number: CN110674355B
Application number: CN201910909874.1A
Authority: CN
Inventors: 王红宾; 吕昊; 韩奕; 李梦钰
Original assignee: Shanghai Yitu Information Technology Co ltd
Current assignee: Shanghai Yitu Information Technology Co ltd
Priority date: 2019-09-25
Filing date: 2019-09-25
Publication date: 2022-07-01
Anticipated expiration: 2039-09-25
Also published as: CN110674355A

Abstract

The invention discloses a DSL application system and a method for describing a data labeling task, wherein the DSL application system comprises: the release service module, the marking service module and the export service module; the release service module is used for receiving DSL information of a task publisher and creating a corresponding task instance according to the DSL information; the marking service module is connected with the issuing service module and is used for marking the task instance created by the issuing service module and obtaining a marking result; the export service module is connected with the labeling service module and used for collecting labeling results of the tasks, processing the labeling data and outputting the task result data. The invention can improve the starting efficiency of the labeling task and the expandability of the system.

Description

DSL application system for describing data labeling task and method thereof

Technical Field

The invention belongs to the technical field of data processing, and particularly relates to a DSL application system for describing a data annotation task and a data annotation task description method realized based on the system.

Background

Data labeling refers to a process of describing or labeling data such as texts, pictures and voices, a large amount of labeled data is generally required to be used as a training set when a machine is trained, and the larger the data amount of the labeled data is, the more helpful the training effect is. Currently, the industry generally adopts a common structured data format such as JSON, XML and the like to represent task attributes. The problems of the prior art are as follows: the starting efficiency is low, and the system expandability is poor. Meanwhile, according to research, the DSL for describing data labeling tasks of the same type is not found at present. DSL is a static type programming language that runs on a Java virtual machine. DSL is first of all a tool to help users abstract certain parts from a system. DSL is useful "when you are aware that you need a component, or when you already have a component and you want to simplify the way it is operated. The use of DSL does provide certain benefits. DSL not only improves code legibility to allow developers to better communicate with domain experts, but is also a means to change execution context, for example: switching logic from compile time to run time or switching to declarative computational models when imperative programming is not appropriate. External DSL: unlike languages in which the application system primarily uses languages, custom syntax is typically employed, and the code of the host application parses scripts written by the external DSL using text parsing techniques. Examples are: regular expressions, SQL, AWK, and Struts' profiles, etc. Internal DSL: the specific syntax of the general language, the script written with internal DSL is a legal program, but it has a specific style and only uses a part of the characteristics of the language to deal with a small aspect of the overall system. How to develop an application system for describing a data annotation task based on DSL, which can improve the initiation efficiency of the annotation task and improve the scalability of the system, is a direction that needs to be studied by those skilled in the art.

Disclosure of Invention

The technical problem to be solved by the invention is to provide a DSL application system for describing a data annotation task, which can improve the starting efficiency of the annotation task and the expandability of the system. Therefore, the invention also provides a data annotation task description method realized based on the system.

In order to solve the technical problems, the technical scheme adopted by the invention is as follows:

a DSL application system for describing data annotation tasks, comprising: the release service module, the marking service module and the export service module; the release service module is used for receiving DSL information of a task publisher and creating a corresponding task instance according to the DSL information; the marking service module is connected with the issuing service module and is used for marking the task instance created by the issuing service module and obtaining a marking result; the export service module is connected with the marking service module and used for collecting marking results of the tasks, processing marking data and outputting task result data.

Preferably, in the DSL application system for describing data annotation tasks, the following steps are performed: the task issuing module comprises a task issuing interface, a DSL resolver, a task checking unit and a first safety sandbox; the release task interface is used for receiving DSL information of a task publisher; the DSL resolver is connected with the release task interface and used for resolving the received DSL information to generate a task instance; the task checking unit is connected with the DSL resolver and used for verifying the task instance generated by the DSL resolver; the first safety sandbox is connected with the task checking unit and used for preprocessing the verified task instance and creating a corresponding task instance according to the opportunity defined in the DSL.

More preferably, in the DSL application system for describing data annotation task, the following steps are performed: the annotation service module comprises a data annotation unit, and the data annotation unit is used for annotating the task instance created by the task issuing module.

Further preferably, in the DSL application system for describing data annotation task, the following steps are performed: the export service module comprises a task data collection module, a second safety sandbox, a task data export module and a result export interface; the task data collection module is used for collecting the marking data of each task instance; the second safety sandbox is connected with the task data collection module and used for performing post-processing defined by DSL on the marked data and obtaining task result data; and the task data export module is connected with the second safety sandbox and the result export interface and is used for calling out the task result data to the result export interface.

Compared with the traditional system using the data exchange languages JSON and XML, the DSL system has the following advantages:

more comprehensive data types are supported. The data types supported by JSON and XML comprise numerical values, character strings, Boolean values, ordered lists of values, objects and null types; the DSL provided by the invention not only supports the data types mentioned above, but also supports advanced data types such as LIST/MAP/SET;

a custom method is supported. JSON, XML, and the like cannot be used to implement methods, but the DSL mentioned in the present invention can implement class member methods, such as task preprocessing and post-processing methods;

and supporting the related processing of the task flow. The DSL provided by the invention supports the concepts of parent-child tasks and batch tasks by realizing a task tree structure, and effectively improves the efficiency of starting the data annotation task, which cannot be realized by the traditional annotation task description mode.

Based on the DSL application system, the present invention further provides a method for describing a data annotation task, which includes the following steps:

firstly, the release service module receives DSL information of a task publisher and creates a corresponding task instance according to the DSL information;

secondly, the marking service module receives the task instance created by the issuing service module, marks the task instance and obtains a marking result;

and thirdly, the export service module receives the labeling result from the labeling service module, collects the labeling result of the task, processes the labeling data and outputs the task result data.

As a preferred technical solution of the present invention, the first step specifically comprises the following steps:

s1: a task publisher uploads DSL information through a task publishing interface;

s2: the DSL resolver resolves the received DSL information to generate a task instance;

s3: syntax error detection is carried out on each task instance obtained in the step S2, and the task instance passing the syntax error detection is output to a task checking module;

s4: the task verification module performs task verification on the task instance and outputs the task instance passing the task verification to the first safety sandbox;

s5: and the first safety sandbox executes a preprocessing method on the task instance which passes the task verification and creates a corresponding task instance according to the opportunity defined in the DSL.

As a preferred technical scheme of the present invention, the second step specifically comprises:

s6: and the labeling service module labels the task instance obtained by the publishing service module, generates a labeling question according to the task configuration, and performs labeling by a labeling person to obtain a labeling result.

As a preferred technical solution of the present invention, step S6 specifically includes the following steps: informing the marking service to mark, starting a marking task, judging whether unfinished questions exist or not, and if so, marking by a marker; if not, the export service is notified to collect.

As a preferred technical solution of the present invention, the third step specifically includes the steps of:

s7: the task data collection module collects the labeling data of each task instance obtained by the labeling service module;

s8: the second safety sandbox executes post-processing defined by DSL (digital subscriber line) on the marked data to obtain task result data;

s9: and calling out the task result data obtained in the step S8 to a result exporting interface by the task data exporting module.

Compared with the prior art, the method and the system can describe the task attributes and characteristics more conveniently and quickly, can realize the control of the task flow through self-defining the function in the sandbox, greatly improve the starting efficiency of the labeling task, and endow the labeling system with great expandability. By utilizing the method and the device, the language description of the complex data processing logic is simplified, the comprehensive annotation task grammar is provided, and the annotation task starting efficiency and expandability are improved.

Compared with the traditional system using the data exchange languages JSON and XML, the DSL system provided by the invention has the following advantages:

1. more comprehensive data types are supported. The data types supported by JSON and XML comprise numerical values, character strings, Boolean values, ordered lists of values, objects and null types; the DSL provided by the invention not only supports the data types mentioned above, but also supports advanced data types such as LIST/MAP/SET;

2. a custom method is supported. JSON, XML and the like can not be used for realizing methods, but the DSL mentioned in the invention can realize member methods of classes, such as a task preprocessing method, a post-processing method and the like;

3. and supporting the related processing of the task flow. The DSL provided by the invention supports the concepts of parent-child tasks and batch tasks by realizing a task tree structure, and effectively improves the efficiency of starting the data annotation task, which cannot be realized by the traditional annotation task description mode.

Drawings

The invention is described in further detail in the following description of embodiments with reference to the accompanying drawings:

FIG. 1 is a schematic structural diagram of the present invention.

Fig. 2 is a flow chart of the operation of the present invention.

FIG. 3 is a detailed flow chart of the first step of the method of the present invention.

FIG. 4 is a detailed flow chart of the second step of the method of the present invention.

FIG. 5 is a detailed flow chart of the third step of the method of the present invention.

The corresponding relation between each reference mark and the part name is as follows:

1. a publishing service module; 2. a labeling service module; 3. a exporting service module; 11. issuing a task interface; 12. a DSL resolver; 13. a task checking unit; 14. a first secure sandbox; 21. a data labeling unit; 31. a task data collection module; 32. a second secure sandbox; 33. a task data export module; 34. the result is exported to the interface.

Detailed Description

In order to more clearly illustrate the technical solution of the present invention, the following embodiments will be further described with reference to the accompanying drawings.

As shown in fig. 1, the DSL application system for describing data annotation task of the present invention includes: the system comprises a publishing service module 1, a marking service module 2 and an exporting service module 3; the release service module 1 is configured to receive DSL information of a task publisher and create a corresponding task instance according to the DSL information; the marking service module 2 is connected with the publishing service module 1 and is used for marking the task instance created by the publishing service module 1 and obtaining a marking result; the export service module 3 is connected with the marking service module 2 and used for collecting marking results of tasks, processing marking data and outputting task result data.

Specifically, the method comprises the following steps: the task issuing module 1 comprises a task issuing interface 11, a DSL resolver 12, a task checking unit 13 and a first security sandbox 14; the release task interface 11 is configured to receive DSL information of a task publisher; the DSL parser 12 is connected to the release task interface 11, and configured to parse the received DSL information to generate a task instance; the task checking unit 13 is connected to the DSL resolver 12, and configured to verify a task instance generated by the DSL resolver 12; the first secure sandbox 14 is connected to the task checking unit 13, and is configured to execute a pre-processing method defined in the DSL on the verified task instance, and create a corresponding task instance according to a time defined in the DSL. The annotation service module 2 comprises a data annotation unit 21, and the data annotation unit 21 is used for annotating the task instance created by the task issuing module 1. The export service module 3 comprises a task data collection module 31, a second security sandbox 32, a task data export module 33 and a result export interface 34; the task data collection module 31 is configured to collect annotation data of each task instance; the second secure sandbox 32 is connected to the task data collecting module 31, and is configured to perform DSL-defined post-processing on the tagged data to obtain task result data; the task data export module 33 is connected to the second secure sandbox 32 and the result export interface 34, and is configured to call out the task result data to the result export interface 34.

Since the DSL of the present invention is implemented based on Kotlin (Kotlin is a static programming language for modern multi-platform applications), its language characteristics are inherited.

The implementation process comprises the following steps:

1. an Interface of the annotation task, i.e. a publishing task Interface 11, is defined for receiving DSL information of the task publisher. The DSL information here refers to an API interface, to which the user sends the DSL by a POST method, and the service end receives and stores the received DSL.

2. Defining Data Class related to annotation task, wherein the Data Class refers to created Class only storing Data. Within these classes, some standard functions are often 3 derived from the data machine, implementing a dslmaker annotation class;

4. implementing a DSL language builder

Among them, the DSL language builder refers to a static type builder that creates type safety by combining functional text with a receiver through a method of specification naming. Complex multi-level data structures can be implemented by DSL language builders.

The DSL parser completes the parsing by internally invoking the DSL language builder. In order to implement the DSL language constructor, convenience and extensibility of describing the labeling task need to be fully considered, and particularly in the constructor of the tree task, the construction of the task book needs to be completed in a recursive manner. In the parsing process, after the DSL is received by the release task interface 11, the DSL is parsed by a script engine, and an instance of the tagged task class is output for subsequent processing. The script engine described herein is a script parsing engine provided by Javax.

As shown in fig. 2, a data annotation task description method implemented by a DSL application system based on the description data annotation task of the present invention includes the following steps:

firstly, the release service module (1) receives DSL information of a task publisher and creates a corresponding task instance according to the DSL information;

secondly, the labeling service module (2) receives the task instance created by the publishing service module (1), labels the task instance and obtains a labeling result;

and thirdly, the export service module (3) receives the labeling result from the labeling service module (2), collects the labeling result of the task, processes the labeling data and outputs the task result data.

As shown in fig. 3, the first step specifically includes the following steps:

s1: a task publisher uploads DSL information by calling a task publishing interface 11;

s2: the DSL resolver 12 resolves the received DSL information to generate a task instance;

s3: syntax error detection is carried out on each task instance obtained in the S2, and if syntax errors exist, error information is returned; if there is no syntax error, outputting the task instance passing the syntax error detection to the task checking module 13;

s4: the task checking module 13 performs task verification on the task instance, and if the task instance fails the task verification, returns error information; if the task passes the task verification, outputting the task instance passing the task verification to the first security sandbox 14;

s5: the first security sandbox 14 executes a preprocessing method on the task instance passing the task verification, and creates a corresponding task instance according to the opportunity defined in the DSL; firstly, judging whether a preprocessing method is realized, if so, executing the preprocessing method, and then creating a corresponding task instance according to the time defined in the DSL; if not, directly creating the task.

As shown in fig. 4, the second step specifically includes the following steps:

s6: the annotation service module 2 annotates the task instance obtained in the step S5, generates an annotation question according to task configuration, and performs annotation by an annotator to obtain an annotation result; the method specifically comprises the following steps: informing the marking service to mark, starting a marking task, judging whether unfinished questions exist or not, and if so, marking by a marker; if not, informing the export service to collect;

as shown in fig. 5, the third step specifically includes the following steps:

s7: the task data collection module 31 collects the annotation data of each task instance obtained in S6;

s8: the second secure sandbox 32 performs post-processing defined by the DSL on the marked data to obtain task result data; firstly, judging whether a post-processing method is realized, if so, executing the post-processing method to generate a task result; if not, directly generating a task result.

S9: the task data export module 33 calls out the task result data obtained in S8 to the result export interface 34.

Embodiment 1, single-tasking DSL example:

the above example defines an annotation task a by DSL, the type of which is object _ detection and the task priority 4. The task A defines a preprocessing flow, and the tasks are marked after preprocessing; a defines a post-processing flow, and the post-processing is started after the labeling task is finished.

Example 2: bulk task DSL examples

The above example defines a callout task group by DSL, the type of which is object _ detection. The task group contains three tasks a _1, a _2 and a _3, and the subtasks automatically inherit all basic properties of the task group, with the task type being object _ detection.

Example 3: task tree DSL example

The above example defines an annotation task a by DSL, the type of which is object _ detection and the task priority 4. The task A comprises two subtasks A _1 and A _2, and the subtasks automatically inherit all basic attributes of the parent task A, but the task priorities are different from A and are respectively 3 and 2; the task A _1 comprises two subtasks, namely A _1_1 and A _1_ 2; the task A _2 defines a preprocessing flow, and the labeling task is started after preprocessing; all subtasks are finished, the mark task A is finished, and the task A defines a post-processing flow, so that the post-processing can be started after all the labeling tasks are finished.

The above description is only an embodiment of the present invention, but the scope of the present invention is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present invention are included in the scope of the present invention. The protection scope of the present invention is subject to the protection scope of the claims.

Claims

1. A DSL application for describing data annotation tasks, comprising: the system comprises a publishing service module (1), a marking service module (2) and an exporting service module (3);

the release service module (1) is used for receiving DSL information of a task publisher and creating a corresponding task instance according to the DSL information; the marking service module (2) is connected with the publishing service module (1) and is used for marking the task instance created by the publishing service module (1) and obtaining a marking result; the export service module (3) is connected with the labeling service module (2) and is used for collecting labeling results of tasks, processing labeling data and outputting task result data;

the release service module (1) comprises a release task interface (11), a DSL resolver (12), a task verification unit (13) and a first security sandbox (14);

the task issuing interface (11) is used for receiving DSL information of a task issuer; the DSL resolver (12) is connected with the issuing task interface (11) and used for resolving the received DSL information to generate a task instance; the task verification unit (13) is connected with the DSL resolver (12) and used for verifying the task instance generated by the DSL resolver (12); the first safety sandbox (14) is connected with the task checking unit (13) and used for preprocessing the verified task instances and creating corresponding task instances according to the time defined in the DSL;

the export service module (3) comprises a task data collection module (31), a second safety sandbox (32), a task data export module (33) and a result export interface (34);

the task data collection module (31) is used for collecting the annotation data of each task instance; the second safety sandbox (32) is connected with the task data collection module (31) and used for performing DSL-defined post-processing on the marked data and obtaining task result data; and the task data export module (33) is connected with the second safety sandbox (32) and the result export interface (34) and is used for calling out the task result data to the result export interface (34).

2. The DSL application for describing data annotation tasks of claim 1 wherein: the annotation service module (2) comprises a data annotation unit (21), and the data annotation unit (21) is used for annotating the task instance created by the publishing service module (1).

3. A description method of a data annotation task is characterized by comprising the following steps:

firstly, a release service module (1) receives DSL information of a task publisher and creates a corresponding task instance according to the DSL information;

thirdly, the export service module (3) receives the labeling result from the labeling service module (2), collects the labeling result of the task, processes the labeling data and outputs the task result data;

the first step specifically comprises the following steps:

s1: the task publisher uploads DSL information through a task publishing interface (11);

s2: the DSL resolver (12) resolves the received DSL information to generate a task instance;

s3: syntax error detection is carried out on each task instance obtained in the step S2, and the task instance passing the syntax error detection is output to a task checking module (13);

s4: the task verification module (13) performs task verification on the task instance and outputs the task instance passing the task verification to the first safety sandbox (14);

s5: the first safety sandbox (14) executes a preprocessing method on the task instance which passes the task verification, and creates a corresponding task instance according to the opportunity defined in the DSL;

the third step specifically comprises the following steps:

s7: the task data collection module (31) collects the labeling data of each task instance obtained by the labeling service module (2);

s8: the second safety sandbox (32) executes post-processing defined by DSL on the marked data to obtain task result data;

s9: and the task data export module (33) calls out the task result data obtained in the step S8 to the result export interface (34).

4. The method according to claim 3, characterized in that the second step is embodied as:

s6: and the labeling service module (2) labels the task instance obtained by the publishing service module (1), generates a labeling question according to task configuration, and performs labeling by a labeling person to obtain a labeling result.

5. The method according to claim 4, wherein S6 particularly includes the steps of: informing the marking service to mark, starting a marking task, judging whether unfinished questions exist or not, and if so, marking by a marker; if not, the export service is notified to collect.