CN113435800A

CN113435800A - Method and device for executing labeling task based on big data, electronic equipment and medium

Info

Publication number: CN113435800A
Application number: CN202110990335.2A
Authority: CN
Inventors: 姜敏华
Original assignee: Ping An Technology Shenzhen Co Ltd
Current assignee: Ping An Technology Shenzhen Co Ltd
Priority date: 2021-08-26
Filing date: 2021-08-26
Publication date: 2021-09-24

Abstract

The application relates to the technical field of artificial intelligence, and provides a method, a device, electronic equipment and a medium for executing a labeling task based on big data, wherein the method comprises the following steps: acquiring a target annotator according to the target field and the target annotation rating, and generating a first test question set according to the annotation rating of the target annotator; performing professional ability evaluation on the target annotator based on the first test question set; when the target annotator has the ability to execute the task to be annotated and meets the requirement of a preset trigger event, carrying out psychological ability evaluation on the target annotator; and executing the task to be marked after the psychological test is passed, and evaluating the marking capability of the target marker according to the marking result. According to the method and the device, the target annotator is subjected to capability assessment from three dimensions of professional ability, psychological ability and accuracy of the annotation result, the accuracy of the obtained target annotator is ensured, and then the annotation efficiency and the accuracy are improved.

Description

Method and device for executing labeling task based on big data, electronic equipment and medium

Technical Field

The application relates to the technical field of artificial intelligence, in particular to a method and a device for executing a labeling task based on big data, electronic equipment and a medium.

Background

High-quality labeling data sets are very important resources in the field of computer research and application, and in the process of data labeling, the existing crowdsourcing technology needs to distribute labeling tasks to a plurality of labeling persons for labeling.

However, due to differences in professional levels, psychological states, understanding of problems, and the like of annotators, or problems themselves are difficult to annotate, the annotation is prone to have wrong labels, and the annotation quality and accuracy are low.

Therefore, it is necessary to provide a method for performing the labeling task quickly and accurately.

Disclosure of Invention

In view of the above, it is necessary to provide a method, an apparatus, an electronic device and a medium for executing a labeling task based on big data, which perform capability assessment on a target annotator from three dimensions of professional ability, psychological ability and accuracy of a labeling result, so as to ensure the accuracy of the obtained target annotator and further improve the labeling efficiency and accuracy.

A first aspect of the present application provides a method for executing an annotation task based on big data, where the method includes:

responding to a received labeling task request, and acquiring a task to be labeled;

identifying the target field of the task to be labeled and the target labeling rating of the corresponding label;

determining a target annotator from a preset annotator database according to the target field and the target annotation rating, and generating a corresponding first test question set according to the annotation rating of the target annotator;

performing professional ability evaluation on the target annotator based on the first test question set, and judging whether the target annotator has the ability to execute the task to be annotated according to a professional ability evaluation result;

when the target annotator has the capability of executing the task to be annotated, detecting whether the target annotator meets the requirement of a preset trigger event;

when the target annotator meets the preset trigger event requirement, determining a second test question set matched with the target annotator from a preset psychological state test question library, and performing psychological ability evaluation on the target annotator based on the second test question set;

and when the psychological ability evaluation result of the target annotator is that the psychological test passes, executing the task to be annotated, and evaluating the annotation ability of the target annotator according to the annotation result.

Optionally, the generating a corresponding first test question set according to the labeling rating of the target labeler includes:

identifying the registration information of the target annotator and acquiring a target portrait of the target annotator, wherein the target portrait comprises an annotation rating;

and generating a corresponding first test question set according to the target image of the target annotator and the corpus content in the task to be annotated.

Optionally, the determining, according to the professional ability evaluation result, whether the target annotator has the ability to execute the task to be annotated includes:

acquiring the professional ability test score of the target annotator from the professional ability evaluation result;

comparing the professional ability test score of the target marker with a preset professional ability score threshold value of the target field;

when the professional ability test score of the target annotator is greater than or equal to a preset professional ability score threshold value of the target field, determining that the target annotator has the ability to execute the task to be annotated; or

And when the professional ability test score of the target annotator is smaller than a preset professional ability score threshold value of the target field, determining that the target annotator is incapable of executing the task to be annotated.

Optionally, the preset trigger event includes:

the target annotator executes an annotation task for the first time; or the target annotator does not execute the annotation task within a preset time period; or the historical marking level of the target marker is lower than a preset reference level; or the quality requirement grade of the task to be labeled exceeds the quality requirement grade corresponding to the labeling grade of the target labeling person.

Optionally, the determining a second test question set matched with the target annotator from a preset psychological state test question library, and performing psychological ability assessment on the target annotator based on the second test question set includes:

determining a second test question set matched with the target field and the labeling rating of the target labeling person from a preset psychological state test question library;

performing psychological ability test on the target annotator based on the second test question set, and acquiring a psychological ability test score of the target annotator;

comparing the psychological ability test score of the target marker with a preset psychological ability score threshold value;

when the psychological ability test score of the target annotator is greater than or equal to the preset psychological ability score threshold value, determining that the psychological ability evaluation result of the target annotator is that the psychological test passes; or

And when the psychological ability test score of the target annotator is smaller than the preset psychological ability score threshold value, determining that the psychological ability evaluation result of the target annotator is that the psychological test fails.

Optionally, the executing the task to be annotated and evaluating the annotation capability of the target annotator according to the annotation result includes:

generating a first test question set to be labeled according to the corpus content in the task to be labeled;

collecting a second test question set to be annotated, which is similar to the corpus content in the task to be annotated, from a pre-constructed corpus containing the annotation;

merging the first to-be-annotated test question set and the second to-be-annotated test question set to obtain a target to-be-annotated test question set;

receiving the labeling result of the target test question set to be labeled reported by the target labeling person;

extracting the labeling result of the second test question set to be labeled from the labeling result;

calculating the accuracy of the labeling result of the second test question set to be labeled;

when the accuracy of the labeling result of the second test question set to be labeled is greater than or equal to a preset accuracy threshold, determining that the labeling capacity of the target label reaches the standard; or

And when the accuracy of the labeling result of the second test question set to be labeled is smaller than the preset accuracy threshold, determining that the labeling capability of the target labeling person does not reach the standard.

Optionally, the constructing process of the annotation-containing corpus includes:

collecting a disclosed public corpus from a plurality of preset data sources;

sending an authorization request to a publisher corresponding to the non-public labeled corpus to be labeled, responding to the authorization instruction to collect the non-public labeled corpus to be labeled when detecting the authorization instruction of the publisher, and loading the non-public labeled corpus to be labeled into the public corpus to obtain the corpus containing labels.

A second aspect of the present application provides a big data-based annotation task execution device, including:

the acquisition module is used for responding to the received labeling task request and acquiring the task to be labeled;

the identification module is used for identifying the target field of the task to be labeled and the target labeling grade of the corresponding label;

the generating module is used for determining a target annotator from a preset annotator database according to the target field and the target annotation rating and generating a corresponding first test question set according to the annotation rating of the target annotator;

the judging module is used for evaluating the professional ability of the target annotator based on the first test question set and judging whether the target annotator has the ability to execute the task to be annotated according to a professional ability evaluation result;

the detection module is used for detecting whether the target annotator meets the requirement of a preset trigger event or not when the target annotator has the capability of executing the task to be annotated;

the evaluation module is used for determining a second test question set matched with the target annotator from a preset psychological state test question bank when the target annotator meets the preset trigger event requirement, and performing psychological ability evaluation on the target annotator based on the second test question set;

and the execution module is used for executing the task to be labeled when the psychological ability evaluation result of the target label maker is that the psychological test passes, and evaluating the labeling ability of the target label maker according to the labeling result.

A third aspect of the present application provides an electronic device, which includes a processor and a memory, wherein the processor is configured to implement the big data based annotation task execution method when executing a computer program stored in the memory.

A fourth aspect of the present application provides a computer-readable storage medium having stored thereon a computer program, which, when executed by a processor, implements the method for annotating tasks based on big data.

To sum up, according to the labeling task execution method, the labeling task execution device, the electronic device and the medium based on the big data, on one hand, different professional ability scoring thresholds are preset through professional ability assessment aiming at different fields, and the professional ability test score of the target label is compared with the preset professional ability scoring threshold of the target field, so that the comparison result is more reasonable and accurate, whether the target label has the ability to execute the task to be labeled is determined according to the comparison result, and the accuracy rate of the target label executing the task to be labeled is ensured; on the other hand, when the target annotator meets the preset trigger event requirement, before the task to be annotated is executed, the psychological ability of the target annotator needs to be evaluated, so that the current psychological state of the target annotator can execute the task to be annotated, and the annotation accuracy and efficiency of the task to be annotated are improved; and finally, in the process of carrying out standard evaluation on the marking capability, collecting a second to-be-marked test question set which is similar to the corpus content in the to-be-marked task from a pre-constructed marked corpus, wherein the second to-be-marked test question set comprises standard answers, evaluating the marking capability of the target marker according to the standard answers of the second to-be-marked test question set, instead of evaluating the marking capability of the target marker according to an analysis result after analyzing the marking result of the first to-be-marked test question set, the objective accuracy of evaluation is ensured, the accuracy of the evaluation result is improved, and the subsequent marking quality is improved.

Drawings

Fig. 1 is a flowchart of a method for executing a big data-based annotation task according to an embodiment of the present application.

Fig. 2 is a structural diagram of a big data based annotation task execution device according to a second embodiment of the present application.

Fig. 3 is a schematic structural diagram of an electronic device according to a third embodiment of the present application.

Detailed Description

In order that the above objects, features and advantages of the present application can be more clearly understood, a detailed description of the present application will be given below with reference to the accompanying drawings and specific embodiments. It should be noted that the embodiments and features of the embodiments of the present application may be combined with each other without conflict.

Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein in the description of the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

Example one

In this embodiment, the method for executing a big data-based annotation task may be applied to an electronic device, and for an electronic device that needs to execute a big data-based annotation task, a function provided by the method for applying for the application for executing the big data-based annotation task may be directly integrated on the electronic device, or may be run in the electronic device in a Software Development Kit (SDK) form.

The embodiment of the application can acquire and process related data based on an artificial intelligence technology. Among them, Artificial Intelligence (AI) is a theory, method, technique and application system that simulates, extends and expands human Intelligence using a digital computer or a machine controlled by a digital computer, senses the environment, acquires knowledge and uses the knowledge to obtain the best result.

The artificial intelligence infrastructure generally includes technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation/interaction systems, mechatronics, and the like. The artificial intelligence software technology mainly comprises a computer vision technology, a robot technology, a biological recognition technology, a voice processing technology, a natural language processing technology, machine learning, deep learning and the like.

As shown in fig. 1, the method for executing a labeling task based on big data specifically includes the following steps, and the order of the steps in the flowchart may be changed, and some steps may be omitted according to different requirements.

And S11, responding to the received labeling task request, and acquiring the task to be labeled.

In this embodiment, in the process of performing task labeling, a client sends a labeling task request to a server, specifically, the client may be a smart phone, an IPAD, or other existing intelligent devices, the server may be a task labeling subsystem, and in the task labeling process, if the client sends a labeling task request to the task labeling subsystem, the task labeling subsystem is configured to receive the labeling task request sent by the client and parse the labeling task request to obtain a task to be labeled.

And S12, identifying the target field of the task to be labeled and the target labeling rating of the corresponding label.

In this embodiment, the target field is used to represent a technical field corresponding to the corpus content of the task to be labeled, for example, a financial field or a computer field, labeling ratings corresponding to different fields are different, labeling tasks adapted to different labeling ratings of the labeling persons are also different, and the labeling ratings of different fields are not shared.

In an optional embodiment, the identifying the target domain of the task to be annotated and the target annotation rating of the corresponding annotator comprises:

extracting a plurality of key fields from the task to be labeled, and determining a target field corresponding to the task to be labeled according to the plurality of key fields;

and identifying the task grade of the task to be marked, and determining a target marking grade corresponding to the task to be marked according to the task grade and the target field.

In this embodiment, since labeling tasks adapted to annotators with different labeling ratings are different, the target labeling rating corresponding to the task to be labeled needs to be determined according to the task level and the target field of the labeling task, so as to ensure the accuracy of the determined target labeling rating, and further improve the labeling quality of the labeling task.

And S13, determining a target annotator from a preset annotator database according to the target field and the target annotation rating, and generating a corresponding first test question set according to the annotation rating of the target annotator.

In this embodiment, the first test question set refers to professional leveling test questions, and specifically, the professional leveling test questions are used for measuring the test questions of the professional literacy standard of the target annotator and may be collected through standard test question libraries in each vertical field or ask experts in the corresponding field to specify the questions.

In this embodiment, a annotator database may be preset, where the preset annotator database includes annotators in different fields and different levels, and when a target field and a target annotation level of a to-be-annotated task are obtained, annotators that are matched with the target field and are greater than or equal to the target annotation level are matched from the preset annotator database, and the matched annotators are determined as target annotators.

In an optional embodiment, the determining a target annotator from a preset annotator database according to the target field and the target annotation rating comprises:

screening a plurality of first annotators matched with the target field from the preset annotator database;

and screening a plurality of second annotators which are greater than or equal to the target annotation rating from the plurality of first annotators, and determining the plurality of second annotators as target annotators.

In this embodiment, the preset annotator database includes a plurality of annotators with a plurality of annotation ratings in a plurality of fields, and because the corresponding tasks to be annotated with a large annotation rating and a small annotation rating can be annotated and are compatible downward, a plurality of second annotators with a rating greater than or equal to the target annotation rating are screened from the plurality of first annotators and determined as target annotators.

In an optional embodiment, the generating the corresponding first test question set according to the annotation rating of the target annotator includes:

In this embodiment, the target portrait includes a historical annotation task executed by the target annotator and a professional background of the target annotator, and in this embodiment, a first test question set is generated by considering multiple dimensions of the target portrait of the target annotator, the annotation rating, and the corpus content of the to-be-annotated task, specifically, the first test question set is generated by three dimensions of the professional level angle, the annotation rating angle, and the corpus content of the to-be-annotated task of the target annotator, thus, it is ensured that the generated first test question set can accurately evaluate the professional ability level of the target annotator.

S14, performing professional ability evaluation on the target annotator based on the first test question set, and judging whether the target annotator has the ability to execute the task to be annotated according to a professional ability evaluation result.

In this embodiment, in the process of evaluating the target annotator, the professional ability of the target annotator is evaluated, specifically, the professional ability is used for measuring the professional literacy of the target annotator, and it is determined whether the target annotator has the ability to execute the task to be annotated according to the professional literacy level of the target annotator.

In this embodiment, the first test question set is generated from three dimensions, namely, a professional level angle, a marking rating angle and an angle of a corpus content of the to-be-marked task of the target marker, the marking of the target marker on the first test question set matches the marking result with a correct marking result to obtain a professional ability test score of each target marker, and whether the target marker has the ability to execute the to-be-marked task can be determined according to the professional ability test score.

In an optional embodiment, the determining, according to the professional ability evaluation result, whether the target annotator has the ability to execute the task to be annotated includes:

In this embodiment, different professional ability scoring thresholds are preset for professional ability evaluations in different fields, the professional ability test score of the target annotator is compared with the preset professional ability scoring threshold in the target field, so that the comparison result is more reasonable and accurate, whether the target annotator has the ability to execute the task to be annotated is determined according to the comparison result, and the accuracy of the target annotator executing the task to be annotated is ensured.

S15, when the target annotator has the ability to execute the task to be annotated, detecting whether the target annotator meets the requirement of a preset trigger event.

In this embodiment, since the psychological test is a negative disturbance to the target annotator, when it is determined that the professional ability of the target annotator meets the requirement for executing the task to be annotated, it is necessary to determine whether the target annotator triggers a preset trigger event according to the historical annotation task condition executed by the target annotator.

Specifically, the preset trigger event includes:

In this embodiment, the first-time annotation execution task is used to characterize the first-time annotation execution task of the target annotator; the historical marking level is used for representing a marking level score recorded in a preset marker database after the target marker executes a historical marking task; the quality requirement grade is used for representing the quality requirement of the task to be annotated, and different quality requirements correspond to different quality grades.

And S16, when the target annotator meets the preset trigger event requirement, determining a second test question set matched with the target annotator from a preset psychological state test question library, and performing psychological ability evaluation on the target annotator based on the second test question set.

In this embodiment, when the target annotator executes the annotation task for the first time, the psychological state of the target annotator cannot be determined, so that the psychological ability of the target annotator is evaluated; when the target annotator does not execute the annotation task for a long time, in order to ensure the annotation quality, the psychological ability of the target annotator needs to be evaluated; the historical marking level of the target marker is used for representing the situation of a historical marking task executed by the target marker, if the repeated marking performance of the target marker is poor, the historical marking level of the target marker is determined to be lower and possibly lower than a preset reference level, and the psychological ability assessment of the target marker is determined; and if the quality requirement grade of the task to be marked is strict, determining whether the quality requirement grade corresponding to the marking grade of the target marker reaches the quality requirement grade corresponding to the task to be marked, and determining to evaluate the psychological ability of the target marker when the quality requirement grade corresponding to the marking grade of the target marker does not reach the quality requirement grade corresponding to the task to be marked.

In an optional embodiment, the determining, from a preset psychological state test question library, a second test question set matching the target annotator, and performing psychological ability assessment on the target annotator based on the second test question set includes:

In this embodiment, the psychological state test questions are used for measuring the current psychological state and working attitude of the target annotator, and are constructed by referring to various psychological scale test schemes in corresponding fields, so as to determine whether the psychological state of the target annotator at a specific moment is suitable for executing the annotation task; the second test question set refers to a psychological state test question, the second test question set can be determined according to the labeling grades of different target labeling persons in different fields, and the different labeling grades in different fields correspond to different second test question sets.

In this embodiment, when the target annotator meets the requirement of a preset trigger event, before executing the task to be annotated, the target annotator needs to be psychologically evaluated to ensure that the current psychological state of the target annotator can execute the task to be annotated, thereby improving the annotation accuracy and efficiency of the task to be annotated.

And S17, when the psychological ability assessment result of the target annotator is that the psychological test is passed, executing the task to be annotated, and assessing the annotation ability of the target annotator according to the annotation result.

In this embodiment, after determining that the target annotator passes the psychological test, it is determined that the target annotator can execute the task to be annotated.

In an optional embodiment, the executing the task to be annotated and evaluating the annotation capability of the target annotator according to the annotation result includes:

In this embodiment, an accuracy threshold may be preset, and when the marking accuracy of the target marker exceeds the preset accuracy threshold, it is determined that the marking capability of the target marker reaches the standard.

In the embodiment, in the process of performing the standard-reaching evaluation of the labeling capacity, a second to-be-labeled test question set similar to the corpus content in the to-be-labeled task is collected from a pre-constructed labeled corpus, the second to-be-labeled test question set comprises standard answers, the labeling capacity of the target label is evaluated according to the standard answers of the second to-be-labeled test question set, instead of evaluating the labeling capacity of the target label according to an analysis result after analyzing the labeling result of the first to-be-labeled test question set, objective accuracy of evaluation is ensured, the accuracy of the evaluation result is improved, and further subsequent labeling quality is improved.

Further, the method further comprises:

and when the labeling capability of the target label maker does not meet the standard, adjusting the labeling rating of the target label maker.

Further, the calculating the accuracy of the labeling result of the second test question set to be labeled includes:

and matching the labeling result of the second test question set to be labeled with the standard result of the second test question set to be labeled, and obtaining the accuracy of the standard result of the second test question set to be labeled according to the matching result.

Specifically, the construction process of the annotation-containing corpus includes:

collecting a disclosed public corpus from a plurality of preset data sources;

In the embodiment, in the process of constructing the corpus containing the annotations, two dimensions of the disclosed corpus and the undisclosed corpus to be annotated are considered, so that the diversity and the integrity of the corpus in the constructed corpus containing the annotations are ensured,

further, when the psychological ability assessment result is that the psychological test fails, the task to be annotated is blocked from being executed.

In this embodiment, the mind state and the responsibility degree of the annotator have a large influence on the annotation quality, the psychological state of the target annotator is confirmed before the annotation task starts, when the psychological capability assessment result is that the psychological test fails, it is determined that the target annotator is currently not suitable for executing the annotation task, the execution of the to-be-annotated task is blocked in time, the suitable target annotator is allowed to execute the to-be-annotated task, and the annotation quality of the to-be-annotated task can be effectively ensured.

In the embodiment, based on the corpus containing the labels, the first test question set and the second test question set, the target label is subjected to capability evaluation from three dimensions of professional capability, psychological capability and accuracy of the labeling result, so that the accuracy of the obtained target label is improved, meanwhile, in the whole process evaluation process of the target label, the abnormal state of the target label can be timely found and intervened, the labeling quality of the task to be labeled is ensured, and meanwhile, the labeling efficiency and accuracy are improved.

In summary, in the method for executing a labeling task based on big data according to this embodiment, on one hand, professional ability evaluation is performed on the target annotator based on the first test question set, whether the target annotator has the ability to execute the task to be labeled is judged according to a professional ability evaluation result, different professional ability scoring thresholds are preset according to professional ability evaluations in different fields, and a professional ability test score of the target annotator is compared with a preset professional ability scoring threshold in the target field, so that the comparison result is more reasonable and accurate, and whether the target annotator has the ability to execute the task to be labeled is determined according to the comparison result, thereby ensuring the accuracy of the target annotator executing the task to be labeled; on the other hand, when the target annotator meets the preset trigger event requirement, psychological ability assessment is carried out on the target annotator, when the target annotator meets the preset trigger event requirement, before the task to be annotated is executed, the psychological ability assessment is carried out on the target annotator, so that the current psychological state of the target annotator can execute the task to be annotated, and the annotation accuracy and efficiency of the task to be annotated are improved; and finally, executing the task to be labeled, evaluating the labeling capacity of the target label according to a labeling result, collecting a second test question set to be labeled, which is similar to the corpus content in the task to be labeled, from a pre-constructed corpus containing labels in the process of standard evaluation of the labeling capacity, wherein the second test question set to be labeled comprises standard answers, evaluating the labeling capacity of the target label according to the standard answers of the second test question set to be labeled, instead of evaluating the labeling capacity of the target label according to an analysis result after analyzing the labeling result of the first test question set to be labeled, so that the objective accuracy of evaluation is ensured, the accuracy of the evaluation result is improved, and the subsequent labeling quality is improved.

Example two

In some embodiments, the big data based annotation task execution device 20 can include a plurality of functional modules composed of program code segments. The program codes of the various program segments in the big data based annotation task performing device 20 can be stored in the memory of the electronic equipment and executed by the at least one processor to perform the functions (described in detail in fig. 1) of the big data based annotation task.

In this embodiment, the annotation task execution device 20 based on big data may be divided into a plurality of functional modules according to the functions executed by the annotation task execution device. The functional module may include: the system comprises an acquisition module 201, an identification module 202, a generation module 203, a judgment module 204, a detection module 205, an evaluation module 206 and an execution module 207. A module as referred to herein is a series of computer readable instruction segments stored in a memory capable of being executed by at least one processor and capable of performing a fixed function. In the present embodiment, the functions of the modules will be described in detail in the following embodiments.

The obtaining module 201 is configured to obtain a task to be annotated in response to the received annotation task request.

And the identification module 202 is configured to identify a target field of the task to be annotated and a target annotation rating of a corresponding annotator.

In an optional embodiment, the identifying module 202 identifies the target domain of the task to be annotated and the target annotation rating of the corresponding annotator includes:

The generating module 203 is configured to determine a target annotator from a preset annotator database according to the target field and the target annotation rating, and generate a corresponding first test question set according to the annotation rating of the target annotator.

In an optional embodiment, the determining, by the generating module 203, a target annotator from a preset annotator database according to the target field and the target annotation rating includes:

In an optional embodiment, the generating module 203 generates the corresponding first test question set according to the labeling rating of the target labeler includes:

The judging module 204 is configured to perform professional ability evaluation on the target annotator based on the first test question set, and judge whether the target annotator has the ability to execute the task to be annotated according to a professional ability evaluation result.

In this embodiment, the first test question set is generated from three dimensions, namely, a professional level angle, a marking rating angle and an angle of a corpus content of the to-be-marked task of the target marker, the marking of the target marker on the first test question set matches the marking result with a correct marking result to obtain a professional ability test score of each target marker, and whether the target marker has the ability to execute the to-be-marked task can be determined according to the professional ability test score. In an optional embodiment, the determining module 204 determines whether the target annotator has the ability to execute the task to be annotated according to the professional ability evaluation result, including:

A detecting module 205, configured to detect whether the target annotator meets a preset trigger event requirement when the target annotator has the capability of executing the task to be annotated.

Specifically, the preset trigger event includes:

An evaluation module 206, configured to determine, when the target annotator meets the preset trigger event requirement, a second test question set matching the target annotator from a preset psychological state test question library, and perform psychological ability evaluation on the target annotator based on the second test question set.

In an optional embodiment, the evaluation module 206 determines a second test question set matching the target annotator from a preset psychological state test question bank, and performing psychological ability evaluation on the target annotator based on the second test question set includes:

And the execution module 207 is configured to execute the task to be labeled when the psychological ability evaluation result of the target label maker is that the psychological test passes, and evaluate the labeling ability of the target label maker according to the labeling result.

In an optional embodiment, the executing module 207 executes the task to be annotated, and evaluating the annotation capability of the target annotator according to the annotation result includes:

Further, when the labeling capability of the target annotator does not meet the standard, adjusting the labeling rating of the target annotator.

collecting a disclosed public corpus from a plurality of preset data sources;

In summary, in the annotation task execution device based on big data according to this embodiment, on one hand, professional ability evaluation is performed on the target annotator based on the first test question set, whether the target annotator has the ability to execute the task to be annotated is judged according to a professional ability evaluation result, different professional ability scoring thresholds are preset according to professional ability evaluations in different fields, and a professional ability test score of the target annotator is compared with a preset professional ability scoring threshold in the target field, so that the comparison result is more reasonable and accurate, and whether the target annotator has the ability to execute the task to be annotated is determined according to the comparison result, thereby ensuring the accuracy of the target annotator executing the task to be annotated; on the other hand, when the target annotator meets the preset trigger event requirement, psychological ability assessment is carried out on the target annotator, when the target annotator meets the preset trigger event requirement, before the task to be annotated is executed, the psychological ability assessment is carried out on the target annotator, so that the current psychological state of the target annotator can execute the task to be annotated, and the annotation accuracy and efficiency of the task to be annotated are improved; and finally, executing the task to be labeled, evaluating the labeling capacity of the target label according to a labeling result, collecting a second test question set to be labeled, which is similar to the corpus content in the task to be labeled, from a pre-constructed corpus containing labels in the process of standard evaluation of the labeling capacity, wherein the second test question set to be labeled comprises standard answers, evaluating the labeling capacity of the target label according to the standard answers of the second test question set to be labeled, instead of evaluating the labeling capacity of the target label according to an analysis result after analyzing the labeling result of the first test question set to be labeled, so that the objective accuracy of evaluation is ensured, the accuracy of the evaluation result is improved, and the subsequent labeling quality is improved.

EXAMPLE III

Fig. 3 is a schematic structural diagram of an electronic device according to a third embodiment of the present application. In the preferred embodiment of the present application, the electronic device 3 comprises a memory 31, at least one processor 32, at least one communication bus 33, and a transceiver 34.

It will be appreciated by those skilled in the art that the configuration of the electronic device shown in fig. 3 does not constitute a limitation of the embodiments of the present application, and may be a bus-type configuration or a star-type configuration, and that the electronic device 3 may include more or less hardware or software than those shown, or a different arrangement of components.

In some embodiments, the electronic device 3 is an electronic device capable of automatically performing numerical calculation and/or information processing according to instructions set or stored in advance, and the hardware thereof includes but is not limited to a microprocessor, an application specific integrated circuit, a programmable gate array, a digital processor, an embedded device, and the like. The electronic device 3 may also include a client device, which includes, but is not limited to, any electronic product that can interact with a client through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device, for example, a personal computer, a tablet computer, a smart phone, a digital camera, and the like.

It should be noted that the electronic device 3 is only an example, and other existing or future electronic products, such as those that can be adapted to the present application, should also be included in the scope of protection of the present application, and are included by reference.

In some embodiments, the memory 31 is used for storing program codes and various data, such as the big data-based annotation task execution device 20 installed in the electronic equipment 3, and realizes high-speed and automatic access to programs or data during the operation of the electronic equipment 3. The Memory 31 includes a Read-Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable Programmable Read-Only Memory (EPROM), a One-time Programmable Read-Only Memory (OTPROM), an electronically Erasable rewritable Read-Only Memory (Electrically-Erasable Programmable Read-Only Memory (EEPROM)), an optical Read-Only disk (CD-ROM) or other optical disk Memory, a magnetic disk Memory, a tape Memory, or any other medium readable by a computer capable of carrying or storing data.

In some embodiments, the at least one processor 32 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or may be composed of a plurality of integrated circuits packaged with the same or different functions, including one or more Central Processing Units (CPUs), microprocessors, digital Processing chips, graphics processors, and combinations of various control chips. The at least one processor 32 is a Control Unit (Control Unit) of the electronic device 3, connects various components of the electronic device 3 by using various interfaces and lines, and executes various functions and processes data of the electronic device 3 by running or executing programs or modules stored in the memory 31 and calling data stored in the memory 31.

In some embodiments, the at least one communication bus 33 is arranged to enable connection communication between the memory 31 and the at least one processor 32 or the like.

Although not shown, the electronic device 3 may further include a power supply (such as a battery) for supplying power to each component, and optionally, the power supply may be logically connected to the at least one processor 32 through a power management device, so as to implement functions of managing charging, discharging, and power consumption through the power management device. The power supply may also include any component of one or more dc or ac power sources, recharging devices, power failure detection circuitry, power converters or inverters, power status indicators, and the like. The electronic device 3 may further include various sensors, a bluetooth module, a Wi-Fi module, and the like, which are not described herein again.

It is to be understood that the described embodiments are for purposes of illustration only and that the scope of the appended claims is not limited to such structures.

The integrated unit implemented in the form of a software functional module may be stored in a computer readable medium. The software functional module is stored in a medium and includes several instructions to enable a computer device (which may be a personal computer, an electronic device, or a network device) or a processor (processor) to execute parts of the methods according to the embodiments of the present application.

In a further embodiment, in conjunction with fig. 2, the at least one processor 32 may execute an operating device of the electronic device 3 and various installed application programs (such as the big data based annotation task executing device 20), program codes, and the like, for example, the above modules.

The memory 31 has program code stored therein, and the at least one processor 32 can call the program code stored in the memory 31 to perform related functions. For example, the modules described in fig. 2 are program codes stored in the memory 31 and executed by the at least one processor 32, so as to implement the functions of the modules for the purpose of executing a labeling task based on big data.

Illustratively, the program code may be partitioned into one or more modules/units that are stored in the memory 31 and executed by the processor 32 to accomplish the present application. The one or more modules/units may be a series of computer readable instruction segments capable of performing certain functions, which are used for describing the execution process of the program code in the electronic device 3. For example, the program code may be divided into an acquisition module 201, an identification module 202, a generation module 203, a determination module 204, a detection module 205, an evaluation module 206, and an execution module 207.

In one embodiment of the present application, the memory 31 stores a plurality of computer-readable instructions that are executed by the at least one processor 32 to implement the functionality of big data based annotation task execution.

Specifically, the at least one processor 32 may refer to the description of the relevant steps in the embodiment corresponding to fig. 1, and details are not repeated here.

In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules is only one logical functional division, and other divisions may be realized in practice.

The modules described as separate parts may or may not be physically separate, and parts displayed as modules may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.

In addition, functional modules in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional module.

It will be evident to those skilled in the art that the present application is not limited to the details of the foregoing illustrative embodiments, and that the present application may be embodied in other specific forms without departing from the spirit or essential attributes thereof. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the application being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned. Furthermore, it is obvious that the word "comprising" does not exclude other elements or that the singular does not exclude the plural. A plurality of units or means recited in the present application may also be implemented by one unit or means through software or hardware. The terms first, second, etc. are used to denote names, but not any particular order.

Finally, it should be noted that the above embodiments are only used for illustrating the technical solutions of the present application and not for limiting, and although the present application is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications or equivalent substitutions can be made on the technical solutions of the present application without departing from the spirit and scope of the technical solutions of the present application.

Claims

1. A big data-based annotation task execution method is characterized by comprising the following steps:

2. The big-data-based annotation task execution method of claim 1, wherein the generating a corresponding first set of test questions according to the annotation rating of the target annotator comprises:

3. The method for executing the annotation task based on big data as claimed in claim 1, wherein the determining whether the target annotator has the ability to execute the task to be annotated according to the professional ability evaluation result comprises:

4. The method for performing annotation tasks based on big data according to claim 1, wherein the preset trigger event comprises:

5. The big data based annotation task execution method of claim 1, wherein the step of determining a second test set matching the target annotator from a preset mental state test question bank comprises:

6. The method for executing the annotation task based on big data according to claim 1, wherein the executing the task to be annotated and evaluating the annotation ability of the target annotator according to the annotation result comprises:

7. The big data-based annotation task execution method of claim 6, wherein the construction process of the annotation-containing corpus comprises:

collecting a disclosed public corpus from a plurality of preset data sources;

8. A big data-based labeling task execution device, the device comprising:

9. An electronic device, comprising a processor and a memory, wherein the processor is configured to implement the big data based annotation task execution method according to any one of claims 1 to 7 when executing the computer program stored in the memory.

10. A computer-readable storage medium, on which a computer program is stored, wherein the computer program, when executed by a processor, implements the big data based annotation task execution method according to any one of claims 1 to 7.