CN113934708A - Method, device and equipment for verifying labeled data and storage medium - Google Patents

Abstract

The present disclosure provides a method, an apparatus, a device and a storage medium for verifying labeled data, which relate to the technical field of computers, in particular to the technical field of artificial intelligence such as cloud computing and data labeling. The specific implementation scheme is as follows: adding verification rule information corresponding to the labeling page into the labeling page, wherein the labeling page comprises at least one labeling question, and the verification rule information corresponds to the at least one labeling question; obtaining labeling data of at least one labeling topic; and verifying the marked data according to the verification rule information in the marked page.

Description

Method, device and equipment for verifying labeled data and storage medium

Technical Field

The present disclosure relates to the field of computer technology, and in particular, to the field of artificial intelligence technologies such as cloud computing and data annotation.

Background

Currently, the Artificial Intelligence (AI) industry is developing vigorously, and data tagging is a ring in the AI data service industry that has a strong dependence relationship with the data production link. Under the background of the vigorous development of the whole AI industry, the quality requirements of data labeling of each subdivision industry are higher and higher.

Disclosure of Invention

The disclosure provides a method, a device, equipment and a storage medium for verifying labeled data.

According to an aspect of the present disclosure, there is provided a method of verifying annotation data, including: adding verification rule information corresponding to a labeling page into the labeling page, wherein the labeling page comprises at least one labeling topic, and the verification rule information corresponds to the at least one labeling topic; obtaining the labeling data of the at least one labeling topic; and verifying the labeled data according to the verification rule information in the labeled page.

According to another aspect of the present disclosure, there is provided an apparatus for verifying annotation data, comprising: the adding module is used for adding verification rule information corresponding to the labeling page into the labeling page, wherein the labeling page comprises at least one labeling topic, and the verification rule information corresponds to the at least one labeling topic; the acquisition module is used for acquiring the labeling data of the at least one labeling topic; and the verification module is used for verifying the marked data according to the verification rule information in the marked page.

Another aspect of the present disclosure provides an electronic device including: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of the embodiments of the present disclosure.

According to another aspect of the disclosed embodiments, there is provided a non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method shown in the disclosed embodiments.

According to another aspect of the embodiments of the present disclosure, there is provided a computer program product comprising computer programs/instructions, characterized in that the computer programs/instructions, when executed by a processor, implement the steps of the method shown in the embodiments of the present disclosure.

It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present disclosure, nor do they limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description.

Drawings

The drawings are included to provide a better understanding of the present solution and are not to be construed as limiting the present disclosure. Wherein:

FIG. 1 is a schematic diagram of an application scenario of a method, an apparatus, an electronic device and a storage medium for verifying annotation data according to an embodiment of the present disclosure;

FIG. 2 schematically illustrates a flow diagram of a method of verifying annotation data in accordance with an embodiment of the present disclosure;

FIG. 3 schematically illustrates a flow chart of a method of adding collation rule information to a callout page according to an embodiment of the present disclosure;

FIG. 4 schematically shows a flowchart of a method of obtaining annotation data for at least one annotation topic according to an embodiment of the present disclosure;

FIG. 5 schematically illustrates a flow chart of a method of verifying annotation data in accordance with an embodiment of the present disclosure;

FIG. 6 schematically illustrates a flow chart of a method of verifying annotation data in accordance with another embodiment of the present disclosure;

FIG. 7 schematically illustrates a method of verifying annotation data, in accordance with an embodiment of the present disclosure;

FIG. 8 schematically illustrates a block diagram of an apparatus for verifying annotation data in accordance with an embodiment of the present disclosure;

FIG. 9 schematically shows a block diagram of an example electronic device that may be used to implement embodiments of the present disclosure.

Detailed Description

Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, in which various details of the embodiments of the disclosure are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

An application scenario of the method and apparatus for verifying annotation data provided in the present disclosure will be described below with reference to fig. 1.

Fig. 1 is a schematic view of an application scenario of a method, an apparatus, an electronic device, and a storage medium for verifying annotation data according to an embodiment of the disclosure.

As shown in FIG. 1, the application scenario 100 includes a configuration page 110, an annotation page 120, and a database 130.

According to embodiments of the present disclosure, the configuration page 110 may be used to configure the verification rule information needed to annotate a project/task. In this embodiment, an annotation item can include one or more tasks, each task can correspond to an annotation page 120, and each annotation page 120 can include one or more annotation topics. Illustratively, the annotation title may include, for example, a picture annotation, a video annotation, a voice transcription annotation, an OCR (Optical Character Recognition) annotation, an automatic driving training data annotation, and the like. The labeling means may include, for example, a picture frame, a punctuation, a line, setting an attribute, and the like.

According to the embodiment of the disclosure, the user can configure the verification rule information of each annotation topic in the annotation page 120 through the configuration page 110. The checking rule information may include, for example, a checking type of the label topic, a checking trigger condition, a checking code, and the like.

After the configuration of the check rule information is completed, the check rule information may be stored in the database 130. According to an embodiment of the present disclosure, a rule base summary table and an item/task rule table may be set in the database 130. The rule base summary table may contain all the check rule information, and each record represents an independent check rule, including information such as a check code of the check rule. The project/task rule table can contain the association relationship between the project/task and the check rule, and the check rule required by the project/task can be freely configured through the table.

It should be noted that, besides the rule base summary table and the item/task rule table, the database may also include other data tables, which are not specifically limited by the present disclosure. For example, the database may further include a rule authority table for recording authority information of each check rule, where the authority information may be used to specify which users have access authority or modification authority, and the like.

Illustratively, in this embodiment, the database 130 may be an associative database, such as a MySQL database. It should be noted that the present disclosure does not specifically limit the type of the database 130.

According to the embodiment of the present disclosure, the verification rule information corresponding to each labeling topic in the labeling page 120 may be acquired from the database 130. These check rule information are added to the annotation page 120. When a user marks a marking question in the marking page 120, marking data of the marking question can be obtained, and then the marking data is verified according to the verification rule information in the marking page 120.

In the technical scheme of the disclosure, the collection, storage, use, processing, transmission, provision, disclosure and other processing of the related check data all accord with the regulations of related laws and regulations, and do not violate the good custom of the public order.

FIG. 2 schematically shows a flow chart of a method of verifying annotation data according to an embodiment of the present disclosure.

As shown in fig. 2, the method 200 for verifying annotation data includes adding verification rule information corresponding to an annotation page to the annotation page in operation S210.

According to the embodiment of the disclosure, the labeling page may include at least one labeling topic, and the verification rule information corresponds to the at least one labeling topic. The verification rule information comprises information related to the verification rule corresponding to the labeling topic.

Then, in operation S220, annotation data of at least one annotation title is acquired.

According to an embodiment of the present disclosure, the annotation data may include, for example, points, lines, boxes marked in the annotation title by the user, and attribute values selected by the user, input data, and the like.

In operation S230, the annotation data is verified according to the verification rule information in the annotation page.

According to the embodiment of the disclosure, the verification rule information is added into the annotation page, so that the annotation data is verified according to the verification rule information in the annotation page in the annotation stage, the accuracy of the annotation data can be improved, and the auditing time of the annotation data is reduced.

According to the embodiment of the disclosure, the verification rule information may include, for example, a verification type, a verification trigger condition, and a verification code corresponding to at least one labeling topic in the labeling page. The check rule information supports flexible configuration, and can customize check types, check triggering conditions, check codes and the like according to actual requirements. Based on this, fig. 3 schematically shows a flowchart of a method of adding collation rule information to a callout page according to an embodiment of the present disclosure.

As shown in fig. 3, the method 310 of adding the verification rule information to the markup page includes configuring a verification script for the markup page according to the verification code in operation S311.

According to the embodiment of the disclosure, the verification script suitable for the marked page can be generated according to the verification code. An object, such as a < script > element, in the markup page is then determined for describing the script according to the DOM (document object model) of the markup page. The verification code is then added to the object. Alternatively, an object may be added to the markup page according to the DOM of the markup page, for example, an < script > element is added, and then the verification script is added to the added object.

In operation S312, a trigger script corresponding to each labeling topic is configured for the labeling page according to the verification trigger condition corresponding to each labeling topic.

According to the embodiment of the disclosure, similar to the verification script, the trigger script may be added to an existing object in the markup page, or an object may be newly created in the markup page to store the trigger script. The verification script and the trigger script may be stored in the same object or different objects, and the present disclosure is not limited thereto.

In operation S313, a verification attribute is configured for each annotation question according to the verification type corresponding to each annotation question.

It should be noted that the execution sequence between operations S311 to S313 in fig. 3 is only an example, and in practical applications, operations S311 to S313 may be executed in any sequence. In addition, operations S311 to S313 may be performed in a linear manner or simultaneously in a parallel manner, which is not particularly limited by the present disclosure.

According to the embodiment of the disclosure, a DOM method can be used to configure a verification attribute for a labeling topic in a labeling page, wherein a value of the verification attribute can be used to represent a verification type of the corresponding labeling topic.

According to embodiments of the present disclosure, the check type may include, for example, a page level check type and a topic level check type. And configuring the labeling topics with the association relation during verification as a page-level verification type. For the labeling questions of the page level verification type in the labeling page, the labeling data of the labeling questions can be collected uniformly, and then correlation verification is carried out. The marked topics of the topic level check type can be checked independently. According to an embodiment of the present disclosure, the title level check type may include, for example, a box check (recatmark), a syntax check (parsing), a text check (text), a single check (single), a multi check (multi), and the like. The frame check may be used to check, for example, the size, overlap, and the like of the labeled frame. The syntax check can be used, for example, to check the syntax of the annotation data. The text check can be used, for example, to check the text data in the annotation data.

FIG. 4 schematically shows a flowchart of a method of obtaining annotation data for at least one annotation topic according to an embodiment of the present disclosure.

As shown in fig. 4, the method 420 for obtaining annotation data of at least one annotation question includes, in operation S421, detecting, for each annotation question, whether an operation in an annotation page satisfies a verification trigger condition of the annotation question by using a trigger script corresponding to the annotation question.

In operation S422, under the condition that the trigger condition of the annotation item is satisfied, the annotation data of the annotation item is obtained.

According to an embodiment of the present disclosure, the trigger condition may include, for example, at least one of: a tagging operation for a tagging topic is triggered, a tagging operation for a tagging topic is completed, and a data submission operation is triggered. The labeling operation may include, for example, an operation of labeling a labeling topic. The data submitting operation can comprise, for example, an operation of submitting annotation data corresponding to the annotation item answered in the annotation page.

It should be noted that the trigger condition may also be configured to include other conditions according to actual needs. For example, in the case that a quality inspection link exists after the labeling, the trigger condition may be configured to include that a quality inspection operation is triggered or a quality inspection submission operation is triggered. For another example, in the case that there is an audit link after the annotation is performed, the trigger condition may be configured to include that an audit operation is triggered or an audit submission operation is triggered. For another example, in the case that there is an examination link after the annotation is performed, the trigger condition may be configured to include that the examination operation is triggered or the examination submission operation is triggered.

FIG. 5 schematically shows a flow chart of a method of verifying annotation data according to an embodiment of the present disclosure.

As shown in fig. 5, the method 530 for adding the verification rule information to the annotation page includes, in operation S531, determining, for each annotation topic that meets the trigger condition, a verification script corresponding to the annotation topic according to the verification attribute of the annotation topic.

In operation S532, a verification script is called to verify the tagged data of the tagged title.

According to the embodiment of the disclosure, the verification type corresponding to the labeling topic can be determined through the verification attribute of the labeling topic. Therefore, the verification script corresponding to the verification type can be called to verify the labeling data of the labeling question. For example, through the verification attribute of the labeling topic, it can be determined that the verification type corresponding to the labeling topic is box verification, and then a verification script for the box verification can be called to verify the labeling data of the labeling topic.

According to other embodiments of the disclosure, the verification rule information further includes prompt configuration information in addition to the verification type, the verification trigger condition and the verification code corresponding to at least one labeling topic in the labeling page. Based on this, fig. 6 schematically shows a flow chart of a method of verifying annotation data according to another embodiment of the present disclosure.

As shown in fig. 6, the method 600 includes acquiring verification rule information corresponding to a markup page in operation S610.

In operation S620, a verification script is configured for the markup page according to the verification code.

In operation S630, a trigger script corresponding to each labeling topic is configured for the labeling page according to the verification trigger condition corresponding to each labeling topic.

In operation S640, a verification attribute is configured for each labeling topic according to the verification type corresponding to each labeling topic.

In operation S650, a prompt script is configured for the markup page according to the prompt configuration information.

In operation S660, annotation data of at least one annotation title is acquired.

In operation S670, the tagged data is verified according to the verification rule information in the tagged page.

In operation S680, in a case that the annotation data of any one of the at least one annotation title does not pass the verification, prompt information for the annotation title is generated using the prompt script.

According to the embodiments of the present disclosure, operations S610 to S640 and operations S660 to S670 may not be described herein again with reference to the above, for example.

It should be noted that the execution sequence between operations S620 to S650 in fig. 6 is only an example, and in practical applications, operations S620 to S650 may be executed in any sequence. In addition, operations S620 to S650 may be performed in a linear manner, respectively, or may be performed simultaneously in a parallel manner, which is not particularly limited by the present disclosure.

According to the embodiment of the disclosure, the prompting mode of each labeling topic when the verification fails can be set through prompting the configuration information. The prompting method may include, for example, displaying text prompting information, displaying a prompting window, highlighting a labeled question that fails to pass verification, and the like, which is not specifically limited in the present disclosure.

According to the embodiment of the disclosure, under the condition that the labeling data of the labeling questions do not pass the verification, the prompt script is used for generating the prompt information aiming at the labeling questions, so that the labeling questions which do not pass the verification can be prompted to a user, and the user can conveniently re-label the labeling questions.

The method for verifying the annotation data described above is further described with reference to fig. 7. Those skilled in the art will appreciate that the following example embodiments are only for the understanding of the present disclosure, and the present disclosure is not limited thereto.

FIG. 7 schematically shows a method for verifying annotation data according to an embodiment of the disclosure.

Fig. 7 shows that a user may configure the verification type, the verification trigger condition, the verification code, the prompt configuration information, and the like of each labeling topic in the labeling page through a configuration page, and generate verification rule information corresponding to the labeling page. The verification rule information may then be stored in a database.

Next, check rule information corresponding to the labeled page may be obtained from the database, and check type, check trigger condition, check code, prompt configuration information, and the like of the check rule information are added to the labeled page.

As an optional embodiment, the annotation page comprises annotation titles of page level types, and the trigger condition of the annotation titles of the page level types is configured to be triggered when data is submitted. Based on the method, when the data submission operation in the labeling page is triggered, the labeling data of the labeling questions of all page level types can be obtained, and then the verification scripts corresponding to the page level types are called to carry out unified verification on the labeling data.

As another optional embodiment, the annotation page contains annotation titles of title level types, and the trigger conditions of the annotation titles of these title level types are configured to trigger when annotation is performed. Based on this, when a labeling operation for a certain labeling topic in a labeling page is triggered, the labeling data of the labeling topic can be obtained, then a corresponding verification script is determined according to the configured verification attribute of the labeling data, and then the verification script is called to verify the labeling data.

FIG. 8 schematically shows a block diagram of an apparatus for verifying annotation data in accordance with an embodiment of the present disclosure.

As shown in FIG. 8, the apparatus 800 for verifying annotation data comprises an adding module 810, an obtaining module 820 and a verifying module 830.

The adding module 810 is configured to add verification rule information corresponding to the labeled page, where the labeled page includes at least one labeled topic, and the verification rule information corresponds to the at least one labeled topic.

An obtaining module 820, configured to obtain labeling data of at least one labeling topic.

And the checking module 830 is configured to check the marked data according to the checking rule information in the marked page.

As an alternative embodiment, the verification rule information may include, for example, a verification type, a verification trigger condition, and a verification code corresponding to at least one tagged topic. The joining module may include, for example, a first configuration submodule, a second configuration submodule, and a third configuration submodule. The first configuration submodule can be used for configuring a verification script for the marked page according to the verification code. And the second configuration submodule can be used for configuring the trigger script corresponding to each marking topic for the marking page according to the verification trigger condition corresponding to each marking topic. And the third configuration submodule can be used for configuring the verification attributes for the labeling questions according to the verification type corresponding to each labeling question.

As an alternative embodiment, the obtaining module may include, for example, a detecting sub-module and an obtaining sub-module. The detection submodule can be used for detecting whether the operation in the labeling page meets the verification triggering condition of the labeling question or not by using the triggering script corresponding to the labeling question aiming at each labeling question. And the obtaining submodule can be used for obtaining the marking data of the marking questions under the condition that the triggering conditions of the marking questions are met.

As an alternative embodiment, the trigger condition for labeling the topic may include at least one of the following: a tagging operation for a tagging topic is triggered, a tagging operation for a tagging topic is completed, and a data submission operation is triggered.

As an alternative embodiment, the check module may comprise, for example, a determination submodule and a call submodule. The determining submodule can be used for determining a verification script corresponding to each labeling topic according to the verification attribute of the labeling topic for each labeling topic meeting the triggering condition. And the calling submodule can be used for calling the verification script and verifying the marking data of the marking questions.

As an optional embodiment, the check rule information may further include prompt configuration information, for example. The device for verifying the annotation data further comprises a fourth configuration module and a prompt module. The fourth configuration module may be configured to configure a prompt script for the markup page according to the prompt configuration information. The prompting module can be used for generating prompting information aiming at the labeling questions by utilizing the prompting script under the condition that the labeling data of any one of the at least one labeling question does not pass the verification.

The present disclosure also provides an electronic device, a readable storage medium, and a computer program product according to embodiments of the present disclosure.

FIG. 9 schematically shows a block diagram of an example electronic device 900 that may be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the disclosure described and/or claimed herein.

As shown in fig. 9, the apparatus 900 includes a computing unit 901, which can perform various appropriate actions and processes in accordance with a computer program stored in a Read Only Memory (ROM)902 or a computer program loaded from a storage unit 908 into a Random Access Memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The calculation unit 901, ROM 902, and RAM 903 are connected to each other via a bus 904. An input/output (I/O) interface 905 is also connected to bus 904.

A number of components in the device 900 are connected to the I/O interface 905, including: an input unit 906 such as a keyboard, a mouse, and the like; an output unit 907 such as various types of displays, speakers, and the like; a storage unit 908 such as a magnetic disk, optical disk, or the like; and a communication unit 909 such as a network card, a modem, a wireless communication transceiver, and the like. The communication unit 909 allows the device 900 to exchange information/data with other devices through a computer network such as the internet and/or various telecommunication networks.

The computing unit 901 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various dedicated Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, and so forth. The calculation unit 901 performs the respective methods and processes described above, such as the method of verifying the annotation data. For example, in some embodiments, the method of verifying annotation data can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program may be loaded and/or installed onto device 900 via ROM 902 and/or communications unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the above described method of verifying annotation data may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform the method of verifying the annotation data by any other suitable means (e.g., by means of firmware).

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.

Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions/operations specified in the flowchart and/or block diagram to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

The Server may be a cloud Server, which is also called a cloud computing Server or a cloud host, and is a host product in a cloud computing service system, so as to solve the defects of high management difficulty and weak service extensibility in a traditional physical host and a VPS service (Virtual Private Server, or VPS for short). The server may also be a server of a distributed system, or a server incorporating a blockchain.

It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present disclosure may be executed in parallel, sequentially, or in different orders, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, and the present disclosure is not limited herein.

The above detailed description should not be construed as limiting the scope of the disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present disclosure should be included in the scope of protection of the present disclosure.

Claims (15)

1. A method of verifying annotation data, comprising:

adding verification rule information corresponding to a labeling page into the labeling page, wherein the labeling page comprises at least one labeling topic, and the verification rule information corresponds to the at least one labeling topic;

obtaining the labeling data of the at least one labeling topic; and

and verifying the labeled data according to the verification rule information in the labeled page.

2. The method according to claim 1, wherein the verification rule information includes a verification type, a verification trigger condition and a verification code corresponding to the at least one labeling topic; the adding of the check rule information to the label page includes:

configuring a verification script for the labeling page according to the verification code;

configuring a trigger script corresponding to each labeling question for the labeling page according to a verification trigger condition corresponding to each labeling question; and

and configuring a verification attribute for each marking topic according to the verification type corresponding to each marking topic.

3. The method of claim 2, wherein the obtaining annotation data for the at least one annotation topic comprises:

for each marking topic, detecting whether the operation in the marking page meets the verification triggering condition of the marking topic by using a triggering script corresponding to the marking topic; and

and acquiring the labeling data of the labeling questions under the condition of meeting the triggering conditions of the labeling questions.

4. The method of claim 3, wherein the trigger condition for the annotation topic comprises at least one of:

the labeling operation aiming at the labeling question is triggered;

the labeling operation aiming at the labeling question is completed; and

a data commit operation is triggered.

5. The method of claim 2, wherein the verifying the annotation data according to the verification rule information in the annotation page comprises:

aiming at each marking topic meeting a trigger condition, determining a verification script corresponding to the marking topic according to the verification attribute of the marking topic; and

and calling the verification script to verify the labeling data of the labeling questions.

6. The method of claim 2, wherein the validation rule information further comprises prompt configuration information; the method further comprises the following steps:

configuring a prompt script for the labeling page according to the prompt configuration information; and

and under the condition that the labeling data of any one of the at least one labeling topic is not verified, generating prompt information aiming at the labeling topic by using the prompt script.

7. An apparatus for verifying annotation data, comprising:

the adding module is used for adding verification rule information corresponding to the labeling page into the labeling page, wherein the labeling page comprises at least one labeling topic, and the verification rule information corresponds to the at least one labeling topic;

the acquisition module is used for acquiring the labeling data of the at least one labeling topic; and

and the verification module is used for verifying the marked data according to the verification rule information in the marked page.

8. The apparatus according to claim 7, wherein the verification rule information includes a verification type, a verification trigger condition and a verification code corresponding to the at least one labeling topic; the joining module comprises:

the first configuration submodule is used for configuring a verification script for the marked page according to the verification code;

a second configuration submodule, configured to configure, according to a verification trigger condition corresponding to each of the labeling questions, a trigger script corresponding to each of the labeling questions for the labeling page; and

and the third configuration submodule is used for configuring the verification attributes for the marking questions according to the verification types corresponding to the marking questions.

9. The apparatus of claim 8, wherein the means for obtaining comprises:

the detection submodule is used for detecting whether the operation in the labeling page meets the verification triggering condition of the labeling question or not by using the triggering script corresponding to the labeling question aiming at each labeling question; and

and the obtaining sub-module is used for obtaining the marking data of the marking questions under the condition that the triggering conditions of the marking questions are met.

10. The apparatus of claim 9, wherein the trigger condition for the annotation topic comprises at least one of:

the labeling operation aiming at the labeling question is triggered;

the labeling operation aiming at the labeling question is completed; and

a data commit operation is triggered.

11. The apparatus of claim 8, wherein the verification module comprises:

the determining submodule is used for determining a verification script corresponding to each marking question meeting the triggering condition according to the verification attribute of the marking question; and

and the calling submodule is used for calling the verification script and verifying the marking data of the marking questions.

12. The apparatus of claim 8, wherein the validation rule information further comprises prompt configuration information; the device further comprises:

the fourth configuration module is used for configuring a prompt script for the labeling page according to the prompt configuration information; and

and the prompting module is used for generating prompting information aiming at the labeling questions by utilizing the prompting script under the condition that the labeling data of any one of the at least one labeling question does not pass the verification.

13. An electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

14. A non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of any one of claims 1-6.

15. A computer program product comprising computer program/instructions, characterized in that the computer program/instructions, when executed by a processor, implement the steps of the method according to any of claims 1-6.

Images