CN113569546A

CN113569546A - Intention labeling method and device and electronic equipment

Info

Publication number: CN113569546A
Application number: CN202110669198.2A
Authority: CN
Inventors: 徐国兴
Original assignee: Shanghai Qiyue Information Technology Co Ltd
Current assignee: Shanghai Qiyue Information Technology Co Ltd
Priority date: 2021-06-16
Filing date: 2021-06-16
Publication date: 2021-10-29

Abstract

The invention relates to the technical field of computers, in particular to an intention labeling method, an intention labeling device and electronic equipment, wherein the intention labeling method comprises the following steps: acquiring data to be marked; creating an intention labeling task according to the data to be labeled, and distributing the intention labeling task to a labeling terminal; acquiring a labeling record added to the data to be labeled by the labeling terminal through the intention labeling task, and acquiring labeling records of other labeling terminals corresponding to the intention labeling task; and obtaining the labeling result of the data to be labeled according to the labeling record of each labeling terminal and by combining a final result judgment strategy. According to the invention, the efficiency and accuracy of the labeling intention are improved through modes such as cross labeling and a final result judgment strategy, and the final labeled labeling data is generated and used for the natural language processing model training of the robot, so that the collection hastening and destruction rate of the robot are improved, and meanwhile, the working state, the working efficiency and the like of the labeling personnel can be monitored in real time.

Description

Intention labeling method and device and electronic equipment

Technical Field

The invention relates to the technical field of computers, in particular to an intention labeling method and device and electronic equipment.

Background

With the continuous development of scientific technology, Artificial Intelligence (Artificial Intelligence) technology has been widely applied to various products. One of the great features of artificial intelligence is that intelligent devices can interact with users. For example, the user may chat with the chat robot, or may input a voice command according to his/her will and habit to control the chat robot to perform a corresponding action. In such human-computer interaction processes, the key to the smart device is to identify the user's intent. Therefore, the smart device needs to be trained in advance by using a large amount of training data for labeling intentions. At present, training data is usually labeled manually, but the labeling of the intention manually is inefficient and has low accuracy.

Disclosure of Invention

The invention provides an intention labeling method, an intention labeling device and electronic equipment, which are used for improving the efficiency and accuracy of labeling intentions.

An embodiment of the present specification provides an intention labeling method, including:

acquiring data to be marked;

creating an intention labeling task according to the data to be labeled, and distributing the intention labeling task to a labeling terminal;

acquiring a labeling record added to the data to be labeled by the labeling terminal through the intention labeling task, and acquiring labeling records of other labeling terminals corresponding to the intention labeling task;

and obtaining the labeling result of the data to be labeled according to the labeling record of each labeling terminal and by combining a final result judgment strategy.

Preferably, the method further comprises the following steps:

and monitoring the workload of the intention labeling task corresponding to each labeler in real time, and counting the labeling completion quantity and the labeling accuracy of the intention labeling task corresponding to each labeler so as to monitor the work of each labeler.

Preferably, the acquiring data to be labeled includes:

acquiring all data to be marked in the same batch and in a preset file format;

and importing the data to be annotated in a preset file format to an annotation center, wherein the data to be annotated in the preset file format can be imported only once, and the intention annotation tasks corresponding to all the data to be annotated in the same batch are the same.

Preferably, the creating of the intention labeling task according to the data to be labeled includes:

judging whether the data to be labeled of the batch associated with the created intention labeling task exists or not;

when the data to be annotated of the associated batch exists, creating the intention annotation task according to the data to be annotated;

and when the data to be labeled of the associated batch does not exist, stopping creating the intention labeling task.

Preferably, the creating an intention labeling task according to the data to be labeled further includes:

and creating an intention marking task through all the data to be annotated of the same batch.

Preferably, the allocating the intention tagging task to a tagging terminal includes:

and distributing the data to be labeled with the state to be labeled to the labeling terminals in real time according to the labeling records of each user and by combining the configuration conditions of the intention labeling task, wherein the same data to be labeled can be labeled by a plurality of labeling terminals.

Preferably, the obtaining of the labeling result of the data to be labeled according to the labeling record of each labeling terminal and by combining the final result judgment policy includes:

obtaining a first number of annotation records marked as correct and a second number of annotation records marked as erroneous; wherein the annotation record comprises: correct and wrong;

judging whether the first quantity and the second quantity are equal;

when the first quantity and the second quantity are equal, rechecking the labeling records, and adding the labeling records to the data to be labeled again;

when the first number is larger than the second number, the labeling result corresponding to the intention labeling task is correct;

and when the first number is smaller than the second number, the labeling result corresponding to the intention labeling task is an error.

Preferably, the method for obtaining the labeling result of the intention labeling task according to the labeling records of all the historical users and by combining a final result judgment policy further includes:

when the labeling results of a plurality of labeling terminals on the same data to be labeled are different, sending the data to be labeled to an administrator terminal to add labeling records again;

and taking the labeling record fed back by the administrator terminal as the labeling result of the data to be labeled.

An embodiment of the present specification further provides an intention labeling apparatus, including:

the data acquisition module is used for acquiring data to be marked;

the task allocation module is used for creating an intention labeling task according to the data to be labeled and allocating the intention labeling task to a labeling terminal;

the annotation record acquisition module is used for acquiring annotation records added to the data to be annotated by the annotation terminal through the intention annotation task and acquiring annotation records of other annotation terminals corresponding to the intention annotation task;

and the labeling result acquisition module is used for acquiring the labeling result of the data to be labeled according to the labeling record of each labeling terminal and by combining the final result judgment strategy.

An electronic device, wherein the electronic device comprises:

a processor and a memory storing computer executable instructions that, when executed, cause the processor to perform the method of any of the above.

A computer readable storage medium, wherein the computer readable storage medium stores one or more programs which, when executed by a processor, implement the method of any of the above.

The beneficial effects are that:

according to the invention, the efficiency and accuracy of the labeling intention are improved through modes such as cross labeling and a final result judgment strategy, and the final labeled labeling data is generated, and can be used for natural language processing model training of the robot, the collection hastening rate and the destruction rate of the robot are improved, and meanwhile, the working state and the working efficiency of the labeling personnel are monitored in real time through counting the working tasks of the labeling personnel.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:

FIG. 1 is a schematic diagram illustrating an intent tagging method provided by an embodiment of the present disclosure;

FIG. 2 is a schematic structural diagram of an intention labeling apparatus provided in an embodiment of the present disclosure;

fig. 3 is a schematic structural diagram of an electronic device provided in an embodiment of the present disclosure;

fig. 4 is a schematic diagram of a computer-readable medium provided in an embodiment of the present specification.

Detailed Description

Exemplary embodiments of the present invention will now be described more fully with reference to the accompanying drawings. The exemplary embodiments, however, may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these exemplary embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the invention to those skilled in the art. The same reference numerals denote the same or similar elements, components, or parts in the drawings, and thus their repetitive description will be omitted.

Features, structures, characteristics or other details described in a particular embodiment do not preclude the fact that the features, structures, characteristics or other details may be combined in a suitable manner in one or more other embodiments in accordance with the technical idea of the invention.

In describing particular embodiments, the present invention has been described with reference to features, structures, characteristics or other details that are within the purview of one skilled in the art to provide a thorough understanding of the embodiments. One skilled in the relevant art will recognize, however, that the invention may be practiced without one or more of the specific features, structures, characteristics, or other details.

The diagrams depicted in the figures are exemplary only, and do not necessarily include all of the contents and operations/steps, nor do they necessarily have to be performed in the order depicted. For example, some operations/steps may be decomposed, and some operations/steps may be combined or partially combined, so that the actual execution sequence may be changed according to the actual situation.

The block diagrams shown in the figures are functional entities only and do not necessarily correspond to physically separate entities. I.e. these functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor means and/or microcontroller means.

The term "and/or" and/or "includes all combinations of any one or more of the associated listed items.

Referring to fig. 1, a schematic diagram of an intention labeling method provided in an embodiment of the present specification includes:

s101: acquiring data to be marked;

in a preferred embodiment of the present invention, the data to be annotated can be manually input to the intention annotation platform, or the data to be annotated in the data server can be imported to the intention annotation platform through other data servers, where the importing of the data to be annotated includes, but is not limited to, the above manual importing or data server importing, so that the data acquisition is diversified, and the data acquisition difficulty is reduced.

S102: creating an intention labeling task according to the data to be labeled, and distributing the intention labeling task to a labeling terminal;

in a preferred embodiment of the invention, the intention labeling task corresponding to the data to be labeled is imported in a manual import mode, the intention labeling task can be created according to specific requirements, or the platform can automatically identify the data to be labeled to create the corresponding intention labeling task, and then the created intention labeling task is distributed to at least one labeling terminal.

S103: acquiring a labeling record added to the data to be labeled by the labeling terminal through the intention labeling task, and acquiring labeling records of other labeling terminals corresponding to the intention labeling task;

in a preferred embodiment of the invention, the annotation terminal can display each data to be annotated corresponding to the intention annotation task by searching the intention annotation task, and further, when searching the intention annotation task, the intention annotation task is matched in a fuzzy matching mode, and the annotation page of the annotation terminal has a task time column, so that the scope search can be performed on the creation time of the intention annotation task, the intention annotation task in a time period to be known can be searched, and the searching, the query and the like of the intention annotation task can be conveniently performed by an annotator.

In a preferred embodiment of the invention, the annotator marks the intention of each data to be annotated through the annotation terminal, marks each data to be annotated in a cross-labeling mode, and finally summarizes the annotation records of each annotator on each data to be annotated, so that the accuracy of the annotation of the data to be annotated is ensured by adopting the method.

S104: and obtaining the labeling result of the data to be labeled according to the labeling record of each labeling terminal and by combining a final result judgment strategy.

In a preferred embodiment of the present invention, the annotation record that the annotator annotates the data to be annotated at the annotation terminal is not the final annotation result, and the annotation record should be determined to be the final annotation result of the data to be annotated by combining with the final result determination policy, for example: if the annotation record marked as correct is 100, the annotation record marked as wrong is 90, and the annotation record marked as correct is greater than the annotation record marked as wrong, the annotation result corresponding to the intention annotation task is correct, and further, the annotation result corresponding to the intention annotation task can be manually checked.

For example: if the annotation record marked as correct is 100, the annotation record marked as error is 110, and the annotation record marked as correct is smaller than the annotation record marked as error, the annotation result corresponding to the intention annotation task is an error, and further, the annotation result corresponding to the intention annotation task can be manually spot-checked.

For example: if the annotation record marked as correct is 100, and the annotation record marked as wrong is 100, and at this time, the annotation record marked as correct is equal to the annotation record marked as wrong, it indicates that the annotation result corresponding to the intention annotation task is wrong, and the annotation record needs to be rechecked manually, and the annotation record is added to the data to be annotated again. As can be seen from the above example, the final labeling result is obtained by combining the final result determination policy.

In a preferred embodiment of the present invention, when a history annotation result needs to be changed, the data to be annotated is switched to the annotated state for modification, and the final result judgment policy is triggered while the modification operation is performed, further, when any annotation record changes, the annotation result is dynamically updated in real time according to the final result judgment policy, and each time the annotation terminal marks the data to be annotated corresponding to the intention annotation task, the above steps need to be performed, so as to determine the annotation condition of each current annotation terminal.

Further, still include:

In a preferred embodiment of the present invention, the labeling workload of each labeling operator operating each labeling terminal is monitored in real time by querying the specific date of the labeling time, for example, if the search date is 2020/01/01, the labeling completion amount, the amount to be labeled, and the like of each labeling operator on the intended labeling task through the labeling terminal on the date will be displayed, and at the same time, a certain labeling operator can be designated to search for the labeling completion amount, the labeling accuracy, and the like of the intended labeling task corresponding to the labeling operator. The statistical data such as the labeling completion quantity or the partial completion quantity of the task list can be scanned and counted according to batch data associated with the task, the data size is large, the query range is narrowed when the search is suggested, and the query range is tentatively set within 7 days; because the labeling result is dynamically changed, the labeling accuracy rate also changes along with the change of the labeling result, thereby obtaining more accurate labeling accuracy rate.

Further, the acquiring data to be labeled includes:

acquiring all data to be marked in the same batch and in a preset file format;

In a preferred embodiment of the present invention, data to be labeled of the same batch is obtained, and the data to be labeled is imported to the labeling center in a CSV file format, where each CSV file only has data to be labeled of one batch, and the data to be labeled of the same batch is intelligently poured once, and the data to be labeled of the same batch is not supported to be divided into multiple CSV files and imported to the labeling center, so as to prevent the imported data to be labeled from being repeated.

Further, the creating of the intention labeling task according to the data to be labeled comprises:

In a preferred embodiment of the invention, whether the data to be labeled of the batch associated with the intention labeling task exists is judged, if the data to be labeled of the associated batch exists, the intention labeling task is created, if the data to be labeled of the associated batch does not exist, the intention labeling task cannot be created, and by judging whether the data to be labeled of the associated batch exists, the matching degree of the intention labeling task corresponding to the created data to be labeled is improved.

Further, the creating an intention labeling task according to the data to be labeled further comprises:

In a preferred embodiment of the invention, corresponding intention labeling tasks can be established for all the data to be annotated in the same batch according to specific requirements in a targeted manner, and the adoption of the manner of establishing the intention labeling tasks according to requirements improves the flexibility of processing the data to be annotated.

In the preferred embodiment of the invention, the intention labeling task is divided into three states of in-process, completed and stopped, the in-process task can be edited under the condition that the task is not received, and if the task is matched with data and is received by a label, the editing is failed; when the number of the finished user label records is more than or equal to the number of the cross labels multiplied by the data volume of the task batch (the number of the task setting labels), marking the task as finished, not editable and not operable; and stopping manually pausing the task in progress for the user and stopping distributing the data to be annotated of the task.

Further, the allocating the intention labeling task to a labeling terminal includes:

In a preferred embodiment of the present invention, 20 annotation records meeting the above conditions are allocated to the annotation terminal in real time according to the configuration conditions of the annotation record and the intention annotation task of each user, and when the data to be annotated has data to be annotated in an annotated state, the annotation terminal directly displays the annotated records of the user. The marked state is to obtain the marked record of the current login user, the marking time is invalid for the state to be marked, and the marked state is to mark the record time range for the user to be matched. By the method, the distribution efficiency of the label records can be improved.

Further, obtaining the labeling result of the data to be labeled according to the labeling record of each labeling terminal and by combining a final result judgment strategy, includes:

judging whether the first quantity and the second quantity are equal;

In a preferred embodiment of the present invention, each user writes its own annotation record and simultaneously triggers traversing the annotation records of all users in the history of the statement in the data to be annotated, and summarizes the annotation records as the correct and incorrect numbers. Wherein the annotation record comprises, for example: if the annotation record marked as correct is 30 and the annotation record marked as error is 20, the annotation result corresponding to the intention annotation task is correct if the annotation record marked as correct is larger than the annotation record marked as error.

For example: if the annotation record marked as correct is 30 and the annotation record marked as error is 40, the annotation result corresponding to the intention annotation task is an error if the annotation record marked as correct is smaller than the annotation record marked as error.

For example: and if the annotation record marked as correct is 10 and the annotation record marked as wrong is 10, and the annotation record marked as correct is equal to the annotation record marked as wrong at this time, it indicates that the annotation result corresponding to the intention annotation task is wrong, and the annotation record needs to be rechecked manually, and the annotation record is added to the data to be annotated again. As can be seen from the above example, the final labeling result is obtained by combining the final result determination policy.

Further, obtaining the labeling result of the intention labeling task according to the labeling records of all the historical users and by combining a final result judgment strategy, and further comprising:

In a preferred embodiment of the invention, the cross labeling result is displayed according to the data dimension, a plurality of markers do not accord with the labeling result of the single data to be labeled, the data to be labeled is sent to the administrator terminal, the labeling administrator performs the rechecking operation and adds the labeling record again, and the accuracy of the labeling result is improved by the method. And the marking record at the moment is used as the final marking result of the data to be marked. The label records labeled as correct and incorrect are searched for as: according to the final result, judging the result of strategy judgment, and inquiring (namely, the result in derivation); it can also be changed to have a correct label record or a wrong label record. The state of the intention labeling task can be divided into partial labeling completion or labeling completion, wherein the partial labeling completion means that the user labeling record corresponding to the label is less than the cross labeling times (less than the number of labeling personnel), and the labeling completion means that the user labeling record is greater than the cross labeling times.

In the preferred embodiment of the invention, the export format of the marked data to be marked is CSV, the export must specify the task name, the search condition is all effective when exporting the marked data to be marked, the difference is that the task name or the intention name is fuzzy matching when inquiring, and the export result is abnormal when the match is completely matched for improving the efficiency and the misword and few words are required when exporting; the marked data to be marked are exported logically to summarize the marked records of all users, and the final export result is determined according to the final result judgment strategy and the result confirmation of the administrator terminal; and if the administrator terminal confirms the result, taking the confirmed marking result as a final marking result, and if the administrator terminal does not confirm the result, determining the strategy judgment result as the final marking result by default.

In the preferred embodiment of the present invention, step 1, data to be labeled is imported to a labeling center in a CSV file format; step 2, manually importing an intention labeling task into the data to be labeled, and creating the intention labeling task for the data to be labeled according to requirements; step 3, a plurality of annotators perform cross annotation on the same data to be annotated; step 4, displaying the cross labeling results according to the data dimensions, and rechecking the operation of a plurality of labeling managers with inconsistent labeling results of the single data by the labeling manager; and 5, monitoring the labeling workload of the laborers in real time, and counting the labeling accuracy of the labeling completion quantity of each task of each laborer.

Step 1 may refer to the above embodiment for "obtaining all data to be annotated with a preset file format of the same batch; and importing the data to be annotated in a preset file format to an annotation center, wherein the data to be annotated in the preset file format can be imported only once, and the intention annotation tasks corresponding to all the data to be annotated in the same batch are the same. "in step 2, reference may be made to the above embodiment to" create an intention tagging task according to the data to be tagged, and allocate the intention tagging task to a tagging terminal; in the description of "step 3, reference may be made to the above embodiment for" distributing data to be annotated having a state to be annotated to the annotation terminal in real time according to the annotation record of each user and by combining with the configuration condition of the intent annotation task, where the same data to be annotated may be annotated by a plurality of annotation terminals. In the description of "obtaining the labeling result of the data to be labeled according to the labeling record of each labeling terminal and by combining the final result judgment policy" in step 4, and in the description of "monitoring the workload of the intention labeling task corresponding to each labeler in real time, and counting the labeling completion amount and the labeling accuracy rate of the intention labeling task corresponding to each labeler in step 5, so as to perform work monitoring on each labeler" in the above embodiment.

Fig. 2 is a schematic structural diagram of an intention labeling apparatus provided in an embodiment of the present specification, including:

the data acquisition module 201 is used for acquiring data to be marked;

the task allocation module 202 is used for creating an intention labeling task according to the data to be labeled and allocating the intention labeling task to a labeling terminal;

the annotation record acquisition module 203 is used for acquiring annotation records added to the data to be annotated by the annotation terminal through the intention annotation task, and acquiring annotation records of other annotation terminals corresponding to the intention annotation task;

and the annotation record acquisition module 204 is configured to obtain an annotation result of the data to be annotated according to the annotation record of each annotation terminal and by combining the final result judgment policy.

Based on the same inventive concept, the embodiment of the specification further provides the electronic equipment.

In the following, embodiments of the electronic device of the present invention are described, which may be regarded as specific physical implementations for the above-described embodiments of the method and apparatus of the present invention. Details described in the embodiments of the electronic device of the invention should be considered supplementary to the embodiments of the method or apparatus described above; for details which are not disclosed in embodiments of the electronic device of the invention, reference may be made to the above-described embodiments of the method or the apparatus.

Fig. 3 is a schematic structural diagram of an electronic device provided in an embodiment of the present specification. An electronic device 300 according to this embodiment of the invention is described below with reference to fig. 3. The electronic device 300 shown in fig. 3 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.

As shown in fig. 3, electronic device 300 is embodied in the form of a general purpose computing device. The components of electronic device 300 may include, but are not limited to: at least one processing unit 310, at least one memory unit 320, a bus 330 connecting different device components (including the memory unit 320 and the processing unit 310), a display unit 340, and the like.

Wherein the storage unit stores program code executable by the processing unit 310 to cause the processing unit 310 to perform the steps according to various exemplary embodiments of the present invention described in the above-mentioned processing method section of the present specification. For example, the processing unit 310 may perform the steps as shown in fig. 1.

The storage unit 320 may include readable media in the form of volatile storage units, such as a random access memory unit (RAM)3201 and/or a cache storage unit 3202, and may further include a read only memory unit (ROM) 3203.

The storage unit 320 may also include a program/utility 3204 having a set (at least one) of program modules 3205, such program modules 3205 including, but not limited to: an operating device, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may comprise an implementation of a network environment.

Bus 330 may be one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

The electronic device 300 may also communicate with one or more external devices 400 (e.g., keyboard, pointing device, bluetooth device, etc.), with one or more devices that enable a user to interact with the electronic device 300, and/or with any devices (e.g., router, modem, etc.) that enable the electronic device 300 to communicate with one or more other computing devices. Such communication may occur via an input/output (I/O) interface 350. Also, the electronic device 300 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the internet) via the network adapter 360. Network adapter 360 may communicate with other modules of electronic device 300 via bus 330. It should be appreciated that although not shown in FIG. 3, other hardware and/or software modules may be used in conjunction with electronic device 300, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID devices, tape drives, and data backup storage devices, to name a few.

Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments of the present invention described herein may be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solution according to the embodiment of the present invention can be embodied in the form of a software product, which can be stored in a computer-readable storage medium (which can be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to make a computing device (which can be a personal computer, a server, or a network device, etc.) execute the above-mentioned method according to the present invention. The computer program, when executed by a data processing apparatus, enables the computer readable medium to implement the above-described method of the invention, namely: such as the method shown in fig. 1.

Fig. 4 is a schematic diagram of a computer-readable medium provided in an embodiment of the present disclosure.

A computer program implementing the method shown in fig. 1 may be stored on one or more computer readable media. The computer readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor device, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

The computer readable storage medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable storage medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution apparatus, device, or apparatus. Program code embodied on a readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).

In summary, the invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that some or all of the functionality of some or all of the components in embodiments in accordance with the invention may be implemented in practice using a general purpose data processing device such as a microprocessor or a Digital Signal Processor (DSP). The present invention may also be embodied as apparatus or device programs (e.g., computer programs and computer program products) for performing a portion or all of the methods described herein. Such programs implementing the present invention may be stored on computer-readable media or may be in the form of one or more signals. Such a signal may be downloaded from an internet website or provided on a carrier signal or in any other form.

While the foregoing embodiments have described the objects, aspects and advantages of the present invention in further detail, it should be understood that the present invention is not inherently related to any particular computer, virtual machine or electronic device, and various general-purpose machines may be used to implement the present invention. The invention is not to be considered as limited to the specific embodiments thereof, but is to be understood as being modified in all respects, all changes and equivalents that come within the spirit and scope of the invention.

The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments.

The above description is only an example of the present application and is not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. An intention labeling method, comprising:

acquiring data to be marked;

2. The intention labeling method of claim 1, further comprising:

3. The method for labeling intentions according to claim 1 or 2, wherein the obtaining data to be labeled comprises:

acquiring all data to be marked in the same batch and in a preset file format;

4. The method for labeling intentions according to any one of claims 1 to 3, wherein the creating of the intentions labeling task according to the data to be labeled comprises:

5. The method for labeling intentions according to any one of claims 1 to 4, wherein the allocating the task of labeling intentions to a labeling terminal comprises:

6. The method for labeling intentions according to any one of claims 1 to 5, wherein obtaining the labeling result of the data to be labeled according to the labeling record of each labeling terminal and by combining a final result judgment policy comprises:

judging whether the first quantity and the second quantity are equal;

7. The method for labeling intentions according to any one of claims 1 to 6, wherein the labeling result of the intentions labeling task is obtained according to the labeling records of all users in the history and by combining a final result judgment policy, further comprising:

8. An intention labeling apparatus, comprising:

the data acquisition module is used for acquiring data to be marked;

9. An electronic device, wherein the electronic device comprises:

a processor and a memory storing computer-executable instructions that, when executed, cause the processor to perform the method of any of claims 1-7.

10. A computer readable storage medium, wherein the computer readable storage medium stores one or more programs which, when executed by a processor, implement the method of any of claims 1-7.