CN109684947B - Method and device for monitoring labeling quality, computer equipment and storage medium - Google Patents
Method and device for monitoring labeling quality, computer equipment and storage medium Download PDFInfo
- Publication number
- CN109684947B CN109684947B CN201811507614.3A CN201811507614A CN109684947B CN 109684947 B CN109684947 B CN 109684947B CN 201811507614 A CN201811507614 A CN 201811507614A CN 109684947 B CN109684947 B CN 109684947B
- Authority
- CN
- China
- Prior art keywords
- labeling
- marking
- task
- ratio
- frames
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/41—Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items
Landscapes
- Engineering & Computer Science (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Image Analysis (AREA)
- General Factory Administration (AREA)
Abstract
The computer equipment quantifies the quality of the labeling task of a labeling person by adopting a preset quantification algorithm according to standard labeling data, and monitors the quality of the labeling task of the labeling person according to a quantification result, so that the quality of the labeling task of the labeling person is automatically quantified by the computer equipment, the labeling task is automatically checked, the quality of each labeling task can be visually observed according to the quantification result, the work of the labeling person is conveniently checked, and the efficiency of monitoring the labeling quality is greatly improved.
Description
Technical Field
The present application relates to the field of data annotation technologies, and in particular, to a method and an apparatus for monitoring annotation quality, a computer device, and a storage medium.
Background
In recent years, with the development of a high-performance Graphics Processing Unit (GPU) and further research on a deep learning technology, the application of the deep learning technology to an unmanned perception algorithm is increasing, and for deep learning, high-quality annotation data largely determines the final accuracy of the deep learning algorithm.
In order to improve the sample data of deep learning, a large number of annotators are required to go to the annotation work. In order to improve the labeling quality, a manager must monitor the labeling task quality of a labeler, but the labeling task performed by the labeler usually has no standard value, that is, when the manager checks the labeling task result of the labeler, no standard labeling answer is provided for reference, and only the manager and related professionals can monitor the labeling task quality of the labeler manually according to experience.
Therefore, how to better quantify the quality of the labeling task performed by the labeling personnel and improve the efficiency of monitoring the labeling quality becomes an urgent technical problem to be solved.
Disclosure of Invention
Therefore, it is necessary to provide a labeling quality monitoring method, apparatus, computer device and storage medium, in order to better quantify the quality of the labeling task performed by the labeling staff and improve the efficiency of monitoring the labeling quality, thereby promoting the labeling staff to improve the labeling quality.
In a first aspect, an embodiment of the present invention provides a method for monitoring annotation quality, where the method includes:
according to the standard marking data, quantifying the quality of the marking task of the marker by adopting a preset quantification algorithm to obtain a quantification result;
and monitoring the quality of the labeling task according to the quantification result.
In one embodiment, the quantizing the quality of the labeling task of the labeling staff by using a preset quantization algorithm according to the standard labeling data to obtain a quantization result includes:
acquiring the accuracy, recall rate and tracking rate of the quality of the labeling task by adopting a preset quantitative algorithm;
and determining the quantization result according to the accuracy, the recall rate and the tracking rate.
In one embodiment, the obtaining the accuracy, the recall rate, and the tracking rate of the quality of the labeling task by using a preset quantization algorithm includes:
determining the number of matching frames between the labeling frames in the labeling task of the labeling personnel and the labeling frames in the standard labeling data;
determining the ratio of the number of the matching frames to the number of the labeling frames in the standard labeling data as the recall rate;
determining the ratio of the number of the matching frames to the number of the labeling frames in the labeling task of the labeling staff as the accuracy;
determining the tracking rate according to the number of matching frames with the same number and the number of marking frames with the same number in the marking task of the marker; the same number indicates that the types of the marking frames of the previous and the next frames in the marking task are the same.
In one embodiment, before determining the tracking rate according to the number of matching boxes with the same number and the number of marking boxes with the same number in the marking task of the marker, the method comprises the following steps:
acquiring the number of matching frames with the same number and the number of marking frames with the same number in the marking tasks of the markers;
the determining the tracking rate according to the number of the matching frames with the same number and the number of the labeling frames with the same number in the labeling task of the labeling staff comprises the following steps:
and determining the average value of the ratio of the number of the matching frames with the same number to the number of the labeling frames with the same number in the labeling task of the labeling personnel as the tracking rate.
In one embodiment, before the determining the number of matching boxes between the annotation box in the annotator annotation task and the annotation box in the standard annotation data, the method further comprises:
determining the intersection ratio of a first labeling frame in the labeling task of the annotator and a second labeling frame in the standard labeling data;
and if the intersection ratio is larger than a preset threshold value, determining that the first marking frame and the second marking frame are matched frames.
In one embodiment, the determining the quantization result according to the accuracy rate, the recall rate and the tracking rate includes:
respectively acquiring an accuracy ratio, a recall ratio and a tracking ratio; the sum of the accuracy ratio, the recall ratio and the tracking ratio is 1;
and determining the quantification result according to the accuracy ratio, the recall ratio, the tracking ratio and the accuracy, the recall ratio and the tracking ratio.
In one embodiment, before quantizing the quality of the labeling task of the labeling staff by using a preset quantization algorithm according to the standard labeling data to obtain a quantization result, the method further includes:
receiving the standard marking data according to an operation instruction of a user;
and randomly distributing the standard marking data in the marking task of the marker.
In a second aspect, an embodiment of the present invention provides an annotation quality monitoring apparatus, where the apparatus includes:
the quantification module is used for quantifying the quality of the labeling task of the labeling personnel by adopting a preset quantification algorithm according to the standard labeling data to obtain a quantification result;
and the monitoring module is used for monitoring the quality of the labeling task according to the quantification result.
In a third aspect, an embodiment of the present invention provides a computer device, including a memory and a processor, where the memory stores a computer program, and the processor implements the following steps when executing the computer program:
according to the standard marking data, quantifying the quality of the marking task of the marker by adopting a preset quantification algorithm to obtain a quantification result;
and monitoring the quality of the labeling task according to the quantification result.
In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, where the computer program is executed by a processor to implement the following steps:
according to the standard marking data, quantifying the quality of the marking task of the marker by adopting a preset quantification algorithm to obtain a quantification result;
and monitoring the quality of the labeling task according to the quantification result.
According to the marking quality monitoring method, the device, the computer equipment and the storage medium, the computer equipment quantizes the marking task quality of a marker by adopting a preset quantization algorithm according to standard marking data, and monitors the marking task quality of the marker according to a quantization result, so that the marking task quality of the marker is automatically quantized through the computer equipment, the marking task is automatically checked, the quality of each marking task can be visually observed according to the quantization result, the work of the marker is conveniently checked, and the efficiency of monitoring the marking quality is greatly improved.
Drawings
Fig. 1 is an application environment diagram of a method for monitoring annotation quality according to an embodiment;
FIG. 2 is a flowchart illustrating a method for monitoring annotation quality according to an embodiment;
FIG. 3 is a flowchart illustrating a method for monitoring annotation quality according to an embodiment;
FIG. 4 is a flowchart illustrating a method for monitoring annotation quality according to an embodiment;
FIG. 5 is a flowchart illustrating a method for monitoring annotation quality according to an embodiment;
FIG. 6 is a flowchart illustrating a method for monitoring annotation quality according to an embodiment;
FIG. 7 is a flowchart illustrating a method for monitoring annotation quality according to an embodiment;
FIG. 8 is a flowchart of a method for monitoring annotation quality according to an embodiment;
FIG. 9 is a block diagram of an embodiment of an annotation quality monitoring device;
FIG. 10 is a block diagram illustrating an exemplary embodiment of an annotation quality monitoring device;
FIG. 11 is a block diagram of an embodiment of an annotation quality monitoring device;
FIG. 12 is a block diagram illustrating an embodiment of an annotation quality monitoring apparatus;
fig. 13 is a block diagram illustrating an exemplary embodiment of a tag quality monitoring apparatus;
fig. 14 is a block diagram of an annotation quality monitoring apparatus according to an embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
The annotation quality monitoring method provided by the application can be applied to the application environment shown in fig. 1, and the computer device can be a server, and the computer device comprises a processor, a memory, a network interface and a database which are connected through a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The database of the computer device is used for storing data of the annotation quality monitoring method. The network interface of the computer device is used for communicating with other external devices through network connection. The computer program is executed by a processor to implement a method of annotation quality monitoring.
Embodiments of the present application provide a method and an apparatus for monitoring annotation quality, a computer device, and a storage medium, which are intended to better quantify the quality of an annotation task performed by an annotator, and improve the efficiency of monitoring the annotation quality, thereby promoting the annotator to improve the annotation quality, which is a technical problem to be urgently solved. The following describes in detail the technical solutions of the present application and how the technical solutions of the present application solve the above technical problems by embodiments and with reference to the drawings. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments. It should be noted that, in the method for monitoring labeling quality provided by the present invention, the execution subject is a computer device, wherein the execution subject may also be a device for monitoring labeling quality, and the device may be implemented as part or all of the device for monitoring labeling quality by software, hardware, or a combination of software and hardware.
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In an embodiment, as shown in fig. 2, an embodiment of the present application provides a method for monitoring annotation quality, where this embodiment relates to a specific process in which a computer device quantizes the quality of an annotation task of an annotator, and monitors the quality of the annotation task according to a quantization result, and as shown in fig. 2, the method includes:
s101, quantizing the quality of the labeling task of the labeling personnel by adopting a preset quantization algorithm according to the standard labeling data to obtain a quantization result.
The standard annotation data can be a reference standard for quantifying the annotation task of the annotator in the annotation task queue of the annotator, and the standard annotation data can be randomly inserted into the annotation task of the annotator by the computer equipment. The preset quantization algorithm represents a rule and a calculation method according to which the computer device quantizes the annotating task of the annotator, and may be a ratio algorithm or other algorithms, wherein the rule in the algorithm may be determined by the user according to the actual situation, and the specific content of the preset quantization algorithm is not specified in this embodiment.
In practical application, the computer device quantizes the quality of the labeling task of the labeling member by using the preset quantization algorithm according to the standard labeling data, for example, the way for quantizing the quality of the labeling task by the computer device may be to calculate the score of each labeling task according to the preset quantization algorithm, and then take the score as the quantization result of each labeling task, where this embodiment does not limit the way for calculating the score of each labeling task by the computer device according to the preset quantization algorithm.
And S102, monitoring the quality of the labeling task according to the quantification result.
Based on the above step S101, the computer device quantifies the result of each annotation task of the annotator, and the computer device monitors the quality of each annotation task according to the quantified result, and the monitoring manner may exemplarily be that the computer device records the quantified result of each annotation task into the daily assessment score of the annotator, during monthly/quarterly assessment, the computer device ranks the annotators of the same assessment task according to the score values, and the administrator can appropriately reward or punish the annotators according to the ranking condition, and in the quality inspection at ordinary times, strengthen the management of the ranked annotators.
According to the marking quality monitoring method provided by the embodiment, the computer equipment quantizes the marking task quality of the marker by adopting a preset quantizing algorithm according to the standard marking data, and monitors the marking task quality of the marker according to a quantizing result, so that the marking task quality of the marker is automatically quantized by the computer equipment, the marking task is automatically checked, the quality of each marking task can be visually observed according to the quantizing result, the work of the marker is conveniently checked, and the efficiency of monitoring the marking quality is greatly improved.
On the basis of the foregoing embodiment, an embodiment of the present application further provides a method for monitoring annotation quality, where the embodiment relates to a specific process in which a computer device obtains an accuracy, a recall rate, and a tracking rate of an annotation task quality according to a preset quantization algorithm, so as to determine a quantization result, as shown in fig. 3, the step S101 includes:
s201, acquiring the accuracy, recall rate and tracking rate of the quality of the labeling task by adopting a preset quantization algorithm.
In this step, the computer device obtains the accuracy, the recall rate and the tracking rate of the quality of the labeling task by using a preset quantization algorithm, where the obtaining mode may be that the computer device directly calculates the accuracy, the recall rate and the tracking rate of the quality of the labeling task of the labeling staff by using the preset quantization algorithm, or may be other modes, which is not limited in this embodiment. The recall rate represents the probability that the labeling conforming to the standard is marked in the labeling task of the labeling personnel, the size and the position of the labeling box are proper, and the type is correct. The accuracy rate represents the probability that invalid labels can be filtered in a large amount of label data in the labeling task of a labeling operator, and only the data required by the labeling rule is labeled. The tracking rate indicates the probability that the marker judges whether the objects related to the marker are accurate or not in one marking task of the marker, generally, one marking task comprises a plurality of pictures or point cloud frames, and if the previous and next frames in the marking task are the same object, the marker needs to be marked with the same number.
S202, determining the quantification result according to the accuracy, the recall rate and the tracking rate.
Based on the accuracy, the recall rate and the tracking rate of the quality of the annotation task obtained by the computer device in the step S201, the computer device determines the quantization result of the annotation task according to the accuracy, the recall rate and the tracking rate, wherein the manner for determining the quantization result by the computer device may be to calculate the score value of the annotation task according to the accuracy, the recall rate and the tracking rate, and use the score value as the quantization result.
Optionally, as shown in fig. 4, one implementation manner of determining the quantization result by the computer device in S202 may further include:
s301, respectively acquiring an accuracy ratio, a recall ratio and a tracking ratio; the sum of the accuracy ratio, the recall ratio and the tracking ratio is 1.
The computer equipment respectively acquires the respective occupation ratios of the accuracy, the recall rate and the tracking rate, wherein the sum of the occupation ratios of the accuracy, the recall rate and the tracking rate is 1, and the occupation ratios of the accuracy, the recall rate and the tracking rate in the quantization process are respectively expressed. For example: the percentage of accuracy may be 30%, the percentage of recall rate may be 35%, the percentage of tracking rate may be 35%, and the specific numerical values of the percentages are not limited in this embodiment. It should be noted that the accuracy ratio, the recall ratio and the tracking ratio may be preset by a user, and then stored in the computer device, and the computer device may directly obtain the accuracy ratio, the recall ratio and the tracking ratio when needed.
S302, determining the quantification result according to the accuracy ratio, the recall ratio, the tracking ratio, the accuracy, the recall ratio and the tracking ratio.
Based on the accuracy ratio, the recall ratio and the tracking ratio determined by the computer device in the step S301, the quantization result is determined according to the accuracy ratio, the recall ratio and the tracking ratio of the labeling task, for example, the recall ratio, the accuracy ratio and the tracking ratio are set and respectively marked as c, a and t, and the quantization result is set as S, then the computer device may determine the quantization result by performing calculation according to the formula S-r 1-c + r 2-a + r 3-t, where r1 represents the recall ratio, r2 represents the accuracy ratio, and r3 represents the tracking ratio. For example: when r1 is 35%, r2 is 30%, the tracking ratio is 35%, c is 80%, a is 90%, and t is 85%, the result S is 35% + 80% + 30% + 90% + 35% + 85% + 0.8475.
According to the marking quality monitoring method provided by the embodiment, the computer equipment firstly adopts a preset quantization algorithm to obtain the accuracy, the recall rate and the tracking rate of the quality of the marking task, and then determines the quantization result of the marking task of the marker according to the accuracy, the recall rate, the tracking rate, the ratio of the accuracy to the recall rate, the ratio of the recall rate to the tracking rate, and the ratio of the tracking rate.
For determining recall rate, accuracy rate and tracking rate of quality of an annotation task, an embodiment of the present application provides a method for monitoring annotation quality, where the embodiment relates to a specific process of calculating recall rate, accuracy rate and tracking rate of quality of an annotation task by a computer device according to the number of matching frames between an annotation frame in an annotation task of an annotator and an annotation frame in standard annotation data, as shown in fig. 5, where the step S201 includes:
s401, determining the number of matching frames between the labeling frames in the labeling task of the labeling personnel and the labeling frames in the standard labeling data.
In this embodiment, the annotation box in the annotation task of the annotator indicates the annotation box completed by the annotator in the actual annotation task. And the marking frame in the standard marking data is the marking frame in the correct result of each marking task pre-distributed by the user. The matching box represents the degree of closeness of the marking box actually finished by the marker compared with the marking box in the standard marking data.
In order to better express the concept of matching boxes, in an embodiment, an embodiment of the present application provides a method for monitoring annotation quality, where the embodiment relates to a specific process of a computer device for determining whether a annotation box in an annotation task of an annotator matches with an annotation box in the standard annotation data, as shown in fig. 6, the method includes:
s501, determining the intersection ratio of a first labeling frame in the labeling task of the annotator and a second labeling frame in the standard labeling data.
And the first labeling box represents a labeling box finished in the labeling task of the label maker. The second annotation box represents an annotation box in the standard annotation data. The Intersection-over-Union (IoU) represents the overlap ratio of the generated candidate box and the original marked box, i.e. the ratio of their Intersection to Union. Namely: IoU is equal to the intersection area of two boxes divided by the combined area of the two boxes. When the two frames overlap each other, IoU is larger, and when the two frames completely overlap each other, IoU is 1. In practical application, the computer device determines the intersection and combination ratio of a first labeling frame in the labeling task of the annotator and a second labeling frame in the standard labeling data, and the determination mode can be that the intersection and combination ratio between the first labeling frame and the second labeling frame is directly calculated according to a preset program.
S502, if the intersection ratio is larger than a preset threshold value, determining that the first marking frame and the second marking frame are matched frames.
Based on the intersection ratio of the first labeled frame and the second labeled frame determined by the computer device in the step S501, if the intersection ratio is greater than the preset threshold, the computer device determines that the first labeled frame and the second labeled frame are matched frames, that is, when the intersection ratio of the two frames is greater than a certain threshold and the labeled types are the same, the two frames can be considered to be matched. The preset threshold is set by the user, and this embodiment does not limit this.
According to the marking quality monitoring method provided by the embodiment, the concept of cross-over ratio is introduced, the cross-over ratio between the marking frame in the marking task of the marker and the marking frame in the standard marking data is calculated through the computer equipment, whether the two marking frames are matched or not is determined, and therefore the number of the matching frames is determined, and the computer equipment can determine the number of the matching frames more conveniently and quickly.
S402, determining the ratio of the number of the matching frames to the number of the labeling frames in the standard labeling data as the recall rate.
In this step, based on the number of matching frames between the labeling frame in the labeling task of the annotator and the labeling frame in the standard labeling data determined by the computer device in the step S401, the computer device determines the ratio of the number of matching frames to the number of labeling frames in the standard labeling data as the recall rate. For example, if the number of matching boxes is set to be p, and the number of labeled boxes in the standard labeled data is a, the recall rate is p/a.
S403, determining the ratio of the number of the matching frames to the number of the labeling frames in the labeling task of the labeling personnel as the accuracy.
In this step, based on the number of matching frames between the labeling frames in the annotator labeling task and the labeling frames in the standard labeling data determined by the computer device in the step S401, the computer device determines the ratio of the number of matching frames to the number of labeling frames in the annotator labeling task as the accuracy. For example, the number of matching boxes is set to be p, the number of marking boxes in the marking task of the marker is set to be s, and the recall rate is p/s.
S404, determining the tracking rate according to the number of matching frames with the same number and the number of marking frames with the same number in the marking task of the marker; the same number indicates that the types of the marking frames of the previous and the next frames in the marking task are the same.
In this step, the computer device determines the tracking rate according to the number of matching frames with the same number and the number of marking frames with the same number in the marking task of the marker, wherein the same number indicates that the types of the marking frames of the front and rear frames in the marking task are the same, it can be understood that the description tracking rate is indicated in the front of one marking task of the marker, the marker has the probability of judging whether the front and rear related objects are accurate, generally, one marking task includes a plurality of pictures or point cloud frames, and the same number needs to be marked if the front and rear frames in the marking task are the same object when the marker marks, so the matching frames with the same number and the marking frames with the same number indicate the marking frames of the same object.
Optionally, one implementation manner of determining, by the computer device, the tracking rate according to the number of matching boxes with the same number and the number of labeling boxes with the same number in the labeling task of the labeling staff includes: and determining the average value of the ratio of the number of the matching frames with the same number to the number of the labeling frames with the same number in the labeling task of the labeling personnel as the tracking rate. Optionally, before determining the tracking rate according to the number of matching boxes with the same number and the number of marking boxes with the same number in the marking task of the marker, the method includes: and acquiring the quantity of the matching frames with the same number and the quantity of the labeling frames with the same number in each labeling task of the labeling personnel.
Wherein, the number of the matching frames of the same object marked by the marker, which is represented by the number of the matching frames with the same number, is obtained, and the number of the matching frames of different objects in the marking task of the marker is obtained by obtaining the number of the matching frames with the same number, for example: the number of object 1 match boxes is 6, the number of object 2 match boxes is 8, and the number of object 3 match boxes is 4. Similarly, the number of the labeling boxes with the same number indicates the number of the labeling boxes (including matching and unmatching) of the same object marked by the marker, and the number of the labeling boxes with the same number in each labeling task of the marker is obtained by the number of the labeling boxes of different objects in the labeling task of the marker, for example: the number of the marking frames of the object 1 is 12, the number of the marking frames of the object 2 is 25, and the number of the marking frames of the object 3 is 16.
In practical application, the computer device determines the tracking rate according to the obtained number of the matching frames with the same number and the obtained number of the labeling frames with the same number in each labeling task of the labeling staff, that is, the tracking rate is determined by an average value of the ratio of the number of the matching frames with the same number to the number of the labeling frames with the same number in each labeling task of the labeling staff, for example: if the ratio of the object 1 is 6/12, the ratio of the object 2 is 8/25, and the ratio of the object 3 is 4/16, the tracking ratio is (6/12+8/25+ 4/16)/3-90%.
In the marking quality monitoring method provided by the embodiment, the computer equipment firstly determines the number of matching frames between the marking frame in the marking task of the marker and the marking frame in the standard marking data, and calculates the recall rate, the accuracy rate and the tracking rate of the quality of the marking task according to the number of the matching frames, the number of the marking frame in the marking task of the marker and the number of the marking frame in the standard marking data, so that the recall rate, the accuracy rate and the tracking rate of the quality of the marking task are calculated according to preset rules, the quality of the marking task of the marker is more definitely reflected, the work of the marker is conveniently checked, and the efficiency of monitoring the marking quality is greatly improved.
Considering that when the quality of the annotation task of the annotator is quantified by using a preset quantification algorithm according to the standard annotation data, the standard annotation data is randomly allocated to the user in the annotation task of the annotator, and the annotator does not know which of the current annotation task carries the standard annotation data in the process of completing the annotation task, in an embodiment, the embodiment of the present application provides a method for monitoring the annotation quality, which specifically relates to a specific process of randomly allocating the standard annotation data to the annotation task of the annotator by a computer device according to an operation instruction of the user, as shown in fig. 7, the method further includes:
s601, receiving the standard marking data according to the operation instruction of the user.
In this step, according to the operation instruction of the user, the computer device receives the standard marking data, where the operation instruction of the user may be that the user passes through an external device, for example: the standard marking data input by the keyboard and the like can also be standard marking data transmitted to the computer equipment by a user through other equipment.
S602, randomly distributing the standard annotation data in the annotation task of the annotator.
Based on the standard annotation data received by the computer device in the step S601, the computer device randomly allocates the standard annotation data to the annotation task of the annotator, for example, the computer device randomly inserts the standard annotation data into the annotation task of the annotator in the annotation task queue of the annotator, it should be noted that different data in the standard annotation data correspond to different annotation tasks, so that the standard annotation data is definitely inserted into the annotation task corresponding to the standard annotation data when the computer device inserts the standard annotation data. In addition, the standard marking data can be made manually, and can be data which is marked by people with more experience and determined by the examination of each professional.
In the method for monitoring the labeling quality provided by this embodiment, the computer device receives the standard labeling data according to the operation instruction of the user, and then randomly allocates the standard labeling data to the labeling task of the annotator.
It should be noted that, in the step S101 of the annotation quality monitoring method provided in the embodiment of the present application, the preset quantization algorithm is adopted to quantize the quality of the annotation task of the annotator according to the standard annotation data to obtain a quantization result, which may also be applied to quantizing the assessment result of the new annotator, for example, as shown in fig. 8, the process of quantizing the assessment result of the new annotator by the computer is as follows:
s701, acquiring standard marking data. As described in steps S601-S602 above, the computer device directly obtains the received standard annotation data.
S702, selecting a labeling task suitable for the assessment of a new person from the labeling tasks corresponding to the standard labeling data. The computer equipment selects an annotation task suitable for the assessment of a new person from the obtained standard annotation data, wherein the selection mode is as follows: and the computer equipment sequences the workload of each labeling task and determines the labeling task with small workload as the labeling task most suitable for the evaluation of the newborns. The workload of the annotation task is reflected on the rules and the content of the annotation task, and if the rules and the content of the annotation task are more, the workload of the annotation task is large.
And S703, quantizing the quality of the labeling task of the labeling personnel by the computer equipment according to the standard labeling data and by adopting a preset quantization algorithm to the labeling task submitted by the new person, and obtaining a quantization result. As in the embodiments shown in fig. 2 to fig. 7, the computer device quantizes the labeling task submitted by the new person as the method for quantizing the quality of the labeling task of the labeling person by using the preset quantization algorithm according to the standard labeling data, and the specific quantization process is not repeated.
And S704, determining the assessment result of the new person by the computer equipment according to the quantification result pair. According to the quantization result of the labeling task submitted by the new person determined by the computer equipment, the computer equipment compares the set examination passing threshold (set manually) according to the quantization result, and judges whether the new person passes the examination or not according to the comparison result. If the quantitative value of the newborn is lower than the examination passing threshold value which passes the setting, the newborn is judged not to pass the examination, and then the computer equipment can give an improvement suggestion according to the results of the recall rate, the accuracy rate and the tracking rate, so that the user can make targeted improvement. If the new person does not pass more than a plurality of (for example, 3) examinations, standard marking data is directly displayed for the new person in the computer equipment for the new person to learn, and then the new person can be manually guided.
And finally, the computer equipment records the assessment result of the novice into the novice file for reference in the subsequent marking and examining quality process. In the embodiment, a novice can perform examination in the process of being familiar with the tool, automatically judge whether the examination is passed or not, and display a reference result, so that manpower is released, and the secondary training cost is reduced.
It should be understood that although the various steps in the flow charts of fig. 2-7 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least some of the steps in fig. 2-7 may include multiple sub-steps or multiple stages that are not necessarily performed at the same time, but may be performed at different times, and the order of performance of the sub-steps or stages is not necessarily sequential, but may be performed in turn or alternating with other steps or at least some of the sub-steps or stages of other steps.
In one embodiment, as shown in fig. 9, there is provided an annotation quality monitoring apparatus including: a quantification module 10 and a monitoring module 11, wherein:
the quantification module 10 is configured to quantify the quality of the labeling task of the labeling staff by using a preset quantification algorithm according to the standard labeling data to obtain a quantification result;
and the monitoring module 11 is configured to monitor the quality of the labeling task according to the quantization result.
The implementation principle and technical effect of the marking quality monitoring device provided by the above embodiment are similar to those of the above method embodiment, and are not described herein again.
In one embodiment, as shown in fig. 10, there is provided an annotation quality monitoring apparatus, wherein the quantization module 10 includes: an acquisition unit 101 and a determination unit 102, wherein:
an obtaining unit 101, configured to obtain, by using a preset quantization algorithm, an accuracy, a recall rate, and a tracking rate of the quality of the labeling task;
a determining unit 102, configured to determine the quantization result according to the accuracy, the recall rate, and the tracking rate.
The implementation principle and technical effect of the marking quality monitoring device provided by the above embodiment are similar to those of the above method embodiment, and are not described herein again.
In one embodiment, as shown in fig. 11, there is provided a marking quality monitoring apparatus, where the obtaining unit 101 includes: a matching box number determining sub-unit 1011, a recall rate determining sub-unit 1012, an accuracy rate determining sub-unit 1013, and a tracking rate determining sub-unit 1014, wherein:
a matching frame number determining subunit 1011, configured to determine the number of matching frames between the annotation frame in the annotator annotation task and the annotation frame in the standard annotation data;
a recall rate determining subunit 1012, configured to determine, as the recall rate, a ratio of the number of the matching boxes to the number of the labeled boxes in the standard labeled data;
an accuracy determining subunit 1013, configured to determine, as the accuracy, a ratio of the number of the matching frames to the number of the labeling frames in the labeling task of the labeling staff;
a tracking rate determining subunit 1014, configured to determine the tracking rate according to the number of matching frames with the same number and the number of labeling frames with the same number in the labeling task of the labeling member; the same number indicates that the types of the marking frames of the previous and the next frames in the marking task are the same.
The implementation principle and technical effect of the marking quality monitoring device provided by the above embodiment are similar to those of the above method embodiment, and are not described herein again.
In one embodiment, the apparatus further includes an obtaining subunit, configured to obtain the number of matching boxes with the same number and the number of labeling boxes with the same number in each of the labeler labeling tasks. The tracking rate determining subunit 1014 is specifically configured to determine, as the tracking rate, an average value of a ratio of the number of the matching frames with the same number to the number of the labeling frames with the same number in the labeling task of the labeling member.
The implementation principle and technical effect of the marking quality monitoring device provided by the above embodiment are similar to those of the above method embodiment, and are not described herein again.
In an embodiment, as shown in fig. 12, there is provided a marking quality monitoring apparatus, where the obtaining unit 101 further includes: the cross-over ratio determining subunit 1015 and the match box determining subunit 1016, where:
an intersection ratio determining subunit 1015, configured to determine an intersection ratio between a first annotation box in the annotator annotation task and a second annotation box in the standard annotation data;
a matching frame determining subunit 1016, configured to determine that the first labeling frame and the second labeling frame are matching frames if the intersection ratio is greater than a preset threshold.
The implementation principle and technical effect of the marking quality monitoring device provided by the above embodiment are similar to those of the above method embodiment, and are not described herein again.
In one embodiment, as shown in fig. 13, there is provided an annotation quality monitoring apparatus, wherein the determining unit 102 includes: a duty acquisition subunit 1021 and a quantization result determination subunit 1022, wherein:
an occupation ratio obtaining subunit 1021, configured to obtain an accuracy occupation ratio, a recall occupation ratio, and a tracking occupation ratio, respectively; the sum of the accuracy ratio, the recall ratio and the tracking ratio is 1;
a quantization result determining subunit 1022, configured to determine the quantization result according to the accuracy ratio, the recall ratio, the tracking ratio and the accuracy, the recall ratio, and the tracking ratio.
The implementation principle and technical effect of the marking quality monitoring device provided by the above embodiment are similar to those of the above method embodiment, and are not described herein again.
In one embodiment, as shown in fig. 14, there is provided a marking quality monitoring apparatus, further comprising: a receiving module 12 and a distribution module 13, wherein,
a receiving module 12, configured to receive the standard annotation data according to an operation instruction of a user;
and the distribution module 13 is configured to randomly distribute the standard annotation data to the annotation task of the annotator.
The implementation principle and technical effect of the marking quality monitoring device provided by the above embodiment are similar to those of the above method embodiment, and are not described herein again.
For the specific definition of the labeling quality monitoring device, reference may be made to the above definition of the labeling quality monitoring method, which is not described herein again. All or part of the modules in the labeling quality monitoring device can be realized by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, a computer device is provided, which may be a terminal, the internal structure of which may be as described above in fig. 1. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a method of annotation quality monitoring. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, a key, a track ball or a touch pad arranged on the shell of the computer equipment, an external keyboard, a touch pad or a mouse and the like.
Those skilled in the art will appreciate that the architecture shown in fig. 1 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, a computer device is provided, comprising a memory and a processor, the memory having a computer program stored therein, the processor implementing the following steps when executing the computer program:
according to the standard marking data, quantifying the quality of the marking task of the marker by adopting a preset quantification algorithm to obtain a quantification result;
and monitoring the quality of the labeling task according to the quantification result.
The implementation principle and technical effect of the computer device provided by the above embodiment are similar to those of the above method embodiment, and are not described herein again.
In one embodiment, a computer-readable storage medium is provided, having a computer program stored thereon, which when executed by a processor, performs the steps of:
according to the standard marking data, quantifying the quality of the marking task of the marker by adopting a preset quantification algorithm to obtain a quantification result;
and monitoring the quality of the labeling task according to the quantification result.
The implementation principle and technical effect of the computer-readable storage medium provided by the above embodiments are similar to those of the above method embodiments, and are not described herein again.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (10)
1. A method for monitoring annotation quality, the method comprising:
according to the standard labeling data, quantifying the labeling task quality of a labeling operator by adopting a preset quantification algorithm to obtain a quantification result, wherein the quantification result comprises the following steps: acquiring the accuracy, recall rate and tracking rate of the quality of the labeling task by adopting a preset quantitative algorithm; determining the quantization result according to the accuracy rate, the recall rate and the tracking rate; the recall rate is determined according to the ratio of the number of the matching frames to the number of the marking frames in the standard marking data; the accuracy is determined according to the ratio of the number of the matching frames to the number of the labeling frames in the labeling task of the labeling personnel; the tracking rate is determined according to the number of matching frames with the same number and the number of marking frames with the same number in the marking task of the marker; the matching box represents a labeling box matched with a labeling box in the standard labeling data in the labeling task of the labeling personnel; the standard marking data represents the correct marking result of each marking task and is used as a reference standard for quantifying the marking tasks of the marker; the marking frame in the standard marking data represents a marking frame in the correct result of each marking task pre-distributed by a user;
monitoring the quality of the labeling task according to the quantification result;
the determining the quantization result according to the accuracy rate, the recall rate, and the tracking rate includes:
respectively acquiring an accuracy ratio, a recall ratio and a tracking ratio;
and determining the quantification result according to the accuracy ratio, the recall ratio, the tracking ratio and the accuracy, the recall ratio and the tracking ratio.
2. The method according to claim 1, wherein before said obtaining the accuracy, recall rate and tracking rate of the quality of the labeling task by using a preset quantization algorithm, the method further comprises:
and determining the number of matching frames between the labeling frames in the labeling task of the labeling personnel and the labeling frames in the standard labeling data.
3. The method of claim 2, wherein the method comprises:
acquiring the number of matching frames with the same number and the number of marking frames with the same number in the marking tasks of the markers;
and determining the average value of the ratio of the number of the matching frames with the same number to the number of the labeling frames with the same number in the labeling task of the labeling personnel as the tracking rate.
4. The method of claim 3, wherein prior to said determining the number of matching boxes between a callout box in the annotator annotation task and a callout box in the standard annotation data, the method further comprises:
determining the intersection ratio of a first labeling frame in the labeling task of the annotator and a second labeling frame in the standard labeling data;
and if the intersection ratio is larger than a preset threshold value, determining that the first marking frame and the second marking frame are matched frames.
5. The method according to claim 3 or 4,
the sum of the accuracy ratio, the recall ratio and the tracking ratio is 1.
6. The method according to claim 1, before quantifying the quality of the labeling task of the labeling staff by using a preset quantification algorithm according to the standard labeling data to obtain a quantification result, the method further comprises:
receiving the standard marking data according to an operation instruction of a user;
and randomly distributing the standard marking data in the marking task of the marker.
7. A marking quality monitoring apparatus, the apparatus comprising:
the quantification module is used for quantifying the quality of the labeling task of the labeling personnel by adopting a preset quantification algorithm according to the standard labeling data to obtain a quantification result;
the quantification module is specifically used for acquiring the accuracy, the recall rate and the tracking rate of the quality of the labeling task by adopting a preset quantification algorithm; respectively acquiring an accuracy ratio, a recall ratio and a tracking ratio; determining the quantification result according to the accuracy ratio, the recall ratio, the tracking ratio and the accuracy, the recall ratio and the tracking ratio; the recall rate is determined according to the ratio of the number of the matching frames to the number of the marking frames in the standard marking data; the accuracy is determined according to the ratio of the number of the matching frames to the number of the labeling frames in the labeling task of the labeling personnel; the tracking rate is determined according to the number of matching frames with the same number and the number of marking frames with the same number in the marking task of the marker; the matching box represents a labeling box matched with a labeling box in the standard labeling data in the labeling task of the labeling personnel; the standard marking data represents the correct marking result of each marking task and is used as a reference standard for quantifying the marking tasks of the marker; the marking frame in the standard marking data represents a marking frame in the correct result of each marking task pre-distributed by a user;
and the monitoring module is used for monitoring the quality of the labeling task according to the quantification result.
8. The apparatus of claim 7, further comprising:
and the matching frame quantity determining subunit is used for determining the quantity of the matching frames between the labeling frames in the labeling task of the labeling personnel and the labeling frames in the standard labeling data.
9. A computer device comprising a memory and a processor, the memory storing a computer program, wherein the processor implements the steps of the method of any one of claims 1 to 6 when executing the computer program.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811507614.3A CN109684947B (en) | 2018-12-11 | 2018-12-11 | Method and device for monitoring labeling quality, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811507614.3A CN109684947B (en) | 2018-12-11 | 2018-12-11 | Method and device for monitoring labeling quality, computer equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109684947A CN109684947A (en) | 2019-04-26 |
| CN109684947B true CN109684947B (en) | 2021-03-30 |
Family
ID=66187497
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811507614.3A Active CN109684947B (en) | 2018-12-11 | 2018-12-11 | Method and device for monitoring labeling quality, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109684947B (en) |
Families Citing this family (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110263853B (en) * | 2019-06-20 | 2023-05-23 | 杭州睿琪软件有限公司 | Method and device for checking state of manual client by using error sample |
| CN110378617A (en) * | 2019-07-26 | 2019-10-25 | 中国工商银行股份有限公司 | A kind of sample mask method, device, storage medium and equipment |
| CN110472054B (en) * | 2019-08-15 | 2023-05-23 | 北京爱数智慧科技有限公司 | Data processing method and device |
| CN110796185B (en) * | 2019-10-17 | 2022-08-26 | 北京爱数智慧科技有限公司 | Method and device for detecting image annotation result |
| CN111159167B (en) * | 2019-12-30 | 2024-02-23 | 上海依图网络科技有限公司 | Labeling quality detection device and method |
| CN111178078A (en) * | 2019-12-31 | 2020-05-19 | 厦门快商通科技股份有限公司 | Quality inspection method, device and equipment for entity labeling |
| CN111275079B (en) * | 2020-01-13 | 2023-07-04 | 浙江大学 | Crowd-sourced label presumption method and system based on graph neural network |
| CN111291013A (en) * | 2020-01-17 | 2020-06-16 | 深圳市商汤科技有限公司 | Behavior data processing method and device, electronic equipment and storage medium |
| CN111427873B (en) * | 2020-03-12 | 2023-03-14 | 无码科技（杭州）有限公司 | Data cleaning method and system |
| CN111581195A (en) * | 2020-04-29 | 2020-08-25 | 厦门快商通科技股份有限公司 | Method, system and device for quality inspection marking data |
| CN112650661B (en) * | 2020-12-29 | 2024-07-09 | 北京嘀嘀无限科技发展有限公司 | Data processing quality control method, device, computer equipment and storage medium |
| CN115131813B (en) * | 2022-07-12 | 2023-04-07 | 中国建筑西南设计研究院有限公司 | Method and device for confirming numbering position, electronic equipment and storage medium |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103324620B (en) * | 2012-03-20 | 2016-04-27 | 北京百度网讯科技有限公司 | A kind of method and apparatus that annotation results is rectified a deviation |
| US8837819B1 (en) * | 2012-04-05 | 2014-09-16 | Google Inc. | Systems and methods for facilitating identification of and interaction with objects in a video or image frame |
| CN105975980B (en) * | 2016-04-27 | 2019-04-05 | 百度在线网络技术（北京）有限公司 | The method and apparatus of monitoring image mark quality |
| CN107273891A (en) * | 2017-06-08 | 2017-10-20 | 深圳市唯特视科技有限公司 | A kind of target category detection method based on click supervised training |
| CN108197658B (en) * | 2018-01-11 | 2020-08-14 | 阿里巴巴集团控股有限公司 | Image annotation information processing method, device, server and system |
-
2018
- 2018-12-11 CN CN201811507614.3A patent/CN109684947B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN109684947A (en) | 2019-04-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109684947B (en) | Method and device for monitoring labeling quality, computer equipment and storage medium | |
| JP6762584B2 (en) | Learning model construction device, post-employment evaluation prediction device, learning model construction method and post-employment evaluation prediction method | |
| CN108766545B (en) | Online inquiry department distribution method, device, computer equipment and storage medium | |
| CN108039200A (en) | A kind of information recommendation method, device and storage medium, program product | |
| CN113157951A (en) | Multimedia resource processing method, device, server and storage medium | |
| CN114240494A (en) | Medical project cost accounting method, system, electronic device and storage medium | |
| Böðvarsdóttir et al. | Achieving compromise solutions in nurse rostering by using automatically estimated acceptance thresholds | |
| US10817920B2 (en) | System and method for managing selection of prescription drug plans | |
| US9251461B2 (en) | Interactive method to reduce the amount of tradeoff information required from decision makers in multi-attribute decision making under uncertainty | |
| CN108764553B (en) | User scale prediction method and device and computer equipment | |
| CN111382250A (en) | Question text matching method and device, computer equipment and storage medium | |
| CN109003193B (en) | Method, device, computer equipment and storage medium for predicting insurance risk | |
| CN110097250B (en) | Product risk prediction method, device, computer equipment and storage medium | |
| US20150364050A1 (en) | Computer-implemented content repository and delivery system for online learning | |
| WO2022267364A1 (en) | Information recommendation method and device, and storage medium | |
| JP2020160686A (en) | Information processing apparatus and program | |
| CN110874758A (en) | Potential customer prediction method, device, system, electronic equipment and storage medium | |
| US11562126B2 (en) | Coaching system and coaching method | |
| CN114020429A (en) | Task processing method and device, equipment and storage medium | |
| CN113764077B (en) | Pathological image processing method and device, electronic equipment and storage medium | |
| Love et al. | Over-application and interviewing in the 2021 United States primary care virtual recruitment season | |
| CN110929163A (en) | Course recommendation method and device, computer equipment and storage medium | |
| CN110827160A (en) | Insurance information pushing method and device, computer equipment and storage medium | |
| CN110097265A (en) | Acquisition methods, device and the storage medium of the ready degree of Project Technical | |
| Mammadova et al. | Methods managing for matching of supply and demand on the IT Specialists |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |