CN105404896B - Labeled data processing method and labeled data processing system - Google Patents
Labeled data processing method and labeled data processing system Download PDFInfo
- Publication number
- CN105404896B CN105404896B CN201510744484.5A CN201510744484A CN105404896B CN 105404896 B CN105404896 B CN 105404896B CN 201510744484 A CN201510744484 A CN 201510744484A CN 105404896 B CN105404896 B CN 105404896B
- Authority
- CN
- China
- Prior art keywords
- mark
- annotation results
- data processing
- labeled data
- task
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
Landscapes
- Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a kind of labeled data processing method and labeled data processing systems.The labeled data processing method includes: step S110: calculating the similarity of multiple annotation results relevant to mark task；Step S120: similarity is compared with similarity threshold, if similarity is greater than or equal to similarity threshold, goes to step S130, if similarity is less than the similarity threshold, goes to step S140；Step S130: determine that multiple annotation results pass through quality testing；And step S140: determine that multiple annotation results do not pass through quality testing.The labeled data processing method and labeled data processing system provided according to the present invention, due to detecting the quality of annotation results automatically using similarity, so that mark personnel are possible to the quality of timely learning annotation results, and then it is possible to timely correction marking error, it can effectively improve mark accuracy.
Description
Technical field
The present invention relates to data processing fields, and in particular to a kind of labeled data processing method and labeled data processing system
System.
Background technique
Machine is trained (or saying study) and usually requires a large amount of labeled data as training set, the data of labeled data
It measures bigger more helpful to trained effect, therefore how efficiently and accurately to carry out data mark and have become one and urgently solve
Certainly the problem of.The data mark process of existing data labeling system is usually: publication includes one or more mark units
Mark task is manually marked, carries out manual quality's inspection etc..Existing data labeling system fully relies on manual quality's inspection
It looks into control mark accuracy, therefore may be grown very much from the time interval being accomplished between quality examination is manually marked, it is difficult to
The mistake of timely correction mark personnel.
Summary of the invention
In view of the above problems, the present invention is proposed in order to provide a kind of labeled data at least being partially solved the above problem
Processing method and labeled data processing system.
According to an aspect of the invention, there is provided a kind of labeled data processing method, comprising: step S110: calculate with
The similarity of the relevant multiple annotation results of mark task；Step S120: similarity is compared with similarity threshold, if phase
It is greater than or equal to similarity threshold like degree, then goes to step S130, if similarity is less than similarity threshold, goes to step
S140；Step S130: determine that multiple annotation results pass through quality testing；And step S140: determine that multiple annotation results are not led to
Cross quality testing.
According to another aspect of the present invention, a kind of labeled data processing system, including computing device, similarity-rough set are provided
Device, the first executive device and the second executive device.Computing device is for calculating multiple annotation results relevant to the task that marks
Similarity.Similarity-rough set device is for comparing similarity with similarity threshold, if similarity is greater than or equal to phase
Like degree threshold value, then start the first executive device, if similarity is less than similarity threshold, starts the second executive device.First
Executive device is for determining that multiple annotation results pass through quality testing.Second executive device is for determining that multiple annotation results are not led to
Cross quality testing.
The labeled data processing method and labeled data processing system provided according to the present invention, due to automatic using similarity
Detect the quality of annotation results so that mark personnel are possible to the quality of timely learning annotation results, and then be possible to and
When correct marking error, can effectively improve mark accuracy.
The above description is only an overview of the technical scheme of the present invention, in order to better understand the technical means of the present invention,
And it can be implemented in accordance with the contents of the specification, and in order to allow above and other objects of the present invention, feature and advantage can
It is clearer and more comprehensible, the followings are specific embodiments of the present invention.
Detailed description of the invention
By reading the following detailed description of the preferred embodiment, various other advantages and benefits are common for this field
Technical staff will become clear.The drawings are only for the purpose of illustrating a preferred embodiment, and is not considered as to the present invention
Limitation.And throughout the drawings, the same reference numbers will be used to refer to the same parts.In the accompanying drawings:
Fig. 1 shows the flow chart of labeled data processing method according to an embodiment of the invention；
Fig. 2 shows the flow charts of labeled data processing method in accordance with another embodiment of the present invention；
Fig. 3 shows the flow chart of labeled data processing method in accordance with another embodiment of the present invention；And
Fig. 4 shows the schematic block diagram of labeled data processing system according to an embodiment of the invention.
Specific embodiment
Exemplary embodiments of the present disclosure are described in more detail below with reference to accompanying drawings.Although showing the disclosure in attached drawing
Exemplary embodiment, it being understood, however, that may be realized in various forms the disclosure without should be by embodiments set forth here
It is limited.On the contrary, these embodiments are provided to facilitate a more thoroughly understanding of the present invention, and can be by the scope of the present disclosure
It is fully disclosed to those skilled in the art.
According to an aspect of the invention, there is provided a kind of labeled data processing method.Fig. 1 shows according to the present invention one
The flow chart of the labeled data processing method 100 of a embodiment.
As shown in Figure 1, labeled data processing method 100 includes the following steps.
Step S110: the similarity of multiple annotation results relevant to mark task is calculated.Mark task as described herein
Refer to requiring including object to be marked and mark for task.Object to be marked is referred to as " mark unit ", can be
One or a set of image, video or audio etc..Mark requirement is the letter how indicateing arm note personnel are labeled mark unit
Breath.For example, mark unit can be the image comprising several faces, mark, which requires can be, to be indicated to outline the institute in image
The information for having face or marking out several key points on face etc..Mark personnel can scheme after receiving mark task
Face or mark face key point etc. are outlined as in.Image comprising the face through marking can be used in and answer with all kinds of recognitions of face
With in the training of relevant machine.It certainly, is not pair the above is only the example of mark unit and mark requirement in mark task
Limitation of the invention.Other several examples that mark unit and mark require are described below.Mark unit can also be comprising it
The image of his content e.g. includes the image of the contents such as text (trademark or license plate number etc.), animal, article.Accordingly
Ground, mark require to can be the information of all texts indicated mark out in image, animal or article.Mark unit can also be
Image comprising personage, mark require to be to indicate to determine the sex, race of personage or the information at age etc. in image.Mark unit
Can also be one group of image, including one include object of reference image and multiple images to be selected, mark require can be instruction from
The information of specific image to be selected is selected in all images to be selected, which includes the object same or similar with object of reference
Body.Mark unit can also be a segment of audio, and mark requires to be the information for indicating the text occurred in input audio.Mark unit
It can also be a problem and set of option, mark requires to be that instruction selects appropriate choosing corresponding with problem from set of option
The information of item.
Annotation results refer to that mark personnel require the result data obtained after being labeled to mark unit based on mark.
For example, annotation results may include the information of several face key points marked out about mark personnel, such as each face
The position etc. of key point in the picture.According to an embodiment of the invention, a mark task can be received by multiple mark personnel
And it participates in.Each mark personnel can provide an annotation results for a mark task, thus, it is possible to obtain appointing with mark
It is engaged in relevant multiple annotation results.Then, the similarity between these annotation results can be calculated.For different types of mark
As a result for, the calculation method of similarity may be different.Below by taking two annotation results as an example description similarity calculation method.
A numerical value be can use to describe the similarity between two annotation results, it includes but be not limited to following several method:
If can be calculated between the corresponding points in two annotation results if mark task is that mark is done on the image
Similarity of the summation of Euclidean distance as two annotation results；
If mark task is to mark several polygons on the image, the polygon phase in two annotation results can be calculated
Cross surface product and phase and similarity of the area ratio (IOU) as two annotation results；
If mark task be in multiple options select an option, the consistent similarity of two annotation results be 1, two
The inconsistent similarity of annotation results is 0；
If mark task is to select more than one option in multiple options, can calculate in two annotation results
The quantity of public option (i.e. the intersection of option) (has selected choosing with all options in two annotation results
Union) similarity of the ratio of number as two annotation results.
With reference to the description of the similarity above with respect to two annotation results, it is possible to understand that the phase of more than two annotation results
Like the calculation of degree, it can realize that details are not described herein using routine techniques.
Step S120: similarity is compared with similarity threshold, if similarity is greater than or equal to similarity threshold,
Step S130 is gone to, if similarity is less than similarity threshold, goes to step S140.Similarity threshold can be any suitable
Value, for example, similarity threshold can be greater than or equal to 80%, e.g. 85%, 90% or 95% etc. can according to need
Depending on, the present invention limits not to this.Similarity threshold can be initially set to a default value, then can be according to practical need
Automatically it to be adjusted.For different mark tasks, similarity threshold can be identical or different.For example, for opposite
Simple mark task, similarity threshold can be larger, on the contrary, for relative complex mark task, similarity threshold can be with
It is smaller.Multiple annotation results of the same mark task can be compared, calculate the similarity between them, then, sentence
Whether the similarity between multiple annotation results of breaking is greater than or equal to above-mentioned similarity threshold, as shown in Figure 1.According to similarity with
Size relation selection between similarity threshold executes step S130 and still executes step S140.
Step S130: determine that multiple annotation results pass through quality testing.If the similarity of above-mentioned multiple annotation results is big
In or be equal to similarity threshold, then illustrate multiple mark personnel to it is same mark unit carry out mark be all it is more similar,
Therefore this multiple annotation results is possible to be the higher annotation results of accuracy.Therefore, such case can be considered as these
Annotation results are all the correct marks to mark unit, are the correct processing results of mark task.In this way it can be considered that this is multiple
The quality of annotation results can guarantee, determine that they pass through quality testing.
Step S140: determine that multiple annotation results do not pass through quality testing.If the phase between above-mentioned multiple annotation results
It is less than similarity threshold like degree, then the mark difference for illustrating that multiple mark personnel carry out same mark unit is larger, therefore, right
For this multiple annotation results, wherein a possibility that there are error labels is very big.Therefore, such case can be considered as to these marks
Note result is not the correct mark to mark unit, that is, thinks that the quality of this multiple annotation results not can guarantee, and determines it
Do not pass through quality testing.It is understood that quality testing as described herein is to detect annotation results by similarity
Quality can reflect the quality of annotation results to a certain extent, can also be further by artificial after quality testing
Mode verifies the quality of annotation results.Efficiency is higher by way of quality of the similarity to detect annotation results automatically, and
And mark accuracy can be improved.
The labeled data processing method provided according to the present invention, due to detecting the matter of annotation results automatically using similarity
Amount so that mark personnel are possible to the quality of timely learning annotation results, and then is possible to timely correction marking error, can
To effectively improve mark accuracy.
Fig. 2 shows the flow charts of labeled data processing method 200 in accordance with another embodiment of the present invention.It is shown in Fig. 2
Step S110, step S120, step S130 and step S140 are similar to Fig. 1, repeat no more.In the present embodiment, in step
Before S110, labeled data processing method 200 may further include following steps.
Step S102: certain number of annotation results relevant to mark task are obtained.Mark personnel can be by some
Interactive device initiates to request to labeled data processing system, and labeled data processing system can be from the queue to be marked safeguarded
Selection is suitable for the mark task of mark personnel and the mark task is sent to mark personnel.Then, available mark
The annotation results that note personnel obtain after marking.
Step S104: judging whether given number is equal to quantity threshold relevant to mark task, if given number etc.
In quantity threshold, then step S106 is gone to, otherwise goes to step S102.Different mark tasks can correspond to different number thresholds
Value is that is, related to different quantity thresholds.For relatively simple mark task, mark personnel mark mark unit
It is less susceptible to mistake occur when note, therefore less mark personnel can be required to be labeled.Therefore, simply mark task can
With the lesser quantity threshold of correspondence.On the contrary, for complicated mark task, when mark personnel are labeled mark unit
The probability for mistake occur may be larger, therefore more mark personnel can be required to be labeled.Therefore, complicated mark task
Biggish quantity threshold can be corresponded to.For new mark task, initial quantity threshold can be smaller, and for example, 2, then may be used
To be adjusted according to actual needs to quantity threshold.For a mark task, whenever mark personnel provide a mark
Note is as a result, can store the annotation results.It may determine that whether the number of stored annotation results reaches number
Threshold value, the annotation results for being not up to quantity threshold do not execute step S110, but re-execute the steps S102, for reaching
The annotation results of quantity threshold start to execute step S110.
Step S106: certain number of annotation results are determined as multiple annotation results and go to step S110.Institute as above
It states, when the number of annotation results reaches quantity threshold, i.e., reaches institute when receiving and participate in the mark personnel of same mark task
When needing number, it can start to execute step S110.Furthermore it is also possible to maintenance quality detects queue, it can be by the number of annotation results
The mark task that mesh is equal to quantity threshold removes out queue to be marked and quality testing queue is added, and temporarily no longer marks to other
Personnel send the mark task.
Receiving is controlled above by quantity threshold and the method for participating in the mark personnel of same mark task can be reasonable
Ground avoids extra, meaningless mark, as far as possible using mark manpower so as to effectively improve manpower utilization.
Fig. 3 shows the flow chart of labeled data processing method 300 in accordance with another embodiment of the present invention.It is shown in Fig. 3
Step S102, step S104, step S106, step S110, step S120 and step S130 are similar to Fig. 1, repeat no more.At this
In embodiment, step S140 may include step S1402 and step S1404.Step S1402 is the multiple marks of above-mentioned determination
As a result do not pass through quality inspection steps.In addition to step S1402, step S140 may further include step S1404, that is, increase
Quantity threshold simultaneously goes to step S102.
When determining that multiple annotation results do not pass through quality testing, quantity threshold can be increased, and obtain more marks
As a result for carrying out quality testing.The number of annotation results increases, so that the similarity of all annotation results is possible to increase, it is whole
The quality of body is possible to improve.Depending on the amount that quantity threshold increases every time can according to need, the present invention limits not to this.
It is appreciated that mark task can be removed to mass detection when increasing quantity threshold and obtaining new annotation results
Queue simultaneously rejoins queue to be marked.Meanwhile new mark personnel can be introduced, wherein each mark personnel only mark one
It is secondary.Further, it is also possible to retain last any one or more of the multiple annotation results for carrying out quality testing, or can be with
All using new annotation results.For new mark task, quantity threshold can initially be set to compared with
It is small, for example, 2.For relatively simple mark task, it is possible to no need to increase quantity threshold or only increase fewer number mesh threshold
Value can pass through quality testing.Therefore it for relatively simple mark task, may be carried out eventually by when quality testing
Labeled times it is less, so as to reduce the repeat mark for being directed to simple mark task to the greatest extent, save mark manpower.And it is right
In relative complex mark task, quantity threshold can be increased always, increase the number of the annotation results of acquisition until passing through matter
Amount detection improves mark accuracy in this way, the final mass of the annotation results of complicated mark task can be improved.Therefore, according to
The method of this adjust automatically quantity threshold provided by the invention, can be as far as possible efficiently using mark manpower, while can lead to
Reasonable more people's repeat marks are crossed, achieve the purpose that promote mark quality.This is unlike the prior art.It is marked in existing data
In method, in order to promote mark accuracy, it usually needs the labeled times being multiplied to the same mark task.However, every
A mark task, which is appropriate for how many times mark, can not determine and can not also be reasonably adjusted, can only be according to theoretical or experience
Estimated.Therefore, this is likely to result in for the inappropriate labeled times of certain mark task choosings, to waste people
Power reduces mark accuracy.
Optionally, step S140 may further include: multiple annotation results be sent, to be checked by inspection personnel；
Check feedback information；And send and check feedback information, to inform mark personnel.Multiple annotation results similarity compared with
It is small, so that multiple annotation results can be sent to inspection personnel in the case that annotation results do not pass through quality testing and examined
It looks into.Inspection personnel can provide after checking annotation results and check feedback information (i.e. inspection opinion).For example,
In the mark task being labeled to face, it is possible to which the mark personnel having mark face, some marks in the larger context
Personnel mark face in lesser range, and inspection personnel may indicate that the mark of which mark personnel more meets the requirements and points out
Suitably mark range should be much, and such information is exactly to check feedback information.It is appreciated that inspection personnel ties mark
The process that fruit is checked can also be considered as an annotation process, wherein mark unit is annotation results, and mark requirement can be with
For, such as, it is indicated that the unreasonable place of annotation results.Therefore, inspection personnel can also be considered as mark personnel.Interaction can be passed through
Device will check that feedback information feeds back to labeled data processing system, and labeled data processing system will check that feedback information is fed back to
All mark personnel for participating in the mark task, to be referred to by mark personnel, so that it is more acurrate to instruct mark personnel to provide
Annotation results.
Optionally, step S130 may further include: be averaged to multiple annotation results, to obtain and mark task
Relevant average annotation results.Average annotation results are for spot-check.When multiple annotation results pass through quality testing, can determine
Average annotation results, and average annotation results are stored be used for after spot-check by selective examination personnel.It is understood that
Labeled data processing system can equally safeguard selective examination queue, can will remove out above-mentioned matter by the mark task of quality testing
Amount detection queue simultaneously stores, and selected section mark task can be spot-check from the mark task stored later.
The method of determination of average annotation results is including but not limited to following several:
If the centre of the corresponding points in multiple annotation results can be calculated if mark task is that mark is done on the image
Point is as average annotation results；
If mark task can calculate separately multiple marks to mark several points on every frame image in one section of video
As a result the intermediate point of the corresponding points in correspondence image in is as average annotation results；
If mark task is to mark several polygons on the image, the correspondence that can be calculated in multiple annotation results is polygon
The intermediate point of the intersection of shape or the corresponding points on corresponding polygon is as average annotation results, it should be noted that for passing through
If doing on labelling polygons is come for the mark task of labelling polygons, the mark sequence of different labeled personnel may be different
It causes (such as the mark personnel having mark clockwise, and some mark personnel mark counterclockwise), in such a case, it is possible to press first
The point on polygon that each mark personnel mark out is mapped according to the mark sequence of each mark personnel, then calculates average mark
Infuse result；
If the task of mark is to select an option in multiple options, it is all only that multiple annotation results should be all consistent
One the correct option selects the option as average annotation results；
If mark task is to select more than one option in multiple options, can choose in multiple annotation results
All options in public option or multiple annotation results are as average annotation results, it is to be understood that
It selects public option the accuracy of final annotation results can be made higher as average annotation results, selects multiple marks
As a result all options in keep the accuracy of final annotation results lower as average annotation results, can according to need choosing
Select suitable scheme；
If the task of mark is one section of specific text of input, all texts for including in multiple annotation results can be made
For average annotation results；
If mark task is the range of age for marking the personage in image, the age in multiple annotation results can be calculated
The common range (i.e. the intersection of the range of age) or total range (i.e. the union of the range of age) of range are as average annotation results；
If mark task is facial orientation, number of people orientation or the angle etc. marked in image, multiple marks can be calculated
As a result the average value of facial orientation, number of people orientation or angle etc. in is as average annotation results.
Optionally, labeled data processing method may further include: selection mark task from mark set of tasks
Set；Average annotation results relevant to each mark task in mark task subclass are sent, to be carried out by selective examination personnel
Selective examination；Receive selective examination feedback information；And determine whether mark set of tasks passes through selective examination based on selective examination feedback information.When multiple
When annotation results pass through quality testing, the average annotation results of multiple annotation results and corresponding mark task can be deposited
Storage is got up.Then, a collection of mark task can be selected to combine from all mark tasks stored, mark is formed and appoint
Business set.It can will mark all mark tasks in set of tasks and selective examination queue is added.Then selected from mark set of tasks
Mark task subclass is selected for spot-check.Depending on the selection mode of mark task subclass can according to need, the present invention is not right
This is limited.For example, can be sub as mark task from a certain proportion of mark task of random sampling in set of tasks that marks
Set.The ratio can be preset, such as be set as 10%~50% etc., due to spot-check need to expend time of selective examination personnel at
This, therefore the ratio that can be sampled determine according to actual needs.Then, by with mark task subclass in each mark task
Relevant average annotation results are sent to selective examination personnel and carry out manual examination and verification.By selective examination, it may further determine that and appoint with mark
Whether relevant average annotation results of being engaged in are qualified, to further increase mark accuracy.
Optionally, the mark task marked in set of tasks is that marking types are identical and label time is in preset period of time
Mark task.The identical mark requirement referred in mark task of marking types be it is identical, it is different only to mark unit.Example
Such as, for face mark, if the facial image in different labeled task is different, but mark requires identical, example
Such as it is required to mark 20 key points on face, it may be considered that the mark that these mark tasks belong to same marking types is appointed
Business.Further for example, mark requires to be the personage marked out in image if mark unit is the different image comprising personage
The range of age, then such mark task also belongs to the mark task of same marking types.Label time, which refers to, receives mark
As a result time, that is, mark personnel provide the time of its annotation results.Since practical mark situation may be with the time
Variation, may be unstable, therefore label time can relatively should compare with reference value.
Optionally it is determined that it includes: to obtain selective examination based on selective examination feedback information to pass through that whether mark set of tasks, which passes through selective examination,
Rate；And selective examination percent of pass compares with percent of pass threshold value, if selective examination percent of pass is greater than or equal to percent of pass threshold value, really
Calibration note set of tasks is by selective examination, if selective examination percent of pass is less than percent of pass threshold value, it is determined that mark set of tasks does not pass through
Selective examination.After determining whether mark set of tasks passes through selective examination, labeled data processing method be may further include: if mark
Set of tasks is infused by selective examination, then is determined based on selective examination feedback information relevant to each mark task in mark set of tasks
Final annotation results；And if mark set of tasks does not pass through selective examination, increases similarity threshold and go to step S120.
Selective examination feedback information is the information that selective examination personnel provide, and may include the letter of the value of direct instruction selective examination percent of pass
Breath, such as point out the selective examination percent of pass of certain selective examination is how many.Selective examination feedback information can also include pointing out to receive the every of selective examination
The information how a average annotation results correct with the presence or absence of mistake and/or mistake.It later, can be all according to receiving to spot-check
The correct and error situations of average annotation results calculates selective examination percent of pass.Percent of pass threshold value can be any suitable value, example
As percent of pass threshold value can be greater than or equal to 90% and be less than or equal to 99%.If spot-check percent of pass is less than percent of pass threshold
Value, then the entire mark set of tasks of explanation does not pass through selective examination, can be by all mark tasks in the mark set of tasks from pumping
It looks into queue and removes and rejoin quality testing queue, and increase the similarity threshold of these mark tasks, so as to this
The requirement of the mark accuracy of a little mark tasks becomes higher.If spot-check percent of pass is greater than or equal to percent of pass threshold value, say
Bright entire mark set of tasks passes through selective examination, it is believed that all mark tasks in the mark set of tasks are completed.Then,
Final annotation results can be determined based on selective examination feedback information.For example, if not finding mistake in sampling procedure, it can be direct
Using average annotation results relevant to each mark task in mark set of tasks as final annotation results, if spot-check
Mistake is found in journey, then obtains final annotation results after the error label that can be found in filtering out selective examination.
Optionally, before step S110, labeled data processing method be may further include: be received and mark personnel's phase
The identification information of pass；Mark task is selected from queue to be marked based on identification information, mark task is corresponding with mark personnel；
And mark task is sent, to provide annotation results relevant to mark task by mark personnel；And it receives mark personnel and mentions
The annotation results of confession are as one of multiple annotation results.As described above, queue to be marked can be safeguarded comprising several marks
Task.Mark task can be sent to mark people by interactive device, such as user's interactive interface by labeled data processing system
Member.The interaction of labeled data processing system and mark personnel can be realized further using such as application program (APP).Mark people
Member can open the APP, input its identification information.Identification information can be any identity that can be used in identifying mark personnel
Information, such as the account name and the password that mark personnel etc..Labeled data processing system can be based on identification information identification mark
The identity of personnel, and then it is sent to it the mark task for being suitble to it.For example, the mark task that mark personnel have received and completed
Mark personnel will be never sent to.Further, it is to be appreciated that mark personnel can also actively initiate to request, such as request
Wish that the type of the mark task participated in, labeled data processing system can be suitble to mark people according to its request selecting and transmission
The mark task of member.
As described above, the process that inspection personnel checks annotation results can also be considered as an annotation process,
Inspection personnel can also be considered as mark personnel.Therefore, optionally, above-mentioned mark personnel and inspection personnel can be same lineup.
That is, same people either mark personnel be also possible to inspection personnel, can according to need flexible conversion.In this way may be used
To prevent mark personnel and the unmatched problem of inspection personnel's ratio, examined so as to avoid mark personnel very busy
It looks into the very idle state of personnel or marks that personnel are very idle and state that inspection personnel is very busy.In the implementation
Each step and the step in labeled data processing method described above are almost the same, but when carrying out quality testing, need to infuse
Meaning is labeled for the different people of same mark task choosing and quality testing, that is, prevents the mark personnel of same mark task
It is same people with inspection personnel.
According to a further aspect of the invention, a kind of labeled data processing system is provided.Fig. 4 shows an implementation according to the present invention
The schematic block diagram of the labeled data processing system 400 of example.As shown in figure 4, labeled data processing system 400 includes computing device
410, similarity-rough set device 420, the first executive device 430 and the second executive device 440.
Computing device 410 is used to calculate the similarity of multiple annotation results relevant to the task that marks.Similarity-rough set dress
420 are set for similarity to compare with similarity threshold, if similarity is greater than or equal to similarity threshold, starts first
Executive device 430 starts the second executive device 440 if similarity is less than similarity threshold.First executive device 430 is used
In determining that multiple annotation results pass through quality testing.Second executive device 440 is for determining that multiple annotation results do not pass through quality
Detection.
Appointing in computing device 410, similarity-rough set device 420, the first executive device 430 and the second executive device 440
What one or more can be realized using any suitable hardware, software and/or firmware, such as pass through specific integrated circuit
(ASIC), field programmable gate array (FPGA), Digital Signal Processing (DSP) etc. are realized.Computing device 410, similarity-rough set
Any one or more of device 420, the first executive device 430 and second executive device 440 can be handled with labeled data
Other devices in system 400 are integrated or are realized using individual device.Computing device 410 and similarity-rough set dress
Setting can be connected between 420 by the way of direct or indirect, such as be connected by wired or wireless way.Similarity-rough set dress
Setting 420 and first can also be connected by the way of direct or indirect between executive device 430 or the second executive device 440, example
Such as connected by wired or wireless way.
The labeled data processing system provided according to the present invention, due to detecting the matter of annotation results automatically using similarity
Amount so that mark personnel are possible to the quality of timely learning annotation results, and then is possible to timely correction marking error, can
To effectively improve mark accuracy.
Optionally, it is true to may further include acquisition device, judgment means and annotation results for labeled data processing system 400
Determine device (not shown).Acquisition device is for obtaining certain number of annotation results relevant to the task that marks.Judgment means are used
In judging whether given number is equal to quantity threshold relevant to mark task, if given number is equal to quantity threshold, open
Dynamic annotation results determining device, otherwise starts acquisition device.Annotation results determining device is used for certain number of annotation results
It is determined as multiple annotation results and starts computing device.It is executed with above-mentioned computing device 410, similarity-rough set device 420, first
Device 430 and the second executive device 440 similarly, any one in acquisition device, judgment means and annotation results determining device
It is a or multiple can be realized using any suitable hardware, software and/or firmware.
Optionally, the second executive device 440 can be further used for increasing quantity threshold and start acquisition device.Work as determination
When multiple annotation results do not pass through quality testing, quantity threshold can be increased, and obtain more annotation results for carrying out matter
Amount detection.It is understood that annotation results are being manually checked and are being provided by inspection personnel due to as described above
During checking feedback information, the requirement to the inspection precision of inspection personnel is not too much high, and quasi- to the mark of mark personnel
The requirement of true property is some higher, therefore can be omitted manual inspection, by way of increasing quantity threshold, is directly examined using quality
It surveys to guarantee to mark quality.Differing biggish mark task for multiple annotation results can require to increase quantity threshold, that is, increase
It marks number.The more similar mark task of consistent or most of annotation results obvious for multiple annotation results, can
To be directly averaged to obtain average annotation results, as described above to multiple annotation results.In this way, can be efficient as far as possible
Using mark manpower, while can achieve the purpose that promote mark quality by reasonable more people's repeat marks.
Optionally, the second executive device 440 may further include annotation results sending module, check feedback reception module
Sending module is fed back with checking.Annotation results sending module is for sending multiple annotation results, to be checked by inspection personnel.
Check feedback reception module for checking feedback information.Check that feedback sending module checks feedback information for sending, with
Inform mark personnel.Multiple annotation results similarity it is smaller so that annotation results do not pass through quality testing in the case where, can
It is checked so that multiple annotation results are sent to inspection personnel.Inspection personnel, can be with after checking annotation results
It provides and checks feedback information.Being manually checked by inspection personnel can be with the side of the quality using similarity detection annotation results
Formula combines, to further increase the accuracy of annotation results.
Optionally, the first executive device 430 can be further used for being averaged to multiple annotation results, to obtain and mark
The relevant average annotation results of note task.Average annotation results can be used for spot-check.When multiple annotation results pass through quality testing
When, can determine average annotation results, and average annotation results are stored be used for after spot-check by selective examination personnel.On
Text has illustrated the method for determination of average annotation results, and details are not described herein.
Optionally, labeled data processing system 400 may further include subclass selection device, the first sending device,
First receiving device and selective examination pass through determining device (not shown).Subclass selection device is used to from mark set of tasks select
Mark task subclass.First sending device is relevant average to each mark task marked in task subclass for sending
Annotation results, to be spot-check by selective examination personnel.First receiving device, for receiving selective examination feedback information.Selective examination passes through determination
Device, for determining whether mark set of tasks passes through selective examination based on selective examination feedback information.It is appreciated that labeled data processing system
System 400 may include storage device (not shown), for storing the mark task for passing through quality testing.It can be from the institute stored
Have and a collection of mark task is selected to combine in mark task, forms mark set of tasks.It then can be from mark task-set
It is selected in conjunction, such as random sampling a part mark task is used to spot-check as mark task subclass.Pass through selective examination, Ke Yijin
One step determines whether average annotation results relevant to mark task are qualified, to further increase mark accuracy.
Optionally, the mark task marked in set of tasks is that marking types are identical and label time is in preset period of time
Mark task.Marking types are identical and the relatively more similar mark task of label time between can compare to have and refer to mutually
Value and significance.
Optionally, selective examination may include that percent of pass obtains module and percent of pass comparison module by determining device.Percent of pass
Module is obtained to be used to obtain selective examination percent of pass based on selective examination feedback information.Percent of pass comparison module will be for that will spot-check percent of pass and lead to
The rate threshold value of mistake compares, if selective examination percent of pass is greater than or equal to percent of pass threshold value, it is determined that and mark set of tasks passes through selective examination,
If spot-check percent of pass is less than percent of pass threshold value, it is determined that mark set of tasks does not pass through selective examination.Labeled data processing system 400
It may further include final annotation results determining device and similarity aggrandizement apparatus (not shown).Final annotation results determine dress
It sets, if being determined based on selective examination feedback information by selective examination for mark set of tasks and marking each of set of tasks
The relevant final annotation results of mark task.Similarity aggrandizement apparatus increases if not passing through selective examination for marking set of tasks
Big similarity threshold simultaneously starts similarity-rough set device.
Selective examination percent of pass can directly be provided by selective examination personnel or the correct and mistake according to selective examination personnel to annotation results
To determine, the present invention limits not to this for evaluation.It can determine whether mark set of tasks passes through pumping by spot-check percent of pass
It looks into, i.e., whether average annotation results relevant to each mark task in mark set of tasks are qualified or say and meet the requirements, and
And then it can choose and determine that final annotation results or selection increase similarity threshold and re-start quality testing.
Optionally, labeled data processing system 400 may further include the second reception device, mark task choosing dress
It sets, the second sending device and third reception device (not shown).Second reception device is for receiving mark relevant to the personnel that mark
Know information.Mark task choosing device is used to select mark task from queue to be marked based on identification information, mark task and
Mark personnel are corresponding.Second sending device is relevant to mark task to be provided by mark personnel for sending mark task
Annotation results.Third reception device is used to receive the annotation results of mark personnel offer as one of multiple annotation results.As above
Described in text, labeled data processing system 400 can be suitble to the mark of mark personnel according to the identification information selection of mark personnel
Mark task is simultaneously sent to mark personnel by task, is no longer repeated herein it.
The implementation that Fig. 1 to Fig. 3 describes each step of labeled data processing method provided by the invention is had been combined above
Mode and its advantage etc., those of ordinary skill in the art pass through the detailed description read above for labeled data processing method,
It will be appreciated that structure, the method for operation and the advantage of above-mentioned labeled data processing system 400, therefore which is not described herein again.
Method and apparatus are not inherently related to any particular computer, virtual system, or other device provided herein.
Various general-purpose systems can also be used together with teachings based herein.As described above, it constructs required by this kind of system
Structure be obvious.In addition, the present invention is also not directed to any particular programming language.It should be understood that can use various
Programming language realizes summary of the invention described herein, and the description done above to language-specific is to disclose this hair
Bright preferred forms.
In the instructions provided here, numerous specific details are set forth.It is to be appreciated, however, that implementation of the invention
Example can be practiced without these specific details.In some instances, well known method, structure is not been shown in detail
And technology, so as not to obscure the understanding of this specification.
Similarly, it should be understood that in order to simplify the disclosure and help to understand one or more of the various inventive aspects,
Above in the description of exemplary embodiment of the present invention, each feature of the invention is grouped together into single implementation sometimes
In example, figure or descriptions thereof.However, the disclosed method should not be interpreted as reflecting the following intention: i.e. required to protect
Shield the present invention claims features more more than feature expressly recited in each claim.More precisely, as following
Claims reflect as, inventive aspect is all features less than single embodiment disclosed above.Therefore,
Thus the claims for following specific embodiment are expressly incorporated in the specific embodiment, wherein each claim itself
All as a separate embodiment of the present invention.
Those skilled in the art will understand that in addition at least one in such feature and/or process or unit
It, can be using any combination in this specification (including the accompanying claims, abstract and drawings) except excluding each other
Disclosed all features and so disclosed any method or all process or units of device are combined.Unless in addition
It is expressly recited, each feature disclosed in this specification (including adjoint claim, abstract and attached drawing) can be by offer phase
The alternative features of same, equivalent or similar purpose replace.
In addition, it will be appreciated by those of skill in the art that although some embodiments described herein include other embodiments
In included certain features rather than other feature, but the combination of the feature of different embodiments mean it is of the invention
Within the scope of and form different embodiments.For example, in the following claims, embodiment claimed is appointed
Meaning one of can in any combination mode come using.
Various component embodiments of the invention can be implemented in hardware, or to run on one or more processors
Software module realize, or be implemented in a combination thereof.It will be understood by those of skill in the art that can be used in practice
Microprocessor or digital signal processor (DSP) realize one in labeled data processing system according to an embodiment of the present invention
The some or all functions of a little modules.The present invention be also implemented as a part for executing method as described herein or
The program of device (for example, computer program and computer program product) of person's whole.It is such to realize that program of the invention be with
It may be stored on the computer-readable medium, or may be in the form of one or more signals.Such signal can from because
It downloads and obtains on spy's net website, be perhaps provided on the carrier signal or be provided in any other form.
It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and ability
Field technique personnel can be designed alternative embodiment without departing from the scope of the appended claims.In the claims,
Any reference symbol between parentheses should not be configured to limitations on claims.Word "comprising" does not exclude the presence of not
Element or step listed in the claims.Word "a" or "an" located in front of the element does not exclude the presence of multiple such
Element.The present invention can be by means of including the hardware of several different elements and being come by means of properly programmed computer real
It is existing.In the unit claims listing several devices, several in these devices can be through the same hardware branch
To embody.The use of word first, second, and third does not indicate any sequence.These words can be explained and be run after fame
Claim.
Claims (16)
1. a kind of labeled data processing method, comprising:
Step S110: the similarity of multiple annotation results relevant to mark task is calculated；
Step S120: the similarity is compared with similarity threshold, if the similarity is more than or equal to described similar
Threshold value is spent, then goes to step S130, if the similarity is less than the similarity threshold, goes to step S140；
Step S130: determine that the multiple annotation results pass through quality testing；And
Step S140: determine that the multiple annotation results do not pass through quality testing；
Wherein, before the step S110, the labeled data processing method further comprises:
Step S102: certain number of annotation results relevant to the mark task are obtained；
Step S104: judging whether the given number is equal to quantity threshold relevant to the mark task, if the spy
Fixed number mesh is equal to the quantity threshold, then goes to step S106, otherwise go to step S102；And
Step S106: the certain number of annotation results are determined as the multiple annotation results and go to the step
S110。
2. labeled data processing method as described in claim 1, wherein the step S140 further comprises: described in increase
Quantity threshold simultaneously goes to the step S102.
3. labeled data processing method as claimed in claim 1 or 2, wherein
The step S140 further comprises:
The multiple annotation results are sent, to be checked by inspection personnel；
Check feedback information；And
The inspection feedback information is sent, to inform mark personnel.
4. labeled data processing method as described in claim 1, wherein the step S130 further comprises: to described more
A annotation results are averaged, to obtain average annotation results relevant to the mark task；
Wherein, the average annotation results are for spot-check.
5. labeled data processing method as claimed in claim 4, wherein the labeled data processing method further comprises:
Mark task subclass is selected from mark set of tasks；
Send average annotation results relevant to each mark task in the mark task subclass, with by spot-check personnel into
Row selective examination；
Receive selective examination feedback information；And
Determine whether the mark set of tasks passes through selective examination based on the selective examination feedback information.
6. labeled data processing method as claimed in claim 5, wherein
Whether the determination mark set of tasks, which passes through selective examination, includes:
Selective examination percent of pass is obtained based on the selective examination feedback information；And
The selective examination percent of pass is compared with percent of pass threshold value, if the selective examination percent of pass is greater than or equal to the percent of pass
Threshold value, it is determined that the mark set of tasks is by selective examination, if the selective examination percent of pass is less than the percent of pass threshold value, really
The fixed mark set of tasks does not pass through selective examination；
After whether the determination mark set of tasks passes through selective examination, the labeled data processing method is further wrapped
It includes:
If the mark set of tasks is by selective examination, based on selective examination feedback information determination and the mark set of tasks
In the relevant final annotation results of each mark task；And
If the mark set of tasks does not pass through selective examination, increases the similarity threshold and go to the step S120.
7. such as labeled data processing method described in claim 5 or 6, wherein the mark task in the mark set of tasks
It is the mark task that marking types are identical and label time is in preset period of time.
8. labeled data processing method as described in claim 1, wherein before the step S110, the labeled data
Processing method further comprises:
Receive identification information relevant to mark personnel；
The mark task, the mark task and the mark personnel are selected from queue to be marked based on the identification information
It is corresponding；
The mark task is sent, to provide annotation results relevant to the mark task by the mark personnel；And
The annotation results of the mark personnel offer are received as one of the multiple annotation results.
9. a kind of labeled data processing system, including computing device, similarity-rough set device, the first executive device and second execute
Device,
The computing device is used to calculate the similarity of multiple annotation results relevant to the task that marks；
The similarity-rough set device for the similarity to be compared with similarity threshold, if the similarity be greater than or
Equal to the similarity threshold, then start the first executive device, if the similarity is less than the similarity threshold, starts
Second executive device；
First executive device is for determining that the multiple annotation results pass through quality testing；And
Second executive device is for determining that the multiple annotation results do not pass through quality testing；
Wherein, the labeled data processing system further comprises acquisition device, judgment means and annotation results determining device:
The acquisition device is for obtaining certain number of annotation results relevant to the mark task；
The judgment means for judging whether the given number is equal to quantity threshold relevant to the mark task, if
The given number is equal to the quantity threshold, then starts the annotation results determining device, otherwise start the acquisition device；
And
The annotation results determining device is used to the certain number of annotation results being determined as the multiple annotation results simultaneously
Start the computing device.
10. labeled data processing system as claimed in claim 9, wherein second executive device is further used for increasing
The quantity threshold simultaneously starts the acquisition device.
11. the labeled data processing system as described in claim 9 or 10, wherein second executive device further comprises:
Annotation results sending module, for sending the multiple annotation results, to be checked by inspection personnel；
Feedback reception module is checked, for checking feedback information；And
Feedback sending module is checked, for sending the inspection feedback information, to inform mark personnel.
12. labeled data processing system as claimed in claim 9, wherein first executive device is further used for institute
It states multiple annotation results to be averaged, to obtain average annotation results relevant to the mark task；
Wherein, the average annotation results are for spot-check.
13. labeled data processing system as claimed in claim 12, wherein the labeled data processing system is further wrapped
It includes:
Subclass selection device, for selecting mark task subclass from mark set of tasks；
First sending device, for sending average mark knot relevant to each mark task in the mark task subclass
Fruit, to be spot-check by selective examination personnel；
First receiving device, for receiving selective examination feedback information；And
Selective examination is by determining device, for determining whether the mark set of tasks passes through pumping based on the selective examination feedback information
It looks into.
14. labeled data processing system as claimed in claim 13, wherein
The selective examination by determining device includes:
Percent of pass obtains module, for obtaining selective examination percent of pass based on the selective examination feedback information；And
Percent of pass comparison module, for the selective examination percent of pass to compare with percent of pass threshold value, if the selective examination percent of pass
More than or equal to the percent of pass threshold value, it is determined that the mark set of tasks is by selective examination, if the selective examination percent of pass is small
In the percent of pass threshold value, it is determined that the mark set of tasks does not pass through selective examination；
The labeled data processing system further comprises:
Final annotation results determining device, if fed back by selective examination based on the selective examination for the mark set of tasks
The determining final annotation results relevant to each mark task in the mark set of tasks of information；And
Similarity aggrandizement apparatus increases the similarity threshold simultaneously if not passing through selective examination for the mark set of tasks
Start the similarity-rough set device.
15. labeled data processing system according to claim 13 or 14, wherein the mark in the mark set of tasks is appointed
Business is the mark task that marking types are identical and label time is in preset period of time.
16. labeled data processing system as claimed in claim 9, wherein the labeled data processing system further comprises:
Second reception device, for receiving identification information relevant to the personnel that mark；
Task choosing device is marked, it is described for selecting the mark task from queue to be marked based on the identification information
Mark task is corresponding with the mark personnel；
Second sending device is related to the mark task to be provided by the mark personnel for sending the mark task
Annotation results；And
Third reception device, for receiving the annotation results of the mark personnel offer as one of the multiple annotation results.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510744484.5A CN105404896B (en) | 2015-11-03 | 2015-11-03 | Labeled data processing method and labeled data processing system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510744484.5A CN105404896B (en) | 2015-11-03 | 2015-11-03 | Labeled data processing method and labeled data processing system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105404896A CN105404896A (en) | 2016-03-16 |
| CN105404896B true CN105404896B (en) | 2019-04-19 |
Family
ID=55470371
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510744484.5A Active CN105404896B (en) | 2015-11-03 | 2015-11-03 | Labeled data processing method and labeled data processing system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105404896B (en) |
Families Citing this family (28)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105975980B (en) * | 2016-04-27 | 2019-04-05 | 百度在线网络技术（北京）有限公司 | The method and apparatus of monitoring image mark quality |
| WO2018000269A1 (en) * | 2016-06-29 | 2018-01-04 | 深圳狗尾草智能科技有限公司 | Data annotation method and system based on data mining and crowdsourcing |
| CN108229772A (en) * | 2016-12-14 | 2018-06-29 | 北京国双科技有限公司 | Mark processing method and processing device |
| CN108537240A (en) * | 2017-03-01 | 2018-09-14 | 华东师范大学 | Commodity image semanteme marking method based on domain body |
| CN108197658B (en) * | 2018-01-11 | 2020-08-14 | 阿里巴巴集团控股有限公司 | Image annotation information processing method, device, server and system |
| CN108154197B (en) * | 2018-01-22 | 2022-03-15 | 腾讯科技（深圳）有限公司 | Method and device for realizing image annotation verification in virtual scene |
| CN108364018A (en) * | 2018-01-25 | 2018-08-03 | 北京墨丘科技有限公司 | A kind of guard method of labeled data, terminal device and system |
| CN108446695B (en) * | 2018-02-06 | 2022-02-11 | 创新先进技术有限公司 | Method and device for data annotation and electronic equipment |
| CN110400029A (en) * | 2018-04-24 | 2019-11-01 | 北京京东尚科信息技术有限公司 | A kind of method and system of mark management |
| CN109034188B (en) * | 2018-06-15 | 2021-11-05 | 北京金山云网络技术有限公司 | Method and device for acquiring machine learning model, equipment and storage medium |
| CN108960297B (en) * | 2018-06-15 | 2021-07-30 | 北京金山云网络技术有限公司 | Picture labeling method, labeling device, equipment and storage medium |
| CN109214343B (en) * | 2018-09-14 | 2021-03-09 | 北京字节跳动网络技术有限公司 | Method and device for generating face key point detection model |
| CN109784381A (en) * | 2018-12-27 | 2019-05-21 | 广州华多网络科技有限公司 | Markup information processing method, device and electronic equipment |
| CN111461154A (en) * | 2019-01-22 | 2020-07-28 | 北京京东尚科信息技术有限公司 | Method and device for labeling data |
| CN110188769B (en) * | 2019-05-14 | 2023-09-05 | 广州虎牙信息科技有限公司 | Method, device, equipment and storage medium for auditing key point labels |
| CN110210526A (en) * | 2019-05-14 | 2019-09-06 | 广州虎牙信息科技有限公司 | Predict method, apparatus, equipment and the storage medium of the key point of measurand |
| CN110263934B (en) * | 2019-05-31 | 2021-08-06 | 中国信息通信研究院 | Artificial intelligence data labeling method and device |
| CN112528609A (en) * | 2019-08-29 | 2021-03-19 | 北京声智科技有限公司 | Method, system, equipment and medium for quality inspection of labeled data |
| CN110750523A (en) * | 2019-09-12 | 2020-02-04 | 苏宁云计算有限公司 | Data annotation method, system, computer equipment and storage medium |
| CN110796185B (en) * | 2019-10-17 | 2022-08-26 | 北京爱数智慧科技有限公司 | Method and device for detecting image annotation result |
| CN110880021B (en) * | 2019-11-06 | 2021-03-16 | 创新奇智(北京)科技有限公司 | Model-assisted data annotation system and annotation method |
| CN114036036A (en) * | 2020-07-21 | 2022-02-11 | 南京行者易智能交通科技有限公司 | Automatic test analysis system and method based on automatic labeling and terminal equipment |
| CN111966674B (en) * | 2020-08-25 | 2024-03-15 | 北京金山云网络技术有限公司 | Method and device for judging eligibility of annotation data and electronic equipment |
| CN111986194B (en) * | 2020-09-03 | 2024-07-05 | 深圳平安智慧医健科技有限公司 | Medical annotation image detection method and device, electronic equipment and storage medium |
| CN112084241B (en) * | 2020-09-23 | 2024-10-01 | 北京金山云网络技术有限公司 | Labeling data screening method and device and electronic equipment |
| CN111932536B (en) * | 2020-09-29 | 2021-03-05 | 平安国际智慧城市科技股份有限公司 | Method and device for verifying lesion marking, computer equipment and storage medium |
| CN112347990B (en) * | 2020-11-30 | 2024-02-02 | 重庆空间视创科技有限公司 | Multi-mode-based intelligent manuscript examining system and method |
| WO2023126280A1 (en) | 2021-12-30 | 2023-07-06 | Robert Bosch Gmbh | A system and method for quality check of labelled images |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104795077A (en) * | 2015-03-17 | 2015-07-22 | 北京航空航天大学 | Voice annotation quality consistency detection method |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103324620B (en) * | 2012-03-20 | 2016-04-27 | 北京百度网讯科技有限公司 | A kind of method and apparatus that annotation results is rectified a deviation |
| CN102867025A (en) * | 2012-08-23 | 2013-01-09 | 百度在线网络技术（北京）有限公司 | Method and device for acquiring picture marking data |
| CN103678685A (en) * | 2013-12-26 | 2014-03-26 | 华为技术有限公司 | Webpage labeling method and device |
| CN104573359B (en) * | 2014-12-31 | 2017-08-08 | 浙江大学 | A kind of mass-rent labeled data integration method of task based access control difficulty and mark person's ability |
| CN104615705B (en) * | 2015-01-30 | 2018-09-18 | 百度在线网络技术（北京）有限公司 | Method and device |
-
2015
- 2015-11-03 CN CN201510744484.5A patent/CN105404896B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104795077A (en) * | 2015-03-17 | 2015-07-22 | 北京航空航天大学 | Voice annotation quality consistency detection method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105404896A (en) | 2016-03-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105404896B (en) | Labeled data processing method and labeled data processing system | |
| US20200356818A1 (en) | Logo detection | |
| CN110046546B (en) | Adaptive sight tracking method, device and system and storage medium | |
| CN109800320B (en) | Image processing method, device and computer readable storage medium | |
| EP3843036B1 (en) | Sample labeling method and device, and damage category identification method and device | |
| EP4372698A3 (en) | Systems and methods for image processing | |
| CN108875734A (en) | Liver canceration localization method, device and storage medium | |
| CN111709421B (en) | Bird identification method, bird identification device, computer equipment and storage medium | |
| EP3057303A3 (en) | Information processing system, information processing apparatus, and information processing method | |
| EP2551796A3 (en) | Image processing device identifying attribute of region included in image | |
| CN112101124B (en) | Sitting posture detection method and device | |
| CN106327546B (en) | Method and device for testing face detection algorithm | |
| EP2497545A3 (en) | Information processing program, information processing system, and information processing method | |
| CN112801976B (en) | Detection system and method for correcting binding mispasting of books and periodicals and computer device | |
| EP2557539A3 (en) | Image processing apparatus, image processing method, and image processing program | |
| US12014829B2 (en) | Image processing and presentation techniques for enhanced proctoring sessions | |
| EP2433685A3 (en) | Information processing apparatus capable of associating data with each other, information processing system, and information processing program | |
| WO2020034672A1 (en) | Method and apparatus for determining region of interest of user in image | |
| CN110210314B (en) | Face detection method, device, computer equipment and storage medium | |
| EP2085930A3 (en) | Image processing apparatus and method thereof | |
| CN109598270A (en) | Distort recognition methods and the device, storage medium and processor of text | |
| CN112328822A (en) | Picture pre-labeling method and device and terminal equipment | |
| CN103593643B (en) | A kind of method and system of image recognition | |
| CN110956656A (en) | Spindle positioning method based on depth target detection | |
| CN110971895B (en) | Video jitter detection method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: 100190 Beijing, Haidian District Academy of Sciences, South Road, No. 2, block A, No. 313 Applicant after: MEGVII INC. Applicant after: Beijing maigewei Technology Co., Ltd. Address before: 100190 Beijing, Haidian District Academy of Sciences, South Road, No. 2, block A, No. 313 Applicant before: MEGVII INC. Applicant before: Beijing aperture Science and Technology Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |