CN110245087A - The state detection method and device at the human customer end for sample audit - Google Patents
The state detection method and device at the human customer end for sample audit Download PDFInfo
- Publication number
- CN110245087A CN110245087A CN201910539076.4A CN201910539076A CN110245087A CN 110245087 A CN110245087 A CN 110245087A CN 201910539076 A CN201910539076 A CN 201910539076A CN 110245087 A CN110245087 A CN 110245087A
- Authority
- CN
- China
- Prior art keywords
- sample
- customer end
- human customer
- recognition result
- audit
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
- G06F11/3692—Test management for test results analysis
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Automatic Analysis And Handling Materials Therefor (AREA)
- Medical Treatment And Welfare Office Work (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The present invention provides the state detection methods and device at a kind of human customer end for sample audit, method includes: to obtain a test sample collection, it concentrates each sample to identify the test sample using a preparatory trained identification model, marks out the recognition result of each sample；Choosing the test sample concentrates preset quantity sample to be revised as wrong identification result as target sample, and by the recognition result that sample each in the target sample is marked；By treated, the test sample collection is sent to human customer end, so that the recognition result of each sample is audited at the human customer end；According to the human customer end to the auditing result of the target sample, check whether the human customer end is in abnormality.It can quickly judge whether human customer end is in abnormality using scheme provided by the invention.
Description
Technical field
The present invention relates to field of artificial intelligence more particularly to a kind of states at the human customer end for sample audit
Inspection method, device, electronic equipment and computer readable storage medium.
Background technique
In artificial intelligence field, before carrying out model training, carry out the mark knot to sample usually using human customer end
Fruit is audited, if human customer end is in abnormality when carrying out sample audit, not can guarantee the mark feelings of sample
Whether condition is up to standard, so as to cause training the recognition accuracy of obtained model not up to standard.Therefore, the model in order to obtain training
Accuracy rate it is up to standard, it is desirable that human customer end when carrying out sample audit be in normal condition, this is just needed to human customer end
State checked.
Currently, can be sentenced by the annotation results for whole samples that the artificial client of inspection is audited according to auditing result
Disconnected human customer end is with the presence or absence of exception, however usually the quantity of sample is very big in sample set, thus needs to spend more
Time could judge whether human customer end is abnormal.
Summary of the invention
The purpose of the present invention is to provide it is a kind of for the state detection method at human customer end of sample audit, device,
Electronic equipment and computer readable storage medium, quickly to judge whether human customer end is in abnormality.Particular technique side
Case is as follows:
In a first aspect, the present invention provides a kind of state detection method at human customer end for sample audit, the side
Method includes:
A test sample collection is obtained, each sample is concentrated to the test sample using a preparatory trained identification model
It is identified, marks out the recognition result of each sample；
Choosing the test sample concentrates preset quantity sample as target sample, and will be each in the target sample
The recognition result that sample is marked is revised as wrong identification result；
By treated, the test sample collection is sent to human customer end, so that the human customer end is to each sample
Recognition result audited；
According to the human customer end to the auditing result of the target sample, check whether the human customer end is in
Abnormality.
Optionally, the recognition result of each sample is audited at the human customer end, comprising:
For treated, the test sample concentrates each sample, and the human customer end judges marked identification knot
Whether fruit is correct；If it is not, then the recognition result marked to the sample is modified.
Optionally, it is described according to the human customer end to the auditing result of the target sample, check the artificial visitor
Whether family end is in abnormality, comprising:
For each sample in the target sample, judge the human customer end whether to the wrong identification of the sample
As a result it is modified；
The ratio for being had modified the sample of recognition result by the human customer end in the target sample is obtained, as the
One ratio；
If first ratio is less than preset threshold, determine that the human customer end is in abnormality.
Optionally, the method also includes:
According to the mark accuracy rate at human customer end described in first ratio-dependent.
Optionally, it is described according to the human customer end to the auditing result of the target sample, check the artificial visitor
Whether family end is in abnormality, comprising:
For each sample in the target sample, judge the human customer end whether by the wrong identification of the sample
Results modification is correct recognition result；
The ratio for being revised as the sample of correct recognition result by the human customer end in the target sample is obtained, is made
For the second ratio；
If second ratio is less than preset threshold, determine that the human customer end is in abnormality.
Optionally, the method also includes:
According to the mark accuracy rate at human customer end described in second ratio-dependent.
Optionally, the minimum value X of the preset threshold is determined according to following formula: 1- (1-X)2=Q；
Wherein, Q indicates the pre-set test after identification model mark and human customer end audit
The target accuracy rate that sample marks in sample set.
Optionally, the preset quantity is more than or equal to smallest sample extraction quantity N；
Wherein, N=Z2×(P×(1-P))/E2；Z indicates that statistic relevant to confidence level, confidence level are equal to the identification
The recognition accuracy of model；E indicates preset sampling error value；P indicates the recognition accuracy of the identification model.
Optionally, the recognition result that sample each in the target sample is marked is revised as wrong identification knot
Fruit, comprising:
The recognition result that sample each in the target sample is marked is revised as the knowledge different from original recognition result
Other result.
Second aspect, it is described the present invention also provides a kind of state inspection apparatus at human customer end for sample audit
Device includes:
Labeling module, for obtaining a test sample collection, using a preparatory trained identification model to the test specimens
The each sample of this concentration is identified, the recognition result of each sample is marked out；
Modified module concentrates preset quantity sample as target sample for choosing the test sample, and will be described
The recognition result that each sample is marked in target sample is revised as wrong identification result；
Auditing module, for will treated that the test sample collection is sent to human customer end, so as to the artificial visitor
Family end is audited by the recognition result of each sample；
It checks module, for the auditing result according to the human customer end to the target sample, checks described artificial
Whether client is in abnormality.
Optionally, the recognition result of each sample is audited at human customer end in the auditing module, comprising:
For treated, the test sample concentrates each sample, and the human customer end judges marked identification knot
Whether fruit is correct；If it is not, then the recognition result marked to the sample is modified.
Optionally, the inspection module, is specifically used for:
For each sample in the target sample, judge the human customer end whether to the wrong identification of the sample
As a result it is modified；Obtain the ratio for being had modified the sample of recognition result by the human customer end in the target sample
Example, as the first ratio；If first ratio is less than preset threshold, determine that the human customer end is in abnormality.
Optionally, described device further include:
First determining module, the mark accuracy rate for the human customer end according to first ratio-dependent.
Optionally, the inspection module, is specifically used for:
For each sample in the target sample, judge the human customer end whether by the wrong identification of the sample
Results modification is correct recognition result；Acquisition is revised as correct recognition result by the human customer end in the target sample
Sample ratio, as the second ratio；If second ratio is less than preset threshold, determine that the human customer end is in
Abnormality.
Optionally, described device further include:
Second determining module, the mark accuracy rate for the human customer end according to second ratio-dependent.
Optionally, the minimum value X of the preset threshold is determined according to following formula: 1- (1-X)2=Q；
Wherein, Q indicates the pre-set test after identification model mark and human customer end audit
The target accuracy rate that sample marks in sample set.
Optionally, the preset quantity is more than or equal to smallest sample extraction quantity N；
Wherein, N=Z2×(P×(1-P))/E2；Z indicates that statistic relevant to confidence level, confidence level are equal to the identification
The recognition accuracy of model；E indicates preset sampling error value；P indicates the recognition accuracy of the identification model.
Optionally, the recognition result that sample each in the target sample is marked is revised as mistake by the modified module
Recognition result, comprising:
The recognition result that sample each in the target sample is marked is revised as the knowledge different from original recognition result
Other result.
The third aspect, the present invention also provides a kind of electronic equipment, including processor, communication interface, memory and communication
Bus, wherein the processor, the communication interface, the memory complete mutual communication by the communication bus；
The memory, for storing computer program；
The processor when for executing the program stored on the memory, is realized described in above-mentioned first aspect
The step of state detection method at the human customer end for sample audit.
Fourth aspect, the present invention also provides a kind of computer readable storage medium, the computer readable storage medium
Inside it is stored with computer program, the computer program realizes that above-mentioned first aspect is stated when being executed by processor described for sample
The step of state detection method at the human customer end of this audit.
Compared with prior art, the present invention is identified using the sample that identification model concentrates test sample, is marked out
Recognition result, and concentrate the recognition result of a part of sample to be deliberately revised as wrong identification as a result, by after processing test sample
Test sample collection issue human customer end and audit, only need to check so artificial client to deliberately mislabel that
Divide auditing result, that is, deducibility human customer end of sample to the mark accuracy rate of entire test sample collection, and then judges artificial visitor
Whether family end is in abnormality, and mark is determined to the audit situation of entire test sample collection without counting human customer end
Accuracy rate is infused, quickly determines whether human customer end is in abnormality to realize, and shorten statistical time, reduces
Expense cost.
Detailed description of the invention
In order to more clearly explain the embodiment of the invention or the technical proposal in the existing technology, to embodiment or will show below
There is attached drawing needed in technical description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this
Some embodiments of invention for those of ordinary skill in the art without creative efforts, can be with
It obtains other drawings based on these drawings.
Fig. 1 is the process of the state detection method at the human customer end for sample audit that one embodiment of the invention provides
Schematic diagram；
Fig. 2 is the structure of the state inspection apparatus at the human customer end for sample audit that one embodiment of the invention provides
Schematic diagram；
Fig. 3 is the structural schematic diagram for the electronic equipment that one embodiment of the invention provides.
Specific embodiment
Below in conjunction with the drawings and specific embodiments to a kind of human customer end for sample audit proposed by the present invention
State detection method, device, electronic equipment and computer readable storage medium are described in further detail.According to claims
With following explanation, advantages and features of the invention will be become apparent from.It should be noted that attached drawing is all made of very simplified form and
Using non-accurate ratio, only for the purpose of facilitating and clarifying the purpose of the embodiments of the invention.
To solve problem of the prior art, the embodiment of the invention provides a kind of human customer ends for sample audit
State detection method, device, electronic equipment and computer readable storage medium.
It should be noted that the state detection method at the human customer end for sample audit of the embodiment of the present invention can answer
The state inspection apparatus at the human customer end for sample audit for the embodiment of the present invention, this is used for the artificial of sample audit
The state inspection apparatus of client can be configured on electronic equipment.Wherein, which can be personal computer, movement
Terminal etc., the mobile terminal can be the hardware device that mobile phone, tablet computer etc. have various operating systems.
Fig. 1 is a kind of state detection method at human customer end for sample audit that one embodiment of the invention provides
Flow diagram.Referring to FIG. 1, a kind of state detection method at the human customer end for sample audit may include walking as follows
It is rapid:
Step S101 obtains a test sample collection, using a preparatory trained identification model to the test sample collection
In each sample identified, mark out the recognition result of each sample.
In the present embodiment, the identification model can be the neural network model established by sample training, can be with
The identification model obtained by any type of sample training, the present embodiment to the sample type of training without limitation.Example
If sample can be bill picture, it can establish bank slip recognition model after training, be also possible to vehicle pictures, face picture, plant
Object picture, paper picture etc., different sample types can establish different identification models by sample training respectively.When described
After identification model trains, the recognition accuracy of the identification model is also determined that.The identification is established by sample training
The mode of the recognition accuracy of the process of model and the determining identification model may refer to the prior art, not do herein superfluous
It states.
The sample type that the test sample is concentrated need to be identical as sample type when identification model training, for example,
The identification model is obtained by bill picture sample training, then it is bill that acquired test sample, which concentrates sample,
Picture.
The sample size that the present embodiment concentrates test sample without limitation, but in order to whether make to judge human customer end
Abnormal judging result is more accurate, and sample size should usually be set as biggish numerical value, such as sample size is 100,1000
Deng.Each sample standard deviation that test sample is concentrated is identified by the identification model, and by the identification of the identification model
As a result it is labeled.Recognition result can be labeled in samples pictures, can also mark the attribute information as samples pictures.
Step S102 chooses the test sample and concentrates preset quantity sample as target sample, and by the target
The recognition result that each sample is marked in sample is revised as wrong identification result.
The recognition result that sample is marked is revised as wrong identification as a result, the original identification knot for being revised as and being marked
The inconsistent or different recognition result of fruit.For example, if the identification model is obtained based on facial image sample training,
The face in facial image is male or women for identification, and the identification model concentrates a certain face to the test sample
The recognition result of image pattern is women, then the recognition result that the facial image sample is marked is women, then by the people
The recognition result that face image sample is marked is revised as wrong identification as a result, being for example revised as male.For another example, if the identification mould
Type is obtained based on plant image sample training, for identification the classification of plant of the plant as in, and the identification model is to institute
Stating test sample and concentrating the recognition result of a certain plant image sample is peach blossom, then the identification knot that the plant image sample is marked
Fruit is peach blossom, then the recognition result that the plant image sample is marked is revised as inconsistent or different recognition result,
Such as it is revised as pear flower.
In the present embodiment, it can be concentrated from the test sample and randomly extract preset quantity sample as target sample
This, the recognition result that target sample is marked is revised as the recognition result of mistake.Since core of the invention thought is to pass through
Human customer end is counted to the audit situation of the wrong identification result of the preset quantity target sample extracted, to infer manually
Client concentrates the audit situation of sample to entire test sample, and then judges whether human customer end is in abnormality, because
This can have following requirement to the quantity of the target sample extracted to guarantee the accuracy of subsequent statistical:
The smallest sample that the preset quantity is more than or equal to sampling statistics extracts quantity N；
Wherein, N=Z2×(P×(1-P))/E2；Z indicates that statistic relevant to confidence level, confidence level are equal to the identification
The recognition accuracy of model；E indicates preset sampling error value；P indicates the recognition accuracy of the identification model.
Z and the corresponding relationship of confidence level are as follows: when confidence level is 90%, Z=1.64；When confidence level is 95%, Z=
1.96；When confidence level is 95.45%, Z=2；When confidence level is 99%, Z=2.68；When confidence level is 99.73, Z=3；More than
Data can be obtained by query statistic table.The confidence interval of the present embodiment is 90%~99.99%, that is to say, that thinks described
The recognition accuracy of identification model should be in the range of 90%~99.99% probability drops into P, and the present embodiment can use
95% confidence level.
In the present embodiment, sampling error value E be may be set between ± 5%, and P is probability value, can set it as 90%,
I.e. the test sample concentrates the accuracy rate of sample mark to need to reach 90% after identification model mark.
It is equal to 100 if smallest sample is calculated by above-mentioned calculation formula and extracts quantity N, the preset quantity can
To set any number for being more than or equal to 100.It can also be concentrated from the test sample and extract a certain proportion of sample as mesh
Standard specimen sheet, as long as guaranteeing that the quantity of extracted target sample is more than or equal to smallest sample and extracts quantity N.
Step S103, by treated, the test sample collection is sent to human customer end, so as to the human customer end
The recognition result of each sample is audited.
The human customer end can the recognition result to the identification model carry out audit processing, audit processing includes: needle
To treated, test sample concentrates each sample, judges whether marked recognition result is correct；If it is determined that incorrect, also
The recognition result that can be marked to the sample is modified.It should be noted that since treated, test sample concentrates packet
Containing two class samples, that is, it is labeled with the sample of the recognition result of the identification model, and is extracted simultaneously intentional marking error identification knot
The sample of fruit, human customer end will not distinguish these two types of samples in audit, but will be extracted and deliberately mark wrong
The target sample of misrecognition result is equally considered as the sample for being labeled with the recognition result of the identification model.
For example, still by taking the citing in above-mentioned steps S102 as an example, if the recognition result that a certain sample is marked is female
Property, and the recognition result that human customer end determines that the sample is marked after audit is wrong, and determines after the identification of itself
The recognition result of the sample should be male, then the recognition result that can be marked to the sample is revised as the knowledge itself determined
Other result.
In fact, human customer end may not identify for being extracted simultaneously the deliberately a certain target sample of marking error
The sample is marked mistake out, is determined as the recognition result of the sample correctly so as to cause human customer end.Human customer end
Identification (the mark that sample is concentrated at human customer end to entire test sample has been reacted to the audit situation of the target sample deliberately mislabeled
Note) situation, and then by checking that artificial client can infer the mark at human customer end to the audit situation of this kind of sample
Accuracy rate or audit accuracy rate, and judge human customer end with the presence or absence of abnormal.
Step S104 checks the human customer according to the human customer end to the auditing result of the target sample
Whether end is in abnormality.
In one implementation, it is described according to the human customer end to the auditing result of the target sample, check
Whether the human customer end is in abnormality, comprising:
For each sample in the target sample, judge the human customer end whether to the wrong identification of the sample
As a result it is modified；
The ratio for being had modified the sample of recognition result by the human customer end in the target sample is obtained, as the
One ratio；
If first ratio is less than preset threshold, determine that the human customer end is in abnormality.
It is understood that usually, if human customer end can know the mistake of the sample of intentional marking error
Other result is modified, it may be considered that human customer end can correctly be marked the sample of the intentional marking error.
If the ratio for being had modified the sample of recognition result in target sample by human customer end is more than or equal to preset threshold, it is believed that people
There is not exception in work client, conversely, if being had modified the ratio of the sample of recognition result in target sample by human customer end
Less than preset threshold, then it represents that exception occurs in human customer end.Further, can also by target sample by artificial visitor
Family end has modified the ratio of the sample of recognition result, come infer human customer end to the mark accuracy rate of entire test sample collection,
Such as using the first ratio as the mark accuracy rate at human customer end.
Specifically, determining the ratio for being had modified the sample of recognition result in the target sample by the human customer end
Example, can there is following two ways:
Mode one: it for the test sample collection after the audit of human customer end, obtains first and is extracted and deliberately marks
The target sample for infusing wrong identification result, then judges the sample that human customer end is modified the recognition result marked
Quantity, and then obtain the ratio for being had modified the sample of recognition result in the target sample by the human customer end；
Mode two: for the test sample collection after the audit of human customer end, it may determine that human customer end first
The recognition result marked to which sample is modified, and it is to belong to be extracted and mark which, which is then counted in these samples,
The target sample of wrong identification result is infused, and then obtains the sample for being had modified recognition result in target sample by the human customer end
This ratio.
In another implementation, it is described according to the human customer end to the auditing result of the target sample, inspection
Look into whether the human customer end is in abnormality, comprising:
For each sample in the target sample, judge the human customer end whether by the wrong identification of the sample
Results modification is correct recognition result；
The ratio for being revised as the sample of correct recognition result by the human customer end in the target sample is obtained, is made
For the second ratio；
If second ratio is less than preset threshold, determine that the human customer end is in abnormality.
In this implementation, if the ratio of the sample of correct recognition result is revised as in target sample by human customer end
Example is more than or equal to preset threshold, it is believed that exception does not occur in human customer end, conversely, if by human customer in target sample
The ratio that the sample of correct recognition result is revised as at end is less than preset threshold, then it represents that exception occurs in human customer end.Into one
Step, it can also be by the way that the ratio of the sample of correct recognition result be revised as in target sample by human customer end, to infer people
Work client is such as accurate as the mark at human customer end using the second ratio to the mark accuracy rate of entire test sample collection
Rate.
It is artificial for judging according to the ratio for the sample for being revised as correct recognition result in target sample by human customer end
Client is with the presence or absence of abnormal, and for characterizing the mark accuracy rate at human customer end, more compared to a kind of upper implementation
It is accurate to add.
Similar, determine the ratio for being revised as the sample of correct recognition result in the target sample by the human customer end
Example, can there is following two ways:
Mode one: it for the test sample collection after the audit of human customer end, obtains first and is extracted and deliberately marks
Then the target sample for infusing wrong identification result judges that the recognition result marked is revised as correctly identifying knot by human customer end
The quantity of the sample of fruit, and then obtain the sample for being revised as correct recognition result in the target sample by the human customer end
Ratio；
Mode two: for the test sample collection after the audit of human customer end, it may determine that human customer end first
The recognition result which sample is marked is revised as correct recognition result, then count in these samples which be belong to by
The target sample of simultaneously marking error recognition result is extracted, and then obtains and is revised as correctly in target sample by the human customer end
The ratio of the sample of recognition result.
When judging that the first ratio is greater than preset threshold or the second ratio greater than preset threshold, it is possible to determine that artificial visitor
Family end is in abnormality, while also illustrating that desired value is not achieved in the mark accuracy rate at human customer end, therefore can be to artificial
Client is modified, and is met the requirements so that it marks accuracy rate.
Wherein, the minimum value X of the preset threshold can be determined according to following formula: 1- (1-X)2=Q；Q indicates preparatory
The test sample after identification model mark and human customer end audit being arranged concentrates the target of sample mark
Accuracy rate.The preset threshold can be set to arbitrarily be equal to the numerical value greater than X, and the present embodiment does not limit this.
As Q=99%, X=90% is calculated by above-mentioned formula, i.e. the mark accuracy rate needs at human customer end reach
To 90% or more.In the present embodiment, if it is desired to the survey after identification model mark and human customer end audit
The accuracy rate of sample this concentration sample mark reaches 99% or more, then mark of the human customer end in the error sample deliberately mislabeled
Note accuracy rate needs to reach 90% or more.
In conclusion compared with prior art, test sample is concentrated using identification model in the present embodiment sample into
Row identification, marks out recognition result, and concentrate the recognition result of a part of sample to be deliberately revised as wrong identification test sample
As a result, will treated that test sample collection issues human customer end audits, only need to check so artificial client to therefore
The auditing result i.e. deducibility human customer end of that a part of sample that mislabels anticipate to the mark accuracy rate of entire test sample collection,
And then judge whether human customer end is in abnormality, entire test sample collection is examined without counting human customer end
Core situation determines mark accuracy rate, quickly determines whether human customer end is in abnormality to realize, and shorten
Statistical time, reduces expense cost.
Corresponding to the state detection method embodiment at the above-mentioned human customer end for sample audit, one embodiment of the invention
A kind of state inspection apparatus at human customer end for sample audit is additionally provided, Fig. 2 is that one embodiment of the invention provides
A kind of structural schematic diagram of the state inspection apparatus at the human customer end for sample audit.Referring to FIG. 2, a kind of be used for sample
The state inspection apparatus at the human customer end of audit may include:
Labeling module 201, for obtaining a test sample collection, using a preparatory trained identification model to the test
Each sample is identified in sample set, marks out the recognition result of each sample；
Modified module 202 concentrates preset quantity sample as target sample for choosing the test sample, and by institute
It states the recognition result that each sample is marked in target sample and is revised as wrong identification result；
Auditing module 203, for will treated that the test sample collection is sent to human customer end, so as to described artificial
Client audits the recognition result of each sample；
Check that module 204 checks the people for the auditing result according to the human customer end to the target sample
Whether work client is in abnormality.
Optionally, the recognition result of each sample is audited at human customer end in the auditing module 203, comprising:
For treated, the test sample concentrates each sample, and the human customer end judges marked identification knot
Whether fruit is correct；If it is not, then the recognition result marked to the sample is modified.
Optionally, the inspection module 204, is specifically used for:
For each sample in the target sample, judge the human customer end whether to the wrong identification of the sample
As a result it is modified；Obtain the ratio for being had modified the sample of recognition result by the human customer end in the target sample
Example, as the first ratio；If first ratio is less than preset threshold, determine that the human customer end is in abnormality.
Optionally, described device further include:
First determining module, the mark accuracy rate for the human customer end according to first ratio-dependent.
Optionally, the inspection module 204, is specifically used for:
For each sample in the target sample, judge the human customer end whether by the wrong identification of the sample
Results modification is correct recognition result；Acquisition is revised as correct recognition result by the human customer end in the target sample
Sample ratio, as the second ratio；If second ratio is less than preset threshold, determine that the human customer end is in
Abnormality.
Optionally, described device further include:
Second determining module, the mark accuracy rate for the human customer end according to second ratio-dependent.
Optionally, the minimum value X of the preset threshold is determined according to following formula: 1- (1-X)2=Q；
Wherein, Q indicates the pre-set test after identification model mark and human customer end audit
The target accuracy rate that sample marks in sample set.
Optionally, the preset quantity is more than or equal to smallest sample extraction quantity N；
Wherein, N=Z2×(P×(1-P))/E2；Z indicates that statistic relevant to confidence level, confidence level are equal to the identification
The recognition accuracy of model；E indicates preset sampling error value；P indicates the recognition accuracy of the identification model.
Optionally, the recognition result that sample each in the target sample is marked is revised as by the modified module 202
Wrong identification result, comprising:
The recognition result that sample each in the target sample is marked is revised as the knowledge different from original recognition result
Other result.
Compared with prior art, the present embodiment is identified using the sample that identification model concentrates test sample, is marked
Recognition result out, and concentrate the recognition result of a part of sample to be deliberately revised as wrong identification as a result, will processing test sample
Test sample collection afterwards is issued human customer end and is audited, only need to check so artificial client to deliberately mislabel that
The auditing result of part sample, that is, deducibility human customer end judges artificial the mark accuracy rate of entire test sample collection
Whether client is in abnormality, determines without counting human customer end to the audit situation of entire test sample collection
Accuracy rate is marked, quickly determines whether human customer end is in abnormality to realize, and shorten statistical time, drops
Low expense cost.
One embodiment of the invention additionally provides a kind of electronic equipment, and Fig. 3 is a kind of electronics that one embodiment of the invention provides
The structural schematic diagram of equipment.Referring to FIG. 3, a kind of electronic equipment includes processor 301, communication interface 302,303 and of memory
Communication bus 304, wherein processor 301, communication interface 302, memory 303 complete mutual lead to by communication bus 304
Letter,
Memory 303, for storing computer program；
Processor 301 when for executing the program stored on memory 303, realizes following steps:
A test sample collection is obtained, each sample is concentrated to the test sample using a preparatory trained identification model
It is identified, marks out the recognition result of each sample；
Choosing the test sample concentrates preset quantity sample as target sample, and will be each in the target sample
The recognition result that sample is marked is revised as wrong identification result；
By treated, the test sample collection is sent to human customer end, so that the human customer end is to each sample
Recognition result audited；
According to the human customer end to the auditing result of the target sample, check whether the human customer end is in
Abnormality.
Using scheme provided in this embodiment, it is only necessary to check artificial client to that a part of sample deliberately mislabeled
Whether auditing result, that is, deducibility human customer end judges human customer end to the mark accuracy rate of entire test sample collection
In abnormality, determine that mark is accurate to the audit situation of entire test sample collection without counting human customer end
Rate quickly determines whether human customer end is in abnormality to realize, and shortens statistical time, reduces expense
Cost.
Specific implementation and relevant explanation content about each step of this method may refer to above-mentioned method shown in FIG. 1
Embodiment, this will not be repeated here.
In addition, the mark at human customer end that processor 301 executes the program stored on memory 303 and realizes is accurate
Other implementations of the determination method of rate, it is identical as implementation mentioned by preceding method embodiment part, here also not
It repeats again.
The communication bus that above-mentioned electronic equipment is mentioned can be Peripheral Component Interconnect standard (Peripheral Component
Interconnect, PCI) bus or expanding the industrial standard structure (Extended Industry Standard
Architecture, EISA) bus etc..The communication bus can be divided into address bus, data/address bus, control bus etc..For just
It is only indicated with a thick line in expression, figure, it is not intended that an only bus or a type of bus.
Communication interface is for the communication between above-mentioned electronic equipment and other equipment.
Memory may include random access memory (Random Access Memory, RAM), also may include non-easy
The property lost memory (Non-Volatile Memory, NVM), for example, at least a magnetic disk storage.Optionally, memory may be used also
To be storage device that at least one is located remotely from aforementioned processor.
Above-mentioned processor can be general processor, including central processing unit (Central Processing Unit,
CPU), network processing unit (Network Processor, NP) etc.；It can also be digital signal processor (Digital Signal
Processing, DSP), it is specific integrated circuit (Application Specific Integrated Circuit, ASIC), existing
It is field programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic device, discrete
Door or transistor logic, discrete hardware components.
One embodiment of the invention additionally provides a kind of computer readable storage medium, the computer readable storage medium memory
Computer program is contained, which realizes the above-mentioned human customer end for sample audit when being executed by processor
The method and step of state detection method.
Using scheme provided in this embodiment, it is only necessary to check artificial client to that a part of sample deliberately mislabeled
Whether auditing result, that is, deducibility human customer end judges human customer end to the mark accuracy rate of entire test sample collection
In abnormality, determine that mark is accurate to the audit situation of entire test sample collection without counting human customer end
Rate quickly determines whether human customer end is in abnormality to realize, and shortens statistical time, reduces expense
Cost.
Described it should be noted that each embodiment in this specification is all made of relevant mode, each embodiment it
Between same and similar part may refer to each other, each embodiment focuses on the differences from other embodiments.
For device, electronic equipment, computer readable storage medium embodiment, implement since it is substantially similar to method
Example, so being described relatively simple, the relevent part can refer to the partial explaination of embodiments of method.
Herein, relational terms such as first and second and the like be used merely to by an entity or operation with it is another
One entity or operation distinguish, and without necessarily requiring or implying between these entities or operation, there are any this reality
Relationship or sequence.Moreover, the terms "include", "comprise" or its any other variant are intended to the packet of nonexcludability
Contain, so that the process, method, article or equipment for including a series of elements not only includes those elements, but also including
Other elements that are not explicitly listed, or further include for elements inherent to such a process, method, article, or device.
In the absence of more restrictions, the element limited by sentence "including a ...", it is not excluded that including the element
Process, method, article or equipment in there is also other identical elements.
Foregoing description is only the description to present pre-ferred embodiments, not to any restriction of the scope of the invention, this hair
Any change, the modification that the those of ordinary skill in bright field does according to the disclosure above content, belong to the protection of claims
Range.
Claims (20)
1. a kind of state detection method at the human customer end for sample audit, which is characterized in that the described method includes:
A test sample collection is obtained, concentrates each sample to carry out the test sample using a preparatory trained identification model
Identification, marks out the recognition result of each sample；
Choosing the test sample concentrates preset quantity sample as target sample, and by sample each in the target sample
The recognition result marked is revised as wrong identification result；
By treated, the test sample collection is sent to human customer end, the knowledge so as to the human customer end to each sample
Other result is audited；
According to the human customer end to the auditing result of the target sample, check whether the human customer end is in abnormal
State.
2. the state detection method for the human customer end of sample audit as described in claim 1, which is characterized in that described
Human customer end is audited by the recognition result of each sample, comprising:
For treated, the test sample concentrates each sample, and the human customer end judges that marked recognition result is
It is no correct；If it is not, then the recognition result marked to the sample is modified.
3. the state detection method for the human customer end of sample audit as claimed in claim 2, which is characterized in that described
According to the human customer end to the auditing result of the target sample, check whether the human customer end is in abnormal shape
State, comprising:
For each sample in the target sample, judge the human customer end whether to the wrong identification result of the sample
It is modified；
The ratio for being had modified the sample of recognition result by the human customer end in the target sample is obtained, as the first ratio
Example；
If first ratio is less than preset threshold, determine that the human customer end is in abnormality.
4. the state detection method for the human customer end of sample audit as claimed in claim 3, which is characterized in that described
Method further include:
According to the mark accuracy rate at human customer end described in first ratio-dependent.
5. the state detection method for the human customer end of sample audit as claimed in claim 2, which is characterized in that described
According to the human customer end to the auditing result of the target sample, check whether the human customer end is in abnormal shape
State, comprising:
For each sample in the target sample, judge the human customer end whether by the wrong identification result of the sample
It is revised as correct recognition result；
The ratio for being revised as the sample of correct recognition result by the human customer end in the target sample is obtained, as the
Two ratios；
If second ratio is less than preset threshold, determine that the human customer end is in abnormality.
6. the state detection method for the human customer end of sample audit as claimed in claim 5, which is characterized in that described
Method further include:
According to the mark accuracy rate at human customer end described in second ratio-dependent.
7. the state detection method at the human customer end for sample audit as claimed in claim 3 or 5, which is characterized in that
The minimum value X of the preset threshold is determined according to following formula: 1- (1-X)2=Q；
Wherein, Q indicates the pre-set test sample after identification model mark and human customer end audit
Concentrate the target accuracy rate of sample mark.
8. the state detection method for the human customer end of sample audit as described in claim 1, which is characterized in that described
Preset quantity is more than or equal to smallest sample and extracts quantity N；
Wherein, N=Z2×(P×(1-P))/E2；Z indicates that statistic relevant to confidence level, confidence level are equal to the identification model
Recognition accuracy；E indicates preset sampling error value；P indicates the recognition accuracy of the identification model.
9. the state detection method for the human customer end of sample audit as described in claim 1, which is characterized in that described
The recognition result that sample each in the target sample is marked is revised as wrong identification result, comprising:
The recognition result that sample each in the target sample is marked is revised as to the identification knot different from original recognition result
Fruit.
10. a kind of state inspection apparatus at the human customer end for sample audit, which is characterized in that described device includes:
Labeling module, for obtaining a test sample collection, using a preparatory trained identification model to the test sample collection
In each sample identified, mark out the recognition result of each sample；
Modified module concentrates preset quantity sample as target sample for choosing the test sample, and by the target
The recognition result that each sample is marked in sample is revised as wrong identification result；
Auditing module, for will treated that the test sample collection is sent to human customer end, so as to the human customer end
The recognition result of each sample is audited；
Check that module checks the human customer for the auditing result according to the human customer end to the target sample
Whether end is in abnormality.
11. the state inspection apparatus for the human customer end of sample audit as claimed in claim 10, which is characterized in that institute
Human customer end in auditing module is stated to audit the recognition result of each sample, comprising:
For treated, the test sample concentrates each sample, and the human customer end judges that marked recognition result is
It is no correct；If it is not, then the recognition result marked to the sample is modified.
12. the state inspection apparatus for the human customer end of sample audit as claimed in claim 11, which is characterized in that institute
Inspection module is stated, is specifically used for:
For each sample in the target sample, judge the human customer end whether to the wrong identification result of the sample
It is modified；The ratio for being had modified the sample of recognition result by the human customer end in the target sample is obtained, is made
For the first ratio；If first ratio is less than preset threshold, determine that the human customer end is in abnormality.
13. the state inspection apparatus for the human customer end of sample audit as claimed in claim 12, which is characterized in that institute
State device further include:
First determining module, the mark accuracy rate for the human customer end according to first ratio-dependent.
14. the state inspection apparatus for the human customer end of sample audit as claimed in claim 11, which is characterized in that institute
Inspection module is stated, is specifically used for:
For each sample in the target sample, judge the human customer end whether by the wrong identification result of the sample
It is revised as correct recognition result；Obtain the sample for being revised as correct recognition result by the human customer end in the target sample
This ratio, as the second ratio；If second ratio is less than preset threshold, determine that the human customer end is in abnormal
State.
15. the state inspection apparatus for the human customer end of sample audit as claimed in claim 14, which is characterized in that institute
State device further include:
Second determining module, the mark accuracy rate for the human customer end according to second ratio-dependent.
16. the state inspection apparatus at the human customer end for sample audit as described in claim 12 or 15, feature exist
In the minimum value X of the preset threshold is determined according to following formula: 1- (1-X)2=Q；
Wherein, Q indicates the pre-set test sample after identification model mark and human customer end audit
Concentrate the target accuracy rate of sample mark.
17. the state inspection apparatus for the human customer end of sample audit as claimed in claim 10, which is characterized in that institute
It states preset quantity and extracts quantity N more than or equal to smallest sample；
Wherein, N=Z2×(P×(1-P))/E2；Z indicates that statistic relevant to confidence level, confidence level are equal to the identification model
Recognition accuracy；E indicates preset sampling error value；P indicates pre-set through identification model mark and the people
The test sample concentrates the target accuracy rate of sample mark after the audit of work client.
18. the state inspection apparatus for the human customer end of sample audit as claimed in claim 10, which is characterized in that institute
It states modified module and the recognition result that sample each in the target sample is marked is revised as wrong identification result, comprising:
The recognition result that sample each in the target sample is marked is revised as to the identification knot different from original recognition result
Fruit.
19. a kind of electronic equipment, which is characterized in that including processor, communication interface, memory and communication bus, wherein described
Processor, the communication interface, the memory complete mutual communication by the communication bus；
The memory, for storing computer program；
The processor when for executing the program stored on the memory, realizes any side claim 1-9
Method step.
20. a kind of computer readable storage medium, which is characterized in that be stored with computer in the computer readable storage medium
Program realizes claim 1-9 described in any item method and steps when the computer program is executed by processor.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910539076.4A CN110245087B (en) | 2019-06-20 | 2019-06-20 | State checking method and device of manual client for sample auditing |
| PCT/CN2020/096645 WO2020253740A1 (en) | 2019-06-20 | 2020-06-17 | Manual client status check method and device for sample verification |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910539076.4A CN110245087B (en) | 2019-06-20 | 2019-06-20 | State checking method and device of manual client for sample auditing |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110245087A true CN110245087A (en) | 2019-09-17 |
| CN110245087B CN110245087B (en) | 2023-04-18 |
Family
ID=67888373
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910539076.4A Active CN110245087B (en) | 2019-06-20 | 2019-06-20 | State checking method and device of manual client for sample auditing |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN110245087B (en) |
| WO (1) | WO2020253740A1 (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111833296A (en) * | 2020-05-25 | 2020-10-27 | 中国人民解放军陆军军医大学第二附属医院 | Automatic detection and verification system and method for bone marrow cell morphology |
| WO2020253740A1 (en) * | 2019-06-20 | 2020-12-24 | 杭州睿琪软件有限公司 | Manual client status check method and device for sample verification |
| CN113138916A (en) * | 2021-04-06 | 2021-07-20 | 青岛以萨数据技术有限公司 | Automatic testing method and system for picture structuring algorithm based on labeled sample |
| CN116307948A (en) * | 2023-05-23 | 2023-06-23 | 飞狐信息技术（天津）有限公司 | Method, device, equipment and storage medium for detecting auditing quality |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113793134B (en) * | 2021-09-26 | 2024-02-13 | 上汽通用五菱汽车股份有限公司 | Vehicle alarm method, device and computer readable storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105975980A (en) * | 2016-04-27 | 2016-09-28 | 百度在线网络技术（北京）有限公司 | Method of monitoring image mark quality and apparatus thereof |
| CN109492549A (en) * | 2018-10-24 | 2019-03-19 | 杭州睿琪软件有限公司 | A kind of processing of training sample set, model training method and system |
| WO2019071662A1 (en) * | 2017-10-09 | 2019-04-18 | 平安科技（深圳）有限公司 | Electronic device, bill information identification method, and computer readable storage medium |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11074494B2 (en) * | 2016-09-09 | 2021-07-27 | Cylance Inc. | Machine learning model for analysis of instruction sequences |
| CN109697537A (en) * | 2017-10-20 | 2019-04-30 | 北京京东尚科信息技术有限公司 | The method and apparatus of data audit |
| CN110245087B (en) * | 2019-06-20 | 2023-04-18 | 杭州睿琪软件有限公司 | State checking method and device of manual client for sample auditing |
-
2019
- 2019-06-20 CN CN201910539076.4A patent/CN110245087B/en active Active
-
2020
- 2020-06-17 WO PCT/CN2020/096645 patent/WO2020253740A1/en active Application Filing
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105975980A (en) * | 2016-04-27 | 2016-09-28 | 百度在线网络技术（北京）有限公司 | Method of monitoring image mark quality and apparatus thereof |
| WO2019071662A1 (en) * | 2017-10-09 | 2019-04-18 | 平安科技（深圳）有限公司 | Electronic device, bill information identification method, and computer readable storage medium |
| CN109492549A (en) * | 2018-10-24 | 2019-03-19 | 杭州睿琪软件有限公司 | A kind of processing of training sample set, model training method and system |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2020253740A1 (en) * | 2019-06-20 | 2020-12-24 | 杭州睿琪软件有限公司 | Manual client status check method and device for sample verification |
| CN111833296A (en) * | 2020-05-25 | 2020-10-27 | 中国人民解放军陆军军医大学第二附属医院 | Automatic detection and verification system and method for bone marrow cell morphology |
| CN111833296B (en) * | 2020-05-25 | 2023-03-10 | 中国人民解放军陆军军医大学第二附属医院 | Automatic detection and verification system and method for bone marrow cell morphology |
| CN113138916A (en) * | 2021-04-06 | 2021-07-20 | 青岛以萨数据技术有限公司 | Automatic testing method and system for picture structuring algorithm based on labeled sample |
| CN113138916B (en) * | 2021-04-06 | 2024-04-30 | 青岛以萨数据技术有限公司 | Automatic testing method and system for picture structuring algorithm based on labeling sample |
| CN116307948A (en) * | 2023-05-23 | 2023-06-23 | 飞狐信息技术（天津）有限公司 | Method, device, equipment and storage medium for detecting auditing quality |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110245087B (en) | 2023-04-18 |
| WO2020253740A1 (en) | 2020-12-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110245087A (en) | The state detection method and device at the human customer end for sample audit | |
| CN110222170B (en) | Method, device, storage medium and computer equipment for identifying sensitive data | |
| CN110263853A (en) | The method and device of artificial client state is checked using error sample | |
| CN110245716A (en) | Sample labeling auditing method and device | |
| CN108921206A (en) | A kind of image classification method, device, electronic equipment and storage medium | |
| US11335087B2 (en) | Method and system for object identification | |
| CN107633227A (en) | A kind of fine granularity gesture identification method and system based on CSI | |
| CN109239102A (en) | A kind of flexible circuit board open defect detection method based on CNN | |
| CN106022317A (en) | Face identification method and apparatus | |
| CN108491388B (en) | Data set acquisition method, classification method, device, equipment and storage medium | |
| CN110503143A (en) | Research on threshold selection, equipment, storage medium and device based on intention assessment | |
| CN107958230A (en) | Facial expression recognizing method and device | |
| CN107491536B (en) | Test question checking method, test question checking device and electronic equipment | |
| CN106485528A (en) | The method and apparatus of detection data | |
| CN106250825A (en) | A kind of at the medical insurance adaptive face identification system of applications fields scape | |
| CN108875797A (en) | A kind of method of determining image similarity, photograph album management method and relevant device | |
| CN104881675A (en) | Video scene identification method and apparatus | |
| CN109858476A (en) | The extending method and electronic equipment of label | |
| CN111046879A (en) | Certificate image classification method and device, computer equipment and readable storage medium | |
| CN109284700B (en) | Method, storage medium, device and system for detecting multiple faces in image | |
| CN108319888A (en) | The recognition methods of video type and device, terminal | |
| US20210192965A1 (en) | Question correction method, device, electronic equipment and storage medium for oral calculation questions | |
| CN110427962A (en) | A kind of test method, electronic equipment and computer readable storage medium | |
| CN110457677A (en) | Entity-relationship recognition method and device, storage medium, computer equipment | |
| CN109815958A (en) | A kind of laboratory test report recognition methods, device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |