CN109784381A - Markup information processing method, device and electronic equipment - Google Patents

Abstract

This application discloses a kind of markup information processing method, device and electronic equipments, are related to field of computer technology.Sample set has been marked this method comprises: obtaining, and the sample set that marked corresponds to mark person's identity information, and the sample set that marked includes multiple having marked sample；Determine that the accuracy rate for having marked sample set, the accuracy rate have marked the accuracy that sample has been marked in sample set for indicating described；According to the accuracy rate, integrated value corresponding with mark person's identity information is set.It therefore, is that certain integrated value is arranged in the mark person by the accuracy rate of the sample marked according to mark person, then the integrated value can react the accuracy of the mark sample of mark person, to the capability evaluation of mark person, provide reference to the mark ability for improving mark person.

Description

Markup information processing method, device and electronic equipment

Technical field

This application involves field of computer technology, more particularly, to a kind of markup information processing method, device and electronics Equipment.

Background technique

With the development of computer technology, more image detection tasks can be allocated to machine to complete.For example, right In some live videos or the detection of video content etc..Wherein, process of the machine in identification live video or video content In would generally be identified based on certain model.And these models that machine is relied on are normally based on the sample of mark early period Originally it is trained.And there are also to be hoisted for the accuracy and timeliness of the mark of sample.And wherein, the mark of mark person Ability is an important factor for influencing the accuracy and timeliness of sample mark, therefore, the feedback pole of ability to be marked to mark person It is important.

Summary of the invention

Present applicant proposes a kind of markup information processing method, device and electronic equipments, to improve drawbacks described above.

In a first aspect, the embodiment of the present application provides a kind of markup information processing method, comprising: acquisition has marked sample Collection, the sample set that marked correspond to mark person's identity information, and the sample set that marked includes multiple having marked sample；It determines The accuracy rate for having marked sample set, the accuracy rate is for indicating that described marked has marked the accurate of sample in sample set Property；According to the accuracy rate, integrated value corresponding with mark person's identity information is set.

Second aspect, the embodiment of the present application also provides a kind of markup information processing units, comprising: acquiring unit, determination Unit and setting unit.Acquiring unit has marked sample set for obtaining, and the sample set that marked corresponds to mark person's identity letter Breath, the sample set that marked includes multiple having marked sample.Determination unit, for having marked the accurate of sample set described in determination Rate, the accuracy rate have marked the accuracy that sample has been marked in sample set for indicating described.Setting unit, for according to institute It states accuracy rate and integrated value corresponding with mark person's identity information is set.

The third aspect, the embodiment of the present application also provides a kind of electronic equipment, comprising: one or more processors；Storage Device；One or more application program, wherein one or more of application programs are stored in the memory and are configured To be executed by one or more of processors, one or more of programs are configured to carry out the above method.

Markup information processing method, device and electronic equipment provided by the present application, obtain each mark person marked it is more The mark sample set that a sample is constituted, then marked sample set include it is multiple marked sample, and multiple marked sample This corresponds to mark person's identity information.Then, the accuracy rate for having marked sample set is obtained, the accuracy rate is for indicating institute The accuracy for having marked and having marked sample in sample set is stated, is arranged further according to the accuracy rate corresponding with mark person's identity information Integrated value.It therefore, is that certain integrated value is arranged in the mark person by the accuracy rate of the sample marked according to mark person, then The integrated value can react the accuracy of the mark sample of mark person, to the capability evaluation of mark person, to the mark for improving mark person Note ability provides reference.

Detailed description of the invention

In order to more clearly explain the technical solutions in the embodiments of the present application, make required in being described below to embodiment Attached drawing is briefly described, it should be apparent that, the drawings in the following description are only some examples of the present application, for For those skilled in the art, without creative efforts, it can also be obtained according to these attached drawings other attached Figure.

Fig. 1 shows a kind of application network environment map of markup information processing method of the application proposition；

Fig. 2 shows a kind of detection interface schematic diagrams provided by the embodiments of the present application；

Fig. 3 shows a kind of configuration diagram of information system provided by the embodiments of the present application；

Fig. 4 shows a kind of flow chart of markup information processing method of one embodiment of the application offer；

Fig. 5 shows a kind of flow chart for markup information processing method that another embodiment of the application provides；

Fig. 6 shows the flow chart of S504 in a kind of markup information processing method that another embodiment of the application provides；

Fig. 7 shows a kind of flow chart for markup information processing method that the another embodiment of the application provides；

Fig. 8 shows the schematic diagram of label provided by the embodiments of the present application；

Fig. 9 shows a kind of flow chart of markup information processing method of the application another embodiment offer

Figure 10 shows a kind of flow chart of markup information processing method of the application yet another embodiment offer；

Figure 11 shows the module frame chart of markup information processing unit provided by the embodiments of the present application；

Figure 12 shows the structural block diagram of electronic equipment provided by the embodiments of the present application；

Figure 13 shows realizing at according to the information of the embodiment of the present application for saving or carrying for the embodiment of the present application The storage unit of the program code of reason method.

Specific embodiment

In order to make those skilled in the art more fully understand application scheme, below in conjunction in the embodiment of the present application Attached drawing, the technical scheme in the embodiment of the application is clearly and completely described.

Referring to Fig. 1, the application scenario diagram of markup information processing method and processing device provided by the embodiments of the present application is shown, As shown in Figure 1, mark node 10 is used to be labeled sample, what detection node 20 was used to generate after marking mark node 10 The annotation results for having marked sample are detected.It should be noted that mark node 10 may include having multiple tagging equipments, In the tagging equipment can be for smart phone, tablet computer or computer etc..It, can be in tagging equipment as a kind of mode Display user interface is labeled sample so as to user.Wherein, the user interface can in a manner of client into Row display, can also be shown in the form of a web page.

With computer technology combining with mathematical computations, people have given more tasks to machine and have gone to execute.Example Such as, whether the image that configuration machine is gone in identification network direct broadcasting has violation content, then for example configuration machine goes identification network messaging In whether have violation content etc..Wherein it is possible to which the machine for directly executing identification mission is normally based on preparatory trained model It is performed identification mission, and the model is needed to be based in the training process to have marked sample and is trained.Mark in so Fig. 1 What note node 10 executed is exactly the mark for sample, and then exports and marked sample.It is then detected that node 20 can be to having marked The annotation results of sample are detected, to detect whether the annotation results for having marked sample are accurate.For example, mark node 10 Output has that have marked sample A be violation content, has marked sample A for this, and detection node 20 has been marked to further detecting this Infuse whether sample A is really violation content, it specifically, can be to the correct sample of all marks marked in sample and mark The sample of mistake is by specifically label mark and distinguishes.Detection interface as shown in Figure 2 can pass through boundary as shown in Figure 2 Face identifies the different testing results for having marked the annotation results of sample, is beaten for example, having marked sample at some by display The mode of hook marks this to mark sample and belongs to the correct sample of mark, has marked sample by way of display fork at some, This is marked to mark the sample that sample belongs to marking error.

Wherein, as a kind of mode, marking node and detection node can be run in the same server.At this In the case of kind, run in a server including mark node and the information system of detection node, and mark node and Detection node can be regarded as two independently operated software modules in the server.And tagging equipment and detection device can be with It is connect with the same server.

Alternatively mode, different servers can be based respectively on to realize by marking node and detection node. In this case, as shown in figure 3, in information system 1 shown in Fig. 3, mark node 10 includes multiple tagging equipments 111 And annotation server 112.Detection node 20 includes multiple detection devices 121 and detection service device 122.Wherein, it marks Equipment 111 is deposited for being labeled, and then will mark result to be output in annotation server 112 to the sample not marked also Storage.And annotation server 112 can will mark sample and be sent in the detection service device 122 of detection node 20, then detect Server 122, which according still further to certain rule will mark sample and be distributed in multiple detection devices 121, to be detected.

Inventor has found under study for action, the mark ability of mark person can affect the mark of entire sample to be marked and When property and accuracy, and mark person only just knows that whether the sample oneself marked passes through audit after mark is completed, i.e., it is quasi- True rate is up to standard, and can not know the height of the standard capability of oneself, also just can not for ability deficiency and self-teaching, and lead It causes mark ability to be unable to get and is promoted active and effectively, and then affects the timeliness of the mark of entire sample to be marked and accurate Property.

Therefore, in order to overcome drawbacks described above, the embodiment of the present application provides a kind of markup information processing method, such as Fig. 4 institute To show, this method is applied to electronic equipment, which can be above-mentioned detection service device, be also possible to annotation server, Specifically, this method comprises: S401 to S403.

S401: acquisition has marked sample set, and the sample set that marked corresponds to mark person's identity information, described to have marked sample This collection has marked sample including multiple.

Wherein, annotation server is that each mark person distributes a mark task, includes multiple wait mark in the mark task Infuse sample and task description information, wherein the task description information is used for the condition and demand for illustrating to need the target marked, For example, the task description information can be " marking out Asia women ", wherein " Asia " and " women " is exactly to the mesh to be marked Target condition and demand.

Specifically, mark task is sent to tagging equipment by annotation server, then the tagging equipment can be a client End, for example, it may be an application program, is also possible to the interface of a webpage, and mark person passes through account and password login The tagging equipment marks multiple samples to be marked according to the task description demand in the tagging equipment, and will mark sample Originally it is sent to annotation server, detection service device is sent to by annotation server and carries out accuracy rate audit, alternatively, annotation server Oneself audit.

Furthermore, it is contemplated that the substantial amounts of sample to be marked, if obtaining all sample standard deviation to be marked mark completions If reviewing core accuracy rate afterwards, it on the one hand will lead to the excessive cycle of audit, on the other hand will also result in the data of communication link Therefore congestion has marked sample for the ease of timely finding that mark node has generated, server can the real-time or period Property to mark node mark behavior detect, consequently facilitating realize mark node have marked sample generate the case where Lower triggering in time starts subsequent the step of judging the accuracy for having marked sample.

It should be noted that it is different for the framework for including the information system for marking node and detection node, specifically hold The execution equipment that row detects the mark behavior of mark node can be different.As a kind of mode, if mark node and Detection node is run on the same server, in this case, marks node and detection node is two software moulds Block, and then be to be executed to detect the mark behavior of mark node by detection node.In the embodiment of the present application, this method Executing subject be detection node, then specifically, can be detection service device or the detection device in detection node.

If then detect the mark node have marked sample generation, from marked described in generation in sample obtain to Detection has marked sample.

As a kind of mode, the sample that marked is carried out fragment by detection node, obtains multiple fragments, then each fragment Sample set has been marked as one.Wherein, the mark that will be generated can be understood as to the sample progress fragment that marked of generation Sample is divided into multiple portions, and each fragment will include the mark sample of a part.For example, if the sample packet of mark got It includes and has marked sample A, marked sample B, marked sample C, marked sample D, marked sample E and marked sample F. If the sample of mark got is so carried out available first fragment of fragment and the second fragment, wherein first point Piece may include having marked sample A, having marked sample B, marked sample C, and the second fragment may include having to have marked sample D, it has marked sample E and has marked sample F.That is to say, a fragment is not limited to a mark task or one kind has marked Sample.

As a kind of mode, the sample that marked can be subjected to fragment according to the specified monitoring period, obtained multiple Fragment.

It is understood that can be pre-configured with by user for the monitoring period.For example, can be with the configuration monitoring period for 5 Minute, it can also be with the configuration monitoring period for 10 minutes etc..So in this case, subsequent during carrying out fragment, it can To carry out fragment according to the duration in monitoring period, then mark sample then characterizes a monitoring period included by a fragment It is interior generated to have marked sample.Or it sample A has been marked, marks sample B with above-mentioned, marked sample C, marked sample This D, for having marked sample E and having marked sample F.In the case where monitoring the period is 5 minutes, sample has been marked if recognizing This A, marked sample B mark start time from generated in 5 minutes, then just having marked sample A, having marked sample B Be divided into the first fragment, and if recognize marked sample C, marked sample D for mark start time from 5 minutes to 10 minutes It generates, then just having marked sample A, having marked sample B and be divided into the second fragment, and if recognizing and having marked sample E, marked Note generates in 10 minutes to 15 minutes from sample F mark start time, then just having marked sample E, having marked sample F It is divided into third fragment.

Therefore, as mark person is during marking sample to be marked, according to the time cycle, in each time cycle The interior mark sample for obtaining mark person and being completed within the week time, so that obtaining the time cycle corresponding has marked sample Collection, it should be noted that this marked sample set can be mark person the time week in complete it is all marked sample, It can be complete within the week time from mark person all and marked a certain proportion of sample of sampling as having marked sample This collection.

Alternatively mode can also carry out fragment according to the quantity of mark sample.Optionally, user can be pre- First configure the number of thresholds that sample has been marked included by each fragment.In this case, detection node can be based on aforementioned Number of thresholds carry out fragment, the quantity for having marked sample distributed in making previous fragment be equal to number of thresholds with Afterwards, it is further continued for generating next fragment.For example, the sample of mark generated includes to have marked sample A, marked sample B, Mark sample C, sample D has been marked, sample E has been marked and has marked sample F.If the number of thresholds of the fragment configured is 3, So in this case, detection node can be generated including having marked sample A, having marked sample B, marked sample C etc. 3 The first fragment of sample has been marked, then can also have been generated including having marked sample D, having marked sample E and marked sample F etc. 3 have marked the second fragment of sample.If the number of thresholds of the fragment configured is 4, then in this case, detection node It can be generated and marked sample including having marked sample A, having marked sample B, marked sample C and marked sample D etc. 4 The first fragment, then can also generate including 2 second points for having marked sample such as having marked sample E and having marked sample F Piece.

Certainly, above-mentioned acquisition has marked the mode of sample set, is also possible to mark all samples to be marked in user Sample set has been marked at and then from obtaining in completed mark sample, has been made for example, it may be extracting a certain proportion of sample To have marked sample set.

In addition, the sample standard deviation for having marked completion uploaded by each tagging equipment and the mark for logging in the tagging equipment Member's identity information is corresponding, for example, mark person's identity information can be the account of mark person, then the mark that annotation server obtains It infuses sample set and corresponds to mark person's identity information.

S402: determine that the accuracy rate for having marked sample set, the accuracy rate have marked sample set for indicating described In marked the accuracy of sample.

After getting and having marked sample set, sample set will be marked and be sent to detection device, passed through with testing staff The detection device audits each correctness for having marked sample marked in sample set, and specifically, testing staff can lead to It crosses detection device each sample that marked marked in sample set is marked, then if it is correct that this, which has marked sample, , then by this marked sample setting first label, if this marked sample be it is wrong, by this marked sample setting Second label, wherein the first label and the second label can be number, parameter or symbol, for example, it may be to be each correct The sample of mark be arranged the first parameter, be each mistake the sample of mark setting the second parameter, finally count the first parameter It just can determine with the second parameter and marked all the first quantity and all mistakes for correctly having marked sample of sample set The second quantity for having marked sample accidentally.Wherein, the input of the first parameter and the second parameter can pass through pair in each sample Bit positions select the first symbol or the second symbol and input, for example, shown in Fig. 2, which is picture, then in picture Upper selection " √ ", indicate input is the first parameter, and "×" is selected on picture, then it represents that input is the second parameter.

Then, it then calculates the first quantity and has marked the ratio of the total quantity of sample set, it will be able to marked sample The accuracy rate of collection, then accuracy rate has marked the accuracy that sample has been marked in sample set for indicating described, and specifically, this is accurate Rate is for indicating to have marked the ratio that the sample correctly marked in sample set accounts for total number of samples, then the value of this accuracy rate gets over Gao Ze It is higher to indicate that this has marked the correct sample accounting in sample set, then shows also this and has marked the corresponding mark person of sample set Mark ability it is stronger.

S403: integrated value corresponding with mark person's identity information is arranged according to the accuracy rate.

It, can be by way of integral, according to the mark person institute for the more intuitive mark ability for embodying mark person The accuracy rate of the sample of mark be the corresponding mark person's identity information of the mark person certain score value is set, i.e., with the mark person The corresponding integrated value of identity information, then the integrated value can add up, specifically, the mark sample set that mark person submits every time, The accuracy rate that sample set will have been marked according to this is that certain score value is arranged in mark person's identity information, then multiple to have marked The corresponding score value of sample set is exactly the corresponding integrated value of mark person's identity information, then the integrated value can not only show bid The accuracy of the standard of note person, additionally it is possible to show the quantity of the sample of mark person mark, i.e. quantity is more, then inspection has been Mark sample set is more, then cumulative score value is higher, higher with regard to integrated value.

Wherein, the specific embodiment of integrated value corresponding with mark person's identity information is set according to the accuracy rate It can be an accuracy rate threshold value be arranged, if the accuracy rate for having marked sample set acquired is less than or equal to the standard True rate threshold value, then integrated value is a, and if being higher than the accuracy rate threshold value, integrated value b, wherein b is the numerical value greater than 0, For example, b can be 1, wherein a can be with right and wrong positive number, for example, it may be 0, naturally it is also possible to be a negative, for example, -1.

Wherein, accuracy rate threshold value is the numerical value set according to actual demand, then is higher than the accuracy rate threshold value, shows this Marking sample set can be by detection, i.e., this has marked sample set and can export and be used, for example, can be applied to engineering The data set of model creation is practised, then the integrated value of a positive number can be set to mark person's identity information.And if having marked sample The accuracy rate of this collection be less than or equal to the accuracy rate threshold value, then show this marked sample set can not by detection, i.e., this has been marked Note sample set can export the standard for being unsatisfactory for being used, that is, input underproof annotation results, then give mark person's identity information The integral of non-positive number is set, for example, it may be 0 or a negative, for example, -1, then the sample that mark person is marked can not When passing through detection, certain score value can be deducted, can not also be integrated.

In the embodiment of the present application, above-mentioned a is that the numerical value of 0, b can be 1, is also possible to be adjusted according to other strategies It is whole, specifically, subsequent embodiment can be referred to.The sample set then marked for mark person, if accuracy rate passes through detection, Assign positive number score value, i.e. bonus point, and if do not passed through, not bonus point.Then as an implementation, each mark person can be given An integral parameter is arranged in identity information, then integrated value obtained by the above method can update the integral parameter.

In addition, above-mentioned integrated value be it is corresponding says acquisition marked sample set, if marked according to mark person All samples obtained it is multiple marked sample set, for example, above-mentioned multiple fragments.Then mark person's identity information has corresponded to multiple Sample set has been marked, then has obtained the accuracy rate for each having marked sample set, and according to each accuracy rate for having marked sample set The corresponding integrated value of mark person's identity information is set, then all integrated values add up, is obtained obtained by mark person The overall score arrived.Therefore, for mark person's identity information it is corresponding it is multiple marked sample set, can obtain respectively it is each Sample set is marked, and the present processes are executed to each sample set that marked respectively, then, respectively obtains and each has marked sample The corresponding integrated value of this collection, all integrated values are added up, and are obtained mark person and are entirely marked total score in sample.

In addition, obtained integrated value or total score can be pushed to the corresponding client of mark person's identity information, specifically Ground, which can be mark person for marking the client of sample, for example, it may be being mounted on the client in mark terminal End, specifically, client can show acquired score value, exhibition method can be by multiple mark person's identity informations with And corresponding score value is shown, for example, it is illustrated as a table of integrals according to the sequence of score value from high to low, for example, it may be It is shown in the homepage of client, so that each mark person for logging in the client can see the table of integrals.At other In embodiment, each fragment corresponding to mark person's identity information can be obtained, i.e., has each marked the integrated value of sample set, so Afterwards, each task description information for having marked sample set is corresponding with the integrated value of sample set has each been marked to show, then it marks Member can determine that oneself score value under that task is relatively low by the content of the displaying, improve this point so as to corresponding It is worth the corresponding mark ability of lower task.

It can also be mark person according to different accuracys rate in addition to adopting in manner just described furthermore according to accuracy rate assignment Identity information increases different integrated values, specifically, referring to Fig. 5, this method comprises: S501 to S504.

S501: acquisition has marked sample set, and the sample set that marked corresponds to mark person's identity information, described to have marked sample This collection has marked sample including multiple.

S502: determine that the accuracy rate for having marked sample set, the accuracy rate have marked sample set for indicating described In marked the accuracy of sample.

S503: obtaining the corresponding relationship of preset accuracy rate and score value, includes multiple accurate in the corresponding relationship Rate and the corresponding score value of each accuracy rate, and the higher corresponding score value of the accuracy rate is higher.

Specifically, the corresponding relationship of an accuracy rate and score value is preset, then the corresponding relationship, which can store, is detecting It include multiple accuracys rate and the corresponding score value of each accuracy rate in corresponding relationship, as one kind specifically in server Embodiment, the corresponding relationship can be as shown in the table:

Table 1

In above-mentioned table 1, [95%, 100%] indicates the range of an accuracy rate, specifically 95% to 100%, and include Two endpoints 95% and 100%, that is to say, that if the accuracy rate for having marked sample set is 95%, 97% or 100%, 95%, it 97% or 100% is respectively positioned in [95%, 100%] range, then corresponding score value is 2.

In addition, the mission bit stream in above-mentioned table 1 may include task category or task identification, that is to say, that different The corresponding relationship of accuracy rate corresponding to task and score value is different, specifically, if two have marked sample set and have belonged to not With task, but two accuracys rate for having marked sample set are just as or the same accuracy rate range in upper table 1 It is interior, but due to the task of the two difference, then acquired score value is different, for example, under same accuracy rate, first task classification Score value than the second task category is high, based on principle be that task difficulty is bigger, obtained score value is higher.

For example, sample classification and sample characteristics mark, the score value of corresponding same accuracy rate is different, specifically Ground can be under same accuracy rate, and the corresponding score value of the task of sample classification is less than the score value of the task of sample characteristics mark, That is, the task difficulty of sample classification is greater than the task difficulty of sample characteristics mark.

And the corresponding task description of sample set can have been marked getting by having marked task category corresponding to sample set It is obtained in information, for example, the task description information is a text information, by can to the keyword extraction in text information Determine that this has marked the task category of sample set, for example, a task description information is " by all pictures by number of person point Class " then recognizes keyword " classification ", and can determine that this has marked the task category of sample set is sample classification, and if one A task description information is " marking all women in picture ", then recognizes keyword " label ", can determine that this has been marked The task category for infusing sample set is sample classification.Certainly, above-mentioned task category is also possible to distinguishing task for mark person When, it has set, can be written into the task description information in mark task.

In addition, not joining to the corresponding integral of mark person's identity information if the accuracy rate for having marked sample set is too low Number bonus point, specific embodiment can be, and the judgment mechanism to accuracy rate is added in executing S503, that is, obtains and presets Accuracy rate and score value corresponding relationship specific embodiment are as follows: whether the judgement accuracy rate for having marked sample set is lower than Specified numerical value, if it is 0, i.e. the mark person that the corresponding integrated value of mark person's identity information, which is arranged, lower than specified numerical value The corresponding integral parameter of identity information adds 0, that is to say that this does not give mark person's identity information bonus point.And if it is being greater than or equal to Specified numerical value, then execute the corresponding relationship for obtaining preset accuracy rate and score value, it can obtain according to corresponding relationship Mark the corresponding score value of accuracy rate of sample set.

Furthermore all standards of specified numerical value can also be set lower than in the corresponding relationship of above-mentioned accuracy rate and score value The score value of true rate is 0, and specifically, in the corresponding relationship, score value corresponding to the accuracy rate lower than specified numerical value is 0, example Such as, the score value of accuracy rate of the accuracy rate range in above-mentioned table 1 in [0,59%] is 0, if that is, accuracy rate is less than or equal to 59%, then corresponding score value is 0.

S504: searching the first score value corresponding with the accuracy rate for having marked sample set in the corresponding relationship, makees For the corresponding integrated value of mark person's identity information.

In above-mentioned corresponding relationship, lookup is described to have marked score value corresponding to the accuracy rate of sample set, is denoted as first Score value, then using the first score value found as the corresponding integrated value of mark person's identity information.For example, described marked sample The accuracy rate of this collection is 88%, then searches the range of the accuracy rate belonging accuracy rate in table 1 above, i.e., [80%, 94%] within the scope of accuracy rate, then the score value within the scope of the accuracy rate is 1.5, then can determine and mark the accurate of sample set The corresponding score value of rate is 1.5, i.e. the first score value is 1.5.Then, it is used as the corresponding integrated value of mark person's identity information by 1.5, That is this mark person has marked sample set by this, integral 1.5 is obtained.

In addition, one marked in sample set can multiple tasks, then can will mark sample according to multiple tasks This collection be divided into it is multiple marked sample set, then, then determine and each marked the sub- accuracy rate of sample set, then root again Pair for each having marked sample set corresponding accuracy rate and score value is determined according to the classification of each having marked sample set of the task It should be related to, then according to the corresponding relationship for each having marked sample set corresponding accuracy rate and score value, determine that every height is quasi- All sub- score values, are then added by the true corresponding sub- score value of rate, have just obtained having marked the accuracy rate corresponding the of sample set One score value.

Furthermore, it is contemplated that when the sample size difference marked, the difficulty of mark is different, and difficulty is higher, The integrated value assigned should be higher, for example, two accuracys rate for having marked sample set are 80%, then one of them has been marked The quantity of all samples is the first quantity in sample set, and another is the second quantity, then if the first quantity is less than the second number Amount, then the corresponding score value for having marked sample set of the first quantity should be lower than score value corresponding to the second quantity, because quantity is got over It is more, it is more possible to cover the mark field that mark person is bad at.It then can also be according to the quantity tune for having marked sample in sample set Whole first score value obtained above, so as to adjust integrated value corresponding with mark person's identity information, specifically, the tool of S504 Body embodiment is as shown in fig. 6, may include: S5041 to S5044.

S5041: all quantity for having marked sample in sample set have been marked described in obtaining, have been denoted as total number of samples.

In some embodiments, it can be set a variable X, which described has marked institute in sample set for recording There is the quantity for having marked sample, specifically, obtains total number of samples, which is assigned to variable X.

S5042: the first score value corresponding with the accuracy rate for having marked sample set is searched in the corresponding relationship.

In addition, S5042 and S5041 execution sequence do not limit, can first in the corresponding relationship search with it is described Corresponding first score value of accuracy rate of sample set is marked, then obtains and described has marked all numbers for having marked sample in sample set Amount, is denoted as total number of samples.

S5043: first score value is adjusted to obtain the second score value according to the total number of samples, wherein for same How described first score value, the total number of samples the second score value more be higher.

In some embodiments, it is previously provided with the additional corresponding relationship of total number of samples and line bonus, the additional corresponding pass It include that multiple total number of samples and the corresponding line bonus of each total number of samples specifically add corresponding relationship at this in system It inside may include the corresponding line bonus in multiple total number of samples sections and each total number of samples section, it specifically, can be such as 2 institute of table Show:

Table 2

Wherein, the inclusion relation of numerical value and the meaning of mission bit stream represented by the interval range in table 2 can refer to The description in table 1 is stated, details are not described herein.Wherein, [800, ∞) indicate 800 and 800 or more.Then getting described marked After the corresponding total number of samples of sample set, determine which section the total number of samples belongs in table 2, for example, the total number of samples is 1000, then belong in the interval range of [800, ∞], it is corresponding additional to be divided into 0.6.Then in table 2, total number of samples is higher, and institute is right The line bonus answered is higher, i.e., the described total number of samples for having marked sample set is higher, and corresponding line bonus is higher.

Then adjust first score value according to the total number of samples may is that with the specific embodiment for obtaining the second score value The additional corresponding relationship of total number of samples and line bonus is obtained, includes multiple total number of samples and each institute in the additional corresponding relationship State the corresponding line bonus of total number of samples；Searched in the additional corresponding relationship marked sample set total number of samples it is corresponding attached Bonus point.Specifically, in the embodiment party by adding the determining corresponding line bonus of total number of samples for having marked sample set of corresponding relationship In formula, it can judge whether the total number of samples is greater than before executing the additional corresponding relationship for obtaining total number of samples and line bonus Specified threshold, if it is greater, then the step of executing the additional corresponding relationship for obtaining total number of samples and line bonus and subsequent step Suddenly, otherwise, directly using the first score value determined in S5042 as the corresponding integrated value of mark person's identity information.

In other implementations, first score value is adjusted to obtain the specific reality of the second score value according to the total number of samples The mode of applying, which may is that, judges whether the total number of samples is greater than specified threshold, if it is greater, then obtaining total number of samples and specified threshold Difference, and determine the ratio of the difference and specified threshold, line bonus be arranged according to the ratio.For example, it may be will Ratio obtains line bonus multiplied by constant.Wherein, specified threshold can rule of thumb be set, and can also be counted week time In phase, the corresponding total number of samples of all mark person's identity informations determines specified threshold according to multiple total number of samples, for example, can be with Then the average value for obtaining all total number of samples further according to the specified threshold and has been marked using the average value as specified threshold The corresponding total number of samples of sample set determines that this has marked the corresponding line bonus of sample set.Wherein, described to have marked the total of sample set Sample number is higher, and corresponding line bonus is higher.

And if the total number of samples for having marked sample set is less than or equal to specified threshold, it is arranged and has marked sample set pair That answers additional is divided into 0.

Then, it adjusts the first score value further according to obtained line bonus to obtain the second score value, specifically, can be acquisition the The sum of one score value and line bonus, the sum of first score value and line bonus are used as the second score value, specifically, then if currently described It is 1.2 that the first score value corresponding with the accuracy rate for having marked sample set is searched in corresponding relationship, and what is obtained additional is divided into 0.6, then 0.6 is added with 1.2, obtained score value is 1.8, i.e., second score value is 1.8, that is to say, mark person's identity information pair The integrated value answered is 1.8, then not only the high integrated value for having marked sample set of accuracy rate is relatively high, but also while accuracy rate is high Marked the total number of samples of sample set it is also high in the case where, obtained integrated value is higher.

S5044: using second score value as the corresponding integrated value of mark person's identity information.

Then in some embodiments, the acquisition modes of the corresponding integrated value of mark person's identity information may is that

J=b+c,

Wherein, J be the corresponding integrated value of mark person's identity information, b be in the corresponding relationship search with it is described Corresponding first score value of accuracy rate of sample set is marked, c is the line bonus determined according to the total number of samples, the then acquisition of c Mode can be obtained using above-mentioned additional corresponding relationship.

In further embodiments, the acquisition modes of the corresponding integrated value of mark person's identity information may is that

Wherein, J is the corresponding integrated value of mark person's identity information, and d1 is above-mentioned specified threshold, and X is to have marked sample The total number of samples of collection, b are that the first score value corresponding with the accuracy rate for having marked sample set is searched in the corresponding relationship, Then when X is less than or equal to d1, b is assigned a value of J, then when X is greater than d1, the value of (1+ (X-d1)/d1) is obtained, by this Value is assigned to J.

That is, just additionally increase score value in the case where the total number of samples for having marked sample set is greater than specified threshold, And in the case where the total number of samples for having marked sample set is less than or equal to specified threshold, do not increase score value additionally.

It, can also be according to having marked sample set in addition, other than it can determine integrated value according to above-mentioned task category Corresponding degree-of-difficulty factor and determine, then the degree-of-difficulty factor has reacted the difficulty that sample to be marked is marked, then can be according to difficulty Coefficient is higher, then the higher principle of the integrated value obtained is set according to the degree-of-difficulty factor and the accuracy rate for having marked sample set Set integrated value corresponding with mark person's identity information.Specifically, referring to Fig. 7, this method comprises:

S701: acquisition has marked sample set, and the sample set that marked corresponds to mark person's identity information, described to have marked sample This collection has marked sample including multiple.

S702: determine that the accuracy rate for having marked sample set, the accuracy rate have marked sample set for indicating described In marked the accuracy of sample.

S703: the corresponding degree-of-difficulty factor of sample set has been marked described in determining.

The degree-of-difficulty factor has reacted the difficulty that sample to be marked is marked, then degree-of-difficulty factor is higher, then the integrated value obtained Should be higher, specifically, which can be when distributing mark task for mark person, with sample to be marked It is sent in the tagging equipment of mark person, for example, including multiple sample, task description information and difficulties to be marked in the mark task Spend coefficient, wherein the degree-of-difficulty factor can be numerical value, then the numerical value is higher shows that degree-of-difficulty factor is bigger.The then degree-of-difficulty factor Be arranged can the time according to required for the test-object sample of test-object personnel and set, for example, it may be before distributing the task, First the sample to be marked in the task is marked by test-object personnel, time span spent by test-object personnel is determined, is denoted as examination Mark time span.The test-object time span is obtained, degree-of-difficulty factor is set according to the test-object time span, i.e. test-object time span is got over Long, then degree-of-difficulty factor is higher.

In addition, the degree-of-difficulty factor can also be determined according to the object and task description information in sample, specifically, The task description information may include multiple labels, as shown in figure 8, the task description information is " to mark the female of long hair It is raw ", then it include the first label 801 and the second label 802 in the task description information, then the corresponding text of the first label 801 For " long hair ", the corresponding text of the second label 802 is " schoolgirl ", then the corresponding mark condition of each label, for example, the One tag representation needs to find the target of long hair from multiple samples, and specifically, the sample in the application can be picture, then The corresponding mark demand of first label is that the target of long hair is found in picture, for example, the people of long hair.And the second label list Show the people for finding that gender is women in picture.

It then determines that the specific embodiment for having marked the corresponding degree-of-difficulty factor of sample set can be and determines described marked Infuse all labels corresponding to sample set, each corresponding mark condition of the label；Institute is determined based on all labels State degree-of-difficulty factor.

Then in some embodiments, after getting multiple labels, degree-of-difficulty factor can be determined according to the quantity of label, For example, the label the more, indicate that difficulty is bigger, then the basic difficulty value that can define each label is e, then number of labels It is exactly degree-of-difficulty factor multiplied by e.

In further embodiments, degree-of-difficulty factor is also determined with the classification according to label, specifically, different classifications indicates The type of mark demand is different, for example, long hair belongs to hair style classification, and schoolgirl inputs gender class, then looks for from different sexes It finds long hair from different hair styles to women ratio to be easier, because the sex character of common people is obvious, and the length of hair style Compare difficult determination, be on the one hand that can be blocked, long hair on the other hand is calculated for hair style how long, for example, the hair style of the neat cheek calculates length Returning is bob, can there is larger dispute.

The classification for then determining each label determines the basic difficulty value of each label further according to the classification of each label, then The basic difficulty value for dropping all labels is added, and has been marked degree-of-difficulty factor corresponding to sample set.And the type institute of label Corresponding basic difficulty value can be obtained according to the corresponding relationship of preset label classification and difficulty value.

S704: integral corresponding with mark person's identity information is arranged according to the degree-of-difficulty factor and the accuracy rate Value.

As an implementation, the corresponding relationship of multiple accuracys rate and score value, and each corresponding relationship are preset with A corresponding degree-of-difficulty factor, and include multiple accuracys rate and the corresponding score value of each accuracy rate in each corresponding relationship, Then after getting degree-of-difficulty factor, in the corresponding relationship of multiple accuracys rate and score value, search corresponding with the degree-of-difficulty factor Corresponding relationship, then, then in the corresponding relationship, search the corresponding score value of accuracy rate as with mark person's identity information Corresponding integrated value.Wherein, in the case where same accuracy rate, degree-of-difficulty factor is higher, and the corresponding score value of the accuracy rate is higher.

As another embodiment, it can also be and determine corresponding first score value of the accuracy rate；According to the difficulty First score value described in coefficient adjustment is to obtain third score value, wherein is directed to same first score value, the bigger institute of degree-of-difficulty factor It is higher to state the second score value；Using the third score value as the corresponding integrated value of mark person's identity information.

Then determine that the specific embodiment of corresponding first score value of the accuracy rate can be with reference to the embodiment party in earlier figures 5 Formula, details are not described herein.And after getting the first score value, the first score value is adjusted according to the degree-of-difficulty factor, it specifically, can be with It is by degree-of-difficulty factor multiplied by the first score value, using resulting product as third score value, for example, the first score value is 1.2, and difficulty system Number is 1.6, then the third score value obtained is 1.9.Furthermore it is also possible to be the corresponding value of difficulty of the determining degree-of-difficulty factor, obtain The value of difficulty and the first score value and, will be required and as third score value, for example, the corresponding score value of degree-of-difficulty factor is 1.6, And the first score value is 1.2, then the third score value obtained is 2.8.Wherein, the corresponding value of difficulty of the degree-of-difficulty factor can be basis Preset degree-of-difficulty factor and the difficulty of value of difficulty are divided corresponding relationship and are determined, specifically, which divides in corresponding relationship Including multiple degree-of-difficulty factors and the corresponding value of difficulty of each degree-of-difficulty factor, then divide lookup in corresponding relationship described in the difficulty Mark value of difficulty corresponding to the corresponding degree-of-difficulty factor of sample set.

Furthermore it is also possible to determine mark person's identity jointly according to above-mentioned degree-of-difficulty factor, total number of samples and accuracy rate The corresponding integrated value of information can be as an implementation and determine the first score value based on the accuracy rate, further according to described Total number of samples adjusts first score value to obtain the second score value, adjusts the second score value further according to degree-of-difficulty factor and obtains third point Value, then this can be referred to above-mentioned according to the degree-of-difficulty factor according to the mode that degree-of-difficulty factor the second score value of adjustment obtains third score value First score value is adjusted to obtain the embodiment of third score value, then using third score value adjusted as the mark person The corresponding integrated value of identity information.

For example, accuracy rate 80%, then score 1.5 is divided if 1000 pictures, i.e. the first score value is 1.5 points, then, root According to the total number of samples, i.e., 1000, the second score value is obtained to 1.5 points of adjustment, i.e., (1+ (1000-700)/1000)=1.9, then, The second score value is adjusted further according to degree-of-difficulty factor, for example, such as adjustment mode is 1.9*1.5, i.e., degree-of-difficulty factor is 1.5, then The third score value arrived is 2.85, then finally obtained integrated value is 2.85.

And as another embodiment, first the first score value can be adjusted according to degree-of-difficulty factor to obtain third score value, it will The third score value adjusts the first new score value as the first new score value, further according to total number of samples, obtains the second score value, by the Two score values can refer to previous embodiment as the integrated value, specific normal form of implementing, and details are not described herein.

Furthermore the above-mentioned basis referred to is determined in the specific embodiment of the degree-of-difficulty factor based on all labels, It is also conceivable to the classification situation of label, specifically, referring to Fig. 9, showing a kind of markup information processing provided by the present application Method, this method comprises: S901 to S908.

S901: acquisition has marked sample set, and the sample set that marked corresponds to mark person's identity information, described to have marked sample This collection has marked sample including multiple.

S902: determine that the accuracy rate for having marked sample set, the accuracy rate have marked sample set for indicating described In marked the accuracy of sample.

S903: all labels corresponding to sample set have been marked described in determining.

S904: the major category of each label is determined.

As an implementation, it can be during above-mentioned test-object or when distributing mark task, for this The multiple labels of task setting are marked, specifically, can be embodied by marking the task description information of task, for example, above-mentioned " label The first label " long hair " and the second label " schoolgirl " in the schoolgirl of long hair out ".

Furthermore a label classification can be preset, and is corresponding with subtab under label, and the classification of subtab is Subclass specifically can preset and record, and when executing this method, mark is got by detection node and is appointed Multiple labels of business, and search in pre-set label and the corresponding relationship of classification the corresponding major category of the label.

Wherein, definition has marked all labels corresponding to sample set, and the corresponding classification of each label is subclass, and is somebody's turn to do A upper classification for subclass is major category, for example, the first label is long hair, then the corresponding subclass of the first label is length Hair, and the corresponding major category of the first label is hair style, then " hair style " is a upper classification of " long hair ", then the major category Under be also possible that multiple subclass such as bob, plank inch, bald.

S905: determine and described marked the corresponding object of each label in sample set, wherein the label with should The corresponding object of label belongs to the same major category.

After the major category for getting each label, searched and each mark respectively in described marked in sample set Label belong to the object of the same major category.For example, the major category is hair style, and the sample is picture, then will mark sample The hair style in every picture is concentrated to mark, to obtain all objects for belonging to hair style classification.It specifically, can be on picture The object found is marked.

S906: the sub- degree-of-difficulty factor of the label is determined according to object corresponding to each label.

The attribute dimensions of the mark task can be reflected by having marked the corresponding object of each label in sample set, tool Body, the attribute dimensions may include personage's gender, personage nationality, scene complexity etc.) attribute is more, and degree-of-difficulty factor is bigger.Tool Body, the attribute dimensions are related to major category, then the quantity of the different major categories in all labels is more, then attribute dimensions are got over It is high.

Then after getting object, then mark person is when marking sample to be marked according to label, the target Object can be used as chaff interferent, that is, be likely to affect the annotation results of mark person so that mistake using the object as with label Belong to the object of the same subclass, for example, the wrong person's of the being marked mark as long hair of the bob meeting in object, and make Accuracy rate reduces.

Then the object according to corresponding to each label determines that the mode of the sub- degree-of-difficulty factor of the label can be system The quantity of the corresponding object of each label is counted, then the bright interference of quantity more multilist is bigger, then degree-of-difficulty factor is higher, thus according to mesh The chaff interferent quantity of mark object determines the corresponding sub- degree-of-difficulty factor of each label, specifically, is previously provided with chaff interferent quantity and oneself The corresponding relationship of degree-of-difficulty factor is searched in the corresponding relationship according to the corresponding relationship, the number of the corresponding object of each label The corresponding sub- dimension coefficient of amount.

In addition, in further embodiments, it, can also be to the target after getting the corresponding object of each label Subclass described in object is sorted out, so that the quantity of the subclass in the corresponding all objects of each label is counted, then It is that sub- degree-of-difficulty factor is arranged in the label according to the quantity of the subclass, specifically, it is determined that the corresponding object of each label Subclass, wherein the subclass of the object and the subclass of the label belong to the corresponding major category of the label；It obtains Take the classification quantity of the subclass of the corresponding object of each label；The sub- difficulty system is determined according to the classification quantity Number, wherein how described the classification quantity sub- degree-of-difficulty factor more be higher.

By taking above-mentioned first label as an example, the object for belonging to hair style classification is searched in sample set in all marked, as A kind of embodiment, the sample are picture, then the object of input hair style is searched in having marked each picture in sample set, And the subclass of each object is counted, that is, belongs to the classifications such as long hair or bob, plank inch.Then, all objects are counted The classification quantity for the subclass for being included, then the classification quantity can be understood as the type of all subclass, specifically, if should Classification described in the corresponding object of first label includes long hair and bob, then point of the subclass of the corresponding object of the label Class quantity is 2.

Then the corresponding sub- degree-of-difficulty factor of the classification quantity is obtained again, it specifically, can be according to preset classification number It measures the corresponding relationship with sub- degree-of-difficulty factor and determines, it is of course also possible to be one sub- first foundation value of difficulty of setting, then son is difficult It spends coefficient and is equal to classification quantity multiplied by first foundation value of difficulty, then the classification quantity is more, and sub- degree-of-difficulty factor is higher.

S907: it is obtained described having marked the corresponding degree-of-difficulty factor of sample set according to the sub- degree-of-difficulty factor of all labels.

Then the sub- dimension coefficient of each label is successively obtained according to the embodiment of S906, then by all sub- dimension systems Number is added, and obtains described having marked the corresponding degree-of-difficulty factor of sample set.

For example, this has marked corresponding two labels of mark task of sample set, respectively the first label and the second label, then The corresponding first sub- degree-of-difficulty factor of the first label and the corresponding second sub- degree-of-difficulty factor of the second label are obtained, the first son is difficult It spends coefficient and the second sub- degree-of-difficulty factor is summed, the result after summing has marked the corresponding degree-of-difficulty factor of sample set as described.

S908: integral corresponding with mark person's identity information is arranged according to the degree-of-difficulty factor and the accuracy rate Value.

Furthermore it is also possible to degree-of-difficulty factor be determined according to the rank of each label, specifically, referring to Fig. 10, this method packet It includes: S1001 to S1007.

S1001: acquisition has marked sample set, and the sample set that marked corresponds to mark person's identity information, described to have marked Sample set includes multiple having marked sample.

S1002: determine that the accuracy rate for having marked sample set, the accuracy rate have marked sample set for indicating described In marked the accuracy of sample.

S1003: all labels corresponding to sample set have been marked described in determining.

S1004: the number of levels that each label is preset is determined.

Wherein, which be classified for preset one, and specifically, which can be as shown in table 3 below:

Table 3

Wherein, above-mentioned label corresponding level form can be obtains in advance, has been marked according to level form lookup The rank of the corresponding each label of sample set, for example, the first label is long hair, then first label corresponds to third level, i.e., The number of levels of first label is 3.

S1005: the sub- degree-of-difficulty factor of the label is determined based on the number of levels of each label, wherein the number of levels How described the sub- degree-of-difficulty factor more be bigger.

Specifically, it is determined that the corresponding sub- degree-of-difficulty factor of the number of levels is previously provided with number of levels as an implementation With the corresponding relationship of sub- degree-of-difficulty factor, is then searched in the corresponding relationship of the number of levels and sub- degree-of-difficulty factor and marked sample set The corresponding sub- degree-of-difficulty factor of the number of levels of corresponding each label.As another embodiment, the number of levels of label is corresponding Sub- degree-of-difficulty factor calculation can be the product for obtaining number of levels with the second basic difficulty point, using the product as the grade of label Corresponding sub- degree-of-difficulty factor is not counted, and therefore, the number of levels of label is bigger, then corresponding sub- degree-of-difficulty factor is higher.

Then the number of levels of the label can reflect the difficulty gone out on missions, for example, the label is third level, then in mark When, mark person needs first to determine macroscopic features in picture, i.e. the corresponding feature of first level exists, then again from macroscopic features Middle determining hair style feature further searches for second level characteristic point in the corresponding characteristic point of first level, then again from hair style Middle lookup long hair, therefore, substantially more, the content for needing to further search for is more, therefore, bigger by setting number of levels, then Corresponding sub- degree-of-difficulty factor is higher, and when so that mark person completing the mark task of the higher label of rank, obtained score is more It is high.

S1006: it is obtained described having marked the corresponding degree-of-difficulty factor of sample set according to the sub- degree-of-difficulty factor of all labels.

Furthermore it is also possible to according to the number of levels of label and the common difficulty in computation coefficient of attribute, specifically, then according to all marks The sub- degree-of-difficulty factor of label, which obtains the specific embodiment for having marked the corresponding degree-of-difficulty factor of sample set, may also is that according to every The major category of a label determines that the first sub- degree-of-difficulty factor of each label, the number of levels according to each label are true Second sub- degree-of-difficulty factor of fixed each label, the first of each label the sub- degree-of-difficulty factor is added with the second sub- degree-of-difficulty factor The sub- degree-of-difficulty factor of third is obtained, then which is executed as the sub- degree-of-difficulty factor of the label according to institute again from degree-of-difficulty factor There is the sub- degree-of-difficulty factor of label to obtain the operation for having marked the corresponding degree-of-difficulty factor of sample set, wherein according to each described The major category of label determines the first sub- degree-of-difficulty factor of each label and is determined according to the number of levels of each label Second sub- degree-of-difficulty factor of each label can refer to embodiments described above.

S1007: integral corresponding with mark person's identity information is arranged according to the degree-of-difficulty factor and the accuracy rate Value.

Figure 11 is please referred to, it illustrates a kind of structural block diagrams of markup information processing unit provided by the embodiments of the present application should Device may include: acquiring unit 1101, determination unit 1102 and setting unit 1103.

Acquiring unit 1101 has marked sample set for obtaining, and the sample set that marked corresponds to mark person's identity information, The sample set that marked includes multiple having marked sample.

Determination unit 1102, for determining the accuracy rate for having marked sample set, the accuracy rate is for indicating described The accuracy that sample has been marked in sample set is marked.

Setting unit 1103, for integrated value corresponding with mark person's identity information to be arranged according to the accuracy rate.

Specifically, setting unit 1103 is also used to obtain the corresponding relationship of preset accuracy rate and score value, described right It include multiple accuracys rate and the corresponding score value of each accuracy rate in should being related to, and higher corresponding point of the accuracy rate It is worth higher；The first score value corresponding with the accuracy rate for having marked sample set is searched in the corresponding relationship, as described The corresponding integrated value of mark person's identity information.Further, setting unit 1103 described has marked institute in sample set for obtaining There is the quantity for having marked sample, is denoted as total number of samples；It is searched in the corresponding relationship and has marked the accurate of sample set with described Corresponding first score value of rate；First score value is adjusted to obtain the second score value according to the total number of samples, wherein for same The first score value, how described the total number of samples the second score value more be higher；Using second score value as mark person's identity The corresponding integrated value of information.

In addition, setting unit 1103 is also used to, determination is described to have marked the corresponding degree-of-difficulty factor of sample set；According to the difficulty It spends coefficient and integrated value corresponding with mark person's identity information is arranged in the accuracy rate.Further, setting unit 1103 The specific embodiment party of integrated value corresponding with mark person's identity information is set according to the degree-of-difficulty factor and the accuracy rate Formula are as follows: determine corresponding first score value of the accuracy rate；First score value is adjusted according to the degree-of-difficulty factor to obtain third Score value, wherein be directed to same first score value, more big second score value of the degree-of-difficulty factor is higher；By the third score value As the corresponding integrated value of mark person's identity information.

Further, setting unit 1103 determines the specific embodiment for having marked the corresponding degree-of-difficulty factor of sample set Are as follows: all labels corresponding to sample set, each corresponding mark condition of the label have been marked described in determining；Based on described All labels determine the degree-of-difficulty factor.Specifically, setting unit 1103 determines the major category of each label；Described in determination The corresponding object of each label in sample set is marked, wherein label object corresponding with the label belongs to The same major category；The sub- degree-of-difficulty factor of the label is determined according to object corresponding to each label；According to all marks The sub- degree-of-difficulty factor of label has marked the corresponding degree-of-difficulty factor of sample set described in obtaining.

Further, setting unit 1103 can determine the subclass of the corresponding object of each label, wherein described The subclass of object and the subclass of the label belong to the corresponding major category of the label；It is corresponding to obtain each label Object subclass classification quantity；The sub- degree-of-difficulty factor is determined according to the classification quantity, wherein the classification number It is higher to measure the more how described sub- degree-of-difficulty factor.Specifically, setting unit 1103 determines the rank that each label is preset Number；The sub- degree-of-difficulty factor of the label is determined based on the number of levels of each label, wherein how described the number of levels son more be difficult It is bigger to spend coefficient；It is obtained described having marked the corresponding degree-of-difficulty factor of sample set according to the sub- degree-of-difficulty factor of all labels.

It further, further include feedback unit, it is corresponding for the integrated value to be fed back to mark person's identity information Client.

It is apparent to those skilled in the art that for convenience and simplicity of description, foregoing description device and The specific work process of module, can refer to corresponding processes in the foregoing method embodiment, and details are not described herein.

In several embodiments provided herein, the mutual coupling of module can be electrical property, mechanical or other The coupling of form.

It, can also be in addition, can integrate in a processing module in each functional module in each embodiment of the application It is that modules physically exist alone, can also be integrated in two or more modules in a module.Above-mentioned integrated mould Block both can take the form of hardware realization, can also be realized in the form of software function module.

As shown in figure 12, it illustrates the structural block diagrams of a kind of electronic equipment provided by the embodiments of the present application.The electronics is set Standby 100, which can be smart phone, tablet computer, e-book etc., can run the electronic equipment of application program.Implement in the application In, which can be as above-mentioned detection service device 121, be also possible to annotation server 112.

Electronic equipment 100 in the application may include one or more such as lower component: processor 110, memory 120 with And one or more application program, wherein one or more application programs can be stored in memory 120 and be configured as It is executed by one or more processors 110, one or more programs are configured to carry out as described in preceding method embodiment Method.

Processor 110 may include one or more processing core.Processor 110 is whole using various interfaces and connection Various pieces in a electronic equipment 100, by run or execute the instruction being stored in memory 120, program, code set or Instruction set, and the data being stored in memory 120 are called, execute the various functions and processing data of electronic equipment 100.It can Selection of land, processor 110 can use Digital Signal Processing (Digital Signal Processing, DSP), field-programmable Gate array (Field-Programmable Gate Array, FPGA), programmable logic array (Programmable Logic Array, PLA) at least one of example, in hardware realize.Processor 110 can integrating central processor (Central Processing Unit, CPU), in image processor (Graphics Processing Unit, GPU) and modem etc. One or more of combinations.Wherein, the main processing operation system of CPU, user interface and application program etc.；GPU is for being responsible for Show the rendering and drafting of content；Modem is for handling wireless communication.It is understood that above-mentioned modem It can not be integrated into processor 110, be realized separately through one piece of communication chip.

Memory 120 may include random access memory (Random Access Memory, RAM), also may include read-only Memory (Read-Only Memory).Memory 120 can be used for store instruction, program, code, code set or instruction set.It deposits Reservoir 120 may include storing program area and storage data area, wherein the finger that storing program area can store for realizing operating system Enable, for realizing at least one function instruction (such as touch function, sound-playing function, image player function etc.), be used for Realize the instruction etc. of following each embodiments of the method.Storage data area can also store electronic equipment 100 and be created in use Data (such as phone directory, audio, video data, chat record data) etc..

Figure 13 is please referred to, it illustrates a kind of structural frames of computer readable storage medium provided by the embodiments of the present application Figure.Program code is stored in the computer readable storage medium 1300, said program code can be called by processor and be executed State method described in embodiment of the method.

Computer readable storage medium 1300 can be (the read-only storage of electrically erasable of such as flash memory, EEPROM Device), the electronic memory of EPROM, hard disk or ROM etc.Optionally, computer readable storage medium 1300 includes non-volatile Property computer-readable medium (non-transitory computer-readable storage medium).It is computer-readable Storage medium 1300 has the memory space for the program code 1310 for executing any method and step in the above method.These programs Code can read or be written to this one or more computer program from one or more computer program product In product.Program code 1310 can for example be compressed in a suitable form.

Finally, it should be noted that above embodiments are only to illustrate the technical solution of the application, rather than its limitations；Although The application is described in detail with reference to the foregoing embodiments, those skilled in the art are when understanding: it still can be with It modifies the technical solutions described in the foregoing embodiments or equivalent replacement of some of the technical features；And These are modified or replaceed, do not drive corresponding technical solution essence be detached from each embodiment technical solution of the application spirit and Range.

Claims (10)

1. a kind of markup information processing method characterized by comprising

Acquisition has marked sample set, and the sample set that marked corresponds to mark person's identity information, and the sample set that marked includes It is multiple to have marked sample；

Determine the accuracy rate for having marked sample set, the accuracy rate is for indicating that described marked has marked sample in sample set This accuracy；

According to the accuracy rate, integrated value corresponding with mark person's identity information is set.

2. the method according to claim 1, wherein described according to accuracy rate setting and mark person's body The corresponding integrated value of part information, comprising:

The corresponding relationship of preset accuracy rate and score value is obtained, the corresponding relationship is interior including multiple accuracys rate and each The corresponding score value of the accuracy rate, and the higher corresponding score value of the accuracy rate is higher；

The first score value corresponding with the accuracy rate for having marked sample set is searched in the corresponding relationship, as the mark The corresponding integrated value of member's identity information.

3. according to the method described in claim 2, it is characterized in that, described search in the corresponding relationship has marked with described The corresponding score value of the accuracy rate of sample set, as the corresponding integrated value of mark person's identity information, comprising:

All quantity for having marked sample in sample set have been marked described in obtaining, have been denoted as total number of samples；

The first score value corresponding with the accuracy rate for having marked sample set is searched in the corresponding relationship；

First score value is adjusted to obtain the second score value according to the total number of samples, wherein is directed to same first score value, institute It is higher to state more how described second score value of total number of samples；

Using second score value as the corresponding integrated value of mark person's identity information.

4. the method according to claim 1, wherein described according to accuracy rate setting and mark person's body The corresponding integrated value of part information, comprising:

The corresponding degree-of-difficulty factor of sample set has been marked described in determining；

According to the degree-of-difficulty factor and the accuracy rate, integrated value corresponding with mark person's identity information is set.

5. according to the method described in claim 4, it is characterized in that, having marked the corresponding difficulty system of sample set described in the determination Number, comprising:

All labels corresponding to sample set, each corresponding mark condition of the label have been marked described in determining；

The degree-of-difficulty factor is determined based on all labels.

6. according to the method described in claim 5, it is characterized in that, described determine the difficulty system based on all labels Number, comprising:

Determine the major category of each label；

It determines and described has marked the corresponding object of each label in sample set, wherein the label is corresponding with the label Object belong to the same major category；

The sub- degree-of-difficulty factor of the label is determined according to object corresponding to each label；

It is obtained described having marked the corresponding degree-of-difficulty factor of sample set according to the sub- degree-of-difficulty factor of all labels.

7. according to the method described in claim 6, the basis is every it is characterized in that, the major category includes multiple subclass Object corresponding to a label determines the sub- degree-of-difficulty factor of the label, comprising:

The subclass of the corresponding object of each label is determined, wherein the son of the subclass of the object and the label Classification belongs to the corresponding major category of the label；

Obtain the classification quantity of the subclass of the corresponding object of each label；

Determine the sub- degree-of-difficulty factor according to the classification quantity, wherein the more how described sub- degree-of-difficulty factor of the classification quantity more It is high.

8. according to the method described in claim 5, it is characterized in that, described determine the difficulty system based on all labels Number, comprising:

Determine the number of levels that each label is preset；

The sub- degree-of-difficulty factor of the label is determined based on the number of levels of each label, wherein the more how described son of the number of levels Degree-of-difficulty factor is bigger；

It is obtained described having marked the corresponding degree-of-difficulty factor of sample set according to the sub- degree-of-difficulty factor of all labels.

9. a kind of markup information processing unit characterized by comprising

Acquiring unit has marked sample set for obtaining, and the sample set that marked corresponds to mark person's identity information, described to have marked Note sample set includes multiple having marked sample；

Determination unit, for determining that the accuracy rate for having marked sample set, the accuracy rate have marked sample for indicating described This concentration has marked the accuracy of sample；

Setting unit, for integrated value corresponding with mark person's identity information to be arranged according to the accuracy rate.

10. a kind of electronic equipment characterized by comprising

One or more processors；

Memory；

One or more application program, wherein one or more of application programs are stored in the memory and are configured To be executed by one or more of processors, one or more of programs are configured to carry out as claim 1-8 is any Method described in.