CN111444332A - Crowdsourcing worker reliability model establishing method and device under crowdsourcing knowledge verification environment - Google Patents
Crowdsourcing worker reliability model establishing method and device under crowdsourcing knowledge verification environment Download PDFInfo
- Publication number
- CN111444332A CN111444332A CN202010179259.2A CN202010179259A CN111444332A CN 111444332 A CN111444332 A CN 111444332A CN 202010179259 A CN202010179259 A CN 202010179259A CN 111444332 A CN111444332 A CN 111444332A
- Authority
- CN
- China
- Prior art keywords
- knowledge
- crowdsourcing
- users
- qualification
- reliability model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/335—Filtering based on additional data, e.g. user or group profiles
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/36—Creation of semantic tools, e.g. ontology or thesauri
- G06F16/367—Ontology
Abstract
The invention discloses a method and a device for establishing a crowdsourcing worker reliability model in a crowdsourcing knowledge verification environment. The method comprises the following steps: matching a knowledge field set for crowdsourcing users according to a pre-stored strategy, and distributing knowledge in the knowledge field set to the crowdsourcing users to enable the crowdsourcing users to verify the knowledge to obtain knowledge labels; establishing a crowdsourcing worker reliability model based on a reinforcement learning algorithm, and calculating the reward value of the knowledge label through the crowdsourcing worker reliability model so as to update a pre-stored strategy according to the reward value; repeatedly executing the operation until the updating times of the pre-stored strategy reach the preset times, and screening the qualification of the crowdsourcing users according to the latest pre-stored strategy; and after the knowledge verification is completed, adding the verified correct knowledge into the corresponding knowledge graph. The invention can establish a crowdsourcing worker reliability model under a crowdsourcing verification environment based on reinforcement learning, and realizes qualification screening of crowdsourcing workers, thereby improving the efficiency of crowdsourcing knowledge verification.
Description
Technical Field
The invention relates to the technical field of knowledge maps, in particular to a method and a device for establishing a reliability model of crowdsourcing workers in a crowdsourcing knowledge verification environment.
Background
The knowledge graph was proposed by Google corporation (Google), and in order to improve the retrieval efficiency of users, computers can understand massive information like people, and the research and application of the knowledge graph are widely concerned by academic and industrial fields. Since the knowledge graph is a dynamically constructed process, newly added knowledge needs to be continuously verified. Currently, knowledge verification is mainly performed through an online crowdsourcing service, namely, knowledge is delivered to a crowd-sourcing worker, so that a plurality of crowd-sourcing workers verify the knowledge to obtain knowledge tags. The accuracy of knowledge verification depends greatly on the built crowdsourced worker reliability model.
The typical method for establishing the reliability model of crowdsourcing workers is to model a crowdsourcing environment into a Markov Decision Process (MDP) of a limited time domain, use a dynamic programming algorithm (DP) to depict an optimal strategy, provide an effective approximate strategy method for calculating the optimal strategy for selecting instance verification, namely an optimistic knowledge gradient (hereinafter, abbreviated as Opt-KG), calculate the optimal strategy by an Opt-KG method, and then select an instance to verify according to the strategy to obtain a better strategy. When the Opt-KG method is applied to a large-scale knowledge graph, example-worker pairs cannot be freely selected, reliability of crowdsourcing workers can only be evaluated, and qualification of the crowdsourcing workers cannot be screened and updated, so that the crowdsourcing workers can control the proportion of true labels on knowledge to keep the reliability of the knowledge, and false knowledge labels are introduced in a knowledge verification process. In addition, different crowdsourcing workers have different mastery levels and different work attentiveness degrees in different knowledge fields, and the Opt-KG method only depends on the difficulty of labeling knowledge and is not enough to determine the reliability of the crowdsourcing workers.
Disclosure of Invention
The invention provides a method and a device for establishing a crowdsourcing worker reliability model in a crowdsourcing knowledge verification environment, which are used for overcoming the defects in the prior art.
In order to solve the above technical problem, in a first aspect, an embodiment of the present invention provides a method for establishing a reliability model of crowdsourcing workers in a crowdsourcing knowledge verification environment, including:
matching a knowledge field set for crowdsourcing users according to a pre-stored strategy, and distributing knowledge in the knowledge field set to the crowdsourcing users to enable the crowdsourcing users to verify the knowledge to obtain knowledge labels;
establishing a crowdsourcing worker reliability model based on a reinforcement learning algorithm, and calculating the reward value of the knowledge label through the crowdsourcing worker reliability model so as to update the pre-stored strategy according to the reward value;
repeatedly executing the operations until the updating times of the pre-stored strategy reach preset times, and screening the qualification of the crowdsourcing user according to the latest pre-stored strategy;
and after the knowledge verification is completed, adding the correctly verified knowledge into the corresponding knowledge graph.
Further, before the matching a knowledge field set to a crowdsourcing user according to a pre-stored policy and distributing knowledge in the knowledge field set to the crowdsourcing user to enable the crowdsourcing user to verify the knowledge and obtain a knowledge tag, the method further includes:
and according to the input information of the user, performing identity authentication and qualification verification on the user, and determining the user as the crowdsourcing user when the user passes the identity authentication and qualification verification.
Further, the calculating, by the crowd-sourcing worker reliability model, the reward value of the knowledge label specifically includes:
if the knowledge is verified knowledge, calculating the reward value of the knowledge tag according to the real tag of the knowledge;
and if the knowledge is unverified knowledge, calculating the reward value of the knowledge tag according to the historical tag of the knowledge.
Further, the screening qualification of the crowdsourced users according to the latest pre-stored policy specifically includes:
judging whether the adequacy degree of the crowdsourcing users to the knowledge field set reaches a preset threshold value or not according to the latest pre-stored strategy;
if the degree of adequacy of the crowdsourcing users on the knowledge domain set reaches the preset threshold value, judging that the crowdsourcing users have crowdsourcing qualification;
and if the adequacy degree of the crowdsourcing users to the knowledge field set does not reach the preset threshold value, judging that the crowdsourcing users do not have crowdsourcing qualification.
In a second aspect, an embodiment of the present invention provides a device for establishing a reliability model of crowdsourcing workers in a crowdsourcing knowledge verification environment, including:
the knowledge verification module is used for matching a knowledge field set to crowdsourced users according to a pre-stored strategy, and distributing knowledge in the knowledge field set to the crowdsourced users to enable the crowdsourced users to verify the knowledge to obtain knowledge labels;
the strategy updating module is used for establishing a crowdsourcing worker reliability model based on a reinforcement learning algorithm, calculating the reward value of the knowledge label through the crowdsourcing worker reliability model and updating the pre-stored strategy according to the reward value;
the qualification screening module is used for repeatedly executing the knowledge verification module and the strategy updating module until the updating times of the pre-stored strategies reach preset times, and screening the qualification of the crowdsourced users according to the latest pre-stored strategies;
and the knowledge adding module is used for adding the correctly verified knowledge into the corresponding knowledge map after the knowledge verification is finished.
Further, the knowledge verification module is further configured to, before matching a knowledge field set with a crowdsourcing user according to a pre-stored policy and allocating knowledge in the knowledge field set to the crowdsourcing user to enable the crowdsourcing user to verify the knowledge and obtain a knowledge tag, perform identity authentication and qualification verification on the user according to input information of the user, and determine the user as the crowdsourcing user when the user passes the identity authentication and qualification verification.
Further, the calculating, by the crowd-sourcing worker reliability model, the reward value of the knowledge label specifically includes:
if the knowledge is verified knowledge, calculating the reward value of the knowledge tag according to the real tag of the knowledge;
and if the knowledge is unverified knowledge, calculating the reward value of the knowledge tag according to the historical tag of the knowledge.
Further, the screening qualification of the crowdsourced users according to the latest pre-stored policy specifically includes:
judging whether the adequacy degree of the crowdsourcing users to the knowledge field set reaches a preset threshold value or not according to the latest pre-stored strategy;
if the degree of adequacy of the crowdsourcing users on the knowledge domain set reaches the preset threshold value, judging that the crowdsourcing users have crowdsourcing qualification;
and if the adequacy degree of the crowdsourcing users to the knowledge field set does not reach the preset threshold value, judging that the crowdsourcing users do not have crowdsourcing qualification.
The embodiment of the invention has the following beneficial effects:
according to a pre-stored strategy, matching a knowledge field set for crowdsourcing users, distributing knowledge in the knowledge field set to the crowdsourcing users, enabling the crowdsourcing users to verify the knowledge to obtain knowledge labels, further establishing a crowdsourcing worker reliability model based on a reinforcement learning algorithm, calculating reward values of the knowledge labels through the crowdsourcing worker reliability model, updating the pre-stored strategy according to the reward values, repeatedly executing the operation until the updating times of the pre-stored strategy reach preset times, screening the qualification of the crowdsourcing users according to the latest pre-stored strategy, and adding the knowledge which is verified to be correct into the corresponding knowledge map after the knowledge verification is completed. Compared with the prior art, the method and the device have the advantages that knowledge in the knowledge field set and the knowledge field set can be freely matched for the crowdsourcing users through the pre-storage strategy, the crowdsourcing worker reliability model is established based on the strengthening algorithm, the pre-storage strategy can be updated by the aid of the calculated reward values, qualification screening of the crowdsourcing users is achieved, and therefore efficiency of crowdsourcing knowledge verification is improved.
Drawings
Fig. 1 is a schematic flow chart illustrating a method for establishing a reliability model of crowdsourcing workers in a crowdsourcing knowledge verification environment according to a first embodiment of the present invention;
fig. 2 is a schematic structural diagram of a crowdsourcing worker reliability model establishing device in a crowdsourcing knowledge verification environment according to a second embodiment of the present invention.
Detailed Description
The technical solutions in the present invention will be described clearly and completely with reference to the accompanying drawings, and it is obvious that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that, the step numbers in the text are only for convenience of explanation of the specific embodiments, and do not serve to limit the execution sequence of the steps. The method provided by the embodiment can be executed by the relevant server, and the server is taken as an example for explanation below.
As shown in fig. 1, a first embodiment provides a method for establishing a reliability model of crowdsourced workers in a crowdsourced knowledge verification environment, which is characterized by comprising steps S1 to S4:
s1, matching a knowledge field set for crowdsourcing users according to a pre-stored strategy, and distributing knowledge in the knowledge field set to the crowdsourcing users to enable the crowdsourcing users to verify the knowledge to obtain knowledge labels;
s2, establishing a crowdsourcing worker reliability model based on a reinforcement learning algorithm, and calculating the reward value of the knowledge label through the crowdsourcing worker reliability model so as to update a pre-stored strategy according to the reward value;
s3, repeatedly executing the operations until the updating times of the pre-stored strategy reach the preset times, and screening the qualification of the crowdsourced users according to the latest pre-stored strategy;
and S4, adding the correct knowledge into the corresponding knowledge graph after the knowledge verification is completed.
It should be noted that the pre-storage strategy is a crowd-sourced worker adept knowledge field model obtained based on reinforcement learning algorithm training.
In step S1, the knowledge base of the knowledge graph is divided into a plurality of knowledge domains, a knowledge domain adept by crowdsourcing workers trained based on the reinforcement learning algorithm is used as a pre-stored policy, a knowledge domain set is matched to crowdsourcing users from the knowledge graph according to the pre-stored policy, and knowledge in the knowledge domain set is distributed to the crowdsourcing users, so that the crowdsourcing users verify knowledge to obtain knowledge labels, and thus the knowledge domain set can be freely matched to the crowdsourcing users and the knowledge in the knowledge domain set can be distributed.
In step S2, the knowledge verification process is established as a markov decision process, a crowdsourcing worker reliability model is established based on a reinforcement learning algorithm, the knowledge labels are compared with the real labels or history labels corresponding to the knowledge through the crowdsourcing worker reliability model, and the reward values of the knowledge labels are calculated to update the pre-stored policy according to the reward values, so that a more appropriate knowledge field set is matched for the crowdsourcing user next time.
In step S3, the steps S1 to S2 are repeatedly executed until the update times of the pre-stored policies reach the preset times, whether the crowdsourcing users have crowdsourcing qualification is determined according to the latest pre-stored policies, and qualification screening is performed on the crowdsourcing users, so that the crowdsourcing users who do not have crowdsourcing qualification can be prevented from continuing to verify knowledge, the accuracy of knowledge verification is reduced, and the efficiency of the crowdsourcing knowledge verification is improved.
In step S4, after completion of knowledge verification, verified correct knowledge is added to the corresponding knowledge map, so that the knowledge map can be updated.
According to a pre-stored strategy, matching a knowledge field set for crowdsourcing users, distributing knowledge in the knowledge field set to the crowdsourcing users, enabling the crowdsourcing users to verify the knowledge to obtain knowledge labels, further establishing a crowdsourcing worker reliability model based on a reinforcement learning algorithm, calculating reward values of the knowledge labels through the crowdsourcing worker reliability model, updating the pre-stored strategy according to the reward values, repeatedly executing the operation until the updating times of the pre-stored strategy reach preset times, screening the qualification of the crowdsourcing users according to the latest pre-stored strategy, and adding the knowledge which is verified to be correct into the corresponding knowledge map after the knowledge verification is completed.
According to the embodiment, knowledge in the knowledge field set and the distributed knowledge field set can be freely matched for crowdsourcing users through the pre-storing strategy, the crowdsourcing worker reliability model is established based on the strengthening algorithm, the pre-storing strategy can be updated by the aid of the calculated reward value, qualification screening of the crowdsourcing users is achieved, and therefore efficiency of crowdsourcing knowledge verification is improved.
In a preferred embodiment, before the matching a knowledge domain set to a crowdsourcing user according to a pre-stored policy, and distributing knowledge in the knowledge domain set to the crowdsourcing user, so that the crowdsourcing user verifies the knowledge and obtains a knowledge tag, the method further includes: and according to the input information of the user, performing identity authentication and qualification verification on the user, and determining the user as a crowdsourcing user when the user passes the identity authentication and qualification verification.
When a user uses the crowdsourcing platform for the first time, the user needs to input information such as names, nationalities, regions, academic calendars, contact ways, passwords, and skilled knowledge fields to complete user registration. When the registration is successful, the user can input a contact and a password to log in the crowdsourcing platform. Wherein the contact means comprises a telephone or a mailbox.
And performing identity authentication on the user according to input information of the user, such as a contact way and a password, if the contact way and the password of the user are correct and matched, determining that the user passes the identity authentication, otherwise, determining that the user does not pass the identity authentication. After the user passes the identity authentication, the qualification of the user is checked according to the input information of the user, such as the adequacy knowledge field, if the adequacy knowledge field of the user is the knowledge field required by the knowledge verification, the user is considered to pass the qualification check, otherwise, the user is considered to not pass the qualification check. And after the user passes the qualification audit, determining the user as the crowdsourcing user.
In a preferred embodiment, the calculating the reward value of the knowledge label through the crowdsourcing worker reliability model specifically includes: if the knowledge is verified, calculating the reward value of the knowledge label according to the real label of the knowledge; and if the knowledge is unverified knowledge, calculating the reward value of the knowledge tag according to the historical tag of the knowledge.
And when the knowledge distributed to the crowdsourced users is verified knowledge, calculating the reward value of the knowledge label according to the real label of the knowledge. The calculation formula of the reward value at this time is R (S)i,ai)＝I(Ka＝Kr) In the formula, SiRepresenting the field of knowledge, aiRepresenting crowd-sourced users, I representing a reward value, KaRepresents a knowledge tag, KrA real label representing knowledge. That is, if the knowledge tag obtained by the crowdsourcing user verifying knowledge is the same as the real tag of knowledge, the corresponding reward value is obtained.
And when the knowledge distributed to the crowdsourced users is unverified knowledge, calculating the reward value of the knowledge label according to the historical label of the knowledge. The calculation formula of the reward value at this time is R (S)i,ai)＝ui*rtIn the form ofIn, SiRepresenting the field of knowledge, uiRepresents the percentage of knowledge tags in the history tags, rtIndicating the prize value at time t. That is, if crowdsourcing users verify that knowledge gets knowledge tags that account for a larger percentage of history tags, the higher the reward value gets.
In a preferred embodiment, the screening of the qualification of the crowdsourced users according to the latest pre-stored policy specifically includes: judging whether the adequacy degree of the crowdsourcing users to the knowledge field set reaches a preset threshold value or not according to the latest pre-stored strategy; if the adequacy of the crowdsourcing users to the knowledge field set reaches a preset threshold value, judging that the crowdsourcing users have crowdsourcing qualification; and if the adequacy of the crowdsourcing users to the knowledge field set does not reach a preset threshold value, judging that the crowdsourcing users do not have crowdsourcing qualification.
In the knowledge base of the knowledge map, each knowledge domain represents a state SiWhen the knowledge selected by the knowledge base corresponds to which knowledge domain, the environment is Si. In the pre-stored policies, a policy table Q (S)i,ai) Is state SiThe column element is a behavior aiThe value in the Q table is crowd-sourced user aiFor the knowledge domain SiIs good at. A greedy algorithm is adopted, a greedy heart rate is set, the optimal Q value is selected according to the probability, and the state S is assumedi(knowledge domain S)i) And then, probabilistically selecting the optimal action sequence in the Q table for knowledge verification and distributing the knowledge field SiThe strategy updating function is a Q function which can be calculated iteratively, gamma is an incentive coefficient gamma ∈ (0,1), and the user hopes that the adequacy of a crowdsourcing user to the next knowledge field can influence the current Q value, so that the crowdsourcing user with better quality can obtain more tasks, the crowdsourcing user group with the core is stabilized, therefore, the Q function considers the adequacy of the crowdsourcing user to the next knowledge field, and the strategy function is updated to be Q (S)i,ai)←Q(Si,ai)+α[ui*rt+γQ(Si+1,ai)]Wherein, Q (S)i,ai) Value of (A)For crowdsourced users aiFor the knowledge domain SiExcellence of, α is learning rate, uiRepresents the percentage of knowledge tags in the history tags, rtIndicates the reward value at time t, γ is the incentive coefficient, γ ∈ (0, 1). if in the Q-table
ρ is a preset threshold, ρ ∈ (0,1), then the corresponding crowdsourced user is not eligible for crowdsourcing.
As shown in fig. 2, a second embodiment provides a device for establishing a reliability model of crowdsourced workers in a crowdsourced knowledge verification environment, including: the knowledge verification module 21 is configured to match a knowledge field set to crowdsourced users according to a pre-stored policy, and distribute knowledge in the knowledge field set to the crowdsourced users, so that the crowdsourced users verify the knowledge to obtain knowledge labels; the strategy updating module 22 is used for establishing a crowdsourcing worker reliability model based on a reinforcement learning algorithm, calculating the reward value of the knowledge label through the crowdsourcing worker reliability model, and updating the pre-stored strategy according to the reward value; the qualification screening module 23 is configured to repeatedly execute the knowledge verification module 21 and the policy updating module 22 until the update times of the pre-stored policies reach a preset number, and perform qualification screening on crowdsourced users according to the latest pre-stored policies; and the knowledge adding module 24 is used for adding the correct verified knowledge into the corresponding knowledge graph after the knowledge verification is completed.
It should be noted that the pre-storage strategy is a crowd-sourced worker adept knowledge field model obtained based on reinforcement learning algorithm training.
Through the knowledge verification module 21, the knowledge base of the knowledge graph is divided into a plurality of knowledge fields, a crowdsourcing worker adept knowledge field model obtained based on reinforcement learning algorithm training is used as a pre-storage strategy, a knowledge field set is matched for crowdsourcing users from the knowledge graph according to the pre-storage strategy, knowledge in the knowledge field set is distributed to the crowdsourcing users, the knowledge is verified by the crowdsourcing users to obtain knowledge labels, and therefore the knowledge in the knowledge field set and the knowledge in the knowledge field set can be freely matched for the crowdsourcing users.
The knowledge verification process is established as a markov decision process through the strategy updating module 22, a crowdsourcing worker reliability model is established based on a reinforcement learning algorithm, the knowledge tags are compared with real tags or historical tags corresponding to knowledge through the crowdsourcing worker reliability model, the reward values of the knowledge tags are calculated, and the pre-stored strategies are updated according to the reward values, so that a more appropriate knowledge field set is matched for crowdsourcing users next time.
Through qualification screening module 23, repeatedly execute knowledge verification module 21 and strategy update module 22 until the update times of the pre-stored strategy reaches the preset times, judge whether the crowdsourcing user possesses the crowdsourcing qualification according to the latest pre-stored strategy, screen the qualification of the crowdsourcing user, can avoid the crowdsourcing user who does not possess the crowdsourcing qualification to continue to verify knowledge, reduce the accuracy of knowledge verification, thereby improve the efficiency of the crowdsourcing knowledge verification.
Through the knowledge adding module 24, after the knowledge verification is completed, the verified correct knowledge is added to the corresponding knowledge graph, so that the knowledge graph can be updated.
The knowledge verification module 21 is used for matching knowledge field sets with crowdsourced users according to pre-stored strategies, distributing knowledge in the knowledge field sets to the crowdsourced users, enabling the crowdsourced users to verify knowledge to obtain knowledge labels, the strategy updating module 22 is further used for building a crowdsourced worker reliability model based on a reinforcement learning algorithm, the crowdsourced worker reliability model is used for calculating reward values of the knowledge labels, pre-stored strategies are updated according to the reward values, the qualification screening module 23 is used for repeatedly executing the knowledge verification module 21 and the strategy updating module 22 until the updating times of the pre-stored strategies reach the preset times, qualification screening is carried out on the crowdsourced users according to the latest pre-stored strategies, and the knowledge addition module 24 is used for adding the correct knowledge into the corresponding knowledge map after the knowledge verification is completed.
According to the embodiment, knowledge in the knowledge field set and the distributed knowledge field set can be freely matched for crowdsourcing users through the pre-storing strategy, the crowdsourcing worker reliability model is established based on the strengthening algorithm, the pre-storing strategy can be updated by the aid of the calculated reward value, qualification screening of the crowdsourcing users is achieved, and therefore efficiency of crowdsourcing knowledge verification is improved.
In a preferred embodiment, the knowledge verification module is further configured to, before the matching of the knowledge field set to the crowdsourcing user according to the pre-stored policy and the distribution of knowledge in the knowledge field set to the crowdsourcing user to verify knowledge and obtain the knowledge tag, perform identity authentication and qualification verification on the user according to input information of the user, and determine the user as the crowdsourcing user when the user passes the identity authentication and qualification verification.
When a user uses the crowdsourcing platform for the first time, the user needs to input information such as names, nationalities, regions, academic calendars, contact ways, passwords, and skilled knowledge fields to complete user registration. When the registration is successful, the user can input a contact and a password to log in the crowdsourcing platform. Wherein the contact means comprises a telephone or a mailbox.
And performing identity authentication on the user through the knowledge verification module 21 according to input information of the user, such as a contact way and a password, if the contact way and the password of the user are correct and matched, determining that the user passes the identity authentication, and otherwise, determining that the user does not pass the identity authentication. After the user passes the identity authentication, the qualification of the user is checked according to the input information of the user, such as the adequacy knowledge field, if the adequacy knowledge field of the user is the knowledge field required by the knowledge verification, the user is considered to pass the qualification check, otherwise, the user is considered to not pass the qualification check. And after the user passes the qualification audit, determining the user as the crowdsourcing user.
In a preferred embodiment, the calculating the reward value of the knowledge label through the crowdsourcing worker reliability model specifically includes: if the knowledge is verified, calculating the reward value of the knowledge label according to the real label of the knowledge; and if the knowledge is not verified, calculating the reward value of the knowledge tag according to the historical tag of the knowledge.
And when the knowledge distributed to the crowdsourced users is verified knowledge, calculating the reward value of the knowledge label according to the real label of the knowledge. The calculation formula of the reward value at this time is R (S)i,ai)＝I(Ka＝Kr) In the formula, SiRepresenting the field of knowledge, aiRepresenting crowd-sourced users, I TableIndicating a prize value, KaRepresents a knowledge tag, KrA real label representing knowledge. That is, if the knowledge tag obtained by the crowdsourcing user verifying knowledge is the same as the real tag of knowledge, the corresponding reward value is obtained.
And when the knowledge distributed to the crowdsourced users is unverified knowledge, calculating the reward value of the knowledge label according to the historical label of the knowledge. The calculation formula of the reward value at this time is R (S)i,ai)＝ui*rtIn the formula, SiRepresenting the field of knowledge, uiRepresents the percentage of knowledge tags in the history tags, rtIndicating the prize value at time t. That is, if crowdsourcing users verify that knowledge gets knowledge tags that account for a larger percentage of history tags, the higher the reward value gets.
In a preferred embodiment, the screening of the qualification of the crowdsourced users according to the latest pre-stored policy specifically includes: judging whether the adequacy degree of the crowdsourcing users to the knowledge field set reaches a preset threshold value or not according to the latest pre-stored strategy; if the adequacy of the crowdsourcing users to the knowledge field set reaches a preset threshold value, judging that the crowdsourcing users have crowdsourcing qualification; and if the adequacy of the crowdsourcing users to the knowledge field set does not reach a preset threshold value, judging that the crowdsourcing users do not have crowdsourcing qualification.
In the knowledge base of the knowledge map, each knowledge domain represents a state SiWhen the knowledge selected by the knowledge base corresponds to which knowledge domain, the environment is Si. In the pre-stored policies, a policy table Q (S)i,ai) Is state SiThe column element is a behavior aiThe value in the Q table is crowd-sourced user aiFor the knowledge domain SiIs good at. A greedy algorithm is adopted, a greedy heart rate is set, the optimal Q value is selected according to the probability, and the state S is assumedi(knowledge domain S)i) And then, probabilistically selecting the optimal action sequence in the Q table for knowledge verification and distributing the knowledge field SiThe policy update function is an iteratively computable Q function, γ is the incentive coefficient γ ∈ (0,1) it is expected that the adequacy of a crowdsourcing user to the next knowledge field influences the current Q value, so that better crowdsourcing users can obtain more tasks and a core crowdsourcing user population is stabilized, therefore, the Q function considers the adequacy of the crowdsourcing users to the next knowledge field and updates the strategy function to be Q (S)i,ai)←Q(Si,ai)+α[ui*rt+γQ(Si+1,ai)]Wherein, Q (S)i,ai) Is a crowdsourced user aiFor the knowledge domain SiExcellence of, α is learning rate, uiRepresents the percentage of knowledge tags in the history tags, rtIndicates the reward value at time t, γ is the incentive coefficient, γ ∈ (0, 1). if in the Q-table
ρ is a preset threshold, ρ ∈ (0,1), then the corresponding crowdsourced user is not eligible for crowdsourcing.
In summary, the embodiment of the present invention has the following advantages:
according to a pre-stored strategy, matching a knowledge field set for crowdsourcing users, distributing knowledge in the knowledge field set to the crowdsourcing users, enabling the crowdsourcing users to verify the knowledge to obtain knowledge labels, further establishing a crowdsourcing worker reliability model based on a reinforcement learning algorithm, calculating reward values of the knowledge labels through the crowdsourcing worker reliability model, updating the pre-stored strategy according to the reward values, repeatedly executing the operation until the updating times of the pre-stored strategy reach preset times, screening the qualification of the crowdsourcing users according to the latest pre-stored strategy, and adding the knowledge which is verified to be correct into the corresponding knowledge map after the knowledge verification is completed. According to the embodiment, knowledge in the knowledge field set and the distributed knowledge field set can be freely matched for crowdsourcing users through the pre-storing strategy, the crowdsourcing worker reliability model is established based on the strengthening algorithm, the pre-storing strategy can be updated by the aid of the calculated reward value, qualification screening of the crowdsourcing users is achieved, and therefore efficiency of crowdsourcing knowledge verification is improved.
While the foregoing is directed to the preferred embodiment of the present invention, it will be understood by those skilled in the art that various changes and modifications may be made without departing from the spirit and scope of the invention.
It will be understood by those skilled in the art that all or part of the processes of the above embodiments may be implemented by hardware related to instructions of a computer program, and the computer program may be stored in a computer readable storage medium, and when executed, may include the processes of the above embodiments. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), or the like.
Claims (8)
1. A method for establishing a reliability model of crowdsourcing workers in a crowdsourcing knowledge verification environment is characterized by comprising the following steps:
matching a knowledge field set for crowdsourcing users according to a pre-stored strategy, and distributing knowledge in the knowledge field set to the crowdsourcing users to enable the crowdsourcing users to verify the knowledge to obtain knowledge labels;
establishing a crowdsourcing worker reliability model based on a reinforcement learning algorithm, and calculating the reward value of the knowledge label through the crowdsourcing worker reliability model so as to update the pre-stored strategy according to the reward value;
repeatedly executing the operations until the updating times of the pre-stored strategy reach preset times, and screening the qualification of the crowdsourcing user according to the latest pre-stored strategy;
and after the knowledge verification is completed, adding the correctly verified knowledge into the corresponding knowledge graph.
2. The method for establishing a reliability model of crowdsourced workers in a crowdsourced knowledge verification environment as claimed in claim 1, wherein before matching a knowledge domain set to crowdsourced users according to a pre-stored policy and distributing knowledge in the knowledge domain set to the crowdsourced users to enable the crowdsourced users to verify the knowledge and obtain knowledge labels, the method further comprises:
and according to the input information of the user, performing identity authentication and qualification verification on the user, and determining the user as the crowdsourcing user when the user passes the identity authentication and qualification verification.
3. The method for establishing a crowdsourcing worker reliability model in a crowdsourcing knowledge verification environment according to claim 1, wherein the calculating the reward value of the knowledge label through the crowdsourcing worker reliability model specifically comprises:
if the knowledge is verified knowledge, calculating the reward value of the knowledge tag according to the real tag of the knowledge;
and if the knowledge is unverified knowledge, calculating the reward value of the knowledge tag according to the historical tag of the knowledge.
4. The method for establishing a reliability model of crowdsourcing workers in a crowdsourcing knowledge verification environment according to claim 1, wherein the screening of the crowdsourcing users for qualification according to the latest pre-stored policy specifically comprises:
judging whether the adequacy degree of the crowdsourcing users to the knowledge field set reaches a preset threshold value or not according to the latest pre-stored strategy;
if the degree of adequacy of the crowdsourcing users on the knowledge domain set reaches the preset threshold value, judging that the crowdsourcing users have crowdsourcing qualification;
and if the adequacy degree of the crowdsourcing users to the knowledge field set does not reach the preset threshold value, judging that the crowdsourcing users do not have crowdsourcing qualification.
5. The utility model provides a crowdsourcing worker reliability model building device under crowdsourcing knowledge verification environment which characterized in that includes:
the knowledge verification module is used for matching a knowledge field set to crowdsourced users according to a pre-stored strategy, and distributing knowledge in the knowledge field set to the crowdsourced users to enable the crowdsourced users to verify the knowledge to obtain knowledge labels;
the strategy updating module is used for establishing a crowdsourcing worker reliability model based on a reinforcement learning algorithm, calculating the reward value of the knowledge label through the crowdsourcing worker reliability model and updating the pre-stored strategy according to the reward value;
the qualification screening module is used for repeatedly executing the knowledge verification module and the strategy updating module until the updating times of the pre-stored strategies reach preset times, and screening the qualification of the crowdsourced users according to the latest pre-stored strategies;
and the knowledge adding module is used for adding the correctly verified knowledge into the corresponding knowledge map after the knowledge verification is finished.
6. The device for establishing a reliability model of crowdsourced workers in a crowdsourced knowledge verification environment as claimed in claim 5, wherein the knowledge verification module is further configured to, before the crowdsourced users are matched with a knowledge domain set according to a pre-stored policy and are distributed with knowledge in the knowledge domain set, enable the crowdsourced users to verify the knowledge and obtain knowledge labels, perform identity authentication and qualification verification on the users according to input information of the users, and determine the users as the crowdsourced users when the users pass the identity authentication and qualification verification.
7. The device for establishing a crowdsourcing worker reliability model in a crowdsourcing knowledge verification environment according to claim 5, wherein the calculating the reward value of the knowledge label through the crowdsourcing worker reliability model specifically comprises:
if the knowledge is verified knowledge, calculating the reward value of the knowledge tag according to the real tag of the knowledge;
and if the knowledge is unverified knowledge, calculating the reward value of the knowledge tag according to the historical tag of the knowledge.
8. The device for establishing a reliability model of crowdsourcing workers in a crowdsourcing knowledge verification environment according to claim 5, wherein the screening of the crowdsourcing users for qualification according to the latest pre-stored policy comprises:
judging whether the adequacy degree of the crowdsourcing users to the knowledge field set reaches a preset threshold value or not according to the latest pre-stored strategy;
if the degree of adequacy of the crowdsourcing users on the knowledge domain set reaches the preset threshold value, judging that the crowdsourcing users have crowdsourcing qualification;
and if the adequacy degree of the crowdsourcing users to the knowledge field set does not reach the preset threshold value, judging that the crowdsourcing users do not have crowdsourcing qualification.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010179259.2A CN111444332B (en) | 2020-03-13 | 2020-03-13 | Crowdsourcing worker reliability model establishing method and device under crowdsourcing knowledge verification environment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010179259.2A CN111444332B (en) | 2020-03-13 | 2020-03-13 | Crowdsourcing worker reliability model establishing method and device under crowdsourcing knowledge verification environment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111444332A true CN111444332A (en) | 2020-07-24 |
| CN111444332B CN111444332B (en) | 2023-04-18 |
Family
ID=71627534
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010179259.2A Active CN111444332B (en) | 2020-03-13 | 2020-03-13 | Crowdsourcing worker reliability model establishing method and device under crowdsourcing knowledge verification environment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111444332B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114020650A (en) * | 2022-01-10 | 2022-02-08 | 广东拓思软件科学园有限公司 | Crowd test task allocation method and device, electronic equipment and storage medium |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105184653A (en) * | 2015-09-08 | 2015-12-23 | 苏州大学 | Trust-based crowdsourcing worker screening method for social network |
| CN106951963A (en) * | 2017-03-29 | 2017-07-14 | 苏州大学 | The method and device of a kind of knowledge refinement |
-
2020
- 2020-03-13 CN CN202010179259.2A patent/CN111444332B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105184653A (en) * | 2015-09-08 | 2015-12-23 | 苏州大学 | Trust-based crowdsourcing worker screening method for social network |
| CN106951963A (en) * | 2017-03-29 | 2017-07-14 | 苏州大学 | The method and device of a kind of knowledge refinement |
Non-Patent Citations (1)
| Title |
|---|
| 刘辉;李盛恩;: "时空众包环境下基于统计预测的自适应阈值算法" * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114020650A (en) * | 2022-01-10 | 2022-02-08 | 广东拓思软件科学园有限公司 | Crowd test task allocation method and device, electronic equipment and storage medium |
| CN114020650B (en) * | 2022-01-10 | 2022-04-12 | 广东拓思软件科学园有限公司 | Crowd test task allocation method and device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111444332B (en) | 2023-04-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20080209431A1 (en) | System and method for routing tasks to a user in a workforce | |
| US10063562B1 (en) | Flexible access management framework based on measuring application usage behavior | |
| CA3098638C (en) | Threshold determining and identity verification method, apparatus, electronic device, and storage medium | |
| CN109063116B (en) | Data identification method and device, electronic equipment and computer readable storage medium | |
| CN113268336B (en) | Service acquisition method, device, equipment and readable medium | |
| CN112785005B (en) | Multi-objective task assistant decision-making method and device, computer equipment and medium | |
| CN111582500A (en) | Method and system for improving model training effect | |
| CN112711640A (en) | Method and device for configuring business handling process | |
| CN111444332B (en) | Crowdsourcing worker reliability model establishing method and device under crowdsourcing knowledge verification environment | |
| CN113240442A (en) | Agricultural Internet of things agricultural product quality safety tracing method and system | |
| US20170124573A1 (en) | User profile database system | |
| WO2020197661A1 (en) | Efficacy measures for unsupervised learning in a cyber security environment | |
| JP6591945B2 (en) | Information terminal, information processing method, program, and information processing system | |
| CN110879865B (en) | Recommendation method and device for nuclear products | |
| CN114693066A (en) | Earthquake risk analysis method, device, equipment and storage medium | |
| CN113987328A (en) | Topic recommendation method, equipment, server and storage medium | |
| CN111241297B (en) | Atlas data processing method and apparatus based on label propagation algorithm | |
| CN111241297A (en) | Map data processing method and device based on label propagation algorithm | |
| US10049223B1 (en) | Analyzing address book upload requests | |
| CN116578996B (en) | Data processing system for acquiring final vulnerability scanning equipment | |
| CN112784133B (en) | Link visualization method, device, equipment and storage medium | |
| US20240005430A1 (en) | Contextual training recommendations | |
| US11816639B2 (en) | Providing an interface for interview sessions | |
| US20230196294A1 (en) | Automatic employment verification using linked data structure | |
| US20210400050A1 (en) | Dynamic trust connection signal |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |