WO2021174814A1 - Answer verification method and apparatus for crowdsourcing task, computer device, and storage medium - Google Patents

Abstract

An answer verification method and apparatus for a crowdsourcing task, a computer device, and a storage medium. The method comprises: acquiring respective response answers corresponding to a target task to serve as initial answers; performing semantic recognition on the respective initial answers; determining a plurality of reference answers according to semantic recognition results, and determining a credibility value for each type of reference answer; if the credibility value, upon comparison, is less than a preset standard value, acquiring, via a preset model, a sample answer corresponding to the target task; and calculating a similarity value between each reference answer and the sample answer, selecting a similarity value having the greatest numerical value, determining, as a target answer, a reference answer corresponding to the similarity value having the greatest numerical value, and determining that a response answer corresponding to the target answer is a successfully verified response answer. The method can improve the accuracy and efficiency of answer verification for crowdsourcing tasks.

Description

Answer verification method, device, computer equipment and storage medium of crowdsourcing task

This application claims the priority of the Chinese patent application filed with the Chinese Patent Office on March 2, 2020, the application number is 202010135251.6, and the invention title is "Crowdsourcing task answer verification method, device, computer equipment and storage medium", all of which The content is incorporated in this application by reference.

Technical field

This application relates to the field of data processing technology, and in particular to methods, devices, computer equipment, and storage media for verifying answers to crowdsourced tasks.

Background technique

With the rapid development of network technology, in order to obtain more creative information, or to solve some cross-domain problems efficiently and conveniently, some companies or institutions often issue crowdsourcing tasks to Internet objects through the Internet, and solve these problems through crowdsourcing tasks. The problem is that crowdsourcing tasks means that internally executed tasks are distributed to external execution objects for completion in order to shorten the task completion time.

The inventor found that due to the different learning methods and implementation logic of different execution objects, different execution objects have different execution results for the same task. At the same time, because the execution objects have the learning ability, it is impossible to directly determine which execution objects provide The response answer must be better than other execution objects, and the response answer cannot be verified. At present, in order to ensure accurate task answers, it is generally to distribute a crowdsourced task to multiple execution objects, and then obtain the answer answer of each execution object, and then verify the answer answer through manual screening. This method makes the accuracy of answer verification for crowdsourcing tasks low, and the efficiency of answer verification is low. How to improve the efficiency of answer verification for crowdsourcing tasks has become an urgent problem to be solved.

Summary of the invention

The purpose of the embodiments of the present application is to propose an answer verification method for crowdsourced tasks, and solve the problem of low answer verification efficiency in the manner of manually verifying crowdsourced tasks in the prior art.

In order to solve the above technical problems, an embodiment of the present application provides an answer verification method for crowdsourcing tasks, including:

From all the answer answers obtained by the client, obtain each answer answer corresponding to the target task as an initial answer, wherein each answer answer corresponds to a answer object;

By means of natural language semantic recognition, semantic recognition is performed on each of the initial answers to obtain semantic recognition results of N initial answers, where N is the number of response objects, and N is a positive integer;

Combine the semantic recognition results in pairs, and use each combination as a set of results. The similarity calculation method is used to calculate the similarity value between the semantic recognition results in each set of results. If the similarity value is obtained If it is greater than the preset similarity threshold, use the two semantic recognition results in the group as the same type of reference answer to obtain M type of reference answer, where M≤N, and N is a positive integer;

Determine the credibility value of each type of reference answer through a preset consistency check method;

From all the credibility values of the reference answers, select the credibility value with the largest numerical value as the maximum credibility value, and compare the maximum credibility value with a preset standard value to obtain a comparison result;

If the comparison result is that the maximum credibility value is greater than or equal to the preset standard value, the reference answer corresponding to the maximum credibility value is taken as the target answer, and the target answer is The answer is confirmed as the answer that passed the verification.

In order to solve the above technical problems, a technical solution adopted in this application is to provide an answer verification device for crowdsourcing tasks, including:

The initial answer obtaining module is used to obtain each answer answer corresponding to the target task from all the answer answers obtained from the client as an initial answer, wherein each answer answer corresponds to a answer object;

The semantic recognition result module is used to perform semantic recognition on each of the initial answers through natural language semantic recognition to obtain semantic recognition results of N initial answers, where N is the number of response objects, and N is positive Integer

The reference answer classification module is used to combine the semantic recognition results in pairs, and use each combination as a set of results, and use the similarity calculation method to count the similarity values between the semantic recognition results in each set of results , If the obtained similarity value is greater than the preset similarity threshold, the two semantic recognition results in the group are regarded as the same type of reference answer, and M type of reference answer is obtained, where M≤N, and N is a positive integer ；

The credibility value statistics module is used to determine the credibility value of each type of reference answer through a preset consistency check method;

The credibility value comparison module is used to select the credibility value with the largest value from all the credibility values of the reference answers as the maximum credibility value, and compare the maximum credibility value with the predicted value. Set standard value comparison and get the comparison result;

The simulation answer obtaining module is configured to obtain the target task if the comparison result is that the maximum credibility value is less than the preset standard value, and input the target task into the preset model, and pass the The preset model gets the simulated answer;

The answer answer verification module is used to count the similarity values between the reference answer and the simulated answer for each type of the reference answer to obtain M similarity values, and select a numerical value from the M similarity values For the maximum similarity value, the reference answer corresponding to the similarity value with the largest numerical value is used as the target answer, and the response answer corresponding to the target answer is confirmed as the verified response answer.

A computer device includes a memory, a processor, and computer-readable instructions stored in the memory and capable of running on the processor, and the processor implements the following steps when the processor executes the computer-readable instructions:

From all the answer answers obtained by the client, obtain each answer answer corresponding to the target task as an initial answer, wherein each answer answer corresponds to a answer object;

By means of natural language semantic recognition, semantic recognition is performed on each of the initial answers to obtain semantic recognition results of N initial answers, where N is the number of response objects, and N is a positive integer;

Combine the semantic recognition results in pairs, and use each combination as a set of results. The similarity calculation method is used to calculate the similarity value between the semantic recognition results in each set of results. If the similarity value is obtained If it is greater than the preset similarity threshold, use the two semantic recognition results in the group as the same type of reference answer to obtain M type of reference answer, where M≤N, and N is a positive integer;

Determine the credibility value of each type of reference answer through a preset consistency check method;

From all the credibility values of the reference answers, select the credibility value with the largest numerical value as the maximum credibility value, and compare the maximum credibility value with a preset standard value to obtain a comparison result;

If the comparison result is that the maximum credibility value is greater than or equal to the preset standard value, the reference answer corresponding to the maximum credibility value is taken as the target answer, and the target answer is The answer is confirmed as the answer that passed the verification.

A computer-readable storage medium having executable code stored on the computer-readable storage medium, and when the executable code is executed by a processor, the steps of the method for verifying the answer of the crowdsourcing task as described below are realized:

From all the answer answers obtained by the client, obtain each answer answer corresponding to the target task as an initial answer, wherein each answer answer corresponds to a answer object;

By means of natural language semantic recognition, semantic recognition is performed on each of the initial answers to obtain semantic recognition results of N initial answers, where N is the number of response objects, and N is a positive integer;

Combine the semantic recognition results in pairs, and use each combination as a set of results. The similarity calculation method is used to calculate the similarity value between the semantic recognition results in each set of results. If the similarity value is obtained If it is greater than the preset similarity threshold, use the two semantic recognition results in the group as the same type of reference answer to obtain M type of reference answer, where M≤N, and N is a positive integer;

Determine the credibility value of each type of reference answer through a preset consistency check method;

From all the credibility values of the reference answers, select the credibility value with the largest numerical value as the maximum credibility value, and compare the maximum credibility value with a preset standard value to obtain a comparison result;

If the comparison result is that the maximum credibility value is greater than or equal to the preset standard value, the reference answer corresponding to the maximum credibility value is taken as the target answer, and the target answer is The answer is confirmed as the answer that passed the verification.

A method for verifying answers to crowdsourcing tasks in the above scheme. From all the answer answers obtained from the client, the answer answer of each answer object corresponding to the target task is obtained as the initial answer; and semantics of each initial answer Then, the semantic recognition results are combined in pairs, and each combination is used as a set of results. The similarity calculation method is used to calculate the similarity value between the semantic recognition results in each set of results. If the obtained similarity value is greater than Preset the similarity threshold, then use the two semantic recognition results in the group as the same type of reference answer to obtain the M type of reference answer; and determine the credibility value of each type of reference answer through the preset consistency check method ; Then select the highest credibility value from all the credibility values of the reference answers as the maximum credibility value, and compare the maximum credibility value with the preset standard value to obtain the comparison result; if the comparison result If the maximum credibility value is greater than or equal to the preset standard value, the reference answer corresponding to the maximum credibility value is taken as the target answer, and the response answer corresponding to the target answer is confirmed as the verified answer. By determining the maximum credibility value and performing similarity value statistics, the largest numerical similarity value is obtained, and finally the answer answers that have passed the verification are confirmed, which can effectively improve the answer verification efficiency of the crowdsourcing task.

Description of the drawings

In order to explain the solution in this application more clearly, the following will briefly introduce the drawings used in the description of the embodiments of the application. Obviously, the drawings in the following description are some embodiments of the application. Ordinary technicians can obtain other drawings based on these drawings without creative work.

FIG. 1 is a schematic diagram of an application environment of a method for verifying answers to crowdsourced tasks provided by an embodiment of the present application;

FIG. 2 is an implementation flow chart of the method for verifying the answers of crowdsourcing tasks according to an embodiment of the present application;

FIG. 3 is an implementation flow chart of step 2 in the method for verifying answers to crowdsourced tasks provided by an embodiment of the present application;

FIG. 4 is an implementation flowchart of step S21 in the method for verifying answers to crowdsourced tasks provided by an embodiment of the present application;

FIG. 5 is an implementation flow chart after step S5 in the method for verifying answers to crowdsourced tasks provided by an embodiment of the present application;

FIG. 6 is another implementation flowchart after step S5 in the method for verifying answers to crowdsourced tasks provided by an embodiment of the present application;

FIG. 7 is an implementation flowchart of step S57 in the method for verifying answers to crowdsourced tasks provided by an embodiment of the present application;

FIG. 8 is a schematic diagram of an answer verification device for crowdsourcing tasks provided by an embodiment of the present application;

Fig. 9 is a schematic diagram of a computer device provided by an embodiment of the present application.

Detailed ways

Unless otherwise defined, all technical and scientific terms used herein have the same meanings as commonly understood by those skilled in the technical field of the application; the terms used in the specification of the application herein are only for describing specific embodiments. The purpose is not to limit the application; the terms "including" and "having" in the specification and claims of the application and the above-mentioned description of the drawings and any variations thereof are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific sequence.

The reference to "embodiments" herein means that a specific feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art clearly and implicitly understand that the embodiments described herein can be combined with other embodiments.

In order to enable those skilled in the art to better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the accompanying drawings.

The application will be described in detail below with reference to the drawings and implementations.

Referring to FIG. 1, the system architecture 100 may include terminal devices 101, 102, and 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, and so on.

The user can use the terminal devices 101, 102, and 103 to interact with the server 105 through the network 104 to receive or send messages and so on. Various communication client applications, such as web browser applications, search applications, instant messaging tools, etc., may be installed on the terminal devices 101, 102, and 103.

The terminal devices 101, 102, 103 may be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and so on.

The server 105 may be a server that provides various services, for example, a background server that provides support for pages displayed on the terminal devices 101, 102, and 103.

It should be noted that the method for verifying answers for crowdsourcing tasks provided by the embodiments of the present application is generally executed by a server. Accordingly, a device for verifying answers for crowdsourcing tasks is generally set in the server.

It should be understood that the numbers of terminal devices, networks, and servers in FIG. 1 are merely illustrative. There can be any number of terminal devices, networks, and servers according to implementation needs.

Please refer to FIG. 2. FIG. 2 shows a specific implementation of the method for verifying the answers of crowdsourcing tasks.

It should be noted that if there are substantially the same results, the method of the present application is not limited to the sequence of the process shown in FIG. 2, and the method includes the following steps:

S1: Among all the answer answers obtained from the client, each answer answer corresponding to the target task is obtained as an initial answer, where each answer answer corresponds to a answer object.

Specifically, after the target task is generated or obtained, the server pushes the target task to multiple objects. After the object answers the target task, a response answer is obtained, and the response answer is fed back to the server through the client. The network transmission protocol receives the answer sent by each client as the initial answer.

On the server side, the information of each respondent is pre-stored. After the respondent responds to the task and gets a response, the mapping relationship between the response and the respondent is established. After obtaining the initial answer, obtain each initial corresponding response object, that is, obtain a list of all objects participating in the target task and feedback the response answer to the server.

Among them, the response object in this embodiment may specifically be a preset network model (machine learning model or neural network model), or may be a preset computing engine or a big data platform and other individuals with autonomous learning and discrimination capabilities. , There is no specific limitation here. For the same crowdsourcing task, different objects will give their own answers due to their different structures and implementation logic.

It should be noted that the server will send and push multiple tasks at the same time, and may also receive answers from different tasks at the same time period. In order to facilitate the subsequent screening of crowdsourced reference answers, the server will pre-select each task Set a task ID for each. The initial answer corresponding to the target task is selected from all the answer answers through the target task identifier.

For example, in a specific embodiment, all the response answers obtained from the client include three task identifiers, namely task A, task B, and task C. The task identifier of the current crowdsourcing task is task A. Therefore, select The task identifier is the answer to task A, as the initial answer.

Among them, the task identification can be specifically represented by a combination of characters, numbers, or characters and numbers, for example, a task identification is "TPSB5201906280236".

Among them, the target task in this embodiment is a type of statement of objective facts, for example, recognizing and transcribing text on a given image.

In this embodiment, an electronic device (such as the server shown in FIG. 1) on which a method for verifying answers to crowdsourcing tasks runs may be connected via a wired connection or a wireless connection. It should be pointed out that the above-mentioned wireless connection methods can include, but are not limited to, 3G/4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or developed in the future .

S2: Used to perform semantic recognition on each initial answer through natural language semantic recognition, and obtain semantic recognition results of N initial answers, where N is the number of response objects, and N is a positive integer.

Specifically, through the method of natural language semantic recognition, semantic recognition is performed on each initial answer, and the semantic recognition result of each initial answer is obtained.

Among them, natural language semantic recognition is mainly through the use of NLP. NLP (Natural Language Processing) is also called natural language processing. Understanding natural language requires extensive knowledge about the external world and the ability to use and operate this knowledge. Natural language cognition is also regarded as an AI-complete problem. NLP tasks mainly refer to some tasks that involve the semantic understanding or parsing of natural language. Common NLP tasks include but are not limited to: Speech recognition, Chinese word segmentation, Part-of- speech tagging, text categorization, parsing, automatic summarization, question answering, information extraction, etc.

S3: Combine the semantic recognition results in pairs, and use each combination as a set of results. The similarity calculation method is used to calculate the similarity value between the semantic recognition results in each set of results. If the obtained similarity value is greater than the expected value If the similarity threshold is set, the two semantic recognition results in the group are regarded as the same type of reference answer, and M type of reference answer is obtained, where M≤N, and N is a positive integer.

Specifically, by combining the semantic recognition results in pairs, and using each combination as a set of results, the similarity calculation is used to obtain the similarity value between each set of semantic recognition results, and the similarity threshold is preset, if If it is greater than the preset similarity threshold, any two semantic recognition results are classified as the preset similarity threshold.

Among them, the preset similarity threshold is a server-side preset setting, which is used to classify different semantic recognition results. The more reasonable preset similarity thresholds are 0.9, 0.8, 0.7, 0.6, etc., and the specific settings can be based on actual conditions. The situation is set, which is not limited here. Preferably, the preset similarity threshold value in this embodiment is 0.8.

Among them, the similarity calculation methods include, but are not limited to: Minkowski Distance, Manhattan Distance, Euclidean Distance, Chebyshev Distance, and Hamming Distance. Distance) and Mahalanobis Distance (Mahalanobis Distance) and so on.

Preferably, the Euclidean distance is used to calculate the similarity value between the semantic recognition results in each group of results; the Euclidean distance calculation is used to quickly and efficiently calculate the similarity value between the semantic recognition results in each group of results.

S4: Determine the credibility value of each type of reference answer through a preset consistency check method.

Among them, the credibility value refers to the reliability of this type of reference answer.

In one embodiment, the following formula is used to calculate the credibility value of each type of reference answer:

CR ₁ +CR ₂ +…+CR _m =1

P={P ₁ ,P ₂ ,…,P _m }

Among them, P = {P ₁ ,P ₂ ,...,P _m } is the reference answer set, P _m is the number of answer answers contained in the m-th reference answer, and CR _m is the credibility value corresponding to the m-th reference answer , M is the number of reference answer categories, n is the number of initial answers.

S5: Select the credibility value with the largest value from the credibility values of all reference answers as the maximum credibility value, and compare the maximum credibility value with the preset standard value to obtain the comparison result.

Specifically, the credibility value with the largest value is selected from the credibility value of each type of reference answer to compare with the preset standard value, and based on the comparison result, it is determined whether there is a reference answer that meets the requirements among all the classified reference answers.

Among them, the preset standard value is the server-side preset setting, which is used to evaluate whether the credibility value meets the requirements. The reasonable range is (0.5,1). The specific setting can be set according to the actual situation. Here It is not limited, and preferably, the preset standard value in this embodiment is 0.6.

Among them, the comparison result includes that the maximum credibility value is less than the preset standard value, and the maximum credibility value is greater than or equal to the preset standard value.

S6: If the comparison result is that the maximum credibility value is greater than or equal to the preset standard value, the reference answer corresponding to the maximum credibility value is taken as the target answer, and the response answer corresponding to the target answer is confirmed as the verified answer.

Specifically, when the maximum credibility value is greater than or equal to the preset standard value, it indicates that the reference answer corresponding to the maximum credibility value is a trustworthy answer answer, which is taken as the target answer, and the answer answer corresponding to the target answer is confirmed To verify the answer answer passed.

The answer verification result is sent to the terminal devices 101, 102, 103, so that the respondent can learn the answer verification result.

In this embodiment, from all the response answers obtained from the client, the response answer of each response object corresponding to the crowdsourcing task is obtained as the initial answer; semantic recognition is performed on each initial answer, and then the semantic recognition results are divided into two Two combinations, and each combination is used as a set of results. The similarity calculation method is used to calculate the similarity value between the semantic recognition results in each set of results. If the obtained similarity value is greater than the preset similarity threshold, the The two semantic recognition results in the group are regarded as the same type of reference answer, and M type of reference answer is obtained; and the credibility value of each type of reference answer is determined through the preset consistency verification method; then all the reference answers of each type The maximum credibility value of the answer is selected to obtain the comparison result; if the comparison result is that the maximum credibility value is greater than or equal to the preset standard value, the reference answer corresponding to the maximum credibility value is taken as the target answer. And confirm the answer answer corresponding to the target answer as the answer answer that passed the verification. By determining the maximum credibility value, and performing similarity value statistics, the largest numerical similarity value is obtained, and finally the answer answer that has passed the verification is confirmed to be effective Improve the efficiency of answer verification for crowdsourcing tasks.

Please refer to FIG. 3, which shows a specific implementation of step S2. In step S2, semantic recognition is performed on each initial answer by means of natural language semantic recognition, and the semantic recognition results of N initial answers are obtained. Among them, N is the number of response objects, and N is a positive integer. The specific implementation process is described in detail as follows:

S21: Perform word segmentation processing on the initial answer through the preset word segmentation method to obtain the basic word segmentation contained in the initial answer.

S22: Convert the basic word segmentation into word vectors, and cluster the word vectors through a clustering algorithm to obtain a cluster center corresponding to each word vector.

S23: Obtain the preset semantics corresponding to the cluster centers corresponding to each word vector as the semantic recognition result of the initial answer.

In this embodiment, by performing word segmentation processing on the initial answer and clustering the word vector through a clustering algorithm, the semantic recognition result of the initial answer is obtained, so that the semantic recognition of the initial answer is more accurate, and the response to the crowdsourcing task is further improved. The verification accuracy of the answer.

Please refer to FIG. 4, which shows a specific implementation of step S21. In step S21, the initial answer is segmented through a preset word segmentation method to obtain the specific implementation process of the basic word segmentation contained in the initial answer. The details are as follows:

S211: Perform word segmentation analysis on the basic sentence to obtain K word segmentation sequences.

S212: For each word segmentation sequence, calculate the occurrence probability of each word segmentation sequence according to the word sequence data of the preset training corpus to obtain the occurrence probability of K word segmentation sequences.

S213: From the occurrence probabilities of K word segmentation sequences, select the word segmentation sequence corresponding to the occurrence probability that reaches the preset probability threshold as the target word segmentation sequence, and use each word segmentation in the target word segmentation sequence as the basic word segmentation included in the initial answer .

In this embodiment, the word segmentation analysis is performed through the basic sentence, and the occurrence probability of each word segmentation sequence is calculated, so that the basic word segmentation contained in the initial answer can be accurately obtained, so that the initial answer can obtain an accurate semantic recognition result.

Please refer to Figure 5. Figure 5 shows a specific implementation after step S5. The specific implementation process is described in detail as follows:

S51: If the comparison result is that the maximum credibility value is less than the preset standard value, the target task is obtained, and the target task is input into the preset model, and the simulated answer is obtained through the preset model.

Specifically, for each type of target task, a corresponding task model is preset, and the task model does not have the ability to learn independently, but a simulated answer with an accuracy that meets the requirements can be obtained. When the maximum credibility value is less than the preset standard value, the target task is obtained, and the target task is input into the preset model, and the simulated answer is obtained through the preset model.

In a specific embodiment, a crowdsourcing task is text recognition and transliteration in an image, the preset model is an optical character recognition matching model, the credibility value is between 0.65 and 0.75, and the preset credibility value It is 0.55. In step S50, when the calculated credibility value is lower than 0.55, the preset model is used to obtain a simulated answer.

S52: For each type of reference answer, count the similarity values between the reference answer and the simulated answer to obtain M similarity values, select the highest similarity value from the M similarity values, and set the highest similarity value The corresponding reference answer is used as the target answer, and the answer answer corresponding to the target answer is confirmed as the answer answer that has passed the verification.

Specifically, the simulated answer is obtained through the preset model, and the similarity value between each reference answer and the simulated answer is obtained by calculating the similarity between each reference answer and the simulated answer, and the similarity value is ranked from large to small Arrange in order, select the reference answer with the largest similarity value as the target answer, and confirm the answer corresponding to the target answer as the verified answer.

Among them, the similarity calculation method has been shown in step S3, which is not redundant here.

In a specific embodiment, by adopting the Euclidean distance, the Euclidean distance between each reference answer and the simulated answer, such as 0.9, 0.8, 0.5, 0.2, and 0.1, is obtained. Due to the Euclidean distance, the Euclidean distance is larger. If it is small, the similarity value between the reference answer and the simulated answer. Therefore, in the above Euclidean distance, the reference answer corresponding to the Euclidean distance of 0.1 should be selected as the target answer, and the answer corresponding to the target answer should be confirmed as the verified answer Answer the answer.

In this embodiment, the simulated answer is obtained through the preset model, and the verified answer is determined, which can further improve the accuracy of the answer verification of the crowdsourcing task.

Please refer to Figure 6. Figure 6 shows a specific implementation after step S6. The specific implementation process is described in detail as follows:

S53: The response object includes a first object and a second object. The answer of the first object is the first answer, the answer of the second object is the second answer, and both the first object and the second object have preset weights. , And the preset weight of the first object is less than the preset weight of the second object.

Specifically, in the server, the response object includes a first object and a second object, where the answer of the first object is the first answer, the answer of the second object is the second answer, and the first object and the first object Both objects have preset weights corresponding to them, and the preset weight of the first object is smaller than the preset weight of the second object.

Among them, the preset weight is based on the answer to the crowdsourcing task of the respondent in the past, and the answer weight set by the server can be used to evaluate the credibility of the answer of the respondent.

It should be understood that by assigning preset weights to both the first object and the second object, when the calculated credibility value does not meet the requirements (less than the preset standard value), the first object and the second object can be assigned according to the preset weight. The response answers corresponding to the two subjects are weighted, so that the distribution of the reliability value of the response answer is more reasonable, which is beneficial to improve the accuracy of the reliability value.

S54: If the comparison result is that the maximum credibility value is less than the preset standard value, obtain the first preset weight corresponding to the first response answer and the second preset weight corresponding to the second response answer.

Specifically, if the maximum credibility value is less than the preset standard value, the answer answer cannot be verified, and the first answer answer and the second answer answer of the first answer object and the second answer object need to be obtained, and the first answer is assigned The answer and the second answer have different weights, and the credibility of the answer is determined according to the weight assigned.

Among them, the preset weight is the server preset setting, and the reasonable range is (0.1, 1). The specific setting can be set according to the actual situation, and it is not limited here. Preferably, the first preset in this embodiment Set the weight to 0.6, and the second preset weight to 0.8.

S55: Determine the credibility weight of each type of reference answer according to the reference answer category to which the first answer answer and the second answer answer belong, the first preset weight, and the second preset weight.

Specifically, crowdsourcing tasks have different answer question types, and correspondingly, there are also reference answer categories to which the answer should belong. According to the reference answer category to which the first answer answer and the second answer answer belong, identify the first preset weight and the second preset weight of the first answer answer and the second answer answer, and finally determine the credibility weight of each type of reference answer .

Among them, the credibility weight is the credibility ratio given to each type of reference answer by the server.

S56: Determine the weighted credibility value of each type of reference answer according to the credibility weight of each type of reference answer and the preset weight verification method.

Among them, the weighted reliability value is the reliability value corresponding to the answer answer.

In one embodiment, the following calculation method is used to calculate the weighted credibility value of each type of reference answer:

Assuming that the first answer is P ₁ , then the reference answer set is P={P ₁ ,P ₂ ,...,P _m }, and the corresponding weighted credibility value is CR={CR ₁ +0.5,CR ₂ ,... ,CR _m }.

Among them, P _m is the number of answer answers contained in the m-th reference answer, CR is the weighted credibility value corresponding to the m-th reference answer, m is the number of reference answer categories, and n is the number of initial answers.

S57: Select the weighted credibility value with the largest numerical value as the target weighted credibility value, obtain the reference answer corresponding to the target weighted credibility value as the target answer, and confirm the response answer corresponding to the target answer as the verified response Answer.

Specifically, by obtaining the largest weighted credibility value, that is, the reference answer corresponding to the weighted credibility value is the best response answer for the crowdsourcing task, and by using it as the response answer of the crowdsourcing task, and confirming that it has passed the verification The answer to the answer.

In this embodiment, by determining the credibility weight of each type of reference answer and calculating the weighted credibility value of each type of reference answer, the reliability of each type of reference answer is obtained, and the accuracy of the answer verification of the crowdsourcing task is improved. .

Please refer to Figure 7. Figure 7 shows a specific implementation of step S57. In step S57, the weighted credibility value with the largest value is selected as the target weighted credibility value, and the corresponding weighted credibility value of the target is obtained. The reference answer is used as the target answer, and the answer corresponding to the target answer is confirmed as the verified answer. The details are as follows:

S571: Obtain historical response answers of the first object and the second object.

Specifically, the server divides the objects of the crowdsourcing task into a first object and a second object; the first object and the second object refer to different types of objects, rather than specifically one or two of them; through The historical response answers of the first object and the second object to the crowdsourcing task can be obtained, and the response level of the first object and the second object to the crowdsourcing task can be learned.

S572: Judging the proportion of the first subject’s historical response answers and obtaining the response accuracy rate of the first subject, and judging the proportion of the second subject’s historical response answers, obtaining the second subject The response accuracy rate.

Specifically, by judging the ratio of the verified response answers in the historical response answers of the first object and the second object, the accuracy rate corresponding to the responses of the first object and the second object is obtained, and the pair of the first object and the second object is further learned The ability to respond to crowdsourcing tasks.

S573: According to the response accuracy rate and the preset classification threshold, update the preset weight of the first object and the preset weight of the second object to obtain the updated preset weight of the first object and the updated second object. Default weight.

Specifically, the response accuracy rates of the first object and the second object are obtained. If a part of the first object has a higher accuracy rate and the accuracy rate exceeds the preset classification threshold, then the first object of this part is adjusted Is the second object; if a part of the second object has a low accuracy rate, and the accuracy rate is lower than the preset classification threshold, then adjust this part of the second object as the first object.

Among them, the preset classification threshold is the server-side preset setting, and the reasonable range is (0.6, 0.9). The specific setting can be set according to the actual situation, and it is not limited here. Preferably, it is preset in this embodiment. Set the classification threshold to 0.6.

In this embodiment, by judging the response accuracy of the first object and the second object, and updating the preset weight of the first object and the preset weight of the second object, the preset weight of the first object and the second object The preset weights of is updated, so that the respondent pays more attention to the answers to the crowdsourcing task, which further improves the accuracy and efficiency of the crowdsourcing task.

It should be understood that the size of the sequence number of each step in the foregoing embodiment does not mean the order of execution. The execution sequence of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing relevant hardware through computer-readable instructions, which can be stored in a computer-readable storage medium. When the computer-readable instructions are executed, they may include the processes of the above-mentioned method embodiments. Among them, the aforementioned storage medium may be a non-volatile storage medium such as a magnetic disk, an optical disc, a read-only memory (Read-Only Memory, ROM), or a random access memory (Random Access Memory, RAM), etc.

Please refer to FIG. 8. As an implementation of the method shown in FIG. 2, this application provides an embodiment of an answer verification device for crowdsourced tasks. The device embodiment corresponds to the method embodiment shown in FIG. 2. The device can be specifically applied to various electronic devices.

As shown in FIG. 8, the answer verification device for crowdsourcing tasks in this embodiment includes: an initial answer obtaining module 81, a semantic recognition result module 82, a reference answer classification module 83, a credibility value statistics module 84, and credibility The value comparison module 85 and the answer verification module 86. in:

The initial answer obtaining module 81 is used to obtain each answer answer corresponding to the target task from all the answer answers obtained from the client as an initial answer, where each answer answer corresponds to a answer object;

The semantic recognition result module 82 is used to perform semantic recognition on each initial answer by means of natural language semantic recognition to obtain semantic recognition results of N initial answers, where N is the number of response objects, and N is a positive integer;

The reference answer classification module 83 is used to combine the semantic recognition results in pairs and use each combination as a set of results. The similarity calculation method is used to calculate the similarity value between the semantic recognition results in each set of results. If the similarity value of is greater than the preset similarity threshold, the two semantic recognition results in the group are regarded as the same type of reference answer, and M type of reference answer is obtained, where M≤N, and N is a positive integer;

The credibility value statistics module 84 is used to determine the credibility value of each type of reference answer through a preset consistency check method;

The credibility value comparison module 85 is used to select the credibility value with the largest value from the credibility values of all the reference answers as the maximum credibility value, and compare the maximum credibility value with the preset standard value Compare, get the comparison result;

The answer answer verification module 86 is configured to, if the comparison result is that the maximum credibility value is greater than or equal to the preset standard value, the reference answer corresponding to the maximum credibility value is used as the target answer, and the answer answer corresponding to the target answer is confirmed as Verify the answer that passed the verification.

Further, the semantic recognition result module 82 includes:

The basic word segmentation acquisition unit is used to perform word segmentation processing on the initial answer through a preset word segmentation method to obtain the basic word segmentation contained in the initial answer.

The word vector acquisition unit is used to convert the basic word segmentation into word vectors, and cluster the word vectors through a clustering algorithm to obtain the cluster center corresponding to each word vector.

The semantic recognition result unit is used to obtain the preset semantics corresponding to the cluster center corresponding to each word vector as the semantic recognition result of the initial answer.

Further, the basic word segmentation acquisition unit includes:

The word segmentation sequence acquisition subunit is used to perform word segmentation analysis on the basic sentence to obtain K word segmentation sequences.

The word segmentation sequence occurrence probability subunit is used to calculate the occurrence probability of each word segmentation sequence based on the word sequence data of the preset training corpus for each word segmentation sequence to obtain the occurrence probability of K word segmentation sequences.

The target word segmentation sequence determination subunit is used to select the word segmentation sequence corresponding to the occurrence probability of the preset probability threshold from the occurrence probabilities of the K word segmentation sequences as the target word segmentation sequence, and use each word segmentation in the target word segmentation sequence as The basic participle included in the initial answer.

Further, the answer verification device for crowdsourcing tasks also includes:

The simulation answer obtaining module is used to obtain the target task if the comparison result is that the maximum credibility value is less than the preset standard value, and input the target task into the preset model to obtain the simulated answer through the preset model;

The target answer selection module is used to calculate the similarity value between the reference answer and the simulated answer for each type of reference answer to obtain M similarity values, and from the M similarity values, select the highest similarity value, and calculate the value The reference answer corresponding to the largest similarity value is used as the target answer, and the answer answer corresponding to the target answer is confirmed as the answer answer that has passed the verification.

Further, the answer verification device for crowdsourcing tasks also includes:

The preset weight setting module is used for answering objects including a first object and a second object, the answer of the first object is the first answer, the answer of the second object is the second answer, the first object and the second object Each corresponds to a preset weight, and the preset weight of the first object is less than the preset weight of the second object;

The preset weight obtaining module is configured to obtain the first preset weight corresponding to the first response answer and the second preset weight corresponding to the second response answer if the comparison result is that the maximum credibility value is less than the preset standard value;

The credibility weight determination module is used to determine the credibility weight of each type of reference answer according to the reference answer category to which the first answer answer and the second answer answer belong, the first preset weight, and the second preset weight;

The weighted credibility value determination module is used to confirm the weighted credibility value of each type of reference answer according to the credibility weight of each type of reference answer and the preset weight check method;

The weighted credibility value selection module is used to select the weighted credibility value with the largest value as the target weighted credibility value, obtain the reference answer corresponding to the target weighted credibility value, and use it as the target answer. The answer is confirmed as the answer that passed the verification.

Further, the weighted credibility value selection module includes:

The historical response answer obtaining unit is used to obtain the historical response answers of the first object and the second object;

The response accuracy rate verification unit is used to determine the proportion of the response answers that passed the verification in the historical response answers of the first subject, obtain the response accuracy rate of the first subject, and determine the response answers that passed the verification in the historical response answers of the second subject Ratio, get the response accuracy rate of the second object;

The following new object unit is used to update the preset weight of the first object and the preset weight of the second object according to the response accuracy rate and the preset classification threshold to obtain the updated preset weight and updated weight of the first object The preset weight of the second object.

In order to solve the above technical problems, the embodiments of the present application also provide computer equipment. Please refer to FIG. 9 for details. FIG. 9 is a block diagram of the basic structure of the computer device in this embodiment.

The computer device 9 includes a memory 91, a processor 92, and a network interface 93 that communicate with each other through a system bus. It should be pointed out that the figure only shows a computer device 9 with three types of components: memory 91, processor 92, and network interface 93, but it should be understood that it is not required to implement all the components shown, and alternative implementations are possible. More or fewer components. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculation and/or information processing in accordance with pre-set or stored instructions. Its hardware includes, but is not limited to, a microprocessor, a dedicated Integrated Circuit (Application Specific Integrated Circuit, ASIC), Programmable Gate Array (Field-Programmable Gate Array, FPGA), Digital Processor (Digital Signal Processor, DSP), embedded equipment, etc.

Computer equipment can be computing equipment such as desktop computers, notebooks, palmtop computers, and cloud servers. The computer equipment can interact with the user through a keyboard, a mouse, a remote control, a touch panel, or a voice control device.

The memory 91 includes at least one type of readable storage medium. The readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory ( SRAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), programmable read only memory (PROM), magnetic memory, magnetic disks, optical disks, etc. In some embodiments, the memory 91 may be an internal storage unit of the computer device 9, for example, the hard disk or memory of the computer device 9. In other embodiments, the memory 91 may also be an external storage device of the computer device 9, such as a plug-in hard disk, a smart media card (SMC), and a secure digital (Secure Digital, SMC) equipped on the computer device 9. SD) card, flash card (Flash Card), etc. Of course, the memory 91 may also include both the internal storage unit of the computer device 9 and its external storage device. In this embodiment, the memory 91 is generally used to store an operating system and various application software installed in the computer device 9, such as a computer-readable instruction of a crowdsourced task sampling method. In addition, the memory 91 can also be used to temporarily store various types of data that have been output or will be output.

In some embodiments, the processor 92 may be a central processing unit (Central Processing Unit, CPU), a controller, a microcontroller, a microprocessor, or other data processing chips. The processor 92 is generally used to control the overall operation of the computer device 9. In this embodiment, the processor 92 is used to run the process code or process data stored in the memory 91, for example, run a computer-readable instruction of a crowdsourced task sampling method.

The network interface 93 may include a wireless network interface or a wired network interface, and the network interface 93 is generally used to establish a communication connection between the computer device 9 and other electronic devices.

The present application also provides another implementation manner, that is, a computer-readable storage medium is provided with a random inspection process stored in the computer-readable storage medium, and the random inspection process can be executed by at least one processor, so that the at least one processor executes as described above The steps of a sampling method for crowdsourcing tasks. The computer-readable storage medium may be non-volatile or volatile.

Through the description of the above implementation manners, those skilled in the art can clearly understand that the above-mentioned embodiment method can be implemented by means of software plus the necessary general hardware platform, of course, it can also be implemented by hardware, but in many cases the former is better.的实施方式。 Based on this understanding, the technical solution of this application essentially or the part that contributes to the existing technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium (such as ROM/RAM, magnetic disk, The optical disc) includes a number of instructions to enable a terminal device (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods of the various embodiments of the present application.

Obviously, the embodiments described above are only a part of the embodiments of the present application, rather than all of the embodiments. The drawings show preferred embodiments of the present application, but do not limit the scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these examples is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although this application has been described in detail with reference to the foregoing embodiments, for those skilled in the art, it is still possible for those skilled in the art to modify the technical solutions described in each of the foregoing specific embodiments, or equivalently replace some of the technical features. . All equivalent structures made by using the contents of the description and drawings of this application, directly or indirectly used in other related technical fields, are similarly within the protection scope of this application.

Claims (20)

1. A method of answer verification for crowdsourcing tasks, including:

   From all the answer answers obtained by the client, obtain each answer answer corresponding to the target task as an initial answer, wherein each answer answer corresponds to a answer object;

   By means of natural language semantic recognition, semantic recognition is performed on each of the initial answers to obtain semantic recognition results of N initial answers, where N is the number of response objects, and N is a positive integer;

   Combine the semantic recognition results in pairs, and use each combination as a set of results. The similarity calculation method is used to calculate the similarity value between the semantic recognition results in each set of results. If the similarity value is obtained If it is greater than the preset similarity threshold, use the two semantic recognition results in the group as the same type of reference answer to obtain M type of reference answer, where M≤N, and N is a positive integer;

   Determine the credibility value of each type of reference answer through a preset consistency check method;

   From all the credibility values of the reference answers, select the credibility value with the largest numerical value as the maximum credibility value, and compare the maximum credibility value with a preset standard value to obtain a comparison result;

   If the comparison result is that the maximum credibility value is greater than or equal to the preset standard value, the reference answer corresponding to the maximum credibility value is taken as the target answer, and the target answer is The answer is confirmed as the answer that passed the verification.
2. The answer verification method for crowdsourcing tasks according to claim 1, wherein said performing semantic recognition on each of said initial answers by means of natural language semantic recognition to obtain semantic recognition results of N initial answers comprises:

   Perform word segmentation processing on the initial answer through a preset word segmentation method to obtain the basic word segmentation contained in the initial answer;

   Converting the basic word segmentation into word vectors, and clustering the word vectors through a clustering algorithm, to obtain a clustering center corresponding to each of the word vectors;

   Obtain the preset semantics corresponding to the cluster centers corresponding to each of the word vectors as the semantic recognition result of the initial answer.
3. The answer verification method for crowdsourcing tasks according to claim 2, wherein said performing word segmentation processing on said initial answer through a preset word segmentation method to obtain the basic word segmentation contained in said initial answer comprises:

   Perform word segmentation analysis on the basic sentence to obtain K word segmentation sequences;

   For each word segmentation sequence, calculate the occurrence probability of each word segmentation sequence according to the word sequence data of the preset training corpus to obtain the occurrence probability of K word segmentation sequences;

   From the occurrence probabilities of the K word segmentation sequences, select the word segmentation sequence corresponding to the occurrence probability that reaches the preset probability threshold as the target word segmentation sequence, and use each word segmentation in the target word segmentation sequence as the initial answer The basic participle included.
4. The method for verifying the answer of the crowdsourcing task according to claim 1, wherein, among the credibility values of all the reference answers, the credibility value with the largest numerical value is selected as the maximum credibility value, and After comparing the maximum credibility value with the preset standard value, and after obtaining the comparison result, the answer verification method of the crowdsourcing task further includes:

   If the comparison result is that the maximum credibility value is less than the preset standard value, the target task is obtained, and the target task is input into a preset model, and a simulated answer is obtained through the preset model ；

   For each type of the reference answer, count the similarity values between the reference answer and the simulated answer to obtain M similarity values, and select the highest similarity value from the M similarity values, and The reference answer corresponding to the similarity value with the largest numerical value is used as a target answer, and the answer answer corresponding to the target answer is confirmed as a verified answer answer.
5. The method for verifying the answer of the crowdsourcing task according to claim 1, wherein, among the credibility values of all the reference answers, the credibility value with the largest numerical value is selected as the maximum credibility value, and After comparing the maximum credibility value with the preset standard value, and after obtaining the comparison result, the answer verification method of the crowdsourcing task further includes:

   The answer object includes a first object and a second object, the answer of the first object is a first answer, the answer of the second object is a second answer, the first object and the first object Both objects have preset weights corresponding to them, and the preset weight of the first object is less than the preset weight of the second object;

   If the comparison result is that the maximum credibility value is less than the preset standard value, a first preset weight corresponding to the first answer answer and a second preset weight corresponding to the second answer answer are obtained Weights;

   Determine the credibility weight of each type of the reference answer according to the reference answer category to which the first answer answer and the second answer answer belong, the first preset weight and the second preset weight;

   Determine the weighted credibility value of each type of reference answer according to the credibility weight of each type of reference answer and a preset weight check method;

   The weighted credibility value with the largest numerical value is selected as the target weighted credibility value, the reference answer corresponding to the target weighted credibility value is obtained as the target answer, and the response answer corresponding to the target answer is confirmed as Verify the answer that passed the verification.
6. The method for verifying the answer of the crowdsourcing task according to claim 5, wherein the weighted credibility value with the largest value is selected as the target weighted credibility value, and the target weighted credibility value corresponds to After the reference answer is used as the target answer, and after the answer answer corresponding to the target answer is confirmed as a verified answer answer, the answer verification method of the crowdsourcing task further includes:

   Acquiring historical response answers of the first object and the second object;

   Judging the proportion of the first subject’s historical response answers that have passed the verification, obtaining the response accuracy rate of the first subject, and judging the proportion of the second subject’s historical response answers that have passed the verification, Obtaining the response accuracy rate of the second object;

   According to the response accuracy rate and the preset classification threshold, the preset weight of the first object and the preset weight of the second object are updated to obtain the updated preset weight and updated weight of the first object The preset weight of the second object.
7. An answer verification device for crowdsourcing tasks, including:

   The initial answer obtaining module is used to obtain each answer answer corresponding to the target task from all the answer answers obtained from the client as an initial answer, wherein each answer answer corresponds to a answer object;

   The semantic recognition result module is used to perform semantic recognition on each of the initial answers through natural language semantic recognition to obtain semantic recognition results of N initial answers, where N is the number of response objects, and N is positive Integer

   The reference answer classification module is used to combine the semantic recognition results in pairs, and use each combination as a set of results, and use the similarity calculation method to count the similarity values between the semantic recognition results in each set of results , If the obtained similarity value is greater than the preset similarity threshold, the two semantic recognition results in the group are regarded as the same type of reference answer, and M type of reference answer is obtained, where M≤N, and N is a positive integer ；

   The credibility value statistics module is used to determine the credibility value of each type of reference answer through a preset consistency check method;

   The credibility value comparison module is used to select the credibility value with the largest value from all the credibility values of the reference answers as the maximum credibility value, and compare the maximum credibility value with the predicted value. Set standard value comparison and get the comparison result;

   A response answer verification module, configured to, if the comparison result is that the maximum credibility value is greater than or equal to the preset standard value, use the reference answer corresponding to the maximum credibility value as a target answer, and The answer answer corresponding to the target answer is confirmed as the answer answer passed the verification.
8. The answer verification device for crowdsourcing tasks according to claim 7, wherein the semantic recognition result module comprises:

   The basic word segmentation acquisition unit is configured to perform word segmentation processing on the initial answer through a preset word segmentation method to obtain the basic word segmentation contained in the initial answer;

   A word vector acquiring unit, configured to convert the basic word segmentation into a word vector, and cluster the word vector through a clustering algorithm to obtain a cluster center corresponding to each word vector;

   The semantic recognition result unit is used to obtain the preset semantics corresponding to the cluster center corresponding to each of the word vectors as the semantic recognition result of the initial answer.
9. A computer device includes a memory, a processor, and computer-readable instructions stored in the memory and capable of running on the processor, and the processor implements the following steps when the processor executes the computer-readable instructions:

   From all the answer answers obtained by the client, obtain each answer answer corresponding to the target task as an initial answer, where each answer answer corresponds to a answer object;

   By means of natural language semantic recognition, semantic recognition is performed on each of the initial answers to obtain semantic recognition results of N initial answers, where N is the number of response objects, and N is a positive integer;

   Combine the semantic recognition results in pairs, and use each combination as a set of results. The similarity calculation method is used to calculate the similarity value between the semantic recognition results in each set of results. If the similarity value is obtained If it is greater than the preset similarity threshold, use the two semantic recognition results in the group as the same type of reference answer to obtain M type of reference answer, where M≤N, and N is a positive integer;

   Determine the credibility value of each type of reference answer through a preset consistency check method;

   From all the credibility values of the reference answers, select the credibility value with the largest numerical value as the maximum credibility value, and compare the maximum credibility value with a preset standard value to obtain a comparison result;

   If the comparison result is that the maximum credibility value is greater than or equal to the preset standard value, the reference answer corresponding to the maximum credibility value is taken as the target answer, and the target answer is The answer is confirmed as the answer that passed the verification.
10. 9. The computer device according to claim 9, wherein said performing semantic recognition on each of said initial answers by means of natural language semantic recognition to obtain semantic recognition results of N initial answers comprises:

    Perform word segmentation processing on the initial answer through a preset word segmentation method to obtain the basic word segmentation contained in the initial answer;

    Converting the basic word segmentation into word vectors, and clustering the word vectors through a clustering algorithm, to obtain a clustering center corresponding to each of the word vectors;

    Obtain the preset semantics corresponding to the cluster centers corresponding to each of the word vectors as the semantic recognition result of the initial answer.
11. 10. The computer device according to claim 10, wherein said performing word segmentation processing on said initial answer through a preset word segmentation method to obtain the basic word segmentation contained in said initial answer comprises:

    Perform word segmentation analysis on the basic sentence to obtain K word segmentation sequences;

    For each word segmentation sequence, calculate the occurrence probability of each word segmentation sequence according to the word sequence data of the preset training corpus to obtain the occurrence probability of K word segmentation sequences;

    From the occurrence probabilities of the K word segmentation sequences, select the word segmentation sequence corresponding to the occurrence probability that reaches the preset probability threshold as the target word segmentation sequence, and use each word segmentation in the target word segmentation sequence as the initial answer The basic participle included.
12. 9. The computer device according to claim 9, wherein, among the reliability values of all the reference answers, the reliability value with the largest numerical value is selected as the maximum reliability value, and the maximum reliability value is selected as the maximum reliability value. The reliability value is compared with the preset standard value, and after the comparison result is obtained, the processor further implements the following steps when executing the computer-readable instruction:

    If the comparison result is that the maximum credibility value is less than the preset standard value, the target task is obtained, and the target task is input into a preset model, and a simulated answer is obtained through the preset model ；

    For each type of the reference answer, count the similarity values between the reference answer and the simulated answer to obtain M similarity values, and from the M similarity values, select the similarity value with the largest numerical value, and The reference answer corresponding to the similarity value with the largest numerical value is used as a target answer, and the answer answer corresponding to the target answer is confirmed as a verified answer answer.
13. 9. The computer device according to claim 9, wherein, among the reliability values of all the reference answers, the reliability value with the largest numerical value is selected as the maximum reliability value, and the maximum reliability value is selected as the maximum reliability value. The reliability value is compared with the preset standard value, and after the comparison result is obtained, the processor further implements the following steps when executing the computer-readable instruction:

    The answer object includes a first object and a second object, the answer of the first object is a first answer, the answer of the second object is a second answer, the first object and the first object Both objects have preset weights corresponding to them, and the preset weight of the first object is less than the preset weight of the second object;

    If the comparison result is that the maximum credibility value is less than the preset standard value, a first preset weight corresponding to the first answer answer and a second preset weight corresponding to the second answer answer are obtained Weights;

    Determine the credibility weight of each type of the reference answer according to the reference answer category to which the first answer answer and the second answer answer belong, the first preset weight and the second preset weight;

    Determine the weighted credibility value of each type of reference answer according to the credibility weight of each type of reference answer and a preset weight check method;

    The weighted credibility value with the largest numerical value is selected as the target weighted credibility value, the reference answer corresponding to the target weighted credibility value is obtained as the target answer, and the response answer corresponding to the target answer is confirmed as Verify the answer that passed the verification.
14. The computer device according to claim 13, wherein the selected weighted credibility value with the largest value is used as the target weighted credibility value, and the reference answer corresponding to the target weighted credibility value is used as the target Answer, and after confirming that the answer answer corresponding to the target answer is a verified answer answer, the processor further implements the following steps when executing the computer-readable instruction:

    Acquiring historical response answers of the first object and the second object;

    Judging the proportion of the first subject’s historical response answers that have passed the verification, obtaining the response accuracy rate of the first subject, and judging the proportion of the second subject’s historical response answers that have passed the verification, Obtaining the response accuracy rate of the second object;

    According to the response accuracy rate and the preset classification threshold, the preset weight of the first object and the preset weight of the second object are updated to obtain the updated preset weight and updated weight of the first object The preset weight of the second object.
15. A computer-readable storage medium having executable code stored on the computer-readable storage medium, and when the executable code is executed by a processor, the steps of the method for verifying the answer of the crowdsourcing task as described below are realized:

    From all the answer answers obtained by the client, obtain each answer answer corresponding to the target task as an initial answer, wherein each answer answer corresponds to a answer object;

    By means of natural language semantic recognition, semantic recognition is performed on each of the initial answers to obtain semantic recognition results of N initial answers, where N is the number of response objects, and N is a positive integer;

    Combine the semantic recognition results in pairs, and use each combination as a set of results. The similarity calculation method is used to calculate the similarity value between the semantic recognition results in each set of results. If the similarity value is obtained If it is greater than the preset similarity threshold, use the two semantic recognition results in the group as the same type of reference answer to obtain M type of reference answer, where M≤N, and N is a positive integer;

    Determine the credibility value of each type of reference answer through a preset consistency check method;

    From all the credibility values of the reference answers, select the credibility value with the largest numerical value as the maximum credibility value, and compare the maximum credibility value with a preset standard value to obtain a comparison result;

    If the comparison result is that the maximum credibility value is greater than or equal to the preset standard value, the reference answer corresponding to the maximum credibility value is taken as the target answer, and the target answer is The answer is confirmed as the answer that passed the verification.
16. 15. The computer-readable storage medium according to claim 15, wherein said performing semantic recognition on each of said initial answers by means of natural language semantic recognition to obtain semantic recognition results of N initial answers comprises:

    Perform word segmentation processing on the initial answer through a preset word segmentation method to obtain the basic word segmentation contained in the initial answer;

    Converting the basic word segmentation into word vectors, and clustering the word vectors through a clustering algorithm, to obtain a clustering center corresponding to each of the word vectors;

    Obtain the preset semantics corresponding to the cluster centers corresponding to each of the word vectors as the semantic recognition result of the initial answer.
17. 15. The computer-readable storage medium according to claim 16, wherein said performing word segmentation processing on said initial answer through a preset word segmentation method to obtain the basic word segmentation contained in said initial answer comprises:

    Perform word segmentation analysis on the basic sentence to obtain K word segmentation sequences;

    For each word segmentation sequence, calculate the occurrence probability of each word segmentation sequence according to the word sequence data of the preset training corpus to obtain the occurrence probability of K word segmentation sequences;

    From the occurrence probabilities of the K word segmentation sequences, select the word segmentation sequence corresponding to the occurrence probability that reaches the preset probability threshold as the target word segmentation sequence, and use each word segmentation in the target word segmentation sequence as the initial answer The basic participle included.
18. The computer-readable storage medium according to claim 15, wherein, among the reliability values of all the reference answers, the reliability value with the largest numerical value is selected as the maximum reliability value, and all the reliability values The maximum credibility value is compared with the preset standard value, and after the comparison result is obtained, when the computer-readable instruction is executed by the processor, the processor is caused to perform the following steps:

    If the comparison result is that the maximum credibility value is less than the preset standard value, the target task is obtained, and the target task is input into a preset model, and a simulated answer is obtained through the preset model ；

    For each type of the reference answer, count the similarity values between the reference answer and the simulated answer to obtain M similarity values, and select the highest similarity value from the M similarity values, and The reference answer corresponding to the similarity value with the largest numerical value is used as a target answer, and the answer answer corresponding to the target answer is confirmed as a verified answer answer.
19. The computer-readable storage medium according to claim 15, wherein, among the reliability values of all the reference answers, the reliability value with the largest numerical value is selected as the maximum reliability value, and all the reliability values The maximum credibility value is compared with the preset standard value, and after the comparison result is obtained, when the computer-readable instruction is executed by the processor, the processor further executes the following steps:

    The answer object includes a first object and a second object, the answer of the first object is a first answer, the answer of the second object is a second answer, the first object and the first object Both objects have preset weights corresponding to them, and the preset weight of the first object is less than the preset weight of the second object;

    If the comparison result is that the maximum credibility value is less than the preset standard value, a first preset weight corresponding to the first answer answer and a second preset weight corresponding to the second answer answer are obtained Weights;

    Determine the credibility weight of each type of the reference answer according to the reference answer category to which the first answer answer and the second answer answer belong, the first preset weight and the second preset weight;

    Determine the weighted credibility value of each type of reference answer according to the credibility weight of each type of reference answer and a preset weight check method;

    The weighted credibility value with the largest numerical value is selected as the target weighted credibility value, the reference answer corresponding to the target weighted credibility value is obtained as the target answer, and the response answer corresponding to the target answer is confirmed as Verify the answer that passed the verification.
20. 18. The computer-readable storage medium of claim 19, wherein the weighted credibility value with the largest value is selected as a target weighted credibility value, and the reference answer corresponding to the target weighted credibility value , As the target answer, and after confirming that the answer answer corresponding to the target answer is a verified answer answer, when the computer-readable instruction is executed by the processor, the processor further executes the following steps:

    Acquiring historical response answers of the first object and the second object;

    Judging the proportion of the first subject’s historical response answers that have passed the verification, obtaining the response accuracy rate of the first subject, and judging the proportion of the second subject’s historical response answers that have passed the verification, Obtaining the response accuracy rate of the second object;

    According to the response accuracy rate and the preset classification threshold, the preset weight of the first object and the preset weight of the second object are updated to obtain the updated preset weight and updated weight of the first object The preset weight of the second object.

Images