CN110580554A

CN110580554A - data processing method and device of interactive service and electronic equipment

Info

Publication number: CN110580554A
Application number: CN201810490617.4A
Authority: CN
Inventors: 薛研歆
Original assignee: Alibaba Group Holding Ltd
Current assignee: Alibaba Huabei Technology Co ltd
Priority date: 2018-05-21
Filing date: 2018-05-21
Publication date: 2019-12-17
Anticipated expiration: 2038-05-21
Also published as: CN110580554B

Abstract

the embodiment of the invention provides a data processing method and device of an interactive service and electronic equipment, wherein the method comprises the following steps: executing an interactive service, wherein the interactive service comprises a plurality of execution stages in which a user participates; counting the behavior data of the user in the interactive service process, and carrying out staged evaluation processing on the behavior data according to the execution stage of the interactive service; and when the accumulated evaluation result of the user in the interactive service process meets the evaluation threshold, triggering the interactive service to end, and executing feedback processing on the user. The scheme of the embodiment of the invention can effectively carry out feedback processing on the user or process the service data according to the behavior data of the user in the interactive service.

Description

Data processing method and device of interactive service and electronic equipment

Technical Field

The present application relates to the field of computer technologies, and in particular, to a data processing method and apparatus for an interactive service, and an electronic device.

Background

at present, in the field of artificial intelligence, the accuracy of artificial intelligence depends not only on algorithm optimization, but also on the size of a training data set. With the increase of the training data magnitude, the task performance can be obviously improved. Therefore, how to acquire larger-scale training data becomes the key for the development of the artificial intelligence field. In the existing manual marking and on-line effect evaluation platform, how to improve marking efficiency and marking quality and reduce marking cost is particularly important for system development. In the prior art, the improvement for manual labeling mainly focuses on the optimization of the excitation mechanism and the recovery mechanism, which is as follows:

(1) Excitation mechanism

In the prior art, the traditional reward mode in the incentive mechanism is reward according to questions, that is, the data to be annotated is distributed in a test question form, and a user gives a reward with a fixed amount no matter whether the user answers one question or not. Because the quality of the recovered data is poor in the mode, the traditional reward mode is gradually optimized in the later period, on one hand, the marking quality of each user is calculated by adding the verification questions, on the other hand, the reward and the ruled line are set, and only the user reaching the qualified line can obtain the reward. Although the existing reward mechanism improves the marking quality to a certain extent, due to strict passing control, users are easy to lay and answer once answering questions meet the passing line, or users wandering near the passing line are completely denied once failing, which affects the data recovery quality to a certain extent.

(2) recovery mechanism

In the prior art, different users understand test questions differently, so that annotation data are very easy to be abnormal, and an annotation platform generally adopts annotation data of multiple people for the same test question so as to improve the accuracy of the annotation data. Therefore, the existing recovery mechanism optimization strategy is to add weights to different users, for example, the weight of a high-level user is higher, the weight of a common user is lower, and although the data recovery quality is improved to a certain extent, the artificial labeling result is too dependent on the number of the high-level users on one hand, and on the other hand, the labeling result is still influenced by the changes of the mood, the state and the like of the users, and the accuracy of data recovery is still not ideal.

Disclosure of Invention

The invention provides a data processing method and device of an interactive service and electronic equipment, which can effectively perform feedback processing on a user or process service data according to behavior data of the user in the interactive service.

In order to achieve the above purpose, the embodiment of the invention adopts the following technical scheme:

in a first aspect, a data processing method for interactive services is provided, including:

Executing an interactive service, wherein the interactive service comprises a plurality of execution stages in which a user participates;

counting the behavior data of the user in the interactive service process, and carrying out staged evaluation processing on the behavior data according to the execution stage of the interactive service;

and when the accumulated evaluation result of the user in the interactive service process meets the evaluation threshold, triggering the interactive service to end, and executing feedback processing on the user.

in a second aspect, another data processing method for interactive services is provided, including:

and when the accumulated evaluation result of the user in the interactive service process meets the evaluation threshold, triggering the interactive service to end, and executing the processing of the service data in the interactive service.

in a third aspect, another data processing method for interactive services is provided, including:

acquiring response data of a user in a response service, wherein the response service comprises a plurality of response stages divided along with the in-depth state of the user response;

Carrying out staged evaluation processing on the answering data according to the answering stage in the answering service;

And when the evaluation result accumulated by the user in the answer service meets the evaluation threshold, triggering to stop issuing the test questions to the user.

In a fourth aspect, a data processing apparatus for interactive services is provided, including:

The service execution module is used for executing interactive services, and the interactive services comprise a plurality of execution stages in which users participate;

The data evaluation module is used for counting the behavior data of the user in the interactive service process and carrying out staged evaluation processing on the behavior data according to the execution stage of the interactive service;

And the user feedback module is used for triggering the interactive service to end and executing feedback processing on the user when the accumulated evaluation result of the user in the interactive service process meets an evaluation threshold.

In a fifth aspect, there is provided another data processing apparatus for interactive services, including:

and the data processing module is used for triggering the interactive service to end and executing the processing of the service data in the interactive service when the accumulated evaluation result of the user in the interactive service process meets the evaluation threshold.

in a sixth aspect, there is provided another data processing apparatus for interactive services, comprising:

the answer data acquisition module is used for acquiring answer data of a user in an answer service, wherein the answer service comprises a plurality of answer stages divided along with the deep state of the answer of the user;

the answer data evaluation module is used for carrying out staged evaluation processing on the answer data according to the answer stage in the answer service;

And the service processing module is used for triggering and stopping the operation of issuing test questions to the user when the evaluation result accumulated by the user in the answer service meets the evaluation threshold.

In a seventh aspect, an electronic device is provided, including:

a memory for storing a program;

A processor, coupled to the memory, for executing the program for:

in an eighth aspect, an electronic device is provided, including:

a memory for storing a program;

A processor, coupled to the memory, for executing the program for:

In a ninth aspect, there is provided another electronic device, including:

a memory for storing a program;

a processor, coupled to the memory, for executing the program for:

according to the data processing method, the data processing device and the electronic equipment for the interactive service, provided by the invention, in the process of executing the interactive service, behavior data of a user are counted, the behavior data are subjected to staged evaluation processing according to the execution stages of the interactive service, and evaluation results of each execution stage are accumulated; when the accumulated evaluation result of the user in the interactive service process meets the evaluation threshold, the interactive service is triggered to end, and feedback processing of the user or processing of service data in the interactive service is executed. The evaluation result of the user behavior data accumulated in the interactive service process is used as the evaluation basis, so that the feedback processing of the user and the processing of the service data in the interactive service can be effectively realized.

the foregoing description is only an overview of the technical solutions of the present application, and the present application can be implemented according to the content of the description in order to make the technical means of the present application more clearly understood, and the following detailed description of the present application is given in order to make the above and other objects, features, and advantages of the present application more clearly understandable.

drawings

Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the application. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:

FIG. 1 is a logic diagram of a user incentive method according to an embodiment of the present invention;

FIG. 2 is a logic diagram of a data recovery method according to an embodiment of the present invention;

Fig. 3a is a flowchart of a data processing method of an interactive service according to an embodiment of the present invention;

fig. 3b is a flowchart of a data processing method of an interactive service according to an embodiment of the present invention;

FIG. 4 is a first flowchart of a user incentive method according to an embodiment of the present invention;

FIG. 5 is a flowchart of a user incentive method according to an embodiment of the present invention;

fig. 6 is a flowchart of a user incentive method provided in the embodiment of the present invention;

fig. 7 is a flow chart of a data processing method of an interactive service according to an embodiment of the present invention;

FIG. 8 is a first flowchart of a data recovery method according to an embodiment of the present invention;

FIG. 9 is a flowchart of a data recovery method according to an embodiment of the present invention;

Fig. 10 is a fourth flowchart of a data processing method of an interactive service according to an embodiment of the present invention;

Fig. 11 is a first block diagram of a data processing apparatus for interactive services according to an embodiment of the present invention;

fig. 12 is a second block diagram of a data processing apparatus for interactive services according to an embodiment of the present invention;

Fig. 13 is a third structural diagram of a data processing apparatus of an interactive service according to an embodiment of the present invention;

Fig. 14 is a fourth structural diagram of a data processing apparatus of an interactive service according to an embodiment of the present invention;

Fig. 15 is a first schematic structural diagram of an electronic device according to an embodiment of the present invention;

Fig. 16 is a second schematic structural diagram of an electronic device according to an embodiment of the present invention;

Fig. 17 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

Detailed Description

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

Currently, in many interactive service scenarios, a service party needs to trigger other service operations by counting behavior data generated by a user during the process of participating in an interactive service, such as recovering service data generated in the interactive service or performing a feedback operation on the user. For example, in a data annotation (a process of answering a question, which is usually performed in a manner that a user answers a test question) scenario, in order to encourage the user to answer the question carefully, so that the quality of retrieved annotation data is higher, an incentive operation is often performed on the user by using answer data (behavior data) based on the user in the prior art. The basic reward mode (corresponding to the feedback operation performed on the user according to the behavior data of the user) is reward on a per-question basis, that is, the user is rewarded with a fixed amount of money every time the user answers a question, and in this way, for the user, the same reward can be obtained no matter whether the answer is correct or not, so that the data recovery quality is poor. Then, data labeling is developed to adopt a mode of adding check questions in test questions to calculate the labeling quality of a user and setting the answer and ruled lines of the user, so that a traditional incentive mechanism is optimized. Although the data recovery quality is improved to a certain extent by the mode, on one hand, the data recovery quality marked by the incentive mechanism is seriously dependent on the number of advanced users, and meanwhile, even the advanced users have different performances in different types of subjects; on the other hand, due to strict passing line limitation, some users do not answer questions seriously once passing the passing line, or some users who are near the passing line but do not pass the passing line dare not to answer questions continuously, thereby affecting the data recovery quality and efficiency.

In a service data recovery scenario (corresponding to the above-mentioned user-based behavior data for processing service data), currently, a method of scoring different users according to different weights is often adopted to determine whether the conditions for recovering labeled data are met, for example, a common user marks 1 point at a time, a senior user marks 2 points at a time, and when the same labeled sample needs to be recovered in 5 time points, at least the labeled data of two senior users and one common user need to be recovered. Although the data recovery amount is reduced to a certain extent, the user has different understanding degrees on different samples, and the labeling states are influenced by emotion, psychology and the like, so the weights preset for different users cannot accurately reflect the labeling levels of the users under different labeling samples, and the method has no obvious improvement effect on the data recovery efficiency.

The technical scheme starts from the source problems of how to evaluate user behavior data in the interactive service process and how to execute feedback processing of users and processing of service data based on evaluation results, and solves the problems of the prior art, such as the defects of incentive rules of user answers in a data labeling scene, and low service data recovery quality and efficiency.

In the following, the answer data of the user in the data tagging scene is evaluated in stages, and the incentive processing of the user and the recovery processing of the service data are described in detail according to the accumulated evaluation result.

For example, in a data annotation scenario, problem data to be annotated is grouped in the form of test questions, each test question can be allocated to one or more test question groups, the same test question group can contain a plurality of test questions, and the test question groups are distributed to users; the process of the user answering a set of test questions is marked as an interactive service. Meanwhile, the weighted value of each test question is set as the basis for evaluating the answering behavior of the user, and the weighted value can be determined in various ways, such as randomly, according to the difficulty level of the test question and according to the position sequence of the test question in the test question group; and finally, counting the answering conditions of the user by taking the test question group as a unit, wherein the user can be considered as an execution stage for the user to participate in the data annotation service when answering each question in the test question group. If the evaluation results accumulated by the questions in the test question group answered by the user in sequence meet the evaluation threshold, the user is triggered to stop issuing the test questions, and feedback processing can be performed on the user based on the accumulated evaluation results of the questions answered by the user, such as rewarding the user, or business data processing, such as data recovery on completed labeled data (answers to questions).

for example, evaluating the behavior data of the user at each execution stage according to the weight value of each test question, and performing feedback processing on the user according to the accumulated evaluation result may include: if the user answers a wrong question, the evaluation scores of the question answered by the user in the test question group are eliminated, and the test questions are not issued continuously. For example, fig. 1 is a logic diagram of a user incentive method provided by an embodiment of the present invention, including but not limited to that shown in fig. 1, distributing data to be annotated to users in the form of test question groups, where each test question group may be provided with four questions, corresponding to the four answering stages, the weighted values of the test questions can be sequentially increased according to the sequence of the test questions, and the user is evaluated and scored according to the weighted values, as shown in figure 1, the weighted values of the questions can be set as 1, 2 and 4 in sequence according to the position sequence of the test questions, and when the user answers one question each time, corresponding scores can be accumulated according to the weighted value of the question, for example, the user can accumulate 1 point for answering the first question and answer the second question, 1 point can be added again, the third question can be answered, 2 points can be added again, the fourth question can be answered, 4 points can be added again, when the set of questions are all answered, 8 points accumulated by the user in the set of questions are all awarded to the user. If the user answers any question in the answering process, all points accumulated in the group of answers by the user can be cleared, and the issuing of test questions to the user can be stopped after the evaluation scores are accumulated to a certain threshold value or the wrong questions are answered.

in a service data recovery scenario, firstly, with test question groups as units, the confidence scores are performed on the screened test questions according to the evaluation conditions obtained by the user incentive method provided by the scheme, the confidence score is used as a basis for whether to recover labeled data, for example, a confidence score threshold value can be preset, and when the confidence score corresponding to the test question is greater than the preset threshold value, the labeled data is recovered, wherein the labeled data includes both the test question data and answer data of the test question given by the user. For example, fig. 2 is a logic schematic diagram of the data recovery method according to the embodiment of the present invention, including but not limited to that shown in fig. 2, the confidence score of each question may be set according to a weight value of each test question, and the determination manner of the weight value may be various, for example, may be determined randomly, may also be determined according to the difficulty level of the test question, and may also be specifically determined according to the position order of the test question in the test question group. As shown in fig. 2, four test questions may be set in the same test question group, and according to the position order of the test questions in the test question group, the confidence score of the first question may be set to 1, the confidence score of the second question may be set to 1, the confidence score of the third question may be set to 2, and the confidence score of the fourth question may be set to 4. And counting the sum of confidence scores of the same test question in the same or different test question groups, setting a preset confidence score threshold of the test question as 6, and when the received sum of the confidence scores of the test questions is equal to or more than 6 minutes, recovering the labeling data corresponding to the test question and not issuing the test question to the user.

The technical solution of the present application is further illustrated by the following examples.

Example one

as shown in fig. 3a, which is a flowchart of a first data processing method of an interactive service provided in an embodiment of the present invention, the method includes:

s310, interactive service is executed, and the interactive service comprises a plurality of execution stages in which the user participates.

Interactive services, also called interactive services, are services that provide two-way information exchange between users or between a user and a host. For example, session type services, message type services and retrieval type services all belong to interactive services.

The interactive service in the scheme includes a plurality of execution stages in which the user participates, that is, the user may participate in at least two execution stages in the process of executing the interactive service once, and the user may execute at least one interactive operation in each execution stage, for example, clicking a mouse, inputting characters, and the like.

s320, counting the behavior data of the user in the interactive service process, and carrying out staged evaluation processing on the behavior data according to the execution stage of the interactive service.

The behavior data of the user may be data that is generated along with the user performing an interactive operation, such as an operation manner (mouse, keyboard, touch, etc.) of the user, operation contents (an object clicked by the mouse, text content input by the keyboard, content of a touch operation, etc.), or statistical data generated based on statistics of the data.

In the process of the user participating in the interactive service, the behavior data of the user can be evaluated in stages according to the execution stage of the interactive service. For example, under the requirement of performing deeper data processing or other business operations based on the behavior data of the user, the behavior data of the user may be evaluated first, and good behavior data may be screened out for subsequent operations. In order to make the overall quality effect of the screened user behavior data better, the behavior data can be evaluated in stages according to the execution stage of the interactive service process.

For example, in a data tagging scenario in which answer data of a user for a specific question is collected in a question-answering manner, in order to ensure that all screened question answers are obtained by the user in a serious answer condition, behavior data of the user in an answer process may be counted first, and then behavior data of the user may be evaluated in stages according to each execution stage in the answer process. For example, the process of answering questions 1-3 by the user is the first execution phase, the process of answering questions 4-6 is the second execution phase, … …, and so on. The user behavior data of different execution phases are evaluated separately, and the evaluation weight of each execution phase may be increased step by step, for example, the evaluation weight may be increased step by step or decreased step by step. The specific evaluation weights and evaluation procedures can be customized according to the target data to be obtained based on the behavior data or the requirements of the target service.

And S330, when the accumulated evaluation result of the user in the interactive service process meets the evaluation threshold, triggering the interactive service to end, and executing feedback processing on the user.

The evaluation results of the user in one interactive service process can be accumulated while the behavior data of the user in each execution stage in the interactive service process is evaluated, and whether the behavior data of the user meets the requirement of extracting target data or performing target service operation can be judged according to the accumulated evaluation results.

the evaluation threshold may be a condition that meets a certain evaluation requirement, for example, in the above data tagging scenario that answer data of the user for a specified question is collected in a question-and-answer manner, the evaluation threshold may be a total score accumulated, or a total score that needs to be accumulated when the execution reaches a certain execution stage.

The data processing method of the interactive service provided by the invention is characterized in that in the process of executing the interactive service, the behavior data of a user is counted, the behavior data is evaluated in stages according to the execution stages of the interactive service, and the evaluation results of each execution stage are accumulated; and when the accumulated evaluation result of the user in the interactive service process meets the evaluation threshold, triggering the interactive service to end, and executing feedback processing on the user. The evaluation result of the accumulated user behavior data in the interactive service process is taken as a basis, so that the feedback processing of the user can be effectively realized, and the high-quality user is saved.

Further, as shown in fig. 3b, a flowchart 2 of a data processing method of the interactive service provided in real time in the present invention is shown, and on the basis of the method shown in fig. 3a, the embodiment specifically describes the step 320, which includes the following steps:

S321, presetting evaluation weights of user behaviors at different execution stages in the interactive service process.

the evaluation weight is used for describing the weight proportion of behavior data corresponding to a certain execution stage in one interactive service of the user in the evaluation of the behavior data. For example, as the interactive service execution process goes deep, the interactive operation in the subsequent execution stage is more important, the application value of the user behavior data in this stage may be higher, or in order to encourage the user to execute the user behavior in the later stage to ensure the quality, the evaluation weight of each execution stage in the interactive service execution process may be preset to increase gradually along with the development process of the interactive service.

for example, in the data labeling scenario, each test question group includes four questions corresponding to four answering stages, and the weighted values of the questions are sequentially set to be 1, 2, and 4 according to the position sequence of the test questions in the test question group.

And S322, evaluating the behavior data of the user in different execution stages by adopting the evaluation weights of the corresponding execution stages.

for example, in the data labeling scenario, when a user answers a question, a corresponding score can be obtained according to the weight value of the question.

and S323, accumulating the evaluation results of each execution stage of the user in the interactive service process.

And accumulating the evaluation results obtained by the user in each execution stage from front to back in one interactive service to obtain accumulated evaluation. For example, as shown in fig. 1, in the data annotation scenario, scores obtained by the user at each answering stage in the answering process may be accumulated.

In addition, the following contents of the present embodiment also describe expanded contents of performing feedback processing on a user according to an evaluation result of the user based on answer data in a data annotation scene.

Fig. 4 is a first flowchart of a user incentive method according to an embodiment of the present invention, including:

and S410, issuing the data to be annotated to the user in the form of test question groups, wherein each test question group comprises a plurality of test questions.

specifically, the data to be labeled refers to data to be labeled by a user, the data to be labeled is issued to the user in the form of test question groups, the user labels the data to be labeled in the form of answer, each test question group may include a plurality of test questions, the test questions in different test question groups may be arranged and combined at will, and some of the test questions in two test question groups may be repeated.

and S420, counting the answer condition of the user by taking the test question groups as units, and if the user answers all the test questions in one test question group, carrying out user incentive scoring according to the weight value of each test question to be used as a reward basis for the user.

specifically, the above-mentioned criterion for judging whether the user answers or incorrectly answers is based on the preset answer of the test question, and the answer may be preset as a standard correct answer (e.g. correct answer option, correct answer noun, correct number, etc.), or may be preset as a certain answer range or standard, so long as the answer result of the user corresponds to the standard correct answer, or is within the range or meets the standard, the user is considered to answer the question, otherwise, the user is considered to answer the question incorrectly. When the user answers all the test questions in the same test question group, the answer condition of the user in the test question group is excited and scored according to the weight value of each test question, and the weight value can be determined in various ways, for example, the weight value can be determined randomly, can also be determined according to the difficulty degree of the test questions, and can also be specifically determined according to the position sequence of the test questions in the test question group. And (4) according to the weight value of each test question, carrying out incentive scoring on the user, wherein the accumulated integral obtained by the user in the test question group is the reward obtained by the user for all questions of the test question group through all answers.

In an actual application scenario, the determination of the weighted value of the test question can be performed in various ways, for example, the determination can be performed according to the difficulty level of the test question, the weighted value can be set to 1 for the non-judgment question, the weighted value can be set to 2 for the selected question, and the weighted value can be set to 3 for the simple answer question; the weighted values may also be set according to the position order of the test questions in the test question group, for example, the weighted value of the first question is set to 1, and the weighted value of the second question is set to 2.

preferably, the test question group includes a fixed number of test questions, the weight value of each test question increases with the increase of the question number, and the larger the weight value is, the higher the corresponding obtained incentive score is.

Specifically, the number of test questions in each test question group may be a fixed number, and may be set to 4 to 8, for example. The weighted value of each test question can be determined according to the position sequence of the test question in the test question group, that is, the weighted value is increased along with the increase of the question number, the increase of the weighted value is not limited to the increase of the equal step length, for example, the weighted value of the first question is set to be 1, the weighted value of the second question is set to be 1, the weighted value of the third question is set to be 2, the weighted value of the fourth question is set to be 4, that is, the weighted value of the test question is increased along with the increase of the question number, correspondingly, the accumulated incentive score of the user is higher, so the enthusiasm of the user for responding to the challenge is increased along with the progress of the question making, and the enthusiasm of the user for.

Further, fig. 5 is a flowchart of a user incentive method according to an embodiment of the present invention, which may be a refinement of the content of the method shown in fig. 4. As shown in fig. 5, the method includes:

And S510, issuing the data to be annotated to the user in the form of test question groups, wherein each test question group comprises a plurality of test questions. The step S510 is the same as the step S410.

s520, counting the answering conditions of the user by taking the test question groups as units, and if the user does not answer all the test questions in one test question group, clearing the incentive scores of other answered test questions in the test question group.

Specifically, if the user does not answer any of the questions in the test question group, all the accumulated incentive points of the user in the test question group are cleared, and the user cannot obtain the reward for answering the test question group.

in an actual application scene, assuming that one test question group has four questions, the weighted value of each test question is 1, 2 and 4 in sequence according to the question number, the corresponding incentive score can be set to be 1, 2 and 4, if a user answers all the questions in the test question group, the incentive score can be obtained for 8 at one time; if the user answers the first two questions, the current accumulated points are 2 points, and the user answers the wrong question when answering the third question, the accumulated points of the user in the test question group are all cleared, and the user also loses the right to answer the fourth question continuously, and cannot obtain the question answering reward of the test question group. Therefore, the disadvantage that the user answers the following question draft rate because of obtaining the reward which is satisfied after answering a part of questions is avoided, and the quality of data recovery is improved.

Further, fig. 6 is a flowchart of a user incentive method provided by the embodiment of the present invention, which may be a refinement of the content of the method shown in fig. 4 or fig. 5. As shown in fig. 6, the method further includes:

s610, grouping the data to be annotated in a test question mode to form a plurality of test question groups, and distributing the same test question to at least one test question group.

Specifically, step S610 may also be performed before performing step S410 or S510. The data to be labeled are grouped in a test question mode to form a plurality of test question groups, the same test question can be uniquely distributed in a single test question group or repeatedly appear in a plurality of test question groups, and meanwhile, each test question is guaranteed to be distributed to at least one test question group.

After this step is performed, the content of the steps in fig. 4 or fig. 5 may be continued.

The data processing method of the interactive service provided by the invention shows a specific mode of carrying out staged evaluation processing on the behavior data of the user according to the execution stage of the interactive service, so that the evaluation result has higher reference value.

Example two

based on the above-mentioned data processing scheme of interactive services, this embodiment may adopt the evaluation mode of the user behavior data provided in the previous embodiment, and after the evaluation processing is performed on the behavior of the user to obtain an evaluation result, the service data is processed by using the evaluation result. As shown in fig. 7, a flow chart of a data processing method for interactive services provided in the embodiment of the present invention includes the following steps:

s710, executing interactive service, wherein the interactive service comprises a plurality of execution stages participated by a user;

s720, the behavior data of the user in the interactive service process is counted, and the behavior data is evaluated in stages according to the execution stage of the interactive service.

steps S710 to S720 correspond to steps S310 to S320 described above.

And S730, when the accumulated evaluation result of the user in the interactive service process meets the evaluation threshold, triggering the interactive service to end, and executing the processing of the service data in the interactive service.

The data processing method of the interactive service provided by the invention is characterized in that in the process of executing the interactive service, the behavior data of a user is counted, the behavior data is evaluated in stages according to the execution stages of the interactive service, and the evaluation results of each execution stage are accumulated; and when the accumulated evaluation result of the user in the interactive service process meets the evaluation threshold, triggering the interactive service to end, and executing the processing of the service data in the interactive service. The evaluation result of the accumulated user behavior data in the interactive service process is taken as a basis, so that the processing effect of the target service data can be effectively realized.

Further, referring to the content shown in fig. 3b, step S720 in this embodiment may also be embodied as the content of the steps S321 to S323, which is not described herein again.

In addition, the following contents of this embodiment also describe the expanding contents of performing data processing on the interactive service according to the evaluation result of the user based on the answer data in the data annotation scene.

Fig. 8 is a first flowchart of a data recovery method according to an embodiment of the present invention, including:

And 810, counting the answer conditions of the user by taking the test question groups as units, and extracting the test question groups of complete answer pairs, wherein each test question group comprises a plurality of test questions, and the contents of the test questions are data to be labeled.

specifically, according to the user answer condition, the labeling data is counted by taking the test question group as a unit, the labeling data comprises data to be labeled and answers given by the user for the test questions, the data to be labeled is the content of the test questions, and then the test question group of the complete answer pair of the user is extracted.

In a specific application scenario, the answer condition of the user can be judged based on the incentive score obtained by each user answering test question group in the previous embodiment, if the incentive score of the user corresponding to one test question group is 0, the test question group is not completely answered by the current user; if the user incentive score corresponding to one test question group is the preset total incentive score accumulated finally by the test question group, the test question group is completely answered by the current user; if a test question group corresponds to a user incentive score other than 0, then the test question group is fully paired by the current user (corresponding to the case where the cumulative score is cleared to 0 if the questions in the test question group are not fully paired).

And S820, according to the test question group of the complete answer pair, scoring the test question confidence coefficient according to the weight value of each test question to be used as a basis for recovering the label data corresponding to the test question.

Specifically, each test question is preset with a weighted value, and the weighted value can be determined in various ways, for example, the weighted value can be determined randomly, can also be determined according to the difficulty level of the test question, and can also be determined specifically according to the position sequence of the test question in the test question group. And according to the weight values of the test questions in different test question groups, scoring the confidence degree of the test questions, wherein the confidence degree score represents the credibility of the corresponding labeling data, and the higher the confidence degree is, the higher the credibility of the labeling data is, so that the confidence degree score of each test question is used as the basis for judging whether to recycle the labeling data.

Preferably, in the above method, the weight value of each question increases as the question number increases, and the higher the weight value is, the higher the confidence score is obtained accordingly.

specifically, as mentioned in the first embodiment, the weight value of each test question may be determined according to the position sequence of the test question in the test question group, that is, the weight value increases with the increase of the question number, and the increase of the weight value is not limited to be equal to the step increase, for example, the first question weight value may be set to be 1, the second question weight value may be 1, the third question weight value may be 2, and the fourth question weight value may be 4, that is, as the question number increases, the test question weight value tends to increase as a whole, and accordingly, as the weight value increases, the confidence score obtained by the test question also increases.

in an actual application scenario, it is assumed that a first question weight value is set to be 1, a second question weight value is set to be 1, a third question weight value is set to be 2, and a fourth question weight value is set to be 4 in advance according to a question number sequence, when a user answers all questions of the test question group, a confidence score for the second question can be increased by 1, a confidence score for the third question can be increased by 2, and a confidence score for the fourth question can be increased by 4. For the same user, the difficulty of answering the first three questions simultaneously is higher than that of answering the first question, so that a corresponding higher confidence score is set, and the accuracy of data recovery is improved.

Further, fig. 9 is a flowchart of a second data recovery method provided in the embodiment of the present invention, which can be regarded as a refinement of the steps of the method shown in fig. 8, as shown in fig. 9, a specific implementation manner of the step S820 includes the following steps:

s910, for the same test question, if the confidence score corresponding to the test question is greater than the preset confidence threshold, the labeling data corresponding to the test question is recovered.

specifically, a confidence threshold may be preset for each test question, and when the confidence score obtained for the test question is greater than the preset threshold, the annotation data corresponding to the test question is recovered, and the annotation data is no longer provided to the user.

in an actual application scenario, it is assumed that a preset confidence threshold of a certain test question is 6, a first question weight value is 1, a second question weight value is 1, a third question weight value is 2, and a fourth question weight value is 4, which are preset in each test question group according to the question number sequence, and when a user answers all questions in a certain test question group, a confidence score corresponding to the weight values can be obtained. When the cumulative confidence score obtained by the test question in the same or different test question groups reaches 6 minutes, the recovery of the labeling data corresponding to the test question is completed, and the labeling data is not issued to the user.

furthermore, the data recovery method provided by the invention counts the answer condition of each user by taking the test question group as a unit, judges whether to recover the labeled data according to the corresponding confidence score accumulation condition of the test questions aiming at the test question group of the complete answer pair of the user, sets different weight values for the test questions in the test question group, and then sets the confidence score of the test questions according to the weight values, thereby effectively improving the data recovery quality and efficiency.

EXAMPLE III

based on the above-mentioned scheme idea of data processing of interactive services, this embodiment provides a control method for interactive services under data annotation. As shown in fig. 10, a fourth flowchart of a data processing method for interactive services provided in the embodiment of the present invention includes the following steps:

S101, response data of a user in a response service is obtained, wherein the response service comprises a plurality of response stages which are divided along with the in-depth state of the user response.

For example, in the interactive service of data annotation, a service side issues question data to a user, and then behavior data of the user is evaluated according to answer data generated by the user in the answering process. The one-time answer class service may include a plurality of answer stages divided according to the in-depth state of the user's answer. For example, in the above data tagging scenario, a one-time answer service may include one test question group, and each test question group is configured to include four questions corresponding to four answer stages. S102, the answering data is evaluated in stages according to answering stages in the answering service.

for example, the answer data of the user during the answer process may be counted, and then the answer data of the user may be evaluated in stages according to each execution stage during the answer process. For example, the process of answering questions 1-3 by the user is the first execution phase, the process of answering questions 4-6 is the second execution phase, … …, and so on. The user behavior data of different execution phases are evaluated separately, and the evaluation weight of each execution phase may be increased step by step, for example, the evaluation weight may be increased step by step or decreased step by step. The specific evaluation weights and evaluation procedures can be customized according to the target data to be obtained based on the behavior data or the requirements of the target service.

And S103, when the evaluation result accumulated by the user in the answer service meets the evaluation threshold, triggering to stop issuing the test questions to the user.

The method can evaluate the answer data of the user at each execution stage in the process of participating in the answer service, and simultaneously accumulate the evaluation results of the user in the process of one-time answer service, and can judge whether the answer data of the user meets the requirements of extracting target data or performing target service operation according to the accumulated evaluation results.

In addition, the method content in this embodiment may be used as the refinement of some method steps in the above embodiments, and specific details and expansion schemes may refer to the content in the above embodiments, which is not described herein again

The data processing method of the interactive service provided by the invention is characterized in that in the process of executing the answer interactive service, the answer data of the user is counted, the answer data is evaluated in stages according to the answer stages of the answer service, and the evaluation results of each answer stage are accumulated; and when the accumulated evaluation result of the user in the answering service process meets the evaluation threshold, triggering to stop issuing test questions. The evaluation result of the accumulated user behavior data in the interactive service process is taken as a basis, so that the control on the execution process of the answer service can be effectively realized, and the operation cost is reduced.

Example four

Fig. 11 is a block diagram of a data processing apparatus for interactive services according to an embodiment of the present invention, which may be used to execute the method steps shown in fig. 3a, as shown in fig. 11, the apparatus includes:

a service execution module 111, configured to execute an interactive service, where the interactive service includes multiple execution phases in which a user participates;

the data evaluation module 112 is used for counting behavior data of users in the interactive service process and performing staged evaluation processing on the behavior data according to the execution stage of the interactive service;

and the user feedback module 113 is configured to trigger the interactive service to end when the evaluation result accumulated by the user in the interactive service process meets the evaluation threshold, and perform feedback processing on the user.

further, as shown in fig. 12, based on the device structure shown in fig. 11, the data evaluation module 112 may include:

The weight setting unit 121 is configured to preset evaluation weights of user behaviors at different execution stages in the interactive service process;

the phase evaluation unit 122 is configured to perform evaluation processing on behavior data of the user in different execution phases by using evaluation weights of the corresponding execution phases;

And the evaluation accumulation unit 123 is configured to accumulate the evaluation results of each execution stage in the interactive service process of the user.

The device structure shown in fig. 12 may be used to perform the method steps shown in fig. 3 b.

further, the evaluation weight of each execution stage in the interactive service process may be gradually increased along with the development process of the interactive service.

further, the user feedback module 113 described above may be used to perform an incentive process for the user.

the data processing device of the interactive service provided by the invention counts the behavior data of the user in the process of executing the interactive service, carries out staged evaluation processing on the behavior data according to the execution stages of the interactive service, and accumulates the evaluation results of each execution stage; and when the accumulated evaluation result of the user in the interactive service process meets the evaluation threshold, triggering the interactive service to end, and executing feedback processing on the user. The evaluation result of the accumulated user behavior data in the interactive service process is taken as a basis, so that the feedback processing of the user can be effectively realized, and the high-quality user is saved.

EXAMPLE five

fig. 13 is a third block diagram of a data processing apparatus for interactive services according to an embodiment of the present invention, where the apparatus is configured to execute the method steps shown in fig. 7, and as shown in fig. 13, the apparatus includes:

And the data processing module 131 is configured to trigger the end of the interactive service and execute processing on service data in the interactive service when an evaluation result accumulated by the user in the interactive service process meets an evaluation threshold.

Further, on the basis of the apparatus structure shown in fig. 13, the data evaluation module 112 may further include:

Further, the evaluation weight of each execution stage in the interactive service process is gradually increased along with the development process of the interactive service.

Further, the data processing module 131 is specifically configured to perform a recycling process on the service data specified in the interactive service.

The data processing device of the interactive service provided by the invention counts the behavior data of the user in the process of executing the interactive service, carries out staged evaluation processing on the behavior data according to the execution stages of the interactive service, and accumulates the evaluation results of each execution stage; and when the accumulated evaluation result of the user in the interactive service process meets the evaluation threshold, triggering the interactive service to end, and executing the processing of the service data in the interactive service. The evaluation result of the accumulated user behavior data in the interactive service process is taken as a basis, so that the processing effect of the target service data can be effectively realized.

EXAMPLE six

Fig. 14 is a block diagram of a data processing apparatus of an interactive service according to an embodiment of the present invention, where the apparatus is configured to execute the method steps shown in fig. 10, and as shown in fig. 14, the apparatus includes:

the answer data acquiring module 141 is configured to acquire answer data of a user in an answer service, where the answer service includes multiple answer stages divided according to a deep state of user answers;

The answer data evaluation module 142 is used for performing staged evaluation processing on the answer data according to the answer stage in the answer service;

and the service processing module 143 is configured to trigger to stop issuing the test question to the user when the evaluation result accumulated by the user in the answer service satisfies the evaluation threshold.

The data processing device of the interactive service provided by the invention counts the answer data of the user in the process of executing the answer interactive service, carries out staged evaluation processing on the answer data according to the answer stages of the answer service, and accumulates the evaluation results of each answer stage; and when the accumulated evaluation result of the user in the answering service process meets the evaluation threshold, triggering to stop issuing test questions. The evaluation result of the accumulated user behavior data in the interactive service process is taken as a basis, so that the control on the execution process of the answer service can be effectively realized, and the operation cost is reduced.

EXAMPLE seven

the fourth embodiment describes an overall architecture of a data processing apparatus for interactive services, and the functions of the apparatus can be implemented by an electronic device, as shown in fig. 15, which is a schematic structural diagram of the electronic device according to the first embodiment of the present invention, and specifically includes: a memory 151 and a processor 152.

And a memory 151 for storing a program.

In addition to the above-described programs, the memory 151 may also be configured to store other various data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phonebook data, messages, pictures, videos, and so forth.

The memory 151 may be implemented by any type or combination of volatile or non-volatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks.

A processor 152, coupled to the memory 151, for executing programs in the memory 151 for:

Counting behavior data of the user in the interactive service process, and performing staged evaluation processing on the behavior data according to the execution stage of the interactive service;

The above specific processing operations have been described in detail in the foregoing embodiments, and are not described again here.

further, as shown in fig. 15, the electronic device may further include: communication components 153, power components 154, audio components 155, a display 156, and other components. Only some of the components are schematically shown in fig. 15, and it is not meant that the electronic device includes only the components shown in fig. 15.

The communication component 153 is configured to facilitate wired or wireless communication between the electronic device and other devices. The electronic device may access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 153 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 153 further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

a power supply component 154 provides power to the various components of the electronic device. The power components 154 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for an electronic device.

audio component 155 is configured to output and/or input audio signals. For example, audio component 155 includes a Microphone (MIC) configured to receive external audio signals when the electronic device is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals may further be stored in the memory 151 or transmitted via the communication component 153. In some embodiments, audio component 155 also includes a speaker for outputting audio signals.

The display 156 includes a screen, which may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

Example eight

The fifth embodiment describes an overall architecture of a data processing apparatus for interactive services, and the functions of the apparatus can be implemented by an electronic device, as shown in fig. 16, which is a schematic structural diagram of the electronic device according to the second embodiment of the present invention, and specifically includes: a memory 161 and a processor 162.

the memory 161 stores programs.

In addition to the above-described programs, the memory 161 may also be configured to store other various data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phonebook data, messages, pictures, videos, and so forth.

The memory 161 may be implemented by any type or combination of volatile or non-volatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks.

A processor 162, coupled to the memory 161, for executing programs in the memory 161 to:

further, as shown in fig. 16, the electronic device may further include: communication components 163, power components 164, audio components 165, display 166, and other components. Only some of the components are schematically shown in fig. 16, and it is not meant that the electronic device includes only the components shown in fig. 16.

The communication component 163 is configured to facilitate wired or wireless communication between the electronic device and other devices. The electronic device may access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 163 receives a broadcast signal or broadcast associated information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 163 further includes a Near Field Communication (NFC) module to facilitate short-range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

And a power supply component 164 that provides power to the various components of the electronic device. The power components 164 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for an electronic device.

audio component 165 is configured to output and/or input audio signals. For example, audio component 165 includes a Microphone (MIC) configured to receive external audio signals when the electronic device is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals may further be stored in the memory 161 or transmitted via the communication component 163. In some embodiments, audio component 165 further comprises a speaker for outputting audio signals.

The display 166 includes a screen, which may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

Example nine

The fifth embodiment describes an overall architecture of a data processing apparatus for interactive services, and the functions of the apparatus can be implemented by an electronic device, as shown in fig. 17, which is a schematic structural diagram of the electronic device according to the second embodiment of the present invention, and specifically includes: a memory 171 and a processor 172.

the memory 171 stores a program.

In addition to the above programs, the memory 171 may also be configured to store other various data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phonebook data, messages, pictures, videos, and so forth.

The memory 171 may be implemented by any type or combination of volatile or non-volatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks.

A processor 172, coupled to the memory 171, for executing programs in the memory 171 to:

Further, as shown in fig. 17, the electronic device may further include: communication components 173, power components 174, audio components 175, display 176, and the like. Only some of the components are schematically shown in fig. 17, and it is not meant that the electronic device includes only the components shown in fig. 17.

the communication component 173 is configured to facilitate wired or wireless communication between the electronic device and other devices. The electronic device may access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 173 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 173 further includes a Near Field Communication (NFC) module to facilitate short-range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

a power supply component 174 provides power to the various components of the electronic device. The power components 174 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for an electronic device.

Audio component 175 is configured to output and/or input audio signals. For example, audio component 175 includes a Microphone (MIC) configured to receive external audio signals when the electronic device is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals may further be stored in the memory 171 or transmitted via the communication component 173. In some embodiments, audio component 175 also includes a speaker for outputting audio signals.

The display 176 includes a screen, which may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

those of ordinary skill in the art will understand that: all or a portion of the steps of implementing the above-described method embodiments may be performed by hardware associated with program instructions. The program may be stored in a computer-readable storage medium. When executed, the program performs steps comprising the method embodiments described above; and the aforementioned storage medium includes: various media that can store program codes, such as ROM, RAM, magnetic or optical disks.

Finally, it should be noted that: the above embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method of interactive service is characterized by comprising the following steps:

2. the method of claim 1, wherein the step of counting behavior data of users during the interactive service process and performing a staged evaluation process on the behavior data according to the execution stage of the interactive service comprises:

presetting evaluation weights of user behaviors at different execution stages in the interactive service process;

Evaluating the behavior data of the user in different execution stages by adopting the evaluation weights of the corresponding execution stages;

And accumulating the evaluation results of the user at each execution stage in the interactive service process.

3. The method of claim 2, wherein the evaluation weight of each execution stage in the interactive service process is increased gradually along with the development of the interactive service.

4. The method according to any one of claims 1-3, wherein the performing feedback processing on the user comprises:

an incentive process for the user is performed.

5. a data processing method of interactive service is characterized by comprising the following steps:

6. The method of claim 5, wherein the step of counting behavior data of users during the interactive service process and performing a staged evaluation process on the behavior data according to the execution stage of the interactive service comprises:

7. the method of claim 6, wherein the evaluation weight of each execution stage in the interactive service process is increased gradually along with the development of the interactive service.

8. the method according to any of claims 5-7, wherein said performing processing of service data in interactive services comprises:

And executing the recovery processing of the service data specified in the interactive service.

9. a data processing method of interactive service is characterized by comprising the following steps:

10. A data processing apparatus for interactive services, comprising:

11. The apparatus of claim 10, wherein the data evaluation module comprises:

The weight setting unit is used for presetting the evaluation weights of the user behaviors at different execution stages in the interactive service process;

The stage evaluation unit is used for evaluating the behavior data of the user in different execution stages by adopting the evaluation weights of the corresponding execution stages;

and the evaluation accumulation unit is used for accumulating the evaluation results of the user at each execution stage in the interactive service process.

12. The apparatus of claim 11, wherein the evaluation weight of each execution stage in the interactive service process is increased gradually along with the development of the interactive service.

13. the apparatus of any one of claims 10-12, wherein the user feedback module is configured to perform an incentive process for a user.

14. a data processing apparatus for interactive services, comprising:

15. The apparatus of claim 14, wherein the data evaluation module comprises:

16. The apparatus of claim 15, wherein the evaluation weight of each execution stage in the interactive service process is increased gradually along with the development of the interactive service.

17. the apparatus according to any of claims 14-16, wherein the data processing module is specifically configured to perform a recycling process on the service data specified in the interactive service.

18. a data processing apparatus for interactive services, comprising:

19. an electronic device, comprising:

a memory for storing a program;

A processor, coupled to the memory, for executing the program for:

20. an electronic device, comprising:

a memory for storing a program;

a processor, coupled to the memory, for executing the program for:

21. an electronic device, comprising:

a memory for storing a program;

A processor, coupled to the memory, for executing the program for: