JP2024088466A - Information processing device, information processing method, and information processing program - Google Patents

Abstract

[Problem] To provide an information processing device, information processing method, and information processing program that can promote crowdsourcing. [Solution] The information processing device according to the present application includes an acquisition unit, an outlier determination unit, and a suitability determination unit. The acquisition unit acquires the results of a task executed by a user using a terminal device. The outlier determination unit determines whether the execution results acquired by the acquisition unit include an outlier. The suitability determination unit determines the suitability of the execution of the task by the user based on the determination result by the outlier determination unit. [Selected Figure] Figure 4

Description

The present invention relates to an information processing device, an information processing method, and an information processing program.

Conventionally, there is known an outsourcing form in which tasks are requested to be performed by an unspecified number of people via a network such as the Internet. This type of outsourcing form is also called crowdsourcing.

As a crowdsourcing technique, Patent Literature 1 proposes a technique for estimating the amount of free time a user has between a given point in time and a deadline based on the behavioral history of the users who are candidates for performing a task, and for selecting a user who can complete the task by the deadline based on the deadline and the free time.

JP 2022-039121 A

However, the technology described in Patent Document 1 is limited to selecting users to perform tasks, and there is room for improvement in terms of further promoting crowdsourcing. For example, in crowdsourcing, tasks are requested from an unspecified number of users, and some users may not perform the tasks seriously. As a result, the task execution results obtained from users may include inappropriate execution results, which may hinder the promotion of crowdsourcing.

The present application has been made in consideration of the above, and aims to provide an information processing device, an information processing method, and an information processing program that can promote crowdsourcing.

The information processing device according to the present application includes an acquisition unit, an outlier determination unit, and a suitability determination unit. The acquisition unit acquires the results of a task executed by a user using a terminal device. The outlier determination unit determines whether the execution results acquired by the acquisition unit include an outlier. The suitability determination unit determines the suitability of the execution of the task by the user based on the determination result by the outlier determination unit.

According to one aspect of the embodiment, it is possible to promote crowdsourcing.

FIG. 1 is a diagram illustrating an example of information processing according to an embodiment. FIG. 2 is a diagram illustrating an example of the configuration of an information processing system according to the embodiment. FIG. 3 is a diagram illustrating an example of the configuration of a terminal device according to the embodiment. FIG. 4 is a diagram illustrating an example of the configuration of the information processing device according to the embodiment. FIG. 5 is a diagram illustrating an example of a user information table stored in a user information storage unit according to the embodiment. FIG. 6 is a diagram illustrating an example of a task information table stored in the task information storage unit according to the embodiment. FIG. 7 is a diagram illustrating an example of a task execution result table stored in the task execution result storage unit according to the embodiment. FIG. 8 is a flowchart illustrating an example of information processing by the processing unit of the information processing device according to the embodiment. FIG. 9 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device and the terminal device according to the embodiment.

Below, the information processing device, information processing method, and information processing program according to the present application will be described in detail with reference to the drawings. Note that the information processing device, information processing method, and information processing program according to the present application are not limited to these embodiments. Furthermore, the embodiments can be appropriately combined as long as they do not cause inconsistencies in the processing content. Furthermore, the same parts in the following embodiments will be given the same reference numerals, and duplicated explanations will be omitted.

[1. An example of information processing]
First, an example of information processing according to the embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of information processing according to the embodiment, which is executed by an information processing device 1.

The information processing device 1 shown in FIG. 1 provides various services via a network such as the Internet. The information processing device 1 provides various online services such as a crowdsourcing intermediary service, a web search service, a schedule management service, a route guidance service, a route information service, a video distribution service, a music distribution service, a map information service, and an e-commerce service.

The following mainly describes the crowdsourcing intermediation service among the services provided by the information processing device 1. The crowdsourcing intermediation service is a service that connects the ordering party and the receiving party in crowdsourcing.

Crowdsourcing is a form of outsourcing in which a business operator, researcher, or other party that receives a task outsources the execution of the task to an unspecified number of people via a network such as the Internet, and obtains the results of the task execution. In what follows, the user that places an order in the crowdsourcing intermediation service will be referred to as the "orderer O," and the user that receives the task in the crowdsourcing intermediation service will be referred to as the "user U."

In a crowdsourcing intermediation service, the information processing device 1 receives task information from the client O, including information for causing the user U to execute a task, and requests the user U to execute the task based on the task information, causing the user U to execute the task. The information processing device 1 then receives the results of the task execution by each user U from the terminal device 2 of each user U, and provides the received execution results to the client O.

A task is a task that is executed by a user U using a terminal device 2, such as a questionnaire-type task, annotation, or operation-type task. The terminal device 2 is, for example, a smartphone, a tablet terminal, a notebook PC (Personal Computer), or a desktop PC, but may also be a wearable device such as VR (Virtual Reality) goggles or a smartwatch.

A questionnaire-type task is a task that has the user U input an answer to each of a number of questions. Examples of questionnaire-type tasks include, but are not limited to, a task that has the user U input an answer to each of a number of questions for a survey of the user U's attitude, a task that has the user U input an answer to each of a number of questions that test the user U's knowledge, and a task that includes article content and has the user U input an answer to each of a number of questions related to the article content.

Annotation is, for example, the task of tagging data used in machine learning, and is performed, for example, by inputting a tag indicating whether the data contains a specific target, or by selecting a specific target contained in image data. When the terminal device 2 is a smartphone or tablet terminal, the selection of the specific target is performed, for example, by a tap operation on the area of the specific target, an operation of tracing the outer edge of the specific target with a finger or a stylus pen, or an operation of filling in the specific target with a finger or a stylus pen.

Tasks are not limited to the examples mentioned above, but may be, for example, creating a web page, creating article content such as news, etc.

In addition, if the terminal device 2 is a notebook PC or a desktop PC, the specific object is selected, for example, by clicking on the area of the specific object, tracing the outer edge of the specific object by moving the cursor, or filling in the specific object by moving the cursor.

An operation-type task is, for example, a task that requires a user interface (UI) operation in content that includes a UI. Operations required in an operation-type task include, for example, a tap operation, a double tap operation, a flick operation, a swipe operation, a pinch operation, etc. when the terminal device 2 is a smartphone or a tablet terminal, and a click operation, a double click operation, a scroll operation, a cursor movement operation, a drag-and-drop operation, etc. when the terminal device 2 is a notebook PC or a desktop PC.

As shown in FIG. 1, the information processing device 1 receives multiple pieces of task information transmitted from the terminal device 3 of the client O (step S1). The multiple pieces of task information received in step S1 may be task information from the same client O, or may be task information from different clients O.

Task information is information for each task, and the information processing device 1 accepts multiple pieces of task information. The task information includes execution information, which includes information for causing the user U to execute the task, and condition information, which indicates the conditions for executing the task.

When the task is a questionnaire-type task, the execution information includes, for example, information indicating the question, a UI in which the user U inputs an answer to the question (hereinafter, the answer UI), and a UI that the user U operates when deciding to send the input answer (hereinafter, the decision UI). The answer UI is, for example, a text box, a check box, a radio button, a combo box, etc., but is not limited to these examples. The decision UI is, for example, a button. When the decision UI is operated, the answer information input by the user U is transmitted from the terminal device 2 to the information processing device 1.

In addition, when the task is an annotation, the execution information includes, for example, the data to be tagged (e.g., image data, document data, sound data, etc.), the UI used to tag the data, the decision UI described above, etc.

When the task is an operational task, the execution information is content including a UI, such as web content or an application program. Web content is, for example, content such as news articles, SNS (Social Networking Service) content, game site content, shopping site content (e.g., a shopping page), and avatars placed in three-dimensional space.

The condition information includes information that specifies the context of user U when performing a task. User U's context is the situation of user U, and includes the situation around user U, the location of user U, user U's state, etc.

The surrounding conditions of user U are the physical environment in which user U is placed, such as the brightness, temperature, humidity, volume and frequency of sound around user U, wind strength, the height and backrest angle of the electric chair in which user U is sitting, and the height of the tabletop of the lift desk being used by user U. In addition, if user U is outdoors, the surrounding conditions of user U also include the weather in the location where user U is located.

The location where user U is located is identified, for example, by the latitude and longitude or the name of the location where user U is currently located. The name of the location where user U is currently located is, for example, a restaurant, park, home, station, hospital, city hall, road, underground passage, train, car, airplane, ship, etc., but is not limited to such examples.

The state of user U includes, for example, the motion state of user U and the emotions of user U. The motion state of user U is, for example, sitting, sleeping, walking, running, standing, etc.

Then, the information processing device 1 requests the user U to execute a task corresponding to the task information acquired in step S1 (step S2). For example, when the user U accesses the information processing device 1 using the terminal device 2, the information processing device 1 requests the user U to execute a task corresponding to the task information acquired in step S1 by sending request information including list information including information showing a list of tasks corresponding to the task information acquired in step S1 to the terminal device 2.

The request information sent by the information processing device 1 includes information indicating an overview of the task corresponding to the task information acquired in step S1, as well as information indicating the context defined by the condition information included in the task information. For example, the request information includes information for displaying the character string "Please perform the task in the restaurant" on the terminal device 2.

When the terminal device 2 receives the request information transmitted from the information processing device 1, it displays the received request information. The request information displayed on the terminal device 2 includes, for example, an acceptance button for each task, and when such an acceptance button is selected by the user U by clicking or tapping, the terminal device 2 transmits a task request including information for identifying the task selected by the user U to the information processing device 1 (step S3).

Next, when the information processing device 1 receives a task request sent from the terminal device 2, it sends the execution information included in the task information to the terminal device 2 of the user U in order to have the user U of the terminal device 2 that sent the task request execute the task (step S4).

When the terminal device 2 receives the execution information transmitted from the information processing device 1, it performs display and the like for the user U to execute the task based on the execution information. This allows the user U to execute the task using the terminal device 2.

Then, the terminal device 2 transmits the result of the task execution by the user U to the information processing device 1 (step S5). If the task is a questionnaire-type task, the result of the task execution includes, for example, information indicating the answers given by the user U to each question and information on the operation history of the user U. The information on the operation history includes, for example, information indicating the timing of the answers given by the user U to the questions, the speed of the answers, the time taken to answer, etc.

Furthermore, if the task is an annotation, the execution result of the task includes, for example, information on tags input by the user U and information on the operation history by the user U. The operation history information includes, for example, information indicating the speed at which the user U tagged, the position of the tag, the range of the tag, the timing of tagging, the time of tagging, etc.

In addition, for example, if the task is an operation-type task, the task execution result includes information on the operation history of the user U. The operation history includes, for example, a history of the contents (e.g., operation position, operation timing, operation speed, operation distance, number of operations, etc.) of various operations (e.g., tap operation, flick operation, swipe operation, pinch operation, click operation, cursor movement operation, drag-and-drop operation, etc.).

Next, when the information processing device 1 obtains the execution result of the task from the terminal device 2, it determines whether the execution result includes an outlier (step S6). An outlier is a value that is extremely large or extremely small compared to other values, and includes statistical outliers and outliers whose causes are known, such as input errors. Outliers whose causes are known, such as input errors, are called abnormal values. In the following, the execution result that is the subject of the determination of whether it includes an outlier may be referred to as the execution result to be determined.

The information processing device 1 compares statistical information on the results of tasks executed by multiple users U using the terminal device 2 with the execution result to be judged, and determines whether the execution result to be judged contains an outlier based on the comparison result.

Statistical information on execution results is, for example, information related to the statistics of the execution results, and is not limited to information obtained by statistically processing the execution results, but also includes information that is statistically processed to find outliers contained in the execution results.

For example, if the task is a questionnaire-type task, the outlier determined in step S6 is an outlier in the answer to the question, and is a statistical outlier or abnormal value. If the task is a questionnaire-type task, the information processing device 1 can determine a statistical outlier based on, for example, the occurrence probability of each answer pattern to multiple questions in multiple execution results.

For example, if the answer to each question is to select one of 1, 2, 3, 4, or 5, and there are 10 questions, the answer patterns for multiple questions are {2, 4, 2, 5, 3, 1, 1, 5, 2, 3} and {4, 1, 5, 2, 2, 3, 3, 1, 2, 1}, etc.

In this case, the information processing device 1 calculates the probability of occurrence of each answer pattern for multiple questions in multiple execution results as statistical information. Then, when the answer pattern indicated in the execution result to be judged is an answer pattern whose occurrence probability is equal to or lower than a threshold, the information processing device 1 judges that the execution result to be judged includes an outlier.

The information processing device 1 also calculates, as statistical information, a response time distribution, which is a distribution of response times to multiple questions in multiple execution results. The response time is the time from when execution of the task becomes possible to when responses to all multiple questions are completed. Then, the information processing device 1 determines that the execution result to be judged includes an outlier when the response time indicated in the execution result to be judged is a response time whose distribution proportion in the response time distribution is equal to or less than a threshold value.

The response time may be the time required to respond to each of the multiple questions. In this case, the information processing device 1 calculates, for each question, a response time distribution, which is a distribution of response times to questions in multiple execution results, as statistical information. Then, the information processing device 1 performs a process for each question to determine that the execution result to be evaluated includes an outlier if the response time indicated in the execution result to be evaluated is a response time whose distribution proportion in the response time distribution is equal to or less than a threshold value.

In addition, when the answer to the question is a character string, the information processing device 1 calculates a character count distribution ratio, which is the distribution ratio for each character count in multiple execution results, as statistical information. Then, when the number of characters in the answer shown in the execution result to be judged is a number of characters for which the character count distribution ratio is equal to or less than a threshold value, the information processing device 1 judges that the execution result to be judged includes an outlier.

Furthermore, if the task is an annotation, the outlier determined in step S6 is a tag outlier, and is a statistical outlier or abnormal value. If the task is an annotation, the information processing device 1 can determine a statistical outlier, for example, based on the distribution of tagging times in multiple execution results. The tagging time is the time from when the task becomes executable to when tagging ends, and may be referred to as the tagging time below.

For example, the information processing device 1 calculates a tagging time distribution, which is a distribution of tagging times in multiple execution results, as statistical information. Then, when the tagging time indicated in the execution result to be judged is a tagging time whose distribution ratio is equal to or less than a threshold value, the information processing device 1 judges that the execution result to be judged includes an outlier.

Furthermore, when tagging is performed by encircling the outer edge of a specific object in an image by tracing the outer edge of the specific object, the information processing device 1 calculates a tagging count distribution, which is the distribution of the number of enclosed ranges (hereinafter, may be referred to as the tagging count), as statistical information. Then, when the tagging count indicated in the execution result to be judged is a tagging count whose distribution ratio is equal to or less than a threshold value, the information processing device 1 judges that the execution result to be judged includes an outlier.

Furthermore, when tagging is an operation-type task, the information processing device 1 calculates the occurrence probability of each operation pattern in multiple execution results as statistical information. The operation pattern is, for example, at least one of the following: the order of operations, the type of operation, and the content of the operation (for example, the operation position, the operation timing, the operation speed, the distance of the operation, the number of operations, etc.). Then, when the operation pattern indicated in the execution result to be judged is an answer pattern whose occurrence probability is equal to or lower than a threshold, the information processing device 1 judges that the execution result to be judged includes an outlier.

In addition, when tagging is an operation-type task, the information processing device 1 calculates an operation time distribution, which is a distribution of operation times in multiple execution results, as statistical information. Then, when the operation time indicated in the execution result to be judged is an operation time whose distribution ratio is equal to or less than a threshold value, the information processing device 1 judges that the execution result to be judged includes an outlier.

The information processing device 1 also determines anomaly information, which is information indicating an anomaly in the execution result of the task, based on the execution information of the task. Then, when the execution result to be judged contains an abnormal value, the information processing device 1 determines that the execution result to be judged contains an outlier.

For example, if the task is a questionnaire-type task and the answer to each question is to select one of 1, 2, or 3, the information processing device 1 will determine that any answer that is not 1, 2, or 3 is an outlier, that is, an abnormal value.

For example, if the answer is "0" for male, "1" for female, or "2" for unknown, then "3" or "4" is an outlier. In this case, if the execution result to be judged contains "3" or "4", the information processing device 1 judges that the execution result to be judged contains an outlier.

In addition, when tagging involves selecting one option as a tag from multiple predetermined options, the information processing device 1 determines that the execution result to be judged includes an outlier, which is an abnormal value, if a tag other than the multiple predetermined options is attached.

For example, if the predetermined multiple options include male "0", female "1", and unknown "2", then "3" and "4" are outliers. In this case, if the execution result to be judged contains "3" or "4", the information processing device 1 judges that the execution result to be judged contains an outlier.

Next, the information processing device 1 determines whether or not the user U should perform the task based on the result of determining whether or not there is an outlier in step S6 (step S7).

For example, if the result of the determination of the presence or absence of an outlier in step S6 satisfies a predetermined condition, the information processing device 1 determines that the performance of the task by the user U is inappropriate, and if not, determines that the performance of the task by the user U is appropriate.

The predetermined condition is that an outlier is present if the number of possible outliers is one, and that the number of possible outliers is equal to or greater than a threshold value TH if the number of possible outliers is two or more.

The predetermined condition may also be that, if the number of possible outliers is two or more, the value obtained by weighting the outliers (if there are multiple outliers, the weighted sum of the multiple outliers) is equal to or greater than the threshold value TH. For example, in step S7, the information processing device 1 can set the threshold value TH to a larger value for a user U with a higher proportion (probability) of the task execution being determined to be inappropriate.

Next, if the information processing device 1 determines in step S7 that the result of the task execution by the user U is inappropriate, it notifies the user U of notification information, which is information indicating that the result of the task execution by the user U is inappropriate (step S8). In step S8, the information processing device 1 notifies the user U of the notification information by sending the notification information to the terminal device 2 of the user U.

The information processing device 1 can also repeatedly acquire execution results during the process of execution of a task by the user U. In this case, the information processing device 1 determines whether or not each of the execution results repeatedly acquired during the process of execution of the task contains an outlier. Based on such a determination result, the information processing device 1 can repeatedly determine the appropriateness of the execution of the task by the user U during the process of execution of the task by the user U.

Then, each time the information processing device 1 determines that the task execution result is inappropriate during the task execution process, it notifies the user U of notification information that indicates that the task execution result is inappropriate during the task execution process.

In addition, if the information processing device 1 determines that the result of the task execution is inappropriate during the task execution process, the information processing device 1 can also update the task that is in the process of being executed. For example, the information processing device 1 can perform an update that changes at least one of the position of the task's UI, the task layout, the task design, the task execution order, etc., and can cause the user U to execute the updated task from the middle of the task execution.

For example, if the information processing device 1 determines that the result of the task execution is inappropriate during the task execution process, and if the task is a questionnaire-type task, it transmits execution information in which the order of the questions answered by the user U during the task execution process is changed to the terminal device 2 of the user U. This allows the information processing device 1 to have the user U re-execute the task in which the order of the options for the questions answered by the user U during the execution process and for which the information processing device 1 determines that the execution result is inappropriate is changed.

Furthermore, if the information processing device 1 determines that the result of a task execution is inappropriate during the task execution process, and if the task is an annotation or operation-type task, for example, the information processing device 1 stops the task being executed by the user U during the task execution process and transmits execution information to the user U's terminal device 2 to have the user U execute the task again from the beginning. This allows the information processing device 1 to have the user U execute the task again, whose execution result has been determined to be inappropriate.

In addition, the information processing device 1 estimates the reward to be granted to the user U based on the result of judging whether the task execution result is appropriate during the task execution process. The information processing device 1 notifies the user U of the reward to be granted to the user U by transmitting estimated reward information, which is information related to the estimated reward, to the terminal device 2 of the user U.

The estimated reward information transmitted from the information processing device 1 to the terminal device 2 includes, for example, the character string "If you don't do better, your reward amount will be reduced" if the estimated reward is lower than the normal reward.

The reward given to the user U may be, for example, electronic money, points, cash, or various benefits. Points are a value that can be used, for example, to purchase products or use services on online sites. Benefits may be, for example, coupons, gift certificates, service vouchers, etc., but are not limited to these examples.

The information processing device 1, for example, rewards user U by adding an amount of electronic money corresponding to the task or points corresponding to the task to user U's account or wallet in the online service. The information processing device 1 also rewards user U by, for example, transmitting reward information, which is information indicating a benefit, to terminal device 2 so that the reward information can be displayed on terminal device 2. If the reward is in cash, the reward is awarded, for example, by depositing the cash via a bank account.

In this way, the information processing device 1 determines whether the results of the task execution by the user U using the terminal device 2 include outliers, and determines the appropriateness of the execution of the task by the user U based on the determination result. This allows the information processing device 1 to, for example, exclude inappropriate execution results from the execution results to be provided to the client O, thereby promoting crowdsourcing.

The following provides a detailed explanation of the configuration of an information processing system that includes the information processing device 1 and terminal devices 2 and 3 that perform such processing. Note that the following does not include an explanation of the processing related to the e-commerce services provided by the information processing device 1 other than the crowdsourcing intermediation service.

2. Configuration of Information Processing System
2 is a diagram showing an example of a configuration of an information processing system according to an embodiment. As shown in FIG. 2, the information processing system 100 according to the embodiment includes an information processing device 1, a plurality of terminal devices 2, and a plurality of terminal devices 3.

The information processing device 1 provides a crowdsourcing intermediation service. The information processing device 1 can also provide various online services such as a web search service, a schedule management service, a route guidance service, a route information service, a video distribution service, a music distribution service, a map information service, and an e-commerce service.

Each of the multiple terminal devices 2 is used by a different user U. Each of the multiple terminal devices 3 is used, for example, by a different orderer O. Each of the terminal devices 2 and 3 is, for example, a smartphone, a tablet terminal, a notebook PC, a desktop PC, etc., but may also be a wearable device such as a smart watch, VR goggles, etc.

The information processing device 1, the terminal device 2, and the terminal device 3 are connected to each other via a network N so that they can communicate with each other by wire or wirelessly. Note that the information processing system 100 shown in FIG. 2 may include multiple information processing devices 1. The network N is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network) such as the Internet.

Each of the terminal devices 2 and 3 can connect to the network N via a wireless communication network such as LTE (Long Term Evolution), 4G (4th Generation), or 5G (5th Generation: 5th generation mobile communication system), or via short-range wireless communication such as Bluetooth (registered trademark) or wireless LAN, and communicate with the information processing device 1.

[3. Terminal device 2]
Fig. 3 is a diagram showing an example of the configuration of the terminal device 2 according to the embodiment. As shown in Fig. 3, the terminal device 2 according to the embodiment includes a communication unit 10, a display unit 11, an operation unit 12, a sensor group 13, a storage unit 14, and a processing unit 15.

3.1. Communication unit 10
The communication unit 10 is realized by, for example, a network interface card (NIC), etc. The communication unit 10 is connected to a network N by wire or wirelessly, and transmits and receives information to and from the information processing device 1 via the network N.

[3.2. Display unit 11
The display unit 11 is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display.

[3.3. Operation unit 12
The operation unit 12 includes, for example, a keyboard including keys for inputting letters, numbers, and spaces, an enter key, arrow keys, etc., a mouse, a power button, etc. In the case where the display unit 11 is a touch panel compatible display, The operation unit 12 includes a touch panel.

3.4. Sensor Group 13
The sensor group 13 includes, for example, a temperature sensor, a humidity sensor, an illuminance sensor, a positioning sensor, a microphone, an acceleration sensor, a gyro sensor, a geomagnetic sensor, and an image sensor. The positioning sensor is a position sensor that detects the position of the terminal device 2.

The temperature sensor is a sensor that detects the temperature around the terminal device 2. The humidity sensor is a sensor that detects the humidity around the terminal device 2. The illuminance sensor is a sensor that detects the illuminance around the terminal device 2. The microphone is a sensor that detects the sound around the terminal device 2.

The acceleration sensor is a sensor that detects the acceleration of the terminal device 2. The gyro sensor is a sensor that detects the attitude, such as the tilt and rotation, of the terminal device 2. The geomagnetic sensor is a sensor that detects geomagnetism. The image sensor is a sensor that captures an image of the surroundings of the terminal device 2.

[3.5. Storage unit 14
The storage unit 14 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk.

The memory unit 14 stores, for example, information transmitted from the information processing device 1 and acquired by the processing unit 15 via the network N and the communication unit 10, and detection information, which is information detected by the sensor group 13.

3.6. Processing Unit 15
The processing unit 15 is a controller, and is realized, for example, by a CPU (Central Processing Unit) or an MPU (Micro Processing Unit) by executing various programs stored in a storage device inside the terminal device 2 using the RAM as a working area.

The processing unit 15 may be realized in part or in whole by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The processing unit 15 includes an information acquisition unit 16, a display processing unit 17, and an information output unit 18.

[3.6.1. Information Acquisition Unit 16
The information acquisition unit 16 acquires various pieces of information transmitted from the information processing device 1 and received by the communication unit 10 via the network N. The information transmitted from the information processing device 1 to the terminal device 2 is, for example, The information acquisition unit 16 acquires, from the sensor group 13, detection information that is information detected by the sensor group 13, such as execution information, request information, notification information, and reward information.

[3.6.2. Display Processing Unit 17]
The display processing unit 17 causes the display unit 11 to display the information acquired by the information acquisition unit 16. For example, the display processing unit 17 causes the display unit 11 to display the execution information, request information, notification information, reward information, and the like acquired by the information acquisition unit 16.

3.6.3. Information output unit 18
The information output unit 18 transmits, for example, operation information corresponding to an operation on the operation unit 12 by the user U to the information processing device 1 via the communication unit 10. In addition, the information output unit 18 transmits detection information acquired by the information acquisition unit 16 to the information processing device 1 via the communication unit 10.

In addition, assume that while the request information is displayed on the display unit 11, the user U operates the operation unit 12 to select a task that the user U wishes to execute. In this case, the information output unit 18 transmits a task request including information for identifying the task that the user U wishes to execute to the information processing device 1 via the communication unit 10.

For example, when the request information includes multiple tasks that the user U can select, the request information displayed on the display unit 11 includes an acceptance button for each task. When the acceptance button is selected by the user U by clicking or tapping, the terminal device 2 transmits a task request to the information processing device 1 to enable the user U to execute the task selected by the user U. Such a task request includes information for identifying the task selected by the user U.

In addition, the information output unit 18 transmits the execution results of the task executed by the user U based on the execution information to the information processing device 1 via the communication unit 10.

4. Configuration of information processing device 1
4 is a diagram showing an example of the configuration of the information processing device 1 according to the embodiment. As shown in FIG. 4, the information processing device 1 includes a communication unit 20, a storage unit 21, and a processing unit 22.

4.1. Communication unit 20
The communication unit 20 is realized by, for example, a NIC. The communication unit 20 is connected to a network N by wire or wirelessly, and transmits and receives information to and from various other devices. For example, the communication unit 20 transmits and receives information to and from the terminal devices 2 and 3 via the network N.

[4.2. Storage unit 21
The storage unit 21 is realized by, for example, a semiconductor memory element such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 21 includes a user information storage unit 30, a task information storage unit 31, and , and a task execution result storage unit 32 .

[4.2.1. User information storage unit 30
The user information storage unit 30 stores various types of information related to the user U. Fig. 5 is a diagram showing an example of a user information table stored in the user information storage unit 30 according to the embodiment.

In the example shown in FIG. 5, the user information table stored in the user information storage unit 30 includes information on items such as "User ID (Identifier)," "Attribute Information," "History Information," "Context Information," and "Settings Information."

"User ID" is an identifier that identifies a user U, and is information assigned to each user U. "Attribute information" is attribute information that indicates the attributes of user U associated with the "user ID." The attributes of user U are, for example, demographic attributes and psychographic attributes. Demographic attributes are demographic attributes, and include multiple attribute items such as age, gender, occupation, place of residence, annual income, and family composition.

Psychographic attributes are psychological attributes and include, for example, multiple attribute items related to lifestyle, values, interests, etc. For example, each of the multiple attribute items in the psychographic attributes is an object of interest to the user U, such as cars, clothes, travel, games, camping, motorcycles, trains, home appliances, or computers.

"History information" is history information that includes information on the behavioral history of user U in online services that is associated with the "user ID." Information on the behavioral history of user U in online services includes, for example, search history information, browsing history information, and transaction history information in online services for user U.

User U's search history information includes, for example, information on the search history by user U in web search services. User U's browsing history information includes, for example, information on the content browsing history by user U in online services. User U's transaction history information includes, for example, information on the product transaction history by user U in online services.

"Context information" is information on the current or future context of user U associated with the "user ID." Such context information includes the context of user U identified based on the detection information transmitted from the terminal device 2, or historical information of the context of user U estimated by the processing unit 22, etc.

The context of user U is the situation of user U, and includes, for example, the situation around user U, the movement state of user U, the posture of user U, the emotions of user U, the current location of user U, the physical environment in which user U is located, etc.

"Settings information" is the settings information of user U associated with the "user ID", such as schedule information and email information set by user U and linked to user U's account.

[4.2.2. Task information storage unit 31]
The task information storage unit 31 stores task information provided by each client O. Fig. 6 is a diagram showing an example of a task information table stored in the task information storage unit 31 according to the embodiment.

In the example shown in FIG. 6, the task information table stored in the task information storage unit 31 includes information on items such as "task ID," "content information," "execution information," "condition information," "statistical information," "abnormality information," and "required number of executions."

"Task ID" is an identifier that identifies a task, and is information that is assigned to each task. "Content information" is information that indicates the content of the task associated with the "task ID", and includes, for example, the type of task, an overview of the task, and information on the context when the task is executed. Condition information may be included in the content information. "Execution information" is information that includes information for causing user U to execute the task associated with the "task ID", and is the execution information described above.

"Condition information" is information indicating the conditions for executing a task associated with a "task ID," and is the condition information described above. "Statistical information" is statistical information on the execution results of a task associated with a "task ID." Statistical information is, for example, information related to the statistics of the execution results, and is not limited to information obtained by statistically processing the execution results, but also includes information that is statistically processed to find outliers included in the execution results.

"Abnormality information" is information about abnormal values of the execution results of the task associated with the "task ID." "Required number of executions" is information indicating the required number of executions of the task associated with the "task ID," and is information set by the client O.

4.2.3. Task execution result storage unit 32
The task execution result storage unit 32 stores the results of the tasks executed by the user U. Fig. 7 is a diagram showing an example of a task execution result table stored in the task execution result storage unit 32 according to the embodiment.

In the example shown in FIG. 7, the task execution result table stored in the task execution result storage unit 32 includes information on items such as "Execution ID," "Task ID," "User ID," "Execution result," and "Detection information."

"Execution ID" is an identifier that identifies the execution result of a task, and is information that is assigned to each execution result of a task. "Task ID" is an identifier that identifies the task of the execution result that corresponds to the "Execution ID", and is the same as the task ID shown in Figure 6.

"User ID" is an identifier that identifies the user U who executed the task with the execution result corresponding to the "Execution ID", and is the same as the user ID shown in FIG. 5. "Execution result" is the execution result corresponding to the "Execution ID". "Detection information" is the detection information transmitted from the terminal device 2 of the user U when the user U executed the task with the execution result corresponding to the "Execution ID".

4.3. Processing Unit 22
The processing unit 22 is a controller, and is realized by a processor such as a CPU or an MPU, which executes various programs (corresponding to an example of an information processing program) stored in a storage device inside the information processing device 1 using a RAM or the like as a working area. The processing unit 22 may be realized in part or in whole by an integrated circuit such as an ASIC, an FPGA, or a GPGPU.

As shown in FIG. 4, the processing unit 22 has an acquisition unit 40, a reception unit 41, a request unit 42, a task unit 43, a calculation unit 44, an outlier determination unit 45, a suitability determination unit 46, a notification unit 47, an update unit 48, a reward granting unit 49, and an estimation unit 50, and realizes or executes the functions and actions of the information processing described below. Note that the internal configuration of the processing unit 22 is not limited to the configuration shown in FIG. 4, and may be other configurations as long as they perform the information processing described below.

[4.3.1. Acquisition unit 40
The acquisition unit 40 acquires various information from external information processing devices and terminal devices 2 and 3 via the communication unit 20 , and stores the acquired information in the storage unit 21 .

For example, the acquisition unit 40 acquires user information, which is information about the user U, from an external information processing device or terminal device 2, etc., via the communication unit 20, and adds the acquired user information to a user information table in the user information storage unit 30.

The acquisition unit 40 also acquires the execution results and detection information transmitted from the terminal device 2. Such execution results are the results of task execution by the user U. Furthermore, the detection information is information detected by the sensor group 13 of the terminal device 2. The acquisition unit 40 adds the acquired execution results and detection information to a task execution result table in the task execution result storage unit 32.

The acquisition unit 40 can also repeatedly acquire execution results and detection information from the terminal device 2 during the process of task execution by the user U, and add the acquired execution results and detection information to the task execution result table in the task execution result storage unit 32.

The acquisition unit 40 also acquires, as context information, information on the context indicated by the detection information transmitted from the terminal device 2 and accepted by the acceptance unit 41. For example, the detection information includes information on the brightness, temperature, humidity, sound volume, and frequency of the surroundings of the user U, information indicating the position of the user U, and information indicating the posture of the terminal device 2. The acquisition unit 40 can acquire information on the brightness, temperature, humidity, sound volume, and frequency of the surroundings of the user U indicated by the detection information as context information, and add the acquired context information to the user information table of the user information storage unit 30.

The acquisition unit 40 also acquires various types of information from the storage unit 21. For example, the acquisition unit 40 acquires user information, which is information about the user U, from the user information storage unit 30 or the like. The user information acquired by the acquisition unit 40 includes, for example, a part or all of at least one of the above-mentioned attribute information, history information, context information, and setting information.

The acquisition unit 40 also acquires task information, which is information about a task to be executed by the user U, from the task information storage unit 31 or the like. The task information acquired by the acquisition unit 40 includes, for example, a part or all of at least one of the above-mentioned content information, execution information, condition information, statistical information, abnormality information, and information on the number of tasks required to be executed.

The acquisition unit 40 also acquires information related to the task execution results from the task execution result table in the task execution result storage unit 32. The information related to the execution results acquired by the acquisition unit 40 includes, for example, part or all of at least one of the above-mentioned task ID, user ID, task execution result, and detection information.

4.3.2. Reception unit 41
The reception unit 41 receives various requests and information from external information processing devices and the terminal devices 2 and 3 via the communication unit 20 .

For example, the reception unit 41 receives order information for each task sent from the terminal device 3. The order information includes the above-mentioned task information and information on the required number of executions. The task information includes the above-mentioned execution information, condition information, and content information.

As described above, the execution information includes information for causing the user U to execute a task. As described above, the condition information includes information that specifies the context of the user U when executing a task. The context of the user U is the situation of the user U, and includes the situation around the user U. The content information includes information indicating the content of the task (e.g., the type of task, an overview of the task, information on the context when the task is executed, etc.). The condition information may be included in the content information. The reception unit 41 adds the received order information to the task information table of the task information storage unit 31.

The reception unit 41 also receives a task list request sent from the terminal device 2. The task list request includes, for example, a user ID. Such a task list request is sent from the terminal device 2 to the information processing device 1 when the user U operates the terminal device 2 and wishes to display a list of tasks.

The reception unit 41 also receives a task request sent from the terminal device 2. The task request includes, for example, a user ID and a task ID. Such a task request is sent from the terminal device 2 to the information processing device 1 when the user U operates the terminal device 2 to select a task that he or she wishes to execute.

[4.3.3. Request Unit 42]
For example, when a task list request is accepted by the accepting unit 41, the request unit 42 requests the user U to execute a task by transmitting request information including task list information to the terminal device 2 that has sent the task list request via the communication unit 20. The task list information is information indicating a list of tasks and is information that allows the user U to select a desired task.

The request unit 42, for example, extracts executable tasks that are tasks that the user U can execute in a situation specified by the task information, and generates information indicating a list of the extracted executable tasks as task list information. The request unit 42 then transmits request information including the generated task list information to the terminal device 2 of the user U.

When there are multiple tasks to be requested to be executed by the user U, the request unit 42 can generate task list information that arranges the multiple tasks in an order according to a predetermined criterion, for example.

For example, the request unit 42 can generate task list information in which a number of tasks are arranged in order of the reward in terms of points. The task with the highest reward in terms of points is ranked highest, and the order of tasks decreases as the reward in terms of points decreases.

The higher the ranking of a task, the more easily it is placed in a position that is easy for the user U to see. For example, if the task list information is information in which multiple tasks are arranged vertically, the topmost position is the topmost position, and the lower the position, the lower the ranking.

Furthermore, when the appropriateness determining unit 46 determines that the result of the task execution by the user U is inappropriate, the request unit 42 requests the user U to re-execute the task. For example, the request unit 42 requests the user U to re-execute the task by sending re-execution request information to the terminal device 2 of the user U.

The execution request information includes, for example, information indicating a request to re-execute the task and an acceptance button that the user U selects when accepting the re-execution, and is displayed on the user U's terminal device 2. When the user U selects the acceptance button displayed on the terminal device 2 by clicking or tapping, the terminal device 2 transmits a task request to the information processing device 1, the task request including information for identifying the target task, which is the task to be re-executed. As a result, execution information for the target task is transmitted from the information processing device 1 to the user U's terminal device 2, and the user U re-executes the target task.

[4.3.4. Task section 43]
The task section 43 causes the user U to execute a task based on the task information. The task section 43 causes the user U to execute a task by transmitting execution information included in the task information to the terminal device 2 of the user U.

For example, when a task request is accepted by the accepting unit 41, the task unit 43 causes the user U to execute the task by transmitting execution information included in the task information of the task specified in the task request to the terminal device 2 of the user U.

4.3.5. Calculation unit 44
The calculation unit 44 calculates statistical information of the execution results based on the execution results of the multiple users U acquired by the acquisition unit 40. The calculation unit 44 adds the calculated statistical information to the task information table in the task information storage unit 31.

Statistical information is, for example, information related to the statistics of execution results, and is not limited to information obtained by statistically processing the execution results, but also includes information that is statistically processed to find outliers in the execution results.

For example, when the task is a questionnaire-type task, the calculation unit 44 calculates, for example, the probability of occurrence of each answer pattern to multiple questions in multiple execution results, or calculates an answer time distribution that is the distribution of answer times to multiple questions in multiple execution results. The answer time is the time from when execution of the task becomes possible to when all multiple questions have been answered. The answer time may also be the time required to answer each of the multiple questions.

In addition, when the task is a questionnaire-type task and the answer to the question is a character string, the calculation unit 44 calculates the character count distribution ratio, which is the distribution ratio for each character count in multiple execution results, as statistical information.

When the task is an annotation, the calculation unit 44 calculates, as statistical information, a tagging time distribution, which is a distribution of tagging times for multiple execution results. When the task is an annotation and tagging is performed by tracing the outer periphery of a specific object in an image to encircle the outer periphery of the specific object, the calculation unit 44 can also calculate, as statistical information, a tagging number distribution, which is a distribution of the number of taggings, which is the number of enclosed ranges.

When the tagging is an operation-type task, the calculation unit 44 calculates the occurrence probability of each operation pattern in multiple execution results as statistical information. The operation pattern is, for example, at least one of the following: the order of operations, the type of operation, and the content of the operation (for example, the operation position, the operation timing, the operation speed, the distance of the operation, the number of operations, etc.).

In addition, when the tagging is an operation-type task, the calculation unit 44 can also calculate an operation time distribution, which is a distribution of operation times for multiple execution results, as statistical information.

[4.3.6. Outlier determination unit 45]
The outlier determination unit 45 determines whether or not an outlier is included in the execution result to be determined, which has been acquired by the acquisition unit 40. The execution result to be determined is the execution result to be determined whether or not an outlier is included. An outlier is a value that is extremely large or extremely small compared to other values, and includes statistical outliers and abnormal values.

For example, the outlier determination unit 45 determines whether or not the execution result to be judged includes an outlier based on the result of comparing the execution result to be judged acquired by the acquisition unit 40 with the statistical information. The statistical information is statistical information on the results of execution of tasks by multiple users U using the terminal devices 2.

For example, when the task is a questionnaire-type task, the outlier determined by the outlier determination unit 45 is an outlier of the answer to the question, and is a statistical outlier or abnormal value. When the task is a questionnaire-type task, the outlier determination unit 45 can determine a statistical outlier based on the occurrence probability of each answer pattern to multiple questions in multiple execution results, for example.

For example, if the answer to each question is to select one of 1, 2, 3, 4, or 5, and there are 10 questions, the answer patterns for multiple questions are {2, 4, 2, 5, 3, 1, 1, 5, 2, 3} and {4, 1, 5, 2, 2, 3, 3, 1, 2, 1}, etc.

In this case, the outlier determination unit 45 determines whether the answer pattern indicated in the execution result to be judged is an answer pattern whose occurrence probability is equal to or less than a threshold value, based on the statistical information acquired by the acquisition unit 40. If the answer pattern indicated in the execution result to be judged is an answer pattern whose occurrence probability is equal to or less than a threshold value, the outlier determination unit 45 determines that the execution result to be judged includes an outlier.

The outlier determination unit 45 also determines whether the response time indicated in the execution result to be judged is a response time whose distribution proportion in the response time distribution is equal to or less than a threshold value, based on the statistical information acquired by the acquisition unit 40. The outlier determination unit 45 determines that the execution result to be judged includes an outlier when the response time indicated in the execution result to be judged is a response time whose distribution proportion in the response time distribution is equal to or less than a threshold value.

The answer time is the time required to answer each of the multiple questions. In this case, the outlier determination unit 45 determines whether the answer time indicated in the execution result to be judged is an answer time whose distribution proportion in the answer time distribution is equal to or less than a threshold value, based on the statistical information acquired by the acquisition unit 40. If the answer time indicated in the execution result to be judged is an answer time whose distribution proportion in the answer time distribution is equal to or less than a threshold value, the outlier determination unit 45 performs a process for each question to determine that the execution result to be judged includes an outlier.

The outlier determination unit 45 also determines whether the number of characters in the answer shown in the execution result to be judged is a number of characters whose character count distribution ratio is equal to or less than a threshold value, based on the statistical information acquired by the acquisition unit 40. Then, the outlier determination unit 45 determines that the execution result to be judged includes an outlier when the number of characters in the answer shown in the execution result to be judged is a number of characters whose character count distribution ratio is equal to or less than a threshold value.

When the task is an annotation, the outlier determination unit 45 can determine a statistical outlier, for example, based on the distribution of tagging times in multiple execution results. The tagging time is the time from when the task becomes executable to when tagging ends.

For example, the outlier determination unit 45 determines whether the tagged time indicated in the execution result to be judged is a tagged time whose distribution ratio is equal to or less than a threshold value based on the statistical information acquired by the acquisition unit 40. Then, the outlier determination unit 45 determines that the execution result to be judged includes an outlier when the tagged time indicated in the execution result to be judged is a tagged time whose distribution ratio is equal to or less than a threshold value.

In addition, when tagging is performed by encircling the outer edge of a specific object in an image by tracing the outer edge of the specific object, the outlier determination unit 45 determines whether the number of taggings shown in the execution result to be judged is a number of taggings whose distribution ratio is equal to or less than a threshold based on the statistical information acquired by the acquisition unit 40. When the number of taggings shown in the execution result to be judged is a number of taggings whose distribution ratio is equal to or less than a threshold, the outlier determination unit 45 determines that the execution result to be judged includes an outlier.

When the tagging is an operation-type task, the outlier determination unit 45 determines whether or not the operation pattern indicated in the execution result to be judged is an answer pattern whose occurrence probability is equal to or lower than a threshold value, based on the statistical information acquired by the acquisition unit 40. Then, when the operation pattern indicated in the execution result to be judged is an answer pattern whose occurrence probability is equal to or lower than a threshold value, the outlier determination unit 45 determines that the execution result to be judged includes an outlier.

When tagging is an operation-type task, the outlier determination unit 45 can also determine whether or not the operation time indicated in the execution result to be judged is an operation time whose distribution ratio is equal to or less than a threshold value, based on the statistical information acquired by the acquisition unit 40. Then, when the operation time indicated in the execution result to be judged is an operation time whose distribution ratio is equal to or less than a threshold value, the outlier determination unit 45 determines that the execution result to be judged includes an outlier.

The outlier determination unit 45 can also determine abnormality information, which is information indicating an abnormality in the execution result of a task, based on the task execution information acquired by the acquisition unit 40. The outlier determination unit 45 adds the determined abnormality information to the task information table of the task information storage unit 31.

The outlier determination unit 45 determines that the execution result to be judged contains an outlier when the execution result to be judged contains an abnormal value indicated by the abnormality information based on the abnormality information acquired by the acquisition unit 40. For example, if the task is a questionnaire-type task and the answer to each question is to select one of 1, 2, or 3, the outlier determination unit 45 determines that an answer that is not 1, 2, or 3 is an outlier that is an abnormal value.

For example, if the answer is "0" for male, "1" for female, or "2" for unknown, then "3" or "4" is an outlier. In this case, the outlier determination unit 45 determines that the execution result to be judged contains an outlier if the execution result to be judged contains "3" or "4", etc.

In addition, when tagging involves selecting one option as a tag from among multiple predefined options, the outlier determination unit 45 determines that the execution result being evaluated contains an outlier, which is an abnormal value, if a tag other than the multiple predefined options is attached.

For example, if the predetermined multiple options are multiple options including male "0", female "1", and unknown "2", "3" and "4" are outliers. In this case, the outlier determination unit 45 determines that the execution result to be judged contains an outlier if the execution result to be judged contains "3" or "4", etc.

In addition, when execution results are repeatedly acquired by the acquisition unit 40 during the execution process of a task by the user U, the outlier determination unit 45 determines whether or not each of the execution results repeatedly acquired during the execution process of the task contains an outlier.

[4.3.7. Suitability determination unit 46]
The suitability determining unit 46 determines whether the user U's execution of the task is appropriate based on the determination result by the outlier determining unit 45 .

For example, if the result of the determination of the presence or absence of an outlier by the outlier determination unit 45 satisfies a predetermined condition, the appropriateness determination unit 46 determines that the execution of the task by the user U is inappropriate, and if not, determines that the execution of the task by the user U is appropriate.

The predetermined condition is that an outlier is present if the number of possible outliers is one, and that the number of possible outliers is equal to or greater than a threshold value TH if the number of possible outliers is two or more.

The predetermined condition may also be that, when the number of possible outliers is two or more, the value obtained by weighting the outliers (when there are multiple outliers, the weighted sum of the multiple outliers) is equal to or greater than the threshold value TH. For example, the suitability determination unit 46 can set the threshold value TH to a larger value for a user U with a higher rate (probability) of task execution being determined to be inappropriate.

In addition, when the outlier determination unit 45 determines whether or not each of the execution results acquired by the repeated acquisition unit 40 during the execution process of the task by the user U contains an outlier, the suitability determination unit 46 can repeatedly determine the suitability of the execution of the task by the user U during the execution process of the task by the user U based on such determination results.

In addition, for example, when the task unit 43 has the same user U execute the same task or part of the same task multiple times, if the execution results are the same each time, the suitability determination unit 46 can determine that the data is not an outlier as rare data even if it falls under the outlier category described above.

For example, if the task is a questionnaire-type task, and, for example, multiple answers from the same user U to the same question are all answers to the same answer option, the appropriateness determination unit 46 can determine that the data is not an outlier as rare data even if it falls under the outlier category described above.

For example, in the case of annotations, if the same task is performed multiple times by the same user U and the number of enclosed areas or the number of tags are all the same, the appropriateness determination unit 46 can determine that the data is rare data and not an outlier, even if it falls under the outlier category described above.

For example, if the task is an operation-type task, for example, when the same task is executed multiple times by the same user U, and the operation pattern, operation time, etc. are all the same, even if the task falls under the outlier described above, the appropriateness determination unit 46 can determine that the data is not an outlier as rare data.

[4.3.8. Notification section 47
When the appropriateness determining unit 46 determines that the execution result of the task by the user U is inappropriate, the notification unit 47 transmits, to the user, notification information indicating that the execution result of the task by the user U is inappropriate. The notification information is sent to U's terminal device 2 to notify user U.

In addition, the notification unit 47 can also notify the user U of the notification information during the task execution process by transmitting notification information indicating that the task execution result is inappropriate to the terminal device 2 of the user U each time the appropriateness determination unit 46 determines that the task execution result is inappropriate during the task execution process.

The notification unit 47 also notifies the user U of the reward to be granted to the user U by transmitting estimated reward information, which is information on the reward estimated by the estimation unit 50, to the terminal device 2 of the user U. If the reward estimated by the estimation unit 50 is lower than the normal reward, the notification unit 47 notifies the user U of the reward to be granted to the user U by transmitting estimated reward information including information such as the character string "If you don't do it better, your reward amount will be reduced" to the terminal device 2 of the user U.

[4.3.9. Update section 48
The update unit 48 updates the tasks based on the determination result by the suitability determination unit 46. The update unit 48 can update the tasks based on the suitability of each task determined by the suitability determination unit 46, for example.

For example, when the suitability is judged on a task-by-task basis by the suitability judgment unit 46, the update unit 48 can also update the task if the inappropriateness rate, which is the rate at which the suitability judgment unit 46 judges that the execution result of the task is inappropriate, is equal to or greater than a threshold value. For example, when a task is executed 100 times, if the suitability judgment unit 46 judges that the execution result of the task is inappropriate 10 times, the inappropriateness rate is 10%.

The update unit 48 updates the execution information by, for example, changing at least one of the position of the UI of the task, the layout of the task, the design of the task, the execution order of the task, etc. The task unit 43 can cause the user U to execute the updated task by transmitting the execution information updated by the update unit 48 to the terminal device 2 of the user U.

The update unit 48 also updates the task based on the judgment result by the appropriateness judgment unit 46 during the execution of the task by the user U. When the appropriateness judgment unit 46 judges that the execution result of the task is not appropriate during the execution of the task, the update unit 48 updates the task that is in the process of being executed. Then, the task unit 43 transmits the execution information updated by the update unit 48 to the terminal device 2 of the user U, thereby enabling the user U to execute the updated task from the middle of the execution of the task.

For example, if the suitability determination unit 46 determines that the result of the task execution is inappropriate during the task execution process, the task unit 43 transmits, if the task is a questionnaire-type task, execution information in which the order of the questions answered by the user U during the task execution process is changed to the terminal device 2 of the user U. This allows the task unit 43 to have the user U re-execute the task in which the order of the options for the questions answered by the user U during the execution process and for which the suitability determination unit 46 determines that the execution result is inappropriate is changed.

Furthermore, if the suitability determination unit 46 determines that the result of a task execution is inappropriate during the task execution process, and if the task is an annotation or operation type task, the task unit 43 transmits execution information to the terminal device 2 of the user U, for example, to cause the user U to stop the task being executed by the user U during the task execution process and to execute the task again from the beginning. This allows the task unit 43 to cause the user U to re-execute the task whose execution result has been determined to be inappropriate by the suitability determination unit 46.

In addition, when the suitability determination unit 46 determines that the result of the task execution is inappropriate during the task execution process, if the task unit 43 updates the task in the process of execution as described above, it can also stop the execution of the task by the user U and have the user U execute the updated task from the beginning of the task.

4.3.10. Reward Granting Unit 49
The reward granting unit 49 grants a reward to the user U when the execution result is acquired by the acquisition unit 40 .

The reward given to the user U may be, for example, electronic money, points, cash, or various benefits. Points are a value that can be used, for example, to purchase products or use services on online sites. Benefits may be, for example, coupons, gift certificates, service vouchers, etc., but are not limited to these examples.

The reward granting unit 49 grants a reward to the user U who performed the task, for example, by adding an amount of electronic money or points corresponding to the task to the account or wallet of the user U who performed the task in the online service. The reward granting unit 49 also grants a reward to the user U who performed the task, for example, by transmitting reward information, which is information indicating a benefit, to the terminal device 2 so that the reward information can be displayed on the terminal device 2. If the reward is cash, the reward granting unit 49 grants the reward to the user U, for example, by depositing the cash via a bank account.

When the suitability determination unit 46 determines that the user U's execution of the task is inappropriate, the reward granting unit 49 determines a lower reward to be granted to the user U compared to a case in which the user U's execution of the task is determined to be appropriate. In addition, when the number of possible outliers is two or more, the reward granting unit 49 can also determine a lower reward to be granted to the user U the greater the number of outliers.

The reward granting unit 49 can also determine the reward to be granted to the user U to be lower, for example, the higher the possibility that the execution result to be judged contains an outlier. The possibility that the execution result to be judged contains an outlier becomes higher, for example, the closer it is to a statistical outlier.

For example, the possibility that the execution result to be judged contains an outlier increases as the occurrence probability of the above-mentioned answer patterns decreases, as the distribution ratio of the answer time decreases, and as the distribution ratio of the number of characters decreases. The possibility that the execution result to be judged contains an outlier is judged, for example, by the suitability judgment unit 46, but may also be judged by the reward granting unit 49.

The reward granting unit 49 determines a higher reward to be granted to the user U, for example, for a task that is executed less frequently or fewer times in absolute or relative terms. A task that is executed relatively fewer times is, for example, a task in which the ratio of the number of executions to the number of executions required is less than the ratio of the elapsed period to the execution result collection period.

For example, in the case of electronic money, the larger the amount, the higher the reward, and in the case of points or benefits, the larger the amount when converted into cash, the higher the reward. Information indicating the reward to be granted to user U may be included in the request information. This can increase the likelihood that user U will complete the task.

Furthermore, when the user U re-executes a task requested by the request unit 42, the reward granting unit 49 grants the user U an additional reward in addition to the reward for executing the task. The additional reward may be the same type of reward as the reward for the previous execution of the task, or may be a different reward from the reward for the previous execution of the task. The type of reward may be, for example, electronic money, points, cash, or various types of benefits.

The reward granting unit 49 can also determine an additional reward based on the difference between the appropriateness of the user U's execution of the task and the appropriateness of the re-execution of the task. For example, the reward granting unit 49 grants a higher reward to the user U as an additional reward the greater the appropriateness difference.

[4.3.11. Estimation unit 50
The estimation unit 50 estimates a reward to be given to the user U in the process of performing the task based on the result of the judgment by the suitability judgment unit 46 of the suitability of the task execution result in the process of performing the task.

For example, the estimation unit 50 estimates that the higher the likelihood that the task execution result in the process of executing the task contains an outlier, the lower the reward to be granted to the user U. The likelihood that the execution result contains an outlier increases, for example, the closer it is to a statistical outlier.

For example, the possibility that the execution result to be judged contains an outlier increases as the occurrence probability of the above-mentioned answer patterns decreases, as the distribution ratio of the answer time decreases, and as the distribution ratio of the number of characters decreases. The possibility that the execution result to be judged contains an outlier is judged, for example, by the outlier judgment unit 45 or the appropriateness judgment unit 46, but may also be judged by the estimation unit 50.

5. Processing Procedure
Next, a procedure of information processing by the processing unit 22 of the information processing device 1 according to the embodiment will be described. Fig. 8 is a flowchart showing an example of information processing by the processing unit 22 of the information processing device 1 according to the embodiment.

As shown in FIG. 8, the processing unit 22 of the information processing device 1 determines whether or not task information transmitted from the terminal device 3 has been acquired (step S10). When the processing unit 22 determines that task information has been acquired (step S10: Yes), it stores the acquired task information in the storage unit 21 (step S11).

When the processing of step S11 is completed or when it is determined that task information has not been acquired (step S10: No), the processing unit 22 determines whether or not the task notification timing has arrived (step S12). The task notification timing may be, for example, a timing that arrives at every predetermined period, when there is a request such as a list request from the terminal device 2 of the user U, or a timing each time task information is acquired, but is not limited to such examples.

When the processing unit 22 determines that the task notification timing has arrived (step S12: Yes), it transmits request information to the terminal device 2 of the user U and requests the user U to execute the task (step S13).

When the processing of step S13 is completed or when it is determined that the task notification timing has not arrived (step S12: No), the processing unit 22 determines whether or not a task request has been accepted (step S14). When the processing unit 22 determines that a task request has been accepted (step S14: Yes), it transmits execution information included in the task information of the task specified by the accepted task request (step S15).

When the processing of step S15 is completed, or when it is determined that a task request has not been received (step S14: No), the processing unit 22 determines whether or not a determination timing has arrived (step S16). The determination timing may be, for example, the timing at which the execution result of the task is acquired, the timing during the execution of the task, or the timing that arrives at each predetermined period, but is not limited to such examples.

When the processing unit 22 determines that the judgment timing has arrived (step S16: Yes), it performs an outlier judgment process (step S17). The outlier judgment process is a process for judging whether or not an outlier is included in the result of the task execution by the user U. Then, the processing unit 22 performs a suitability judgment process based on the result of the outlier judgment process (step S18). The suitability judgment process is a process for judging whether or not the execution of the task by the user U is appropriate.

Then, the processing unit 22 performs a response process based on the suitability judgment process (step S19). For example, if the suitability judgment process judges that the execution of the task by the user U is inappropriate, the response process may request the user U to re-execute the task or notify the user U that the result of the task execution is inappropriate.

When the processing of step S19 is completed or when it is determined that the determination timing has not yet arrived (step S16: No), the processing unit 22 determines whether or not the execution result transmitted from the terminal device 2 that transmitted the execution information in step S19 has been accepted (step S20). When the processing unit 22 determines that the execution result has been accepted (step S20: Yes), it grants a reward to the user U of the terminal device 2 that transmitted the execution result (step S21).

When the processing of step S21 is completed or when it is determined that the execution result has not been received (step S20: No), the processing unit 22 determines whether or not the operation end timing has arrived (step S22). The processing unit 22 determines that the operation end timing has arrived, for example, when the power supply of the information processing device 1 is turned off.

If the processing unit 22 determines that the operation end time has not yet arrived (step S22: No), it transitions to step S10, and if it determines that the operation end time has arrived (step S22: Yes), it terminates the processing shown in FIG. 8.

[6. Other]
If the outlier determination unit 45 determines that the result of task execution by user U contains an abnormal value during the process of task execution by user U, the notification unit 47 can send notification information including information indicating an inquiry regarding the abnormal value during the process of task execution to user U's terminal device 2.

For example, the notification unit 47 can transmit information including the character string "For example, is there a mistake?" as notification information to the terminal device 2 of the user U as information indicating an inquiry about an abnormal value, thereby causing the character string "For example, is there a mistake?" to be displayed on the terminal device 2 of the user U in the process of executing a task.

The calculation unit 44 can also perform, for example, a process of generating statistical information on the results of task execution for each attribute of the user U who executed the task for each task. For example, the calculation unit 44 can generate statistical information on the results of task execution performed by a male user U in his twenties, generate statistical information on the results of task execution performed by a female user U in his twenties who likes idols, or generate statistical information on the results of task execution performed by a female user U in her forties who lives in Minato Ward, Tokyo. The calculation unit 44 identifies the attributes of user U based on the user information acquired by the acquisition unit 40.

The calculation unit 44 can also perform, for example, a process of generating statistical information on the execution results of a task for each context of the user U who executed the task for each task. The estimation unit 50 can estimate the context of the user U at the time of execution of the task based on the detection information acquired by the acquisition unit 40, etc. The calculation unit 44 can perform, for each task, a process of generating statistical information on the execution results of a task for each context of the user U based on the result of the context estimation by the estimation unit 50.

As described above, the estimation unit 50 can estimate the context of the user U, for example, based on the detection information. For example, the estimation unit 50 estimates the location of the user U, the operating state of the user U, the posture of the user U, etc., based on information indicating the position of the user U and information indicating the posture of the terminal device 2, etc., contained in the detection information.

For example, when the estimation unit 50 determines, based on information indicating the location of user U included in the detection information, that the location of user U is moving on the line at a predetermined speed or faster, it estimates that user U is inside a train. In addition, when the estimation unit 50 determines, based on information indicating the location of user U included in the detection information, that user U has remained at the location of a restaurant for a predetermined period of time or longer, it estimates that user U is inside the restaurant.

The processing unit 22 can also function as an extraction unit that extracts information that satisfies a predetermined condition from the results of the execution of a task by each user U using the terminal device 2 as extraction target information. In this case, the calculation unit 44 generates statistical information based on the extraction target information extracted by the extraction unit. The predetermined condition is, for example, a condition that outliers and abnormal values are excluded.

The above-mentioned threshold for determining outliers is, for example, a value outside nσ of the standard deviation, where n is a value equal to or greater than 1. For example, the threshold is a value outside 1σ of the standard deviation or outside 2σ of the standard deviation. It may also be, for example, a value 1/m times the average value. m is, for example, a value equal to or greater than 5.

The update unit 48 can also generate execution information in which a task determined by the suitability determination unit 46 to be inappropriate for execution or a part of the task is included in another task. For example, the update unit 48 can generate execution information in which a question corresponding to an answer determined by the suitability determination unit 46 to be inappropriate for execution is included in another questionnaire-type task. In this case, when multiple answers by the same user U to the same question are all answers to the same option, the suitability determination unit 46 can also determine that the answers are not rare data and are not an outlier, even if they fall under the above-mentioned outlier.

The outlier determination unit also has an outlier degree determination model that receives the execution result of a task as input and outputs an appropriateness score (hereinafter, "score") indicating the appropriateness of the execution of the task, and can determine outliers using the outlier degree determination model. The outlier degree estimation model is generated, for example, by machine learning using learning data that includes the execution information and information indicating whether or not it is an outlier.

For example, the outlier degree estimation model is generated by machine learning using a neural network such as a convolutional neural network, a recurrent neural network, or a deep neural network, but is not limited to such examples. For example, instead of a neural network, the learning model may be generated using machine learning using a learning algorithm such as a gradient boosting decision tree (GBDT), linear regression, or logistic regression.

The request unit 42 can, for example, lower the probability of requesting task execution or reduce the type or number of tasks to be executed for a user U whose execution of a task is determined to be inappropriate by the suitability determination unit 46. The request unit 42 can, for example, increase the probability of requesting task execution or increase the type or number of tasks to be executed for a user U whose execution of a task is determined to be appropriate by the suitability determination unit 46.

In addition, the execution results that are to be judged as outliers by the outlier judgment unit 45 and the execution results that are to be judged as appropriate for task execution by the suitability judgment unit 46 can be selected randomly, for example, or can be increased for users U whose task execution is judged to be inappropriate by the suitability judgment unit 46 at a higher rate.

7. Hardware Configuration
Each of the information processing device 1 and the terminal device 2 according to the embodiment described above is realized by a computer 200 having a configuration as shown in Fig. 9. Fig. 9 is a hardware configuration diagram showing an example of the computer 200 that realizes each function of the information processing device 1 and the terminal device 2 according to the embodiment. The computer 200 has a CPU 201, a RAM 202, a ROM (Read Only Memory) 203, a HDD (Hard Disk Drive) 204, a communication interface (I/F) 205, an input/output interface (I/F) 206, and a media interface (I/F) 207.

The CPU 201 operates based on a program stored in the ROM 203 or the HDD 204, and controls each component. The ROM 203 stores a boot program executed by the CPU 201 when the computer 200 is started, and programs that depend on the hardware of the computer 200.

The HDD 204 stores programs executed by the CPU 201 and data used by such programs. The communication interface 205 receives data from other devices via the network N (see FIG. 2) and sends it to the CPU 201, and transmits data generated by the CPU 201 to other devices via the network N.

The CPU 201 controls output devices such as a display and a printer, and input devices such as a keyboard and a mouse, via the input/output interface 206. The CPU 201 acquires data from the input devices via the input/output interface 206. The CPU 201 also outputs data generated via the input/output interface 206 to the output devices.

The media interface 207 reads a program or data stored in the recording medium 208 and provides it to the CPU 201 via the RAM 202. The CPU 201 loads the program from the recording medium 208 onto the RAM 202 via the media interface 207 and executes the loaded program. The recording medium 208 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable Disc), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

For example, when the computer 200 functions as the information processing device 1 or terminal device 2 according to the embodiment, the CPU 201 of the computer 200 executes a program loaded onto the RAM 202 to realize the functions of the processing unit 22 or processing unit 15. In addition, the HDD 204 stores data in the memory unit 21 or memory unit 14. The CPU 201 of the computer 200 reads and executes these programs from the recording medium 208, but as another example, these programs may be obtained from another device via the network N.

[8. Other]
In addition, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

For example, the information processing device 1 described above may be realized by multiple server computers, and depending on the function, the configuration can be flexibly changed, such as by calling an external platform using an API or network computing.

The above-described embodiments and variations can be combined as appropriate to the extent that they do not cause inconsistencies in the processing content.

9. Effects
As described above, the information processing device 1 according to the embodiment includes an acquisition unit 40 that acquires execution results of a task by a user U using a terminal device 2, an outlier determination unit 45 that determines whether or not an outlier is included in the execution result acquired by the acquisition unit 40, and an appropriateness determination unit 46 that determines the appropriateness of the execution of the task by the user U based on the determination result by the outlier determination unit 45. Thereby, the information processing device 1 can, for example, exclude inappropriate execution results from the execution results to be provided to the client O, thereby promoting crowdsourcing.

The acquisition unit 40 also acquires statistical information on the results of tasks executed by multiple users U using the terminal device 2, and the outlier determination unit 45 determines whether the execution results include an outlier based on the results of a comparison between the execution results acquired by the acquisition unit 40 and the statistical information. This allows the information processing device 1 to accurately determine outliers and promote crowdsourcing.

The acquisition unit 40 also acquires an abnormal value indicating an abnormality in the task execution result, and the outlier determination unit 45 determines that the execution result includes an outlier when the execution result acquired by the acquisition unit 40 includes an abnormal value. This allows the information processing device 1 to accurately determine outliers and promote crowdsourcing.

The information processing device 1 also includes a request unit 42 that requests the user U to re-execute the task when the suitability determination unit 46 determines that the result of the task execution by the user U is not appropriate. This enables the information processing device 1 to increase appropriate execution results and promote crowdsourcing.

The information processing device 1 also includes a notification unit 47 that notifies information indicating that the result of the task execution by the user U is inappropriate when the suitability determination unit 46 determines that the result of the task execution by the user U is inappropriate. This enables the information processing device 1 to, for example, reduce inappropriate execution results and promote crowdsourcing.

The acquisition unit 40 also repeatedly acquires execution results during the process of the user U executing the task, the outlier determination unit 45 determines whether or not each of the execution results repeatedly acquired by the acquisition unit 40 during the process of the user U executing the task contains an outlier, and the suitability determination unit 46 repeatedly determines the suitability of the user U during the process of the user U executing the task based on the determination result by the outlier determination unit 45. This allows the information processing device 1 to, for example, reduce inappropriate execution results and promote crowdsourcing.

In addition, the notification unit 47 notifies information indicating that the execution result is inappropriate each time the appropriateness determination unit 46 determines that the execution result is inappropriate during the execution process. This enables the information processing device 1 to, for example, reduce inappropriate execution results and promote crowdsourcing.

The information processing device 1 also includes an estimation unit 50 that estimates the reward to be granted to the user U based on the judgment result by the appropriateness judgment unit 46 during the execution process, and the notification unit 47 notifies information regarding the reward estimated by the estimation unit 50 during the execution process. This enables the information processing device 1 to reduce inappropriate execution results, for example, and promote crowdsourcing.

The information processing device 1 also includes an update unit 48 that updates the tasks based on the judgment result by the suitability judgment unit 46. This enables the information processing device 1 to, for example, reduce inappropriate execution results and promote crowdsourcing.

The above describes the embodiments of the present application in detail based on the drawings, but this is merely an example, and the present invention can be implemented in other forms that incorporate various modifications and improvements based on the knowledge of those skilled in the art, including the aspects described in the disclosure section of the invention.

The above-mentioned "section, module, unit" can be read as "means" or "circuit." For example, an acquisition unit can be read as an acquisition means or an acquisition circuit.

REFERENCE SIGNS LIST 1 Information processing device 2, 3 Terminal device 10, 20 Communication unit 11 Display unit 12 Operation unit 13 Sensor group 14, 21 Memory unit 15, 22 Processing unit 16 Information acquisition unit 17 Display processing unit 18 Information output unit 30 User information storage unit 31 Task information storage unit 32 Task execution result storage unit 40 Acquisition unit 41 Reception unit 42 Request unit 43 Task unit 44 Calculation unit 45 Outlier determination unit 46 Suitability determination unit 47 Notification unit 48 Update unit 49 Reward grant unit 50 Estimation unit 100 Information processing system N Network

Claims (11)

an acquisition unit that acquires a result of a task executed by a user using a terminal device;
an outlier determination unit that determines whether or not an outlier is included in the execution result acquired by the acquisition unit;
an appropriateness determination unit that determines whether execution of the task by the user is appropriate based on a determination result by the outlier determination unit. The acquisition unit is
acquiring statistical information on the results of execution of the tasks by a plurality of users using terminal devices;
The outlier determination unit is
The information processing apparatus according to claim 1 , further comprising: determining whether or not the execution result includes the outlier based on a comparison result between the execution result acquired by the acquisition unit and the statistical information. The acquisition unit is
acquiring an abnormality value indicating an abnormality in the execution result of the task;
The outlier determination unit is
The information processing apparatus according to claim 1 , wherein, when the abnormal value is included in the execution result acquired by the acquisition unit, it is determined that the execution result includes the outlier. The information processing device according to any one of claims 1 to 3, further comprising a request unit that requests the user to re-execute the task when the suitability determination unit determines that the result of the user's execution of the task is inappropriate. The information processing device according to any one of claims 1 to 3, further comprising a notification unit that, when the appropriateness determination unit determines that the result of the execution of the task by the user is inappropriate, notifies the user of information indicating that the result of the execution of the task by the user is inappropriate. The acquisition unit is
repeatedly acquiring the execution results during the process of execution of the task by the user;
The outlier determination unit is
determining whether or not an outlier is included in each of the execution results repeatedly acquired by the acquisition unit during the process of the task being executed by the user;
The suitability determination unit is
The information processing apparatus according to claim 5 , further comprising: a step of repeatedly judging the suitability of the user during the process of the user's execution of the task, based on a result of the judgment by the outlier judgment unit. The notification unit is
The information processing apparatus according to claim 6 , wherein each time the execution result is determined to be inappropriate by the appropriateness determining unit during the execution process, information indicating that the execution result is inappropriate is notified. an estimation unit that estimates a reward to be given to the user based on a result of the judgment by the suitability judgment unit during the execution process;
The notification unit is
The information processing device according to claim 7 , further comprising: a notification of information regarding the remuneration estimated by the estimation unit during the execution process. 4. The information processing apparatus according to claim 1, further comprising an update unit that updates the task based on a result of the judgment by the suitability judgment unit. 1. A computer-implemented information processing method, comprising:
an acquisition step of acquiring a result of a task executed by a user using a terminal device;
an outlier determination step of determining whether or not the execution result acquired by the acquisition step includes an outlier;
and a suitability determination step of determining whether execution of the task by the user is appropriate based on a determination result from the outlier determination step. An acquisition step for acquiring a result of a task executed by a user using a terminal device;
an outlier determination step of determining whether or not the execution result acquired by the acquisition step includes an outlier;
and a suitability determination procedure for determining whether execution of the task by the user is appropriate based on a determination result of the outlier determination procedure.

Images