JP6677794B1 - Evaluation device, evaluation method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an evaluation device, an evaluation method and a program for efficiently evaluating an answer based on a reliable answerer's answer. An evaluation device includes an inspection target acquisition unit that acquires a plurality of inspection targets that are set to be associated with a concept, and a dummy target that acquires one or more dummy targets that are not set to be associated with a concept. Select from the acquisition unit 102, a plurality of checkers, a presentation unit 103 that presents a question for selecting a target that should not be associated with a concept from a plurality of inspection targets and one or more dummy targets, and a plurality of checkers Acceptance unit 104 that accepts an answer that includes a selected object, an extraction unit 105 that extracts an answer that includes one or more dummy objects from the accepted answers, and an inspection that is included in the extracted answer among a plurality of inspection objects A reporting unit 106 for reporting the target is provided. [Selection diagram] FIG.

Description

  The present invention relates to an evaluation device, an evaluation method, and a program, and more particularly, to evaluation of a response.

  2. Description of the Related Art In recent years, crowdsourcing services have been known in which an unspecified number of people are requested via a network for simple marketing research and work that can be easily processed, and those who receive the request are rewarded as compensation. Crowdsourcing services have the merit of being able to receive requests easily, but there are quite a few contractors who do not work seriously, and there is a concern about the accuracy of such workers.

  It is widely recognized that the attitude toward the work affects the answer of the worker, and there are attempts to reflect the attitude toward the work in the answer. For example, Patent Document 1 discloses a technique in which a questionnaire question group and a dummy question group are prepared, the psychological tendency of the respondent is obtained based on the answer result, and the obtained psychological tendency is reflected in the answer. Is done.

JP 2004-287736 A

  As in Patent Document 1, a method of preparing a question for obtaining intended information and a dummy question and answering both are generally used. In such a method, a respondent seriously answers the question. It is possible to determine whether or not the person has the knowledge to answer correctly. That is, it is possible to determine whether the respondent is a reliable person. However, the number of questions that need to be prepared increases, the number of questions that need to be answered increases, and the burden on respondents increases, so a more efficient method is desired.

  An object of the present invention is to solve the above-described problems, and to provide an evaluation device, an evaluation method, and a program for efficiently evaluating an answer based on an answer of a reliable answerer.

The evaluation device according to the first aspect of the present invention includes:
An inspection target acquisition unit that acquires a plurality of inspection targets set to be linked to a concept,
A dummy object acquisition unit that acquires one or more dummy objects not set to be associated with the concept,
A plurality of checkers, the plurality of test objects, the one or more dummy objects, from the presentation unit for presenting a question to select an object that should not be associated with the concept,
From the plurality of checkers, a receiving unit that receives an answer composed of the selected target,
An extracting unit configured to extract an answer including the one or more dummy targets among the accepted answers;
Of the plurality of inspection targets, a reporting unit that reports an inspection target included in the extracted answer,
It is characterized by having.

Further, in the evaluation device according to the above viewpoint,
The dummy object acquisition unit acquires one of the dummy objects,
The presenting unit presents a question for selecting a plurality of targets that should not be linked to the concept.

Further, in the evaluation device according to the above viewpoint,
The dummy target obtaining unit obtains a plurality of the dummy targets,
The reporting unit,
Based on the number of the dummy object is included in the extracted answer, to determine the accuracy indicating the degree to which the selection by the checker was accurate,
Based on the accuracy, for the test target included in the extracted answer, determine an inappropriate score indicating the degree to which the test target should not be associated with the concept,
A value obtained by integrating the inappropriate scores obtained from the extracted answers among the answers received from the plurality of checkers is reported.

Further, in the evaluation device according to the above viewpoint,
The report unit reports an inspection target having a minimum value obtained by integrating the inappropriate scores as a target to be linked to the concept.

Further, in the evaluation device according to the above viewpoint,
The report unit reports the number of answers that are the extracted answers and that include the inspection target, among the answers received from the plurality of checkers.

Further, in the evaluation device according to the above viewpoint,
The report unit reports the inspection target having the smallest number of the extracted responses among the responses received from the plurality of checkers as a target to be associated with the concept.

Further, in the evaluation device according to the above viewpoint,
The concept is a product,
The inspection target acquisition unit acquires, as the inspection target, a category represented by a hierarchical structure, the category being associated with a node in the same hierarchy as the category to which the product belongs.
The dummy target obtaining unit obtains, as the dummy target, a category that is a node in the same hierarchy as a parent node of a node to which the category to which the product belongs is associated and that is associated with a node different from the parent node. It is characterized by the following.

Further, in the evaluation device according to the above viewpoint,
The concept is a product,
The inspection target acquisition unit acquires, as the inspection target, a tag set to be linked to the product by a shop,
The dummy object acquisition unit acquires, as the dummy object, a tag that is not set to be linked to the product by another shop.

Further, in the evaluation device according to the above viewpoint,
The concept is a tag,
The inspection target acquisition unit acquires, as the inspection target, a product for which a setting for linking to the tag has been performed by a shop,
The dummy target acquisition unit acquires, as the dummy target, a product that is not set to be linked to the tag by another shop.

Further, in the evaluation device according to the above viewpoint,
The concept is a category represented by a hierarchical structure,
The inspection target acquisition unit acquires a product belonging to the category as the inspection target,
The dummy target acquisition unit acquires, as the dummy target, a product belonging to a category that is a node in the same hierarchy as the parent node of the category and that is associated with a node different from the parent node.

Further, in the evaluation device according to the above viewpoint,
The concept is a search word,
The inspection target acquisition unit acquires items included in the first number of search results from the top of the search results of the search word as the inspection target,
The dummy target acquisition unit acquires an item that is not included in the search result of the search word as the dummy target.

Further, in the evaluation device according to the above viewpoint,
The dummy target acquisition unit,
In a text associated with the item to be inspected, a second number of words having the highest appearance frequency is extracted,
The extracted word is specified as a search word, and an item included in a search result in which the search word of the concept is specified as an exclusion word is acquired as the dummy target.

The evaluation method according to the second aspect of the present invention includes:
An evaluation method executed by the evaluation device,
The evaluation device, an inspection target acquisition step of acquiring a plurality of inspection targets set to be associated with a concept,
The evaluation device, a dummy target obtaining step of obtaining one or more dummy targets not set to be associated with the concept;
A presentation step of presenting a question that causes the plurality of checkers to select a target that should not be linked to the concept, from the plurality of test targets and the one or more dummy targets,
The evaluation device, from the plurality of checkers, a receiving step of receiving an answer consisting of the selected target,
An extracting step in which the evaluation device extracts an answer including the one or more dummy targets among the accepted answers;
A report step in which the evaluation device reports an inspection object included in the extracted answer, among the plurality of inspection objects,
It is characterized by having.

A program according to a third aspect of the present invention includes:
Computer
An inspection target acquisition unit that acquires a plurality of inspection targets set to be linked to a concept,
A dummy object acquisition unit that acquires one or more dummy objects not set to be associated with the concept,
A plurality of checkers, a presentation unit that presents a question for selecting an object that should not be linked to the concept from the plurality of inspection objects and the one or more dummy objects;
From the plurality of checkers, a receiving unit that receives an answer consisting of the selected target,
An extracting unit configured to extract an answer including the one or more dummy targets among the accepted answers;
Of the plurality of inspection targets, a reporting unit that reports the inspection targets included in the extracted answer,
It is characterized by functioning as

  The program may be recorded on a non-transitory recording medium. The non-transitory recording medium can be distributed and sold independently of the computer. Here, the non-transitory recording medium refers to a tangible recording medium. The non-transitory recording medium is, for example, a compact disk, a flexible disk, a hard disk, a magneto-optical disk, a digital video disk, a magnetic tape, a semiconductor memory, or the like. In addition, the temporary (transitory) recording medium indicates a transmission medium (propagation signal) itself. The temporary recording medium is, for example, an electric signal, an optical signal, an electromagnetic wave, or the like. The temporary storage area is an area for temporarily storing data and programs, and is, for example, a volatile memory such as a RAM (Random Access Memory).

  According to the present invention, it is possible to provide an evaluation device, an evaluation method, and a program for efficiently evaluating an answer based on the answer of a reliable answerer.

It is a figure showing the relation between the evaluation device concerning the embodiment of the present invention, and a checker terminal. FIG. 2 is a diagram illustrating a hardware configuration of an evaluation device according to the embodiment. FIG. 2 is a diagram illustrating a functional configuration of the evaluation device according to the first embodiment. FIG. 4 is a diagram for explaining a question presented to the checker according to the first embodiment. FIG. 4 is a diagram for explaining an answer table according to the first embodiment. FIG. 4 is a diagram for describing a page that reports an inspection target that should not be linked to the concept according to the first embodiment. FIG. 4 is a diagram for explaining an inspection target table according to the first embodiment. FIG. 5 is a diagram for explaining a page that reports an inspection target to be linked to the concept according to the first embodiment. FIG. 4 is a diagram for explaining a hierarchical structure of a category to which a product according to the first embodiment belongs. 5 is a flowchart for explaining an evaluation process performed by the evaluation device according to the first embodiment. It is a figure for explaining the question shown to the checker concerning Embodiment 2. FIG. 14 is a diagram for explaining an answer table according to the second embodiment. FIG. 9 is a diagram for explaining an accuracy table according to the second embodiment. FIG. 14 is a diagram for explaining an inappropriate score table according to the second embodiment. 9 is a flowchart for explaining an evaluation process performed by the evaluation device according to the second embodiment.

(1. Relationship between the evaluation device of the embodiment and the checker terminal)
The evaluation device 100 according to the embodiment of the present invention is connected to a network 300 as shown in FIG. A plurality of checker terminals 200 are connected to the network 300 in addition to the evaluation device 100.

  The evaluation device 100 is a device that provides a question to the checker terminal 200 and evaluates a response received from the checker terminal 200.

  The checker terminal 200 is a terminal used by a checker who answers a question provided by the evaluation device 100.

  The network 300 is a wireless or wired communication network. The network 300 may be any network, for example, the Internet, an intranet, an extranet, a LAN (Local Area Network), a VPN (Virtual Private Network), a telephone network, and the like.

(2. Hardware configuration of evaluation device)
Next, the hardware configuration of the evaluation device 100 will be described.

  As shown in FIG. 2, the evaluation device 100 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM 13, a recording medium 14, an output unit 15, a communication unit 16, an operation unit And 17. Each component is connected by a bus 18.

  The CPU 11 controls the operation of the entire evaluation apparatus 100, is connected to each component, and exchanges control signals and data.

  The ROM 12 stores operating system programs and various data necessary for controlling the operation of the entire evaluation apparatus 100.

  The RAM 13 is for temporarily recording data and programs, and stores programs and data read from the recording medium 14 and other data necessary for communication.

  The recording medium 14 includes a hard disk, a flash memory, and the like, and records data to be processed by the evaluation device 100.

  The output unit 15 includes a display device including an LCD (Liquid Crystal Display) and a backlight, and a sound output device such as a speaker. The output unit 15 outputs, for example, data output from the CPU 11 under the control of the CPU 11.

  The communication unit 16 includes a communication interface for connecting the evaluation device 100 to a computer communication network such as the Internet, and communicates with other information processing devices via the communication unit 16.

  The operation unit 17 includes input devices such as buttons, a keyboard, and a touch panel. The operation unit 17 receives an operation input from a user of the evaluation device 100 and outputs a signal corresponding to the received operation input to the CPU 11.

(3. Functional configuration of the evaluation device according to the first embodiment)
The evaluation device according to the present embodiment presents a plurality of inspection objects and one dummy object, and evaluates an answer based on the selected inspection object and dummy object.

  In the following, the administrator of the evaluation device 100 determines whether or not it is appropriate to link the targets i1 to i5 set to be linked to the concept X to the five checkers 1 to 5 with the concept X. The function of the evaluation device 100 will be described with reference to a case where an operation for making a determination is requested as an example.

  FIG. 3 is a diagram illustrating a functional configuration of the evaluation device 100. The evaluation device 100 functionally includes an inspection target acquisition unit 101, a dummy target acquisition unit 102, a presentation unit 103, a reception unit 104, an extraction unit 105, and a report unit 106. These functions are realized by turning on the power of the evaluation apparatus 100 and executing a program stored in the ROM 12.

  The inspection target acquisition unit 101 acquires a plurality of inspection targets set to be linked to a concept.

  The concept is a thing to which a plurality of objects are linked, and can be arbitrarily set by the administrator of the evaluation device 100. For example, the concept is “product”, “search word”, “tag”, “category”, or the like.

  The inspection target is a target to be set to be linked to the concept and to check whether the linking is appropriate, and can be arbitrarily set by the administrator of the evaluation device 100. . For example, when the concept is “product”, the inspection target is “tag” or the like, and when the concept is “tag”, the inspection target is “product” or the like.

  When the administrators of the evaluation device 100 specify the targets i1 to i5 linked to the concept X as the inspection targets, the inspection target acquisition unit 101 acquires the targets i1 to i5 as the inspection targets.

  The dummy target acquisition unit 102 acquires one dummy target that is not set to be associated with a concept.

  The dummy target is different from the inspection target and has not been set to be linked to the concept, and can be arbitrarily set by the administrator of the evaluation device 100.

  For example, when the administrator of the evaluation device 100 specifies a target d1 not linked to the concept X as a dummy target, the dummy target obtaining unit 102 obtains the target d1 as a dummy target.

  The presentation unit 103 presents, to the plurality of checkers, a question for selecting a target that should not be linked to a concept from a plurality of test targets and a dummy target.

  The checker is a person who has been requested by the administrator of the evaluation device 100 to perform a task of determining whether or not the concept is linked to the inspection target, and has accepted the task.

  For example, as illustrated in FIG. 4, the presentation unit 103 includes a page 410 including a question 411 for asking an option that should not be linked to the concept X, and an option 412 having the inspection target i1 to i5 and the dummy target d1 as options. The work is transmitted to the checker terminals 200 of the checkers 1 to 5 that have accepted the work.

  The accepting unit 104 accepts, from a plurality of checkers, an answer including the selected target.

  For example, when the checker 1 checks the check boxes of i1, i2 and d1 among the options 412 in the checker terminal 200 and clicks the button 413, the accepting unit 104 returns an answer specifying i1, i2 and d1. To receive. Similarly, the receiving unit 104 receives an answer from the checker terminals 200 of the checkers 2 to 5.

  The content of the answer accepted by the accepting unit 104 is registered in, for example, an answer table 500a shown in FIG. In the answer table 500a of FIG. 5, a checker ID for identifying a checker and a selected target are registered in association with each other. The answer table 500a is stored in the recording medium 14 of the evaluation device 100.

  The extracting unit 105 extracts an answer including a dummy target from the received answers.

  For example, the extraction unit 105 refers to the answer table 500a of FIG. 5 and extracts the answers of the checker IDs “1”, “2”, “4”, and “5” as the answers from which the dummy target d1 is selected. In the answer table 500a of FIG. 5, the answer of the checker ID “3” is not extracted by the extracting unit 105 because the dummy object d1 is not included.

  The report unit 106 reports the inspection target included in the extracted answer among the plurality of inspection targets.

  The answer extracted by the extracting unit 105 is an answer including a dummy target. Since the question presented to the checker is to select an object that should not be linked to the concept, the answer including the dummy object is the correct answer. Therefore, the checker who gave the answer including the dummy object is considered to be a person who is serious about answering the question or has sufficient knowledge to answer the question correctly, that is, a reliable checker. Can be Therefore, the test target included in the extracted answer is a target that has not been determined by a reliable checker to be associated with a concept like a dummy target. Therefore, the report unit 106 reports the inspection target that is determined not to be linked to the concept by the reliable checker.

  For example, the report unit 106 specifies the inspection target i1, i2, i4 selected together with the dummy target with reference to the answer table 500a of FIG. Then, as shown in FIG. 6, the report unit 106 displays a page 420 that notifies that i1, i2, and i4 should not be linked to the concept X on the screen of the evaluation device 100. The administrator can see from the page 420 that i1, i2, and i4 should not be linked to the concept X.

  In addition, the report unit 106 reports the number of replies extracted from among the replies received from the plurality of checkers, the replies including the inspection target.

  The answer that is the extracted answer and includes the inspection target is an answer in which the inspection target is selected together with the dummy target. Therefore, the number of extracted answers that include the test target indicates the number of times that the test target is selected together with the dummy target.

  For example, the report unit 106 displays the inspection target table 600a on the screen of the evaluation device 100, as shown in FIG. In the inspection target table 600a, the inspection target presented to the checker and the number of times the inspection target is selected together with the dummy target are registered in association with each other. The inspection target table 600a is stored in the recording medium 14 of the evaluation device 100. By looking at the inspection target table 600a, the administrator can know which inspection target is highly unlikely to be associated with the concept X.

  In addition, the report unit 106 may report the number of accepted responses and the number of responses including the dummy target. For example, when the answer table 500a of FIG. 5 is obtained, the report unit 106 displays on the screen that the number of accepted answers is “5” and the number of answers for which the dummy target is selected is “4”. I do. Since the answer for which the dummy object is selected is the correct answer, knowing the ratio of the answers for which the dummy object is selected makes it possible to know the difficulty level of the question for the checker.

  In addition, the reporting unit 106 may report the number of checkers who have answered the question and the number of checkers who have selected the dummy target. For example, when the answer table 500a of FIG. 5 is obtained, the report unit 106 determines that the number of checkers that answered the question is “5” and that the checker selected the dummy target (checkers 1, 2, 4, and 5) Is displayed on the screen to the effect that the number is "4". As a result, it is possible to know how many people in the checker are seriously working on answering the question or have knowledge enough to answer the question correctly.

  In addition, the reporting unit 106 may report the number of checkers who answered the question and the number of checkers for which no dummy target was selected. For example, when the answer table 500a of FIG. 5 is obtained, the report unit 106 determines that the number of checkers that answered the question is “5” and the number of checkers (checker 3) that did not select a dummy target is “5”. "1 person" is displayed on the screen. As a result, it is possible to know how many people in the checker are not seriously working on answering the question or do not have the knowledge to answer the question correctly.

  In addition, the reporting unit 106 reports, as the target to be associated with the concept, the inspection target having the smallest number of extracted responses among the responses received from the plurality of checkers.

  The test object with the least number of extracted answers is the test object with the least number of selections together with the dummy object, and the number of times that is selected as a target that should not be linked to a concept by a reliable checker It is the least inspected object. Therefore, such an inspection target is considered to be an object that is appropriately linked to a concept.

  For example, the reporting unit 106 refers to the inspection target table 600a in FIG. 7 and identifies the inspection targets having the smallest number of times selected together with the dummy targets as i3 and i5. Then, as shown in FIG. 8, the report unit 106 displays a page 430 notifying that i3 and i5 are targets to be linked to the concept X on the screen of the evaluation device 100. The administrator can see from the page 430 that i3 and i5 are targets to be linked to the concept X.

  Further, the presentation unit 103 may present a question for selecting a plurality of targets that should not be linked to a concept.

  For example, as shown in the second row of the answer table 500a in FIG. 5, if only the dummy object is selected, the answer itself is correct, and the checker 2 is considered to be a reliable answerer. Therefore, it is not possible to judge the suitability of the inspection target. Therefore, by using a plurality of questions, it is possible to reduce the number of answers in which only the dummy object is selected, and determine the suitability of the inspection object by reflecting the answers of more reliable respondents.

  Note that the function of the evaluation device 100 according to the present embodiment has been described above by taking as an example the case where the dummy object acquisition unit 102 acquires one dummy object. However, the present invention is not limited to this. A plurality of dummy objects may be obtained. In this case, the extraction unit 105 extracts an answer including one or more dummy targets among the received responses, and the reporting unit 106 reports the inspection target included in the extracted answer among the plurality of inspection targets. I do.

  Hereinafter, examples of the inspection target and the dummy target when “product”, “search word”, “tag”, or “category” is adopted for the concept will be described.

  When the concept is “product”, the inspection target obtaining unit 101 obtains, as the inspection target, a category represented by a hierarchical structure, which is associated with a node in the same hierarchy as the category to which the product belongs. The acquisition unit 102 acquires, as a dummy target, a category that is a node in the same hierarchy as the parent node of the node to which the category to which the product belongs is associated and that is associated with a node different from the parent node.

  For example, assume that the hierarchical structure of categories is represented as shown in FIG. This hierarchical structure shows the inclusion relation of the category, and the product is included in any node of this hierarchical structure. The hierarchical structure below each of the nodes “A2” to “Ana”, “B2” to “Bnb”, and “C2” to “Cnc” (na, nb, nc: natural numbers) is omitted.

  For example, suppose that the shop Z that handles the product “Y” does not know the exact category of the product “Y” and sets the category of the product “Y” to “(A1-B1-C1-) other”. However, the product "Y" is not a category of "others", but categories "(A1-B1-C1-) D1" to "(A1-B1-C1-) Dnd" (nd: (Natural numbers) may be correct. Therefore, “D1” to “Dnd” are to be inspected, and the nodes are in the same hierarchy as the category “(A1-B1-) C1” of the “other” parent node, and are associated with nodes different from “C1”. One or more of the categories “(A1-B1-) C2” to “(A1-B1-) Cnc” are set as dummy objects. Then, the presentation unit 103 presents to the checker a question for selecting a category to which the product “Y” should not belong from the options, and an option including a category to be inspected and a category to be dummy.

  Since shop Z deals with the product "Y", it has some knowledge of the product "Y", and the classification of the parent node of the "other" category up to the category "C1" may be correct. It is considered high. That is, it is considered inappropriate to classify the product "Y" into the categories "C2" to "Cnc" that are nodes in the same hierarchy as the category "C1" and are associated with nodes other than "C1". Therefore, by setting any one of “C2” to “Cnc” as a dummy target, the respondent who has selected the dummy target has a person who has accurate knowledge of the classification of the product “Y”, or seriously. Can be regarded as responding.

  The category of the inspection target selected by the person who can trust the answer together with the dummy object may not be appropriate as the category of the product “Y”, and the inspection that has the least number of times included in the answer reported by the evaluation device 100 By confirming the target category, the most appropriate category to which the product “Y” belongs can be known.

  Also, when the concept is “product”, the inspection target acquisition unit 101 acquires, as an inspection target, a tag set to be linked to the product by the shop, and the dummy target acquisition unit 102 sets the tag associated with the product to another. Get tags that have not been set by the shop as dummy objects.

  For example, the shop Z that handles the product “Y” has set a plurality of tags “E1” to “Ene” (ne: natural number) to be linked to the product “Y”. When it is desired to confirm whether the tag is correct or not, the tags to be inspected are the tags “E1” to “Ene”, and the tags “F1” to “Fnf” that are not linked to the product “Y” by another shop handling the product “Y” (Nf: natural number) as one or more dummy targets. The presentation unit 103 presents to the checker a question for selecting a tag that should not be linked to the product “Y” from options, and an option including a tag to be inspected and a tag to be dummy.

  A tag that is not linked to the product “Y” by another shop is likely to be a tag that is not appropriate to be linked to the product “Y”. Therefore, by setting any one of the tags “F1” to “Fnf” as a dummy target, the respondent who has selected the dummy target has knowledge that can determine an accurate tag associated with the product “Y”. Alternatively, it can be considered that the answer is serious.

  The tag to be inspected selected by the person whose answer is reliable together with the dummy object may not be suitable as the tag of the product “Y”. Therefore, by confirming the tag of the inspection target reported by the evaluation device 100, , It is possible to check whether or not the tagging of the product “Y” by the shop Z is correct.

  When the concept is “tag”, the inspection target acquisition unit 101 acquires, as an inspection target, a product set to be linked to the tag by the shop, and the dummy target acquisition unit 102 sets the item linked to the tag to another. Get products that have not been sold by the shop as dummy objects.

  For example, the shop Z is configured to link the tag “E” to the products “Y1” to “Yny” (ny: natural number), and to link the tag “E” to the products “Y1” to “Yny”. If it is desired to confirm whether or not the item is correct, the inspection target is the products "Y1" to "Yny", and the product "W1" which is not linked to the tag "E" by another shop that links the tag "E" to the product ”To“ Wnw ”(nw: natural number) are set as dummy objects. The presentation unit 103 presents to the checker a question for selecting a product to which the tag “E” should not be linked from the options, and an option including a product to be inspected and a product to be dummy.

  It is considered that a product that is not linked to the tag “E” by another shop is likely to be a product that is not appropriate to be linked to the tag “E”. Therefore, by setting any of the products “W1” to “Wnw” as a dummy target, the respondent who has selected the dummy target has a knowledge of determining an accurate product linked with the tag “E”, or It can be regarded as a person who answers seriously.

  Then, the product to be inspected selected by the person whose answer is reliable together with the dummy object may not be suitable as the product to which the tag “E” is linked, so the product to be inspected reported by the evaluation device 100 is confirmed. This makes it possible to check whether the linking of the tag “E” by the shop Z is correct.

  When the concept is a “category” represented by a hierarchical structure, the inspection target acquisition unit 101 acquires a product belonging to the category as an inspection target, and the dummy target acquisition unit 102 determines a node of the same hierarchy as the parent node of the category. Then, a product belonging to a category associated with a node different from the parent node is acquired as a dummy target.

  For example, if the shop Z has set products “Y1” to “Yny” as products belonging to the category “D1” in FIG. 9, the inspection target is the products “Y1” to “Yny”, and Goods "V1" to "Vnv" belonging to the following categories "C2" to "Cnc" which are nodes in the same hierarchy as the category "C1" of the parent node and corresponded to nodes different from "C1" (nv: natural number) ) Is set as one or more dummy objects. The presenting unit 103 presents to the checker a question for selecting a product that should not belong to the category “D1” from options, and an option including a product to be inspected and a product to be dummy.

  The products "V1" to "Vnv" belonging to the categories "C2" to "Cnc" belong to different categories in the higher hierarchy of the category "D1", and thus are unlikely to be mistaken for the products belonging to "D1". It is. Therefore, by setting any of the products “V1” to “Vnv” as the dummy target, the respondent who has selected the dummy target has knowledge that can accurately determine the product to be included in the category “D1”. Alternatively, it can be regarded as a person who answers seriously.

  Then, since the product to be inspected selected by the person whose answer is reliable together with the dummy object may not be appropriate as a product belonging to the category “D1”, the product to be inspected reported by the evaluation device 100 is confirmed. This makes it possible to check whether or not the classification of the product into the category “D1” is correct.

  When the concept is “search word”, the inspection target acquisition unit 101 acquires items included in the first number of search results from the top of the search results of the search word as inspection targets, and the dummy target acquisition unit 102 Then, items that are not included in the search result of the search word are acquired as dummy targets.

  Here, in the text associated with the item to be inspected, the dummy target acquisition unit 102 extracts the second number of words having the highest appearance frequency, specifies the extracted word as a search word, and searches for a concept. Retrieve items included in the search result with the word specified as an excluded word as a dummy target.

  The first number and the second number are numbers that can be arbitrarily set by the administrator.

  For example, when it is desired to check whether or not the search algorithm is correct, an arbitrary word ("gold") is used as a search word, and ten products "P1" to "10" are ranked from the top of the search result of the search word "gold". “P10” is the inspection target. In addition, the dummy target acquisition unit 102 extracts, for example, ten words from the highest frequency of appearance in the product description or the product tag of the ten products “P1” to “P10”. When one of the extracted words is “necklace”, the dummy target acquisition unit 102 obtains a search result obtained by searching “necklace” as a search word and “gold” as an exclusion word. Then, the dummy target obtaining unit 102 obtains, as a dummy target, at least one of the products “Q1” to “Qnq” (nq: natural number) included in the search result. The presenting unit 103 presents to the checker a question for selecting a product that should not be included in the search result of the search word “gold” from options, and an option including a product to be inspected and a product to be dummy.

  The products “Q1” to “Qnq” are items that are included in the top of the search results when “necklace” that frequently appears in the search results of “gold” is used as a search word, but are not included in the search results of “gold” Item. In other words, by setting the items "Q1" to "Qnq" as the dummy targets, the respondents who have selected the dummy targets have knowledge of accurately determining the items to be included in the search results of the search word "gold", or , It can be regarded as a person who answers seriously.

  If a person who can trust the answer has a large number of times that the products “P1” to “P10” ranked higher in the search result with “gold” as the search word together with the dummy object are selected, the products “P1” to “P10” are selected. It turns out that the search result of “gold” is not appropriate. Therefore, it is possible to check whether or not the algorithm for calculating the score of the search is appropriate.

  In the above example, the item is a commodity, but the present invention is not limited to this, and the item includes all items obtained by a search using a search engine.

(4. Operation of evaluation apparatus of Embodiment 1)
The operation of the evaluation device 100 according to the present embodiment will be described. For example, when the power is turned on, the evaluation device 100 starts the evaluation processing illustrated in FIG.

  The inspection target acquisition unit 101 acquires a plurality of inspection targets set to be linked to a concept (step S101).

  For example, the inspection target acquisition unit 101 acquires, as the inspection target, the objects i1 to i5 specified by the administrator of the evaluation device 100 and set to be associated with the concept X.

  Next, the dummy target obtaining unit 102 obtains one or more dummy targets that are not set to be associated with the concept (Step S102).

  For example, the dummy target acquisition unit 102 acquires the target d1 specified by the administrator of the evaluation device 100 as a dummy target.

  The presentation unit 103 presents, to the plurality of checkers, a question for selecting an object that should not be linked to a concept from a plurality of inspection objects and one or more dummy objects (step S103).

  For example, the presenting unit 103 asks the checker terminals 200 of the checkers 1 to 5 a question 411 for asking an option that should not be linked to the concept X, and a page 410 including an option 412 of an inspection object i1 to i5 and a dummy object d1 (FIG. 4) is sent.

  The receiving unit 104 determines whether an answer including the selected target has been received from a plurality of checkers (Step S104). When the accepting unit 104 determines that an answer has been accepted from a plurality of checkers (Step S104; Yes), the extracting unit 105 extracts an answer including one or more dummy targets from the accepted answers (Step S105). On the other hand, when the receiving unit 104 determines that no answer has been received from the plurality of checkers (step S104; No), the receiving unit 104 stands by as it is.

  For example, when receiving the answer from the checker terminal 200 of the checkers 1 to 5, the receiving unit 104 registers the answer in the answer table 500a (FIG. 5) stored in the recording medium 14, and the extracting unit 105 determines the answer in FIG. With reference to the table 500a, the answers of the checker IDs "1", "2", "4", and "5" are extracted as the answers from which the dummy object d1 is selected. On the other hand, when the receiving unit 104 determines that the replies have not been received from the checker terminals 200 of all checkers that have requested the work, the reception unit 104 stands by as it is. Note that the reception unit 104 may proceed to step S105 after a predetermined time set by the administrator has elapsed, even if responses from all checkers have not been received.

  The report unit 106 reports the inspection target included in the extracted answer among the plurality of inspection targets (Step S106).

  For example, referring to the answer table 500a (FIG. 5), the report unit 106 specifies the inspection target i1, i2, i4 selected together with the dummy target, and the page 420 for notifying the specified i1, i2, i4. (FIG. 6) is displayed on the screen of the evaluation device 100.

  In step S106, the report unit 106 reports the number of replies extracted from among the replies received from the plurality of checkers and which includes the test target. The least test object may be reported as the object to be linked to the concept.

  Further, the processing in step S102 may be performed before the processing in step S101.

  According to the present embodiment, depending on whether or not to select a dummy target not set to be linked to a concept, whether the respondent has the knowledge to correctly answer the question, or It is possible to determine whether or not the user is working on a response. Also, it can be shown that the inspection target selected together with the dummy target may not be appropriate as a target linked to the concept. Conventionally, a question for obtaining information to be sought and a dummy question are prepared, and both are answered to determine whether or not to answer seriously. However, according to the present embodiment, it is possible to know whether the respondent has the knowledge to correctly answer or whether he / she is answering seriously simply by answering one question, and Since it is possible to know an inappropriate target as a target to be linked to, the answer can be efficiently evaluated.

(5. Functional configuration of evaluation device according to second embodiment)
The evaluation device according to the present embodiment presents a plurality of inspection objects and a plurality of dummy objects, and obtains a degree to which the inspection object should not be linked to a concept based on the obtained answer.

  The evaluation device 100 functionally includes an inspection target acquisition unit 101, a dummy target acquisition unit 102, a presentation unit 103, a reception unit 104, an extraction unit 105, and a report unit 106. In the present embodiment, the functions of the inspection target acquisition unit 101, the reception unit 104, and the extraction unit 105 have the same functions as those of the first embodiment, and thus description thereof will be omitted.

  The dummy target acquisition unit 102 acquires a plurality of dummy targets.

  For example, the dummy target acquisition unit 102 acquires, from the administrator of the evaluation device 100, the objects d1, d2, and d3 that are not associated with the concept X as dummy objects.

  As illustrated in FIG. 11, the presentation unit 103 asks a question 441 asking an option that should not be linked to the concept X, and an option 442 that sets the inspection targets i1 to i5 and the dummy objects d1 to d3 linked to the concept X as options. Is transmitted to the checker terminals 200 of the checkers 1 to 5 that have accepted the work.

  The receiving unit 104 receives a response from each checker terminal 200. The received answer is registered in the answer table 500b shown in FIG.

  The extracting unit 105 refers to the answer table 500b in FIG. 12 and, as an answer in which one of the dummy objects d1 to d3 is selected, the answer of the checker ID “1”, “2”, “4”, “5”. Extract.

  The report unit 106 determines the degree of accuracy indicating the degree to which the checker has made a correct selection, based on the number of the extracted answers that include the dummy object.

  Accuracy indicates the degree to which the checker has correctly selected an object that should not be associated with a concept.

  For example, if there are multiple dummy objects, the checker who can select all the dummy objects determines that he / she is seriously working on answering the question or has knowledge enough to answer the question correctly. it can. On the other hand, a checker who can select only a part of the dummy objects has a lower degree of seriousness or less knowledge of answering the question correctly than a checker who can select all the dummy objects. I can judge. Therefore, it can be considered that the greater the number of selected dummy objects that can be presented, the higher the accuracy, and the smaller the number, the lower the accuracy.

  For example, the reporting unit 106 refers to the answer table 500b in FIG. 12 and registers the number of times each checker selects a dummy target in the answer in the accuracy table 700a illustrated in FIG. In the accuracy table 700a, a checker ID for identifying the checker and the accuracy are registered in association with each other. The accuracy table 700a is stored in the recording medium 14 of the evaluation device 100. In the answer table 500b of FIG. 12, the checkers 1 and 2, from which all of the dummy objects d1 to d3 can be selected, have the highest accuracy. Further, since the checker 3 that has transmitted the answer not extracted by the extraction unit 105 cannot select any dummy object, “0” is registered as the accuracy of the checker 3.

  The report unit 106 determines an inappropriate score indicating the degree to which the test target should not be linked to the concept, for the test target included in the extracted answer, based on the accuracy.

  The inappropriate score is a score required for the inspection target selected together with the dummy target, and is a score indicating a degree to which the inspection target should not be linked to the concept. The higher the inappropriate score, the higher the degree of inadequacy of the test target as a target linked to the concept.

  For example, the inappropriate score of the inspection target is the accuracy obtained for the checker that has selected the inspection target. For example, when the reporting unit 106 obtains the accuracy of the checker 1 as “3” with reference to the accuracy table 700a of FIG. 13, the reporting unit 106 sets the inappropriate scores of the inspection targets i1 and i2 selected by the checker 1 to “3”. Is determined and registered in the inappropriate score table 800a shown in FIG. In the unsuitable score table 800a of FIG. 14, the inspection target, the unsuitable score obtained from the accuracy of each checker, and the integrated value obtained by integrating the unsuitable score by all checkers are registered in association with each other. . The inappropriate score table 800a is stored in the recording medium 14 of the evaluation device 100.

  The checker 2 has selected all the dummy objects d1 to d3 and thus has an accuracy of “3”. However, since the inspection object was not selected together with the dummy object in the answer, all the inappropriate scores of the inspection objects i1 to i5 are It is "0". The checker 3 does not select a dummy target in the answer, so the accuracy is “0”, and the inappropriate scores of the test targets i1 to i5 are all “0”. Since the checker 4 selects the dummy objects d1 and d2 in the answer, the accuracy is “2”, and the inappropriate score of the inspection object i2 selected together with the dummy objects d1 and d2 is “2”. The checker 5 has selected the dummy target d1 in the answer, so the accuracy is “1”, and the inappropriate score of the test target i4 selected together with the dummy target d1 is “1”.

  The report unit 106 reports the inappropriate score obtained from the extracted answers among the answers received from the plurality of checkers, and a value obtained by integrating the inappropriate scores.

  For example, the reporting unit 106 calculates a value obtained by integrating the inappropriate scores of the checkers, and registers the calculated value in the inappropriate score table 800a (FIG. 14) as the integrated value. Then, the report unit 106 displays an inappropriate score table 800a (FIG. 14) in which the inappropriate score and the integrated value are described on the screen of the evaluation device 100.

  The report unit 106 may report the number of accepted responses and the number of responses including the dummy target. In addition, the report unit 106 may report the number of checkers who have answered the question and the number of checkers that have selected dummy targets or the number of checkers that have not selected dummy targets.

  For example, the report unit 106 displays the number of accepted answers “5” and the number of answers including the dummy target “4” on the screen together with the inappropriate score table 800a (FIG. 14). Also, the report unit 106 did not select the number of checkers that answered the question “5” and the number of checkers that selected the dummy target “4” or the dummy target, together with the inappropriate score table 800a (FIG. 14). The number “1” of checkers is displayed on the screen.

  In addition, the reporting unit 106 reports the inspection target having the smallest value obtained by integrating the inappropriate scores as the target to be linked to the concept.

  For example, the report unit 106 refers to the inappropriate score table 800a in FIG. 14 and specifies the test target having the minimum integrated value as i3 and i5, and notifies the specified i3 and i5 as illustrated in FIG. Is displayed on the screen of the evaluation device 100.

(6. Operation of evaluation device of Embodiment 2)
The operation of the evaluation device 100 according to the present embodiment will be described. For example, when the power is turned on, the evaluation device 100 starts the evaluation processing illustrated in FIG. Steps S201, S204, and S205 in FIG. 15 are the same processes as steps S101, S104, and S105 in FIG. 10, and a description thereof will be omitted.

  When the inspection target is obtained in step S201, the dummy target obtaining unit 102 obtains a plurality of dummy targets (step S202).

  For example, the dummy target acquisition unit 102 acquires, from the administrator of the evaluation device 100, the objects d1, d2, and d3 that are not associated with the concept X as dummy objects.

  The presentation unit 103 presents, to the plurality of checkers, a question for selecting an object that should not be linked to a concept from a plurality of inspection objects and a plurality of dummy objects (step S203).

  For example, the presentation unit 103 asks the checker terminals 200 of the checkers 1 to 5 a question 441 for asking an option that should not be linked to the concept X, and a page 440 including an inspection target i1 to i5 and an option 442 of the dummy targets d1 to d3. (FIG. 11) is transmitted.

  In step S205, when the extraction unit 105 extracts a response including the dummy target from the response received by the reception unit 104, the report unit 106 determines the accuracy based on the number of the extracted responses including the dummy target. It is determined (step S206).

  For example, the reporting unit 106 refers to the answer table 500b in FIG. 12 and registers the number of times each checker selects a dummy target in the answer in the accuracy table 700a illustrated in FIG.

  Next, the report unit 106 determines an inappropriate score for the inspection target included in the extracted answer based on the accuracy (Step S207).

  For example, the reporting unit 106 obtains the accuracy of each checker with reference to the accuracy table 700a of FIG. 13, and uses the obtained accuracy as an inappropriate score of the inspection target selected by each checker, as shown in FIG. It is registered in the appropriate score table 800a.

  The report unit 106 reports a value obtained by integrating the inappropriate scores obtained from the extracted answers among the answers received from the plurality of checkers (Step S208).

  For example, the reporting unit 106 calculates the integrated value of the inappropriate score, registers the integrated value in the inappropriate score table 800a (FIG. 14), and displays the inappropriate score table 800a in which the integrated value is registered on the screen of the evaluation device 100. indicate.

  According to the present embodiment, a plurality of dummy objects are presented, and it is determined that the greater the number of selected dummy objects, the more accurate the answer of the respondent. Thereby, based on the degree of the accuracy of the answer, it is possible to know the degree to which the inspection target is inappropriate as a target to be linked to the concept.

  In the first and second embodiments, instead of displaying the report content on the screen, the report unit 106 may output the report content to a file that can be processed by statistical processing software or spreadsheet software.

  The present invention is capable of various embodiments and modifications without departing from the broad spirit and scope of the invention. Further, the above-described embodiment is for describing the present invention, and does not limit the scope of the present invention. That is, the scope of the present invention is shown not by the embodiments but by the claims. Various modifications made within the scope of the claims and the scope of the invention equivalent thereto are considered to be within the scope of the present invention.

11 CPU
12 ROM
13 RAM
14 recording medium 15 output unit 16 communication unit 17 operation unit 18 bus 100 evaluation device 101 inspection object acquisition unit 102 dummy object acquisition unit 103 presentation unit 104 reception unit 105 extraction unit 106 report unit 200 checker terminal 300 network 410, 420, 430, 440 pages

Claims (14)

An inspection target acquisition unit that acquires a plurality of inspection targets set to be linked to a concept,
A dummy object acquisition unit that acquires one or more dummy objects not set to be associated with the concept,
A plurality of checkers, the plurality of test objects, the one or more dummy objects, from the presentation unit for presenting a question to select an object that should not be associated with the concept,
From the plurality of checkers, a receiving unit that receives an answer composed of the selected target,
An extracting unit configured to extract an answer including the one or more dummy targets among the accepted answers;
Of the plurality of inspection targets, a reporting unit that reports an inspection target included in the extracted answer,
An evaluation device comprising: The dummy object acquisition unit acquires one of the dummy objects,
The evaluation device according to claim 1, wherein the presentation unit presents a question for selecting a plurality of targets that should not be associated with the concept. The dummy target obtaining unit obtains a plurality of the dummy targets,
The reporting unit,
Based on the number of the dummy object is included in the extracted answer, to determine the accuracy indicating the degree to which the selection by the checker was accurate,
Based on the accuracy, for the test target included in the extracted answer, determine an inappropriate score indicating the degree to which the test target should not be associated with the concept,
The evaluation apparatus according to claim 1, wherein a value obtained by integrating the inappropriate scores obtained from the extracted answers is reported from among the answers received from the plurality of checkers. The evaluation device according to claim 3, wherein the reporting unit reports an inspection target having a minimum value obtained by integrating the inappropriate scores as a target to be associated with the concept. 5. The report unit according to claim 1, wherein the report unit reports the number of replies that are the extracted replies and that include the inspection target, among the replies received from the plurality of checkers. 6. The evaluation device according to claim 1. The said report part reports the said test object with the smallest number of the said extracted answer among the answers received from the said several checkers as an object which should be linked | linked with the said concept. 6. The evaluation device according to any one of 5. The concept is a product,
The inspection target acquisition unit acquires, as the inspection target, a category represented by a hierarchical structure, the category being associated with a node in the same hierarchy as the category to which the product belongs.
The dummy target obtaining unit obtains, as the dummy target, a category that is a node in the same hierarchy as a parent node of a node to which the category to which the product belongs is associated and that is associated with a node different from the parent node. The evaluation device according to claim 1, wherein: The concept is a product,
The inspection target acquisition unit acquires, as the inspection target, a tag set to be linked to the product by a shop,
The evaluation device according to any one of claims 1 to 6, wherein the dummy target obtaining unit obtains, as the dummy target, a tag that is not set to be linked to the product by another shop. . The concept is a tag,
The inspection target acquisition unit acquires, as the inspection target, a product for which a setting for linking to the tag has been performed by a shop,
The evaluation device according to any one of claims 1 to 6, wherein the dummy target acquisition unit acquires, as the dummy target, a product that is not set to be associated with the tag by another shop. . The concept is a category represented by a hierarchical structure,
The inspection target acquisition unit acquires a product belonging to the category as the inspection target,
The dummy object acquiring unit acquires, as the dummy object, a product belonging to a category that is a node in the same hierarchy as the parent node of the category and that is associated with a node different from the parent node. The evaluation device according to any one of claims 1 to 6. The concept is a search word,
The inspection target acquisition unit acquires items included in the first number of search results from the top of the search results of the search word as the inspection target,
The evaluation device according to claim 1, wherein the dummy target acquisition unit acquires an item that is not included in the search result of the search word as the dummy target. The dummy target acquisition unit,
In a text associated with the item to be inspected, a second number of words having the highest appearance frequency is extracted,
The evaluation apparatus according to claim 11, wherein the extracted word is specified as a search word, and an item included in a search result in which the search word of the concept is specified as an exclusion word is acquired as the dummy target. . An evaluation method executed by the evaluation device,
The evaluation device, an inspection target acquisition step of acquiring a plurality of inspection targets set to be associated with a concept,
The evaluation device, a dummy target obtaining step of obtaining one or more dummy targets not set to be associated with the concept;
A presentation step of presenting a question that causes the plurality of checkers to select a target that should not be linked to the concept, from the plurality of test targets and the one or more dummy targets,
The evaluation device, from the plurality of checkers, a receiving step of receiving an answer consisting of the selected target,
An extracting step in which the evaluation device extracts an answer including the one or more dummy targets among the accepted answers;
A report step in which the evaluation device reports an inspection object included in the extracted answer, among the plurality of inspection objects,
An evaluation method, comprising: Computer
An inspection target acquisition unit that acquires a plurality of inspection targets set to be linked to a concept,
A dummy object acquisition unit that acquires one or more dummy objects not set to be associated with the concept,
A plurality of checkers, a presentation unit that presents a question for selecting an object that should not be linked to the concept from the plurality of inspection objects and the one or more dummy objects;
From the plurality of checkers, a receiving unit that receives an answer consisting of the selected target,
An extracting unit configured to extract an answer including the one or more dummy targets among the accepted answers;
Of the plurality of inspection targets, a reporting unit that reports the inspection targets included in the extracted answer,
A program characterized by functioning as a program.

Images