JP2020115292A - Evaluation support program, evaluation support method, and information processing device - Google Patents

Abstract

To provide an evaluation support program, an evaluation support method, and an information processing device capable of supporting validity of a determination result derived through a machine learning.SOLUTION: A prediction result screen 800 is a screen to display "hardware" categorized with prediction types by corresponding them to input item names "AP service." The input item names correspond to input data of objects to be determined. The prediction types correspond to categories in which the input data of the objects to be determined are determined using a learning model MD. On the prediction result screen 800, an item name "P server" and "P server" are displayed by corresponding to an item name "AP service." The item name "P server" and "P server" are two upper input data with larger similarity to the input data "AP service" of the object to be determined among learnt input data (item name) belonging to the category "hardware" and information forming basis of the prediction result derived using the learning model MD.SELECTED DRAWING: Figure 8

Description

The present invention relates to an evaluation support program, an evaluation support method, and an information processing device.

In recent years, AI (Artificial Intelligence) has been used to automate business decisions. Business decisions are not always made into rules, but are often made based on human experience and know-how. For such business judgment, for example, the input data of the past character string is expressed as the feature amount, and the supervised learning using the judgment result as the label is performed to automate the labeling (class classification) for the new input data. Is being done.

As the prior art, for example, the exceptional degree of the input data is determined, the learning model is selected from the model storage unit based on the determination result, and the data stored in the model execution history storage unit and the inter-model dependency relationship is used. There is one that updates the learned model.

JP, 10-074188, A

However, with the conventional technology, it is difficult to evaluate the validity of the judgment result obtained by the method based on machine learning. For example, a person may not be able to evaluate the validity of the judgment result unless the basis for judging the label by machine learning is shown.

In one aspect, the present invention aims to assist in evaluating the validity of a judgment result obtained by machine learning.

In one embodiment, a learning model that determines the category to which the input data to be determined is determined using a learning model that determines the category to which the input data belongs from the feature amount of the input data, and the learning used when generating the learning model. From the already input data, other input data that is different from the determination target input data belonging to the determined category is extracted, and is associated with the determination target input data, and the determined category and the extracted An evaluation support program that outputs other input data is provided.

According to one aspect of the present invention, it is possible to support the evaluation of the validity of the judgment result obtained by machine learning.

FIG. 1 is an explanatory diagram showing an example of the evaluation support method according to the embodiment. FIG. 2 is an explanatory diagram showing a system configuration example of the information processing system 200. FIG. 3 is a block diagram showing a hardware configuration example of the information processing apparatus 101. FIG. 4 is an explanatory diagram showing an example of the storage contents of the learning data DB 220. FIG. 5 is a block diagram showing a functional configuration example of the information processing apparatus 101. FIG. 6 is an explanatory diagram showing an example of calculating the feature amount vector. FIG. 7 is an explanatory diagram showing an example of the stored contents of the similarity table 700. FIG. 8 is an explanatory diagram (No. 1) showing a screen example of the prediction result screen. FIG. 9 is an explanatory view (No. 2) showing an example of the prediction result screen. FIG. 10 is an explanatory diagram (part 3) showing an example of the prediction result screen. FIG. 11 is a flowchart showing an example of a learning processing procedure of the information processing apparatus 101. FIG. 12 is a flowchart (No. 1) showing an example of a first prediction processing procedure of the information processing apparatus 101. FIG. 13 is a flowchart (No. 2) showing an example of the first prediction processing procedure of the information processing apparatus 101. FIG. 14 is a flowchart (No. 1) illustrating an example of the second prediction processing procedure of the information processing apparatus 101. FIG. 15 is a flowchart (No. 2) showing an example of the second prediction processing procedure of the information processing apparatus 101.

Embodiments of an evaluation support program, an evaluation support method, and an information processing apparatus according to the present invention will be described in detail below with reference to the drawings.

(Embodiment)
FIG. 1 is an explanatory diagram showing an example of the evaluation support method according to the embodiment. In FIG. 1, the information processing apparatus 101 is a computer that supports evaluation of the validity of a determination result obtained by a method based on machine learning. Machine learning is learning that creates a learning model (prediction model) from various data and predicts the result.

Here, there are an increasing number of cases where AI is used to automate business decisions. As an example of the work determination, there is a case where the accounting staff assigns the expense code Y1 to the purchase request product X1, and the purchase reception staff sets the staff Y2 to the purchase case X2. .. With regard to such business judgment, by labeling past input data as a feature amount and performing supervised learning using the judgment result as a label, labeling of new input data can be automated. In the above-mentioned example, the purchase request product X1 and the purchase item X2 correspond to “input data”, and the expense code Y1 and the person in charge Y2 correspond to “label”.

It is difficult to get a 100% correct answer rate by automation by machine learning, but it becomes possible to apply it to work by going through the task of human beings to confirm whether the label obtained by judgment is correct or not. It can be expected that work efficiency will be improved compared to the case where the judgment is made from scratch. When a person confirms that the label is wrong, it is necessary to correct the label.

However, it may be difficult for a human to evaluate the validity of the judgment result unless the information that is the basis for judging the label by the machine learning is presented. If humans cannot properly evaluate the validity of the judgment result, the label may be erroneously corrected or the label error may be overlooked.

As a machine learning algorithm capable of presenting the basis of the judgment result, class classification using a decision tree can be considered. In the class classification using the decision tree, the learning model can be expressed as a tree structure in which each node has a rule (logical expression), so that the basis can be explained as a sequence of rules.

However, rules are created only by machines for classification, and each rule is often not easy for humans to understand. In addition, since a maximum of tree depth rules are applied to certain input data, the number of rules that must be understood in order to understand the grounds becomes enormous.

Further, since the method itself using the decision tree is one implementation of the machine learning algorithm, if the accuracy is not high in this method, the underlying rule itself is not reliable and cannot be applied. Therefore, there is a demand for a technique that enables human evaluation of the validity of the judgment result while ensuring the accuracy of the judgment result obtained by machine learning.

Therefore, in the present embodiment, when outputting a category obtained by machine learning to data input as a determination target, by showing another example that is determined to belong to the category, machine learning is performed. An evaluation support method for supporting the evaluation of the validity of the judgment result obtained by will be described. Hereinafter, a processing example of the information processing apparatus 101 will be described.

(1) The information processing apparatus 101 uses the learning model 110 to determine the category to which the input data to be determined belongs. Here, the learning model 110 is a prediction model that determines the category to which the input data belongs from the feature amount of the input data. The learning model 110 is represented by, for example, mathematical formulas or tree structure data of decision trees.

The input data is data that is input as a determination target of which category it belongs to, and is, for example, the purchase request product X1 or the purchase project X2 described above. The category is a type for classifying the input data, and is, for example, the expense code Y1 or the person in charge Y2 described above. The characteristic amount of the input data is a numerical value of the characteristic of the input data. For example, the feature amount of the input data is represented by an N-gram feature amount vector.

In the example of FIG. 1, it is assumed that the input data to be determined is “input data X”, and “category Y” is determined as the category to which the input data X belongs.

(2) The information processing apparatus 101 extracts, from the learned data 120, input data belonging to the determined category and different from the input data to be determined. Here, the learned data 120 includes the learned input data used when the learning model 110 is generated.

Specifically, the learned data 120 is a set of teacher data used for generating the learning model 110. The teacher data is data about “example” and “answer” given in supervised learning, and is a pair of input data (example) and a category (answer) to which the input data belongs.

In the example of FIG. 1, it is assumed that “input data X′” that belongs to category Y and is different from the input data X to be determined is extracted from the learned data 120.

(3) The information processing apparatus 101 outputs the determined category and the extracted input data in association with the input data to be determined. Specifically, for example, the information processing apparatus 101 outputs the determined category Y and the extracted input data X′ in association with the determination target input data X.

As described above, according to the information processing apparatus 101, when outputting a category obtained by machine learning to the input data to be determined, another case (a learned input is determined to belong to the category) is output. Data) can be shown. As a result, it is possible to present information that serves as the basis for determining the category, and it is possible to support the evaluation of the validity of the determination result obtained by machine learning.

In the example of FIG. 1, the category Y and the input data X′ are output in association with the input data X to be determined. Thus, if the user can determine that the input data X and the input data X′ are not of the same type without knowing the correct categories of the input data X and X′, for example, the determination on the input data X is made. It can be noticed that the result (category Y) may be incorrect. As a result, the user can check the determination result carefully, and it is possible to prevent the mistake of the label (category) from being overlooked. In the following description, "category" may be referred to as "label".

(Example of system configuration of information processing system 200)
Next, a system configuration example of the information processing system 200 including the information processing apparatus 101 shown in FIG. 1 will be described. The information processing system 200 is applied to, for example, a computer system for automating various business decisions in a company.

FIG. 2 is an explanatory diagram showing a system configuration example of the information processing system 200. In FIG. 2, the information processing system 200 includes an information processing device 101 and a client device 201. In the information processing system 200, the information processing apparatus 101 and the client apparatus 201 are connected via a wired or wireless network 210. The network 210 is, for example, a LAN (Local Area Network), a WAN (Wide Area Network), or the Internet.

Here, the information processing apparatus 101 has a learning data DB (Data base) 220. The information processing device 101 is, for example, a server. The storage contents of the learning data DB 220 will be described later with reference to FIG.

The client device 201 is a computer used by a user of the information processing system 200. The user of the information processing system 200 is, for example, a person in charge of accounting in a company, a person in charge of purchasing, or the like who makes various business decisions. The client device 201 is, for example, a PC (Personal Computer), a tablet PC, or the like.

Although the information processing apparatus 101 and the client apparatus 201 are separately provided in the above description, the present invention is not limited to this. For example, the information processing device 101 may be realized by the client device 201. Further, in the example of FIG. 2, only one client device 201 is displayed, but the present invention is not limited to this. For example, the information processing system 200 may include the client device 201 for each user.

(Example of hardware configuration of information processing apparatus 101)
FIG. 3 is a block diagram showing a hardware configuration example of the information processing apparatus 101. 3, the information processing apparatus 101 includes a processor 301, a memory 302, a disk drive 303, a disk 304, a communication I/F (Interface) 305, a portable recording medium I/F 306, and a portable recording medium. And 307. In addition, each component is connected by a bus 300.

Here, the processor 301 controls the entire information processing apparatus 101. The processor 301 may have a plurality of cores. The processor 301 is, for example, a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit).

The memory 302 includes, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), and a flash ROM. Specifically, for example, the flash ROM stores an OS (Operating System) program, the ROM stores an application program, and the RAM is used as a work area of the processor 301. The program stored in the memory 302 causes the processor 301 to execute the coded processing by being loaded into the processor 301.

The disk drive 303 controls reading/writing of data with respect to the disk 304 under the control of the processor 301. The disk 304 stores the data written under the control of the disk drive 303. Examples of the disk 304 include a magnetic disk and an optical disk.

The communication I/F 305 is connected to the network 210 via a communication line, and is connected to an external computer (for example, the client device 201 shown in FIG. 2) via the network 210. Then, the communication I/F 305 administers an interface between the network 210 and the inside of the apparatus, and controls the input/output of data from an external computer. For the communication I/F 305, for example, a modem or a LAN adapter can be adopted.

The portable recording medium I/F 306 controls reading/writing of data with respect to the portable recording medium 307 under the control of the processor 301. The portable recording medium 307 stores the data written under the control of the portable recording medium I/F 306. Examples of the portable recording medium 307 include a CD (Compact Disc)-ROM, a DVD (Digital Versatile Disk), and a USB (Universal Serial Bus) memory.

Note that the information processing apparatus 101 may include, for example, an SSD (Solid State Drive), an input device, a display, and the like, in addition to the above-described components. In addition, the information processing apparatus 101 may not include, for example, the disk drive 303, the disk 304, the portable recording medium I/F 306, and the portable recording medium 307 among the above-described components. Further, the client device 201 shown in FIG. 2 can also be realized by the same hardware configuration as the information processing device 101. However, the client device 201 has an input device, a display (for example, a display 810 shown in FIG. 8 described later), and the like, in addition to the above-described components.

(Memory contents of learning data DB 220)
Next, the storage content of the learning data DB 220 included in the information processing apparatus 101 will be described with reference to FIG. The learning data DB 220 is realized by a storage device such as the memory 302 and the disk 304 shown in FIG. 3, for example.

FIG. 4 is an explanatory diagram showing an example of the storage contents of the learning data DB 220. In FIG. 4, the learning data DB 220 has fields of product name and type, and stores learning data (for example, learning data 400-1 to 400-3) as records by setting information in each field.

Here, the product name is the name of the product and corresponds to input data (example) used for learning with a teacher. The type is a category to which a product name belongs, that is, a category to which a product having the product name belongs, and corresponds to a label (answer) used for supervised learning. For example, the learning data 400-1 includes the product name “P server” and the type “hardware”, and indicates that the type (category) to which the “P server” belongs is “hardware”.

(Example of functional configuration of information processing apparatus 101)
FIG. 5 is a block diagram showing a functional configuration example of the information processing apparatus 101. In FIG. 5, the information processing apparatus 101 includes an acquisition unit 501, a learning processing unit 502, a reception unit 503, a prediction processing unit 504, an extraction unit 505, an output unit 506, an updating unit 507, and a storage unit 510. And, including. Specifically, for example, the acquisition unit 501 to the update unit 507 cause the processor 301 to execute a program stored in a storage device such as the memory 302, the disk 304, and the portable recording medium 307 illustrated in FIG. Alternatively, the function is realized by the communication I/F 305. The processing result of each functional unit is stored in a storage device such as the memory 302 or the disk 304, for example. The storage unit 510 is realized by a storage device such as the memory 302 and the disk 304, for example.

The acquisition unit 501 acquires learning data. Here, the learning data is information used for machine learning (learning with a teacher), and includes input data and a category to which the input data belongs. For example, the learning data is a set of a product name (input data) and a type (category) to which the product name belongs.

Specifically, for example, the acquisition unit 501 acquires the received learning data (product name, type) by receiving the learning data (product name, type) from the client device 201. Further, the acquisition unit 501 may acquire the learning data (product name, type) by an operation input of the user using an input device (not shown) of the information processing device 101.

The acquired learning data (product name, type) is stored in the learning data DB 220 shown in FIG. 4, for example.

The learning processing unit 502 generates a learning model MD based on the acquired learning data. Here, the learning model MD is a prediction model that determines the category to which the input data belongs from the feature amount of the input data. That is, the learning model MD labels the input data (multiclass classification). The learning model MD may be expressed by a mathematical expression, or may be expressed by tree structure data of a decision tree. The learning model 110 shown in FIG. 1 corresponds to, for example, the learning model MD.

Specifically, for example, first, the learning processing unit 502 acquires a learning model MD that is a base. The learning model MD as a base is, for example, created in advance and stored in a storage device such as the memory 302 or the disk 304. Next, the learning processing unit 502 acquires learning data from the learning data DB 220. Then, the learning processing unit 502 calculates the feature quantity vector of the product name (input data) of the acquired learning data.

Next, the learning processing unit 502 represents the acquired learning data as a set of a feature amount vector and a label. Then, the learning processing unit 502 stores the set of the feature amount vector and the label in the input information IN_D. At this time, the learning processing unit 502 stores the set of the feature amount vector and the label of each product name (input data) in the input information IN_D in association with each product name (input data), for example.

Here, an example of calculating the feature amount vector of the product name (input data) will be described with reference to FIG. Here, a case where a Tri-gram feature amount vector is calculated as the feature amount vector of the product name (input data) will be described as an example.

FIG. 6 is an explanatory diagram showing an example of calculating the feature amount vector. In FIG. 6, as examples of product names (input data), “P server”, “Q software”, and “P server” are shown. Here, it is assumed that there are three product names (input data) in the learning data DB 220: “P server”, “Q software”, and “P server”.

In this case, the learning processing unit 502 calculates the feature amount vector of each product name (input data) by taking “0” or “1” depending on the presence or absence of the partial character string in each product name (input data). To do. Each partial character string is a product name (input data) divided into three character segments. However, $ indicates a blank character.

Taking "P server" as an example, "P$P, $P server, P server, server, server $, server $$, $$Q, $Q software, Q software, software, software $, server $" A feature amount vector is calculated according to the presence or absence of each partial character string of "$, $$P, $P server, P server, server, overbar, bar $, -$$". Note that some of the partial character strings are omitted in FIG.

For example, the partial character string “$$P” is included in “P server”. Therefore, the value corresponding to the partial character string “$$P” in the feature amount vector of “P server” is “1”. In addition, the partial character string “$$Q” is not included in “P server”. Therefore, the value corresponding to the partial character string “$$Q” in the feature value vector of “P server” is “0”.

In this way, by taking "0" or "1" depending on the presence or absence of each partial character string, the feature amount vector "1,1,1,1,1,1,0," of the "P server" is obtained. ..." can be calculated. Similarly, the feature quantity vector “0,0,0,0,0,0,1,...” of “Q server” and the feature quantity vector “1,1,1,1,0,1” of “P server”. , 0,... ”can be calculated.

Further, the learning processing unit 502 may express the feature amount vector of each product name (input data) as a sparse vector. For example, the feature quantity vector of each product name (input data) tends to have more “0” elements than “1” elements. Therefore, when the feature quantity vectors of all product names (input data) are represented by a matrix (see FIG. 6 ), the feature quantity vector of each product name (input data) is defined as what row and column of “1” is. It may be expressed as information.

For example, the feature amount vector of "P server" may be represented by a list structure of "(1,1), (1,2), (1,3),..., (1,6)". .. As a result, compared to the case where all the elements of the feature amount vector of each item name (input data) are stored, it is possible to suppress the memory used when storing the feature amount vector of each item name (input data).

Returning to the description of FIG. 5, the learning processing unit 502 updates the learning model MD by performing supervised learning based on the set of the feature amount vector and the label (teacher data) stored in the input information IN_D. More specifically, for example, the learning processing unit 502 adjusts the parameters in the mathematical expression by performing multiple regression analysis, logistic regression, or the like, and updates (corrects, adds, or deletes) the rules of each node of the tree structure data. Etc.) and the learning model MD is updated.

The receiving unit 503 receives input data to be determined. Here, the input data to be judged is the data to be judged to which category (label) it belongs. The input data to be judged is, for example, a product name. The category is, for example, the type to which the product name belongs.

Specifically, for example, the receiving unit 503 receives the input data (product name) to be determined from the client device 201, and thus receives the input data (product name) to be received. Further, the acquisition unit 501 may accept the input data (product name) to be determined by a user's operation input using an input device (not shown) of the information processing apparatus 101.

The prediction processing unit 504 determines the category to which the input data to be determined belongs, using the learning model MD. Specifically, for example, the prediction processing unit 504 calculates a feature amount vector of the received determination target input data. More specifically, for example, the prediction processing unit 504 calculates a Tri-gram feature amount vector as the feature amount vector of the input data to be determined.

As an example, the input data to be determined is “AP service”. In this case, the prediction processing unit 504 determines whether or not each partial character string “$$P, $P server, P server, server, server $, server $$, $$Q,..., —$” exists. According to the above, the feature amount vector of the input data “AP service” to be determined is calculated. The partial character string is a partial character string of the input data (P server, Q software, P server) that has been learned as described above.

For example, the partial character string “$$P” is not included in the “AP service”. Therefore, the value corresponding to the partial character string “$$P” in the feature amount vector of “AP service” is “0”. In addition, the partial character string “P server” is included in the “AP service”. Therefore, the value corresponding to the partial character string “P server” in the feature amount vector of “AP service” is “1”. In this way, by taking “0” or “1” depending on the presence or absence of each partial character string, the feature quantity vector “0,0,1,0,0 of the input data “AP service” to be judged is obtained. , 0, 0,...” can be calculated.

Then, the prediction processing unit 504 uses the learning model MD updated by the learning processing unit 502 to predict the label for the calculated feature amount vector of the input data to be determined. That is, the prediction processing unit 504 determines the category (label) to which the input data to be determined belongs by inputting the feature amount vector of the input data (product name) to be determined to the learning model MD.

In the following description, the category to which the determination target input data belongs may be referred to as “category #”.

The extraction unit 505 extracts, from the learned input data used when generating the learning model MD, other input data belonging to the determined category # and different from the determination target input data. Specifically, for example, the extraction unit 505 may randomly extract K other input data from the input data belonging to the category # of the learned input data. K can be set arbitrarily, and is set to a value of about 1 to 10, for example.

The extraction unit 505 may also calculate the degree of similarity between the input data to be determined and the learned input data. Then, the extraction unit 505 may extract other input data belonging to the category # from the learned input data based on the calculated similarity.

Here, the degree of similarity is an index value indicating the degree of similarity between input data. As the similarity, for example, the cosine similarity between the feature amount vector of the input data to be determined and the feature amount vector of the learned input data can be used. The cosine similarity is used to evaluate the degree of similarity between data based on the closeness (angle) of the vector directions of the data.

More specifically, for example, the extraction unit 505 calculates the number a in which the values of both input data are 1 among all the elements of the feature amount vector. Further, the extraction unit 505 calculates the number b in which the value of the input data of any one of all the elements of the feature amount vector is 1. Then, the extraction unit 505 divides the calculated number a by the number b to calculate the cosine similarity (a/b) between the feature amount vector of the input data to be determined and the feature amount vector of the learned input data. calculate. In this case, the maximum value of the similarity is "1" and the minimum value is "0".

The feature amount vector of the learned input data is specified from the input information IN_D, for example. When the feature amount vector of the learned input data is represented as a sparse vector, the extraction unit 505 restores the sparse vector to the feature amount vector, and then extracts the feature amount vector of the input data to be determined. , The cosine similarity with the feature amount vector of the learned input data is calculated.

The calculated similarity is stored in the similarity table 700 as shown in FIG. 7, for example. The similarity table 700 is realized by a storage device such as the memory 302 and the disk 304, for example. Here, the storage contents of the similarity table 700 will be described.

FIG. 7 is an explanatory diagram showing an example of the stored contents of the similarity table 700. In FIG. 7, the similarity table 700 has fields of item name, type, cosine similarity and similarity order, and by setting information in each field, similarity information (for example, similarity information 700-1 to 700-700). -3) is stored as a record.

Here, the product name is the learned input data. The type is a category to which the product name (learned input data) belongs. The cosine similarity is the cosine similarity between the feature amount vector of the input data to be judged and the feature amount vector of the learned input data. In the example of FIG. 7, the input data to be determined is the product name “A server”. The similarity rank is a rank when the learned input data are arranged in descending order of the cosine similarity with the input data to be determined.

For example, the similarity information 700-1 indicates the type “hardware” of the learned input data “P server”, the cosine similarity “0.500”, and the similarity order “1”.

Returning to the description of FIG. 5, the extraction unit 505, for example, from the learned input data, among the input data belonging to the determined category #, the top N pieces (N: natural number) having a high similarity to the input data to be determined. Other input data may be extracted. N can be set arbitrarily, and is set to a value of about 1 to 5, for example.

Specifically, for example, the extraction unit 505 refers to the similarity table 700 illustrated in FIG. 7 and determines from the learned input data (product name) as the determination target input data among the input data belonging to the category #. The upper N other input data (product names) having a large cosine similarity of are extracted. For example, when N is “N=1”, the extraction unit 505 extracts, from the input data belonging to the category #, other input data having the maximum cosine similarity with the input data to be determined.

The output unit 506 outputs the determined category # and the other extracted input data in association with the input data to be determined. Further, the output unit 506 may output the similarity between the other extracted input data and the determination target input data in association with the determination target input data.

The output format of the output unit 506 includes, for example, transmission by the communication I/F 305 to another computer (for example, the client device 201), display on a display not shown, print output to a printer not shown, and the like.

Specifically, for example, it is assumed that the input data (product name) to be determined is received from the client device 201. In this case, the output unit 506 may display the prediction result screen on the client device 201. Here, the prediction result screen is a screen that displays the category # judged by the prediction processing unit 504 and other input data extracted by the extraction unit 505 in association with the input data to be judged.

A screen example of the prediction result screen will be described later with reference to FIG.

Further, when the similarity between the other extracted input data and the determination target input data is equal to or less than the threshold value α, the output unit 506 outputs the predetermined alert AL in association with the determination target input data. You may decide. The threshold value α can be set arbitrarily. For example, when the similarity is the cosine similarity (0 or more and 1 or less), the threshold value α is set to a value of about 0.3.

Specifically, for example, the output unit 506 refers to the similarity table 700, and among the input data belonging to the category #, the cosine similarity of the other input data having the maximum cosine similarity to the input data to be determined. Identify the degree. Then, when the identified cosine similarity is equal to or lower than the threshold value α, the output unit 506 outputs a predetermined alert AL in association with the input data to be determined.

The alert AL conveys that the determination result of the category to which the input data to be determined belongs may be incorrect, and calls attention. For example, the alert AL may be a warning message or a warning image. That is, the lower the degree of similarity of the other input data having the highest degree of similarity with the input data to be determined, the higher the possibility that the determination result will be erroneous, and the user is warned.

An output example of the alert AL will be described later with reference to FIG.

Further, the output unit 506 does not include other extracted input data in the upper M (M: natural number) input data having a high degree of similarity with the input data to be determined among the learned input data. In this case, a predetermined alert AL may be further output in association with the determination target data. M is set to a value of about 1 to 5, for example.

Specifically, for example, the output unit 506 refers to the similarity table 700 and, among the input data belonging to the category #, the similarity order of the other input data having the maximum cosine similarity to the input data to be determined. Specify. Then, when the identified similarity order is equal to or lower than the threshold β (where β=M), the output unit 506 further outputs a predetermined alert AL in association with the input data to be determined.

That is, the lower the similarity rank of the other input data having the highest similarity to the input data to be determined, the higher the possibility that the determination result is incorrect, and the user is warned. The alert AL that is output when the similarity rank is less than or equal to the threshold value β is, for example, a warning message such as “Please note that the similarity rank is low”.

Further, the receiving unit 503 may receive correctness information indicating whether or not the category # is correct as a result of the category # determined by the prediction processing unit 504 being output in association with the input data to be determined. .. Specifically, for example, the reception unit 503 receives the correct/incorrect information by receiving the correct/incorrect information from the client device 201. Further, the acquisition unit 501 may accept the correctness information by a user's operation input using an input device (not shown) of the information processing device 101.

Then, the reception unit 503 may record the similarity between the other input data extracted and the input data to be determined in the storage unit 510 in association with the received correctness information. The other input data is, for example, other input data having the maximum degree of similarity with the input data to be determined among the input data belonging to the category #.

The updating unit 507 updates the threshold value α based on the pair of similarity and correctness information recorded in the storage unit 510. Specifically, for example, the updating unit 507 updates the threshold value α by performing supervised learning (machine learning) using the pair of similarity and correctness information recorded in the storage unit 510 as teacher data.

As an example, when the threshold α is “α=0.3”, the pair of similarity and correctness information is <0.144, false>, <0.188, correct>, <0.8, correct>. Is recorded. In this case, the updating unit 507 performs supervised learning using each pair of <0.144, erroneous>, <0.188, correct>, <0.8, correct> as teacher data, and, for example, the threshold α Is changed to “α=0.18”.

(Screen example of prediction result screen)
Next, screen examples of the prediction result screen displayed on the client device 201 will be described with reference to FIGS. 8 to 10. The prediction result screen is displayed on the display 810 of the client device 201, for example, under the control of the information processing device 101, in accordance with the input data of the determination target received from the client device 201.

The display 810 is a display device that displays data such as a document, an image, and functional information in addition to a cursor, an icon, or a tool box. As the display 810, for example, a liquid crystal display, an organic EL (Electroluminescence) display, or the like can be adopted.

FIG. 8 is an explanatory diagram (No. 1) showing a screen example of the prediction result screen. In FIG. 8, the prediction result screen 800 is a screen that displays the prediction type “hardware” of the input product name “AP service” in association with the input product name “AP service”. Here, the input product name corresponds to the input data to be judged. The prediction type corresponds to the category determined to belong to the input data to be determined.

On the prediction result screen 800, the product name “P server” and the similarity “0.500” are displayed in association with the input product name “AP service”. Further, on the prediction result screen 800, the product name “P server” and the similarity “0.154” are displayed in association with the input product name “AP service”.

Here, the product names “P server” and “P server” are the top two of the learned input data (product name) belonging to the category “hardware” that have the highest similarity to the input data “AP service” to be determined. Input data. The similarities “0.500” and “0.154” are cosine similarities between the input data “AP service” to be judged and the product names “P server” and “P server”.

The judgment result (prediction type, prediction basis) in the prediction result screen 800 is input by the user's operation using an input device (not shown) of the client device 201 to input the input data to be judged in the input box 801. It is displayed by selecting the determination button 802.

According to the prediction result screen 800, when outputting the category “hardware” obtained by machine learning to the input data “AP service” to be determined, other cases (P server, P Server). As a result, it is possible to present information that serves as a basis for determining the category and support the evaluation of the validity of the determination result obtained by the machine learning.

For example, if the user finds that the input product name “AP service” is of a type different from the product name “P server” or the product name “P server”, the predicted type “hardware” may be incorrect. You can notice that.

FIG. 9 is an explanatory diagram (part 2) showing a screen example of the prediction result screen. In FIG. 9, the prediction result screen 900 is a screen for displaying the prediction type “hardware” of the input product name “AP service” in association with the input product name “AP service”.

Further, on the prediction result screen 900, the product name “P server”, the similarity “0.144”, and the similarity order “1” are displayed in association with the input product name “AP service”. Here, the product name “P server” is the input data having the maximum degree of similarity with the input data “AP service” to be determined, among the learned input data (product name) belonging to the category “hardware”.

The similarity “0.144” is the cosine similarity between the input data “AP service” to be determined and the product name “P server”. The similarity order “1” indicates that the product name “P server” has the highest degree of similarity with the input data “AP service” to be determined, among the learned input data. For the sake of explanation, a value different from the example shown in FIG. 8 is used as the similarity between the input data “AP service” to be judged and the product name “P server”.

In addition, an alert AL1 is displayed on the prediction result screen 900. The alert AL1 is to inform the user that the prediction type may be incorrect because the past data (P server) that is most similar to the input product name has a low similarity.

According to the prediction result screen 900, when the degree of similarity with the input product name does not satisfy a certain level, a warning is issued to the user and the judgment result obtained by machine learning may be incorrect. can do. This allows the user to carefully check the determination result obtained by machine learning, and prevent a mistake in the label (category) from being overlooked.

FIG. 10 is an explanatory diagram (part 3) showing an example of the prediction result screen. In FIG. 10, the prediction result screen 1000 is a screen for displaying the prediction type “hardware” of the input product name “AP service” in association with the input product name “AP service”.

On the prediction result screen 1000, the product name “P server”, the similarity “0.144”, the similarity order “1”, and the alert AL1 are displayed in association with the input product name “AP service”.

According to the prediction result screen 1000, when the degree of similarity with the input product name does not satisfy a certain level, a warning is issued to the user and the judgment result obtained by machine learning may be incorrect. can do. For example, the user can confirm whether or not the prediction type “hardware” is correct by referring to the product name “P server”, the similarity “0.144”, and the similarity order “1”.

When the correct button 1001 is selected by the user's operation input on the prediction result screen 1000, the correctness information indicating that the prediction type is correct can be transmitted from the client device 201 to the information processing device 101. Further, when the wrong button 1002 is selected by the user's operation input on the prediction result screen 1000, the correctness information indicating that the prediction type is wrong can be transmitted from the client device 201 to the information processing device 101.

As a result, it is possible to notify the information processing apparatus 101 of the correctness of the prediction type (category #) for the input product name (input data to be determined), and the threshold value α can be adjusted by feedback from the user. The correct button 1001 and the erroneous button 1002 may be included in the prediction result screens 800 and 900 shown in FIGS. 8 and 9.

(Various processing procedures of information processing apparatus 101)
Next, various processing procedures of the information processing apparatus 101 will be described. First, the learning processing procedure of the information processing apparatus 101 will be described with reference to FIG.

FIG. 11 is a flowchart showing an example of a learning processing procedure of the information processing apparatus 101. In the flowchart of FIG. 11, first, the information processing apparatus 101 acquires a learning model MD that is a base (step S1101). Next, the information processing apparatus 101 acquires unacquired learning data that has not been acquired from the learning data DB 220 (step S1102).

Then, the information processing apparatus 101 calculates the feature quantity vector of the product name (input data) of the acquired learning data (step S1103). Next, the information processing apparatus 101 stores the set of the calculated feature amount vector and the label (type) of the learning data in the input information IN_D in association with the product name (input data) (step S1104).

Then, the information processing apparatus 101 performs supervised learning based on the set of the feature amount vector and the label stored in the input information IN_D, and updates the learning model MD (step S1105). Next, the information processing apparatus 101 determines whether or not there is unacquired learning data that has not been acquired from the learning data DB 220 (step S1106).

Here, when there is unacquired learning data (step S1106: Yes), the information processing apparatus 101 returns to step S1102. On the other hand, when there is no unacquired learning data (step S1106: No), the information processing apparatus 101 ends the series of processes according to this flowchart.

Accordingly, it is possible to generate the learning model MD that determines the category to which the input data belongs from the feature amount of the input data.

Next, the first prediction processing procedure of the information processing apparatus 101 will be described with reference to FIGS. 12 and 13. The first prediction processing procedure outputs the category determined to belong to the input data, the learned input data most similar to the input data, the predetermined alert AL, etc. in association with the input data to be determined. Processing.

12 and 13 are flowcharts showing an example of the first prediction processing procedure of the information processing apparatus 101. In the flowchart of FIG. 12, first, the information processing apparatus 101 determines whether or not the input data to be determined has been received (step S1201). Here, the information processing apparatus 101 waits for receiving the input data to be determined (step S1201: No).

Then, when the information processing apparatus 101 receives the determination target input data (step S1201: Yes), the information processing apparatus 101 calculates the feature amount vector of the received determination target input data (step S1202). Next, the information processing apparatus 101 uses the updated learning model MD to determine the label (category) for the calculated feature amount vector of the input data to be determined (step S1203).

Then, the information processing apparatus 101 calculates the similarity between the input data to be determined and the learned input data (step S1204). Specifically, for example, the information processing apparatus 101 calculates the cosine similarity between the feature amount vector of the input data to be determined and each feature amount vector stored in the input information IN_D.

Next, the information processing apparatus 101 sets “i” to “i=1” (step S1205) and extracts learned input data having the i-th highest similarity from the learned input data (step S1206). .. Then, the information processing apparatus 101 determines whether or not the determined label and the extracted label of the learned input data match (step S1207).

Here, when the labels match (step S1207: Yes), the information processing apparatus 101 proceeds to step S1301 shown in FIG. On the other hand, when the labels do not match (step S1207: No), the information processing apparatus 101 determines whether or not there is unextracted learned input data that has not been extracted among the learned input data ( Step S1208).

If there is unextracted learned input data (step S1208: Yes), the information processing apparatus 101 increments “i” (step S1209) and returns to step S1206. On the other hand, if there is no unextracted input data that has been learned (step S1208: No), the information processing apparatus 101 outputs the prediction result screen (step S1210), and ends the series of processes according to this flowchart.

The prediction result screen output in step S1210 is a screen that displays the label (category #) determined in step S1203 in association with the input data to be determined.

In the flowchart of FIG. 13, first, the information processing apparatus 101 determines whether the similarity of the learned input data extracted in step S1206 is equal to or greater than the threshold value α (step S1301). Here, when the degree of similarity is less than the threshold value α (step S1301: No), the information processing apparatus 101 proceeds to step S1304.

On the other hand, when the similarity is equal to or higher than the threshold value α (step S1301: Yes), the information processing apparatus 101 determines whether “i” is equal to or lower than the threshold value β (step S1302). Here, when “i” is equal to or smaller than the threshold β (step S1302: Yes), the information processing apparatus 101 sets the extracted learned input data as a normal value (step S1303), and proceeds to step S1305. ..

On the other hand, when “i” is larger than the threshold β (step S1302: No), the information processing apparatus 101 sets the extracted learned input data as an abnormal value (step S1304). Then, the information processing apparatus 101 outputs the prediction result screen (step S1305) and ends the series of processes according to this flowchart.

The prediction result screen output in step S1305 is a screen that displays the label (category #) determined in step S1203 and the learned input data extracted in step S1206 in association with the input data to be determined. Is. When the learned input data is set as an abnormal value, the alert AL is displayed on the prediction result screen.

Thereby, when outputting the category obtained by machine learning to the input data to be determined, it is possible to show another example that is determined to belong to the category. In addition, if the similarity or the order of similarity with the input data to be judged does not meet a certain level, a warning (alert AL) is issued to the user, and the judgment result obtained by machine learning may be incorrect. Can be suggested.

Next, the second prediction processing procedure of the information processing apparatus 101 will be described with reference to FIGS. 14 and 15. The second prediction processing procedure outputs the category determined to belong to the input data and the top N learned input data having a high degree of similarity to the input data in association with the input data to be determined. Processing.

14 and 15 are flowcharts showing an example of the second prediction processing procedure of the information processing apparatus 101. In the flowchart of FIG. 14, first, the information processing apparatus 101 determines whether or not the input data to be determined has been received (step S1401). Here, the information processing apparatus 101 waits for receiving the input data to be determined (step S1401: No).

Then, when the information processing apparatus 101 receives the determination target input data (step S1401: Yes), the information processing apparatus 101 calculates a feature amount vector of the received determination target input data (step S1402). Next, the information processing apparatus 101 uses the updated learning model MD to determine the label (category) for the calculated feature amount vector of the input data to be determined (step S1403).

Then, the information processing apparatus 101 calculates the similarity between the determination target input data and the learned input data (step S1404), and proceeds to step S1501 shown in FIG. Specifically, for example, the information processing apparatus 101 calculates the cosine similarity between the feature amount vector of the input data to be determined and each feature amount vector stored in the input information IN_D.

In the flowchart of FIG. 15, first, the information processing apparatus 101 sets “i” to “i=1” (step S1501) and extracts learned input data having the i-th highest similarity from the learned input data. Yes (step S1502). Then, the information processing apparatus 101 determines whether or not the label determined in step S1403 matches the label of the extracted learned input data (step S1503).

Here, if the labels do not match (step S1503: No), the information processing apparatus 101 moves to step S1506. On the other hand, when the labels match (step S1503: Yes), the information processing apparatus 101 adds the extracted learned input data to the list (step S1504). Then, the information processing apparatus 101 determines whether or not the number of data items in the list is “N” (step S1505).

Here, when the number of data in the list becomes “N” (step S1505: Yes), the information processing apparatus 101 moves to step S1508. On the other hand, when the number of data in the list is not “N” (step S1505: No), the information processing apparatus 101 has unextracted learned input data that has not been extracted among the learned input data. It is determined whether or not (step S1506).

If there is unextracted learned input data (step S1506: Yes), the information processing apparatus 101 increments “i” (step S1507) and returns to step S1502. On the other hand, when there is no unextracted learned input data (step S1506: No), the information processing apparatus 101 outputs the prediction result screen (step S1508) and ends the series of processes according to this flowchart.

The prediction result screen output in step S1507 is a screen that displays the label determined in step S1403 and the maximum N learned input data registered in the list in association with the input data to be determined. Is.

As a result, when outputting a category (label) obtained by machine learning to the input data to be determined, other cases (up to N pieces of maximum similarity in descending order of similarity) are determined to belong to the category. The learned input data) can be shown.

As described above, according to the information processing apparatus 101 according to the embodiment, it is possible to determine the category to which the input data to be determined belongs by using the learning model MD. The learning model MD is a prediction model that determines the category to which the input data belongs from the feature amount of the input data. Further, according to the information processing device 101, other input data belonging to the determined category and different from the input data to be determined can be extracted from the learned input data used when generating the learning model MD. it can. Then, according to the information processing apparatus 101, it is possible to output the determined category and the other extracted input data in association with the determination target input data.

Accordingly, when outputting a category (label) obtained by machine learning to the input data to be determined, another example that is determined to belong to the category can be shown. Therefore, it is possible to present the user with the information that is the basis for determining the category, and it is possible to support the evaluation of the validity of the determination result obtained by the machine learning.

Further, according to the information processing apparatus 101, the similarity between each of the input data to be determined and the learned input data is calculated, and the learned input data belongs to the determined category based on the calculated similarity. Other input data can be extracted.

This makes it possible to select the learned input data based on which the category to which the determination target input data belongs is determined in consideration of the degree of similarity with the determination target input data.

Further, according to the information processing apparatus 101, the top N other input data having a high degree of similarity with the input data of the determination target among the input data belonging to the determined category can be extracted from the learned input data. it can.

As a result, it is possible to select, as the learned input data based on which the category to which the determination target input data belongs is determined, input data having a high degree of similarity to the determination target input data. For this reason, it is possible to present a basis that is intuitively understandable to humans, and it is possible to easily evaluate the validity of the judgment result obtained by machine learning.

Further, according to the information processing apparatus 101, it is possible to extract, from the learned input data, other input data having a maximum degree of similarity with the input data to be determined among the input data belonging to the determined category. Then, according to the information processing apparatus 101, when the similarity between the extracted other input data and the determination target input data is equal to or less than the threshold value α, the predetermined alert AL is further associated with the determination target input data. Can be output.

As a result, when outputting the category obtained by machine learning to the input data to be judged, the learned input data that belongs to the same category as the category and is the most similar to the input data to be judged is output. Can be presented. In addition, if the similarity with the input data to be judged does not satisfy a certain level, a warning is issued to the user and it may be suggested that the judgment result obtained by machine learning may be incorrect. it can. Therefore, the user can carefully check the determination result obtained by machine learning, and it is possible to prevent the mistake of the label (category) from being overlooked.

Further, according to the information processing apparatus 101, it is possible to extract, from the learned input data, other input data having a maximum degree of similarity with the input data to be determined among the input data belonging to the determined category. Then, according to the information processing apparatus 101, when the extracted other input data is not included in the upper M pieces of input data having a large similarity to the input data to be determined among the learned input data, the determination is made. A predetermined alert AL can be further output in association with the target data.

As a result, when outputting the category obtained by machine learning to the input data to be judged, the learned input data that belongs to the same category as the category and is the most similar to the input data to be judged is output. Can be presented. In addition, if the similarity order of the learned input data to be presented does not meet a certain level, a warning is issued to the user and the judgment result obtained by machine learning may be incorrect. You can Therefore, the user can carefully check the determination result obtained by machine learning, and it is possible to prevent the mistake of the label (category) from being overlooked.

Further, according to the information processing apparatus 101, as a result of outputting the determined category in association with the determination target input data, the correctness information indicating whether the determined category is correct is accepted and determined as other extracted input data. The received correctness information can be recorded in the storage unit 510 in association with the similarity with the target input data. Then, according to the information processing apparatus 101, the threshold value α can be updated based on the pair of the similarity and the correctness information recorded in the storage unit 510.

Accordingly, the threshold value α can be adjusted based on the result of the user's evaluation of the validity of the judgment result (category) for the input data to be judged.

Further, according to the information processing apparatus 101, the similarity between the other input data extracted and the input data to be determined can be output in association with the input data to be determined.

As a result, the reliability of the ground can be judged from the similarity between the learned input data presented as the ground and the input data to be judged. For example, the user can determine that the basis is more reliable as the similarity with the input data to be determined is higher.

From these things, according to the information processing apparatus 101 according to the embodiment, it is possible to apply an arbitrary machine learning algorithm to secure the accuracy of the judgment result obtained by the machine learning, and to confirm the validity of the judgment result by the human. Can allow evaluation. As a result, it is possible to prevent the user from accidentally correcting the label or overlooking the mistake of the label while reducing the user's load on the business decision in the company or the like.

The evaluation support method described in the present embodiment can be realized by executing a prepared program on a computer such as a personal computer or a workstation. This evaluation support program is recorded in a computer-readable recording medium such as a hard disk, a flexible disk, a CD-ROM, a DVD, or a USB memory, and is executed by being read from the recording medium by the computer. The evaluation support program may be distributed via a network such as the Internet.

The information processing apparatus 101 described in the present embodiment can also be realized by a special-purpose IC such as a standard cell or a structured ASIC (Application Specific Integrated Circuit) or a PLD (Programmable Logic Device) such as an FPGA.

The following supplementary notes will be disclosed regarding the above-described embodiment.

(Supplementary Note 1) The category to which the input data to be determined belongs is determined using a learning model that determines the category to which the input data belongs from the feature amount of the input data.
From the learned input data used when generating the learning model, extract other input data belonging to the determined category and different from the determination target input data,
Outputting the determined category and the extracted other input data in association with the determination target input data,
An evaluation support program characterized by causing a computer to execute processing.

(Supplementary Note 2) The computer is caused to execute a process of calculating the similarity between the input data to be determined and the learned input data.
The process of extracting is
The additional input data belonging to the determined category and different from the determination target input data is extracted from the learned input data based on the calculated degree of similarity. Evaluation support program.

(Supplementary Note 3) The extraction processing is
From the learned input data, upper N (N: natural number) other input data having a high degree of similarity with the input data of the determination target among the input data belonging to the determined category are extracted. The evaluation support program described in Appendix 2.

(Supplementary Note 4) The extraction processing is
From the learned input data, other input data having a maximum degree of similarity with the input data to be determined is extracted from the input data belonging to the determined category,
The output process is
When the similarity between the extracted other input data and the input data of the determination target is less than or equal to a threshold value, the predetermined input is further associated with the input data of the determination target, and a predetermined alert is output. The evaluation support program described in 2 or 3.

(Supplementary Note 5) The extraction processing is
From the learned input data, other input data having a maximum degree of similarity with the input data to be determined is extracted from the input data belonging to the determined category,
The output process is
If the extracted other input data is not included in the upper M (M: natural number) input data having a high degree of similarity with the input data to be determined among the learned input data, the determination target The evaluation support program according to any one of appendices 2 to 4, wherein a predetermined alert is further output in association with the input data.

(Supplementary Note 6) As a result of outputting the determined category in association with the input data to be determined, correctness information indicating whether or not the determined category is correct is accepted,
Corresponding to the similarity between the extracted other input data and the input data to be determined, the received correctness information is recorded in the storage unit,
Updating the threshold value based on a pair of the similarity and the correctness information recorded in the storage unit,
5. The evaluation support program described in appendix 4, which causes the computer to execute a process.

(Supplementary Note 7) The output processing is
In any one of appendices 2 to 6, wherein the degree of similarity between the other input data extracted and the input data of the determination target is output in association with the input data of the determination target. Evaluation support program described.

(Supplementary note 8) The evaluation according to any one of supplementary notes 2 to 7, characterized in that the similarity between each of the determination target input data and each of the learned input data is represented by a cosine similarity. Support program.

(Supplementary Note 9) The category to which the input data to be determined belongs is determined using a learning model that determines the category to which the input data belongs from the feature amount of the input data,
From the learned input data used when generating the learning model, extract other input data belonging to the determined category and different from the determination target input data,
Outputting the determined category and the extracted other input data in association with the determination target input data,
An evaluation support method characterized in that a computer executes the processing.

(Supplementary Note 10) A prediction processing unit that determines a category to which the input data to be determined belongs by using a learning model that determines the category to which the input data belongs from the feature amount of the input data,
From the learned input data used when generating the learning model, an extraction unit that extracts other input data belonging to the category determined by the prediction processing unit, which is different from the determination target input data,
An output unit that outputs the category determined by the prediction processing unit and the other input data extracted by the extraction unit in association with the input data to be determined.
An information processing device comprising:

101 information processing device 110, MD learning model 120 learned data 200 information processing system 201 client device 210 network 220 learning data DB
300 bus 301 processor 302 memory 303 disk drive 304 disk 305 communication I/F
306 Portable recording medium I/F
307 Portable recording medium 501 Acquisition unit 502 Learning processing unit 503 Reception unit 504 Prediction processing unit 505 Extraction unit 506 Output unit 507 Update unit 510 Storage unit 700 Similarity table 800, 900, 1000 Prediction result screen

Claims (8)

Using the learning model that determines the category to which the input data belongs from the feature amount of the input data, the category to which the input data to be judged belongs is judged,
From the learned input data used when generating the learning model, extract other input data belonging to the determined category and different from the determination target input data,
Outputting the determined category and the extracted other input data in association with the determination target input data,
An evaluation support program characterized by causing a computer to execute processing. Calculating the degree of similarity between the input data to be judged and each of the learned input data, causing the computer to execute a process,
The process of extracting is
The other input data belonging to the determined category and different from the determination target input data is extracted from the learned input data based on the calculated similarity. Evaluation support program described. The process of extracting is
From the learned input data, upper N (N: natural number) other input data having a high degree of similarity with the input data of the determination target among the input data belonging to the determined category are extracted. The evaluation support program according to claim 2. The process of extracting is
From the learned input data, other input data having a maximum degree of similarity with the input data to be determined is extracted from the input data belonging to the determined category,
The output process is
When the similarity between the extracted other input data and the input data of the determination target is less than or equal to a threshold value, a predetermined alert is further output in association with the input data of the determination target. The evaluation support program according to Item 2 or 3. The process of extracting is
From the learned input data, other input data having a maximum degree of similarity with the input data to be determined is extracted from the input data belonging to the determined category,
The output process is
If the extracted other input data is not included in the upper M (M: natural number) input data having a high degree of similarity with the input data to be determined among the learned input data, the determination target The evaluation support program according to any one of claims 2 to 4, wherein a predetermined alert is further output in association with the input data of. As a result of outputting the determined category in association with the determination target input data, accepting correctness information indicating whether the determined category is correct,
Corresponding to the similarity between the extracted other input data and the input data to be determined, the received correctness information is recorded in the storage unit,
Updating the threshold value based on a pair of the similarity and the correctness information recorded in the storage unit,
The evaluation support program according to claim 4, which causes the computer to execute a process. Using the learning model that determines the category to which the input data belongs from the feature amount of the input data, the category to which the input data to be judged belongs is judged,
From the learned input data used when generating the learning model, extract other input data belonging to the determined category and different from the determination target input data,
Outputting the determined category and the extracted other input data in association with the determination target input data,
An evaluation support method characterized in that a computer executes a process. Using a learning model that determines the category to which the input data belongs from the feature amount of the input data, a prediction processing unit that determines the category to which the input data to be judged belongs,
From the learned input data used when generating the learning model, an extraction unit that extracts other input data belonging to the category determined by the prediction processing unit, which is different from the determination target input data,
An output unit that outputs the category determined by the prediction processing unit and the other input data extracted by the extraction unit in association with the input data to be determined.
An information processing device comprising:

Images