JP7477744B2 - Information processing device, control method, and program - Google Patents

Description

The present invention relates to technology for creating learning data and generating learning models according to the characteristics of the data.

In order to present users with appropriate search results, there are conventional techniques that calculate the relevance between search criteria and terms (character strings extracted using certain criteria such as morphological analysis or N-Grams) contained in each document in a document set as statistical values. These techniques are called similarity search (hereinafter, in the explanation of this invention, this technique will be referred to uniformly as similarity search, and will be distinguished from the search by rank learning described later in this invention).

There is also a ranking learning technology that uses machine learning to model features when the learning data and the group of documents to be searched are similar, and when new search conditions are specified, adjusts the rankings based on the learning model, thereby improving the accuracy of similarity searches.

Ranking learning requires a large amount of learning data, but collecting learning data is difficult. It is possible to collect learning data from users' search logs after similarity search is put into operation as a system, but evaluating search results can place a burden on users, so it is not certain that a sufficient amount of logs can be collected. Furthermore, before operation begins, learning data is limited to data created by developers for testing.

Patent Document 1 provides a technology that finds the most similar question (a question sentence from the learning data) to a user's inquiry from a set of pre-prepared answers (a group of FAQ documents, so to speak) and returns the corresponding answer, and improves the accuracy of topic estimation even when there are only a few question sentences.

Specifically, the questions in the training data are expanded by replacing words that appear in the questions with the corresponding words in the answers, i.e., the number of training data items is increased. In addition, to eliminate unnatural questions from the expanded questions, a probabilistic language model is used to calculate the probability of a question's existence, and only those questions that exceed a certain threshold are used as training data.

JP 2017-37588 A

However, in the technology of Patent Document 1, a probabilistic language model is used to determine whether the expanded question sentence is appropriate, but the replaced words are merely those included in the answers prepared in advance, and there is a possibility that technical terms or terms specific to a certain organization are used. In such cases, there may be cases where the probabilistic language model does not have enough examples and the question sentence is not expanded appropriately.

Furthermore, the technology of Patent Document 1 aims to improve the learning effect by expanding the number of questions used as learning data. However, as the number of learning data items increases, the calculation time required for learning becomes enormous, and this may become impractical.

An object of the present invention is to provide a technique for creating learning data based on data classified into groups containing similar data .

The present invention is an information processing device that manages a learning model, and is characterized by comprising a classification means that classifies data used for learning into groups, a determination means that determines feature data indicative of data classified into the groups based on the data classified into the groups, a combination means that combines groups based on the feature data indicative of the data classified into the groups to create new groups, and a registration means that registers a learning model generated using data classified into the combined group and the feature data related to the determined group.

The present invention makes it possible to provide a technique for creating learning data based on data classified into groups containing similar data .

FIG. 2 is a diagram illustrating an example of a functional configuration according to an embodiment of the present invention. 1 is a block diagram showing an example of a hardware configuration applicable to an information processing apparatus 100 according to an embodiment of the present invention. 13 is an example of a user interface for inputting search conditions and specifying a correct answer from search results according to an embodiment of the present invention. 13 is an example of a search result of a similarity search according to the embodiment of the present invention. 13 is an example of a result of clustering search results according to an embodiment of the present invention. 13 is a diagram showing an example of a result of drilling down using clustering according to an embodiment of the present invention. 4 is an example of a structure of learning data registered in a learning data storage unit according to the embodiment of the present invention. 1 is an example of a flowchart illustrating a learning model generation process according to an embodiment of the present invention. FIG. 1 is a diagram illustrating an example of group similarity and grouping of learning data according to an embodiment of the present invention. 1 is an example of a diagram illustrating a data structure for storing a learning model according to an embodiment of the present invention and a similarity calculation for a cluster selected during a search. 1 is a flowchart illustrating a search process according to an embodiment of the present invention.

In this invention, machine learning is used to re-assign the search ranking of conventional document search results. This is called rank learning. For the sake of convenience in this invention, the process of determining a learning model in advance is called "learning model generation," and the process of actually re-assigning the ranking of search results based on user search criteria, etc., using the generated learning model is called "re-ranking."

The present invention has three features. First, it is about how to store learning data for a group of documents that does not have classification information in order to generate a learning model (described below). Second, it is about how to limit the group of documents that will become the learning data to a portion based on the above-mentioned learning data to generate a learning model. Third, it is about how to select an appropriate learning model from among multiple learning models when re-ranking a group of documents that have been dynamically clustered during a search.

The following describes in detail the embodiments of the present invention with reference to the drawings.

Figure 1 is a diagram showing an example of a functional configuration according to an embodiment of the present invention. A search condition receiving unit 101 receives search conditions (character strings) from a search user or another program, and sends them to a similarity search unit 102. The similarity search unit 102 searches the document storage unit 121 to obtain search results that match the conditions described in the search conditions, i.e., a list of documents. This search process involves calculating the similarity between the search conditions and documents based on the frequency of word appearances, and other factors, and then creating a document list of the top documents as search results. There are various well-known techniques for this search process, and a detailed description will be omitted here.

The list of documents resulting from the search is passed to the clustering unit 103, which divides the documents into clusters based on the similarity of each document using natural language processing. Clustering is also a well-known technique, and so a detailed explanation is omitted. In addition, there are methods for clustering in which one document is always classified into one cluster, and methods in which one document is allowed to be included in multiple clusters, and either method may be used in the present invention.

The display unit 104 displays the clusters and allows the user to select one of the clusters, for example. It accepts the cluster selected by the user and displays the documents classified into that cluster (part of the document list of the search results). At this time, the learning model selection unit 105 selects an appropriate learning model from the learning model storage unit 123 according to the cluster selected by the user, re-ranks the documents according to the learning model, and displays them on the display unit 104.

The display unit 104 also allows the user to select, for example, one document from the group of documents included in the cluster selected by the user, and passes that document to the learning data registration unit 107 as the correct answer to the search criteria, and the learning data registration unit 107 then composes learning data and registers it in the learning data storage unit 122.

The learning model generation unit 108 generates a learning model for reranking using the learning data stored in the learning data storage unit 122. Instead of generating one learning model using all the learning data, the generation model determination unit 109 groups the learning data using information about the document group included in the cluster when the learning data was registered, and the learning model generation unit 108 generates a learning model for each group based on the groups.

Figure 2 is a block diagram showing an example of a hardware configuration that can be applied to the information processing device 100 according to an embodiment of the present invention.

As shown in FIG. 2, the information processing device 100 has a configuration in which a CPU (Central Processing Unit) 201, a RAM (Random Access Memory) 202, a ROM (Read Only Memory) 203, an input controller 205, a video controller 206, a memory controller 207, a communication I/F controller 208, etc. are connected via a system bus 204.

The CPU 201 provides overall control over the devices and controllers connected to the system bus 204.

The ROM 203 or external memory 211 also stores the BIOS (Basic Input/Output System) and OS (Operating System), which are control programs for the CPU 201, as well as various programs (described below) required to realize the functions executed by each server or each PC. Information required to implement the present invention is also stored. The external memory may be a database.

RAM 202 functions as the main memory, work area, etc. of CPU 201. CPU 201 loads programs and the like required for executing processing from ROM 203 or external memory 211 into RAM 202, and executes the loaded programs to realize various operations.

The input controller 205 also controls input from a keyboard (KB) 209 and a pointing device such as a mouse (not shown).

The video controller 206 controls the display on a display device such as a display 210. The display device may be a liquid crystal display or other display device. These are used by the administrator as necessary.

The memory controller 207 controls access to an external memory 211 such as an external storage device (hard disk (HD)) that stores boot programs, various applications, font data, user files, edited files, various data, etc., a flexible disk (FD), or a compact flash (registered trademark) memory connected via an adapter to a PCMCIA (Personal Computer Memory Card International Association) card slot.

The communication I/F controller 208 connects to and communicates with external devices via a network, and executes communication control processing on the network. For example, communication using TCP/IP (Transmission Control Protocol/Internet Protocol) is possible.

The CPU 201 can display on the display 210 by, for example, executing a process of expanding (rasterizing) an outline font in a display information area in the RAM 202. The CPU 201 also enables user instructions using a mouse cursor (not shown) on the display 210.

The various programs described below for implementing the present invention are recorded in the external memory 211 and are executed by the CPU 201 by being loaded into the RAM 202 as required.

Next, an overview of searching will be described with reference to Figures 3 to 6. Figure 3 shows an example of a user interface for inputting search conditions and specifying the correct answer from the search results according to an embodiment of the present invention.

The condition search input screen 301 is a screen where a search user can start a search by entering "Otani's batting record" in the question field 311 and pressing the search button 312.

The list of document data that the search user ultimately wants to view is explained on the document viewing screen 302. The display area 321 displays the document ID "0028", the title, the text, etc.

In addition, if the search user views the document displayed in the display area 321 and determines that it is exactly the information he or she wanted to see, the user can press the correct answer button 322 to have it registered in the learning data storage unit 122.

However, to reach document ID "0028", the search result list 400 in Figure 4 is first presented. Generally, the search result list 400 is often presented to the search user, but some commercial systems may automatically classify and limit the set of documents presented to the search user. This depends on the number of registered documents, but when hundreds of hits are generated for the search criteria, it can be difficult for the search user to reach the information they are looking for, so by having the search user specify some criteria again, for example, limiting the results to a few dozen makes it easier to reach the information they are looking for.

One method is to cluster the documents that match the search criteria and present documents with similar content as groups called clusters. The search user first selects the cluster that contains the information they want to obtain, thereby limiting the documents they check to a portion of the total, as described above.

In the search result list in Figure 4, for example, information about Ichiro Otani would likely be divided into clusters for his high school years, his time in professional baseball in Japan, and his performance in the major leagues. In Figure 5, these classifications are presented to the search user as an example cluster list 501.

When a search user selects "Major League Baseball" from cluster 503, a list of documents classified into that cluster (600 in FIG. 6) is displayed. While the search result list 400 in FIG. 4 contained a variety of information about player Ichiro Otani, document list 600 in FIG. 6 shows an example in which only articles about his achievements in Major League Baseball are included. Limiting the document list in this way makes it possible for the search user to easily arrive at the information they are looking for (document ID "0028" in FIG. 3).

In addition, when clustering, if the documents that match the query (311 in Figure 3) specified by the search user are different, they will naturally be classified into different groups. If you search for the query "baseball in America" as in 502 in Figure 5, you will not necessarily find information about Ichiro Otani, but will find documents that are different from the example in Figure 4. If you then cluster these documents, different clusters will be generated. In the example of 502, a cluster called Major League (504) is generated, but even if the heading is the same, it will contain many documents that are completely different from 503. For example, it may contain information about the position of Major League Baseball in American baseball, records of famous players other than Ichiro Otani, etc. In this way, even if the documents have the same title, the title "Major League" does not have much meaning, and it is more important to have a list of document IDs that are included in the same cluster as a representation of the linguistic characteristics of these clusters.

Figure 7 shows an example of the structure of learning data according to an embodiment of the present invention. Learning data for generating a learning model in ranking learning is obtained, for example, by specifying a document that matches the user's search intent when the user actually searches. Therefore, at least the user's search conditions and information for identifying the document selected by the user must be registered in pairs. In the learning data 701, these are stored as a question sentence 703 and a correct answer document ID 704.

The learning data 701 (a to f) stored in the learning data storage unit 122 in the present invention is characterized in that, in addition to the question sentence 703 and the correct answer document ID 704, a same cluster document ID list 705 is stored.

For example, Figure 6 shows that a search user who entered the search criteria "Otani's batting record" selected "Major League" from cluster 503. This cluster contains document IDs "0005", "0006", ..., "0028", so the same cluster document ID list 705 in the training data 701a lists these document IDs as is.

Next, the process of generating a learning model will be described with reference to Fig. 8 and Fig. 9. Fig. 8 is an example of a flowchart for explaining the process of generating a learning model according to an embodiment of the present invention. Each step of the flowchart in Fig. 8 is executed by the CPU 201 on the information processing device 100.
<Embodiment 1>
In step S801, all learning data groups stored in the learning data storage unit 122 are read. The read data is shown in learning data document ID vector (example 1) 901 in Fig. 9. This table is essentially the same as Fig. 7, but whereas the document ID list in Fig. 7 is expressed as a list, this table expresses each document ID (for example, if document IDs range from 1 to 500) as a vector arranged in a row. Each element of the vector is set to "1" if the document ID is included in the same cluster document ID list 705, and "0" if it is not included.

In step S802, among the training data (901), those with the same correct answer document ID (911) are grouped together as in the training data document ID vector (example 2) 902 in FIG. 9. For example, there are two correct answer document IDs for "0028", "L0001" and "L0003". These are grouped together as in the first line of 902. Specifically, these two training data IDs are listed in the training data ID list 916, and the document ID vector of 901 indicates how many IDs were present (1) or absent (0) in total. For example, the value corresponding to the document ID "0005" is "1" in 914, but "2" in 917. It goes without saying that this simple sum is merely an example, and there are various calculation methods.

In step S803, the learning data 902 is grouped. The purpose is to determine the learning data to be used when generating one learning model. In other words, if the document contains some classification information, it is possible to group the learning data by collecting documents with the same classification information indicated by the correct answer document ID. However, the present invention assumes a case where such classification information is not available or cannot be used, and therefore assumes that the learning data that contains similar groups of documents when drilled down as in FIG. 6 is used to generate one learning model.

As a method for grouping learning models, vector clustering is a well-known technique. In Example 2 of Figure 9, 500-dimensional vectors, which corresponds to the number of documents, are compared and clustered. There is also a technique known as overlap clustering, which allows the same vector to be included in multiple clusters. In any case, it goes without saying that any method that can divide these vector groups into clusters is acceptable.

As a result, even if there is no classification information in advance for the document group as described above, it is possible to obtain the effect of dividing the learning data into appropriate groups and generating multiple pieces of learning data. An example of the grouped learning models is shown as group 903 (a, b) in Figure 9.

The repeated processing from S804 to S806 is performed on each set of learning data grouped in S803.

S805 focuses on each of the grouped learning data (e.g., 903a and 903b in FIG. 9 ) and generates a learning model from the learning data. In the case of ranking learning, this can be realized by an SVM (support vector machine) or the like. The generated learning model is stored in the learning model storage unit 123.

This completes the explanation of how to generate a learning model in the present invention using the flowchart in Figure 8.

Figure 10 is an example of a diagram illustrating the data structure when storing a learning model according to an embodiment of the present invention, and the similarity calculation of a cluster selected during a search. First, the learning models stored in the learning model storage unit 123 will be described. In this figure, it is assumed that two learning models are stored.

The learning model body generated by an SVM (support vector machine) or the like is stored in the learning model storage unit 123. However, in conventional technology, information about the conditions under which these learning models are used is not included, and the application (or user) that uses the learning models selects which learning model to use from among multiple available models. However, as a premise of the present invention, fixed information equivalent to classification information is not prepared for document groups, and the clusters in FIG. 5 are also dynamically generated, so which learning model to use must be determined at the time of search.

A feature of the present invention is that it includes linguistic features 1003 of the learning model that are associated with each learning model and indicate in what context the learning model is to be used.

As an example of setting data 1005 for linguistic features 1003, the following shows a case where it is expressed as a document ID vector (sum). Document ID vector (sum) 1006 is a simple sum of the document ID vectors of the training data included in group 903a in FIG. 9. In other words, the training data used when generating this training model describes the tendency of what kind of documents and how often they appeared in the same cluster as the correct answer document. In similarity search and document clustering, this represents a kind of linguistic feature, and is the linguistic feature of the training model in this invention.

FIG. 11 is an example of a flowchart illustrating a search process according to an embodiment of the present invention. Each step of the flowchart in FIG. 11 is executed by the CPU 201 on the information processing device 100.

In step S1101, search conditions are accepted from a search user or an application, and in step S1102, a group of documents that match the search conditions are obtained from the document storage unit 121.

In step S1103, the documents acquired in step S1102 are clustered. An example of this is shown in FIG. 5, with three clusters.

In step S1104, the list of clusters is presented to the user, and in step S1105, the user selects one of the presented clusters, i.e., drills down. Let us assume that the search user selects the cluster "Major League Baseball" from 501 in FIG. 5.

In step S1106, a list of document IDs contained in the cluster selected by the user ("Major League Baseball" in this example) is generated as a vector. This is the document ID vector 1007 in FIG. 10 (document ID vector contained in the cluster automatically generated during search). Next, of the document ID vectors (sum) 1006 of 1001a to 1001b, the one with the highest similarity to vector 1007 is identified.

However, even if the similarity is the highest, it may be inappropriate to use it as a learning model. Therefore, in step S1107, a comparison is made with a threshold value stored in a memory unit (not shown), and if there is no learning model whose vector similarity exceeds the threshold value (if NO), the ranking of the similarity search results in step S1102 may be presented to the user as is without using a learning model. If there is an appropriate learning model, in step S1108, the results of the similarity search in step S1102 are re-ranked, and in step S1109, they are presented to the user as search results.

Finally, in step S1110, if the user designates one document from the presented search results as being appropriate for the user's own search, that document is accepted as the correct selection, and in step S1111, it is registered as new learning data in the learning data storage unit 122. This learning data will be used the next time a learning model is generated.

This completes the explanation of the process of selecting an appropriate learning model from clustering and drill-down, re-ranking documents that appear in a cluster, and presenting the re-ranked documents to the user using the flowchart in FIG.
<Embodiment 2>
Another embodiment will be described below. In the flowchart of Fig. 8, learning data having the same correct answer document ID are grouped together in step S802, but this process does not have to be performed.

In that case, training data having the same correct answer document ID will be included in different training data groups. Even if the correct answer document is the same, if the user's original search intent is different, the linguistic features of the search conditions will be different, and other document IDs included in the same cluster may be completely different. There is no need to forcibly group such training data together and use them to generate the same training data. By using them to generate different training models, it is possible to obtain the effect of generating training models that better reflect the user's intent.
<Embodiment 3>
In the example of FIG. 10, the setting data 1005 of the linguistic features 1003 of the learning model is the document ID vector (sum) 1006, but it goes without saying that anything that represents the linguistic features of the cluster may be used.

For example, feature words (such as important words) may be extracted by natural language processing from the “question sentence” of the learning model used to generate the corresponding learning model and the text in the correct answer document, and stored in 1005.

In this case, characteristic words may be extracted from documents contained in the cluster selected by the search user during the search and compared with 1005.

Moreover, it does not have to be the word itself. There is a model of a well-known technology. "Tomas Mikolov, Kai Chen, and Jeffrey Dean, Efficient estimation of word representation in vector space, CoRR, Vol. abs/1301.3781,, 2013"
This technology learns to represent words that appear in a large number of documents as feature vectors of, for example, 200 dimensions. Furthermore, the contents of a document can be considered as the sum of these feature vectors. Therefore, the characteristics of the questions involved in generating the learning model and the documents contained in the clusters can be represented as feature vectors, and during a search, feature vectors can be generated from the documents contained in a cluster selected by the user and compared for similarity.

This has the effect of making it possible to select learning models that are more semantically similar, rather than simply using the similarity of a numerical vector composed of document IDs or the similarity of feature words.

It goes without saying that any feature can be used as long as it is a method for selecting a learning model that optimally reranks the list of documents contained in a cluster.

In addition, while this embodiment focuses on documents, it can also be applied to various types of data, such as images that can be searched, categorized, and evaluated as data.

It goes without saying that the structure and content of the various data described above are not limited to those described above, and may be structured and configured in various ways depending on the application and purpose.

Although several embodiments have been described above, the present invention can be embodied, for example, as a system, device, method, computer program, or recording medium, and specifically, may be applied to a system made up of multiple devices, or may be applied to a device made up of a single device.

The computer program of the present invention is a computer program that enables a computer to execute the processing methods of the flowcharts shown in Figures 8 and 11, and the storage medium of the present invention stores a computer program that enables a computer to execute the processing methods of Figures 8 and 11. Note that the computer program of the present invention may be a computer program for each processing method of each device in Figures 8 and 11.

As described above, it goes without saying that the object of the present invention can be achieved by supplying a recording medium on which a computer program that realizes the functions of the above-mentioned embodiments is recorded to a system or device, and having the computer (or CPU or MPU) of that system or device read and execute the computer program stored on the recording medium.

In this case, the computer program read from the recording medium itself realizes the novel functions of the present invention, and the recording medium on which the computer program is stored constitutes the present invention.

Recording media for supplying computer programs include, for example, flexible disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, CD-Rs, DVD-ROMs, magnetic tapes, non-volatile memory cards, ROMs, EEPROMs, silicon disks, solid-state drives, etc.

Furthermore, it goes without saying that the functions of the above-mentioned embodiments are not only realized by the computer executing a computer program that has been read out, but also includes cases where an operating system (OS) running on the computer performs some or all of the actual processing based on the instructions of the computer program, thereby realizing the functions of the above-mentioned embodiments.

Furthermore, it goes without saying that this also includes cases where a computer program read from a recording medium is written into a memory provided on an expansion board inserted into a computer or a function expansion unit connected to a computer, and then a CPU or the like provided on the expansion board or function expansion unit performs some or all of the actual processing based on the instructions of the computer program code, thereby realizing the functions of the above-mentioned embodiments.

The present invention may be applied to a system consisting of multiple devices, or to an apparatus consisting of a single device. Needless to say, the present invention can also be applied to cases where the invention is achieved by supplying a computer program to a system or apparatus. In this case, the system or apparatus can enjoy the effects of the present invention by reading a recording medium containing a computer program for achieving the present invention into the system or apparatus.

Furthermore, by downloading and reading a computer program for achieving the present invention from a server, database, etc. on a network using a communication program, the system or device can enjoy the effects of the present invention.

In addition, the present invention also includes configurations that combine the above-mentioned embodiments and their modified examples.

REFERENCE SIGNS LIST 100 Information processing device 101 Search condition acceptance unit 102 Similarity search unit 103 Clustering unit 104 Display unit 105 Learning model selection unit 106 Re-ranking unit 107 Learning data registration unit 108 Learning model generation unit 109 Generation model determination unit 121 Document storage unit 122 Learning data storage unit 123 Learning model storage unit

Claims (9)

An information processing device for managing a learning model,
A classification means for classifying data used for learning into groups;
A determination means for determining, based on the data classified into the groups, characteristic data indicative of the data classified into the groups;
a combining means for combining the groups to create a new group based on feature data indicating the data classified into the groups;
a registration means for registering a learning model generated using the data classified into the combined group and feature data related to the determined group;
An information processing device comprising: The information processing device according to claim 1, characterized in that the characteristic data is data that characterizes the group. The information processing device according to claim 1 or 2, characterized in that the feature data is used when the group is selected from a plurality of group candidates. The information processing device according to any one of claims 1 to 3, characterized in that the learning model is a ranking learning model that ranks the data. The information processing device according to any one of claims 1 to 4, characterized in that the feature data is data indicating an inclusion relationship between the group and data classified into the group. 5. The information processing apparatus according to claim 1, wherein the feature data is a vector of values indicating whether each piece of data has been classified into the group. The information processing device according to any one of claims 1 to 6, characterized in that the data is document data searched for by search text. A method for controlling an information processing device that manages a learning model, comprising:
A classification step in which classification means classifies data used for learning into groups;
A determination step in which a determination means determines, based on the data classified into the groups, feature data indicative of the data classified into the groups;
a combining step in which combining means combines the groups based on feature data indicating the data classified into the groups to create a new group;
a registration step in which a registration means registers a learning model generated using the data classified into the combined group and feature data related to the determined group;
13. A method for controlling an information processing device, comprising: A program executable by an information processing device that manages a learning model,
The information processing device,
A classification means for classifying data used for learning into groups;
A determination means for determining, based on the data classified into the groups, characteristic data indicative of the data classified into the groups;
a combining means for combining the groups to create a new group based on feature data indicating the data classified into the groups;
a registration means for registering a learning model generated using the data classified into the combined group and feature data related to the determined group;
A program to function as a

Images