JP2021071957A - Information processing apparatus, control method, and program - Google Patents

Abstract

To provide a technology to select an appropriate learning model during a search for data.SOLUTION: The present invention is an information processing apparatus that manages a plurality of learning models, and comprises: receiving means that presents groups in which pieces of data searched for according to a search condition are analyzed and classified, and receives designation of a group; and selection means that selects a learning model according to first characteristic data based on pieces of data classified into the designated group and second characteristics data associated with the respective learning models.SELECTED DRAWING: Figure 1

Description

The present invention relates to a technique for selecting a learning model according to the characteristics of data.

Conventionally, in order to present appropriate search results to the user, the relationship between the search conditions and the terms (morphological analysis, character strings cut out by a certain standard such as N-Gram) included in each document included in the document group is determined. There is a technique to calculate as a statistical value. These techniques are referred to as a similarity search or the like (hereinafter, in the description of the present invention, the techniques will be collectively referred to as a similarity search, and will be distinguished from the search by rank learning described later in the present invention).

In addition, the feature amount when the learning data and the document group to be searched are similar is modeled by machine learning, and when a new search condition is specified, the ranking is adjusted based on the learning model to perform a similar search. There is a ranking learning technique that improves the accuracy of.

Although a large amount of learning data is required for rank learning, it is difficult to collect the learning data. It is conceivable to collect learning data from the user's search log after starting the operation of similar search as a system, but it may be a burden on the user to evaluate the search results, so it is possible to collect a sufficient amount of logs. could not say it all. Also, before the start of operation, it is limited to learning data created by the developer for testing.

Patent Document 1 is a technique for finding the most similar question (question text of learning data) to an inquiry from a user with respect to an answer prepared in advance (so to speak, a group of FAQ documents) and returning the corresponding answer. We provide technology to improve topic estimation accuracy even when there are few question sentences.

Specifically, by substituting the words appearing in the interrogative sentence of the learning data with the words in the corresponding answers, the interrogative sentence of the learning data is expanded, that is, the number of learning data is increased. In addition, in order to exclude unnatural question sentences from the expanded question sentences, the existence probability of the question sentence is calculated using a stochastic language model, and it is used as training data only when the existence probability exceeds a certain threshold.

Japanese Unexamined Patent Publication No. 2017-37588

However, in the technique of Patent Document 1, it is determined whether or not the expanded question sentence is appropriate by using a stochastic language model, but the replaced word is included in the answer prepared in advance. And there is a possibility that technical terms or terms specific to an organization are used. In that case, the stochastic language model lacks cases, and the question text may not be expanded appropriately.

Further, the technique of Patent Document 1 aims to enhance the learning effect by expanding the question sentences used as learning data. However, if the number of training data increases, the calculation time required for learning becomes enormous, which may become impractical.

An object of the present invention is to provide a technique for selecting an appropriate learning model when retrieving data.

The present invention is an information processing device that manages a plurality of learning models, presents a group that analyzes and classifies the data searched according to the search conditions, and accepts the designation of the group, and the designated group. It is characterized by including a first feature data based on the classified data and a selection means for selecting a learning model according to the second feature data associated with each of the learning models.

INDUSTRIAL APPLICABILITY According to the present invention, it is possible to provide a technique for selecting an appropriate learning model when searching data.

It is a figure which shows an example of the functional structure which concerns on embodiment of this invention. It is a block diagram which shows an example of the hardware configuration applicable to the information processing apparatus 100 which concerns on embodiment of this invention. This is an example of a user interface for inputting search conditions and designating a correct answer from the search results according to the embodiment of the present invention. This is an example of a search result of a similar search according to an embodiment of the present invention. This is an example of the result of clustering the search results according to the embodiment of the present invention. This is an example of the result of drilling down using the clustering according to the embodiment of the present invention. This is an example of the structure of the learning data registered in the learning data storage unit according to the embodiment of the present invention. This is an example of a flowchart for explaining the process of generating a learning model according to the embodiment of the present invention. It is an example of the figure explaining the group similarity and grouping of the learning data which concerns on embodiment of this invention. It is an example of a figure explaining the data structure when storing the learning model according to the embodiment of the present invention and the similarity calculation of the cluster selected at the time of retrieval. This is an example of a flowchart for explaining the search process according to the embodiment of the present invention.

In the present invention, the search result of the conventional document is redesignated by machine learning, and the search order is redesignated by using machine learning. This is called ranking learning. In particular, in the present invention, for convenience of explanation, the process of determining the learning model in advance is "generation of learning model", and the search result based on the search conditions such as the user is redesignated by using the generated learning model. We will call the process "reranking".

The features of the present invention are the following three points. The first is how to store learning data in order to generate a learning model, which will be described later, for a group of documents that do not have classification information. The second point is how to generate a learning model by limiting the document group to be the learning data to a part based on the learning data. The third point is how to select an appropriate learning model from a plurality of learning models when re-ranking a group of documents dynamically clustered at the time of searching.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

FIG. 1 is a diagram showing an example of a functional configuration according to an embodiment of the present invention. The search condition receiving unit 101 receives a search condition (character string) from a search user or another program and sends it to the similar search unit 102. The similar search unit 102 searches the document storage unit 121 and acquires a search result that hits the condition described in the search condition, that is, a document list. This search process has various well-known techniques such as calculating the similarity between the search condition and the document based on the frequency of occurrence of words and making the higher-ranking document into a document list of the search result, and the description is omitted.

The document list that is the search result is passed to the clustering unit 103, and is divided into clusters based on the similarity of each document by natural language processing. Clustering is also a well-known technique, and the description thereof will be omitted. Further, in clustering, there are a method of always classifying one document into one cluster and a method of allowing one document to be included in a plurality of clusters, and any of them may be used in the present invention.

The display unit 104 displays the cluster and allows, for example, the user to select one of the clusters. The cluster selected by the user is accepted and the document group classified into the cluster (a part of the document list of the search result) is displayed. At that time, the learning model selection unit 105 selects an appropriate learning model from the learning model storage unit 123 according to the cluster selected by the user, re-ranks the document group according to the learning model, and displays the document group on the display unit 104. indicate.

Further, for example, the user is made to select one document from the document group included in the cluster selected by the user on the display unit 104, the document is passed to the learning data registration unit 107 as a correct answer to the search condition, and the learning data registration is performed. The unit 107 configures the learning data and registers it in the learning data storage unit 122.

The learning model generation unit 108 generates a learning model for re-ranking using the learning data stored in the learning data storage unit 122. Instead of generating one learning model using all the training data, the generation model determination unit 109 groups the training data using the information about the document group included in the cluster when the training data is registered. The learning model generation unit 108 generates a learning model for each group based on the group.

FIG. 2 is a block diagram showing an example of a hardware configuration applicable to the information processing apparatus 100 according to the embodiment of the present invention.

As shown in FIG. 2, the information processing apparatus 100 includes a CPU (Central Processing Unit) 201, a RAM (Random Access Memory) 202, a ROM (Read Only Memory) 203, an input controller 205, and a video controller 206 via the system bus 204. , Memory controller 207, communication I / F controller 208, etc. are connected.

The CPU 201 comprehensively controls each device and controller connected to the system bus 204.

Further, the ROM 203 or the external memory 211 will be described later, which is necessary for realizing the functions executed by the BIOS (Basic Input / Output System) and the OS (Operating System), which are the control programs of the CPU 201, and each server or each PC. Various programs etc. are stored. In addition, information necessary for carrying out the present invention is stored. The external memory may be a database.

The RAM 202 functions as a main memory, a work area, and the like of the CPU 201. The CPU 201 realizes various operations by loading a program or the like necessary for executing the process from the ROM 203 or the external memory 211 into the RAM 202 and executing the loaded program.

Further, the input controller 205 controls input from a pointing device such as a keyboard (KB) 209 or a mouse (not shown).

The video controller 206 controls the display on a display such as the display 210. The display may be a display such as a liquid crystal display. These are used by the administrator as needed.

The memory controller 207 is an external storage device (hard disk (HD)) for storing boot programs, various applications, font data, user files, edit files, various data, etc., a flexible disk (FD), or a PCMCIA (Personal Computer). It controls access to an external memory 211 such as a compact flash® memory connected to a Memory Card International Association card slot via an adapter.

The communication I / F controller 208 connects and communicates with an external device via the network, and executes communication control processing on the network. For example, communication using TCP / IP (Transmission Control Protocol / Internet Protocol) is possible.

The CPU 201 can display the outline font on the display 210 by executing the outline font expansion (rasterization) process on the display information area in the RAM 202, for example. Further, the CPU 201 enables a user instruction by a mouse cursor (not shown) or the like on the display 210.

Various programs described later for realizing the present invention are recorded in the external memory 211, and are executed by the CPU 201 by being loaded into the RAM 202 as needed.

Next, the outline of the search will be described with reference to FIGS. 3 to 6. FIG. 3 is an example of a user interface for inputting search conditions and designating a correct answer from the search results according to the embodiment of the present invention.

The condition search input screen 301 is a screen in which the search user inputs "Otani's batting record" in the question text field 311 and presses the search button 312 to start the search.

A list of document data that the search user finally wants to browse is described on the document browsing screen 302. In the display area 321, the document ID is "0028", the title, the text, and the like are displayed.

Further, if the search user browses the document displayed in the display area 321 and determines that the information is exactly the information he / she wants to see, he / she presses the correct answer button 322 to register the document in the learning data storage unit 122. Can be done.

However, in order to reach the document ID "0028", the search result list 400 of FIG. 4 is first presented. Generally, the search result list 400 is often presented to the search user, but some commercial systems may be automatically classified to limit the document group to be presented to the search user. This depends on the number of registered documents, but when hundreds of hits are made to the search conditions, it is difficult to reach the information requested by the search user, and in some way, for example, the search user is asked again. This is to make it easier to reach the desired information by limiting the number to dozens by specifying some condition.

As one method, a group of documents that hit the search conditions is clustered, and a group of documents having similar contents is presented as a group called a cluster. The search user first selects the cluster of information he / she wants to obtain, thereby limiting the documents to be confirmed to a part as described above.

In the search result list in Fig. 4, for example, even information about Ichiro Otani may be divided into clusters such as high school days, times when he was active in professional baseball in Japan, and results in major leagues. In FIG. 5, these classifications are presented to the search user as Example 501 of the cluster list.

When the search user selects "Major League" of the cluster 503, a list of documents classified into the cluster (600 in FIG. 6) is displayed. The search result list 400 in FIG. 4 contains various information about Ichiro Otani, but the document list 600 in FIG. 6 shows an example in which only articles related to activities in the major leagues are included. By limiting the document list in this way, it is possible to easily reach the information requested by the search user (document ID "0028" in FIG. 3).

Further, clustering is naturally classified into a different group if the document group that hits the question sentence (311 in FIG. 3) specified by the search user is different. If you search for the question "baseball in the United States" as in 502 in FIG. 5, information about Ichiro Otani is not always collected, and a document different from the example in FIG. 4 is hit. Further clustering those documents will generate different clusters. In the example of 502, a cluster (504) called Major League Baseball is generated, which means that many documents with the same heading but completely different from 503 are included. For example, it may include the position of Major League Baseball in American baseball, records of famous players other than Ichiro Otani, and so on. The title "Major League" doesn't make much sense even if it has the same title like this, and the list of what kind of document IDs are included in the same cluster is the language of these clusters. It becomes important as a representation of the characteristic.

FIG. 7 is an example of the structure of the learning data according to the embodiment of the present invention. The learning data for generating a learning model in rank learning is obtained by, for example, designating a document that matches the user's search intention when the user actually searches. Therefore, at least the user's search condition and the information for identifying the document selected by the user need to be registered as a pair. In the learning data 701, they are stored as the question sentence 703 and the correct answer document ID 704.

The learning data 701 (a to f) stored in the learning data storage unit 122 in the present invention is characterized in that the same cluster document ID list 705 is stored in addition to the question sentence 703 and the correct answer document ID 704.

For example, FIG. 6 shows that the search user who entered the search condition “Otani's batting record” selected the “major league” of the cluster 503. Since the cluster includes "0005", "0006" ..., "0028" as the document IDs, the list of these document IDs is described as it is in the same cluster document ID list 705 of the learning data 701a. There is.

Next, the process of generating the learning model will be described with reference to FIGS. 8 to 9. FIG. 8 is an example of a flowchart for explaining the process of generating a learning model according to the embodiment of the present invention. Each step of the flowchart of FIG. 8 is executed by the CPU 201 on the information processing apparatus 100.
<Embodiment 1>
In step S801, all the learning data groups stored in the learning data storage unit 122 are read. The read data is shown in the document ID vector (Example 1) 901 of the learning data of FIG. This table is essentially the same as FIG. 7, but for FIG. 7, which represents a list of document IDs, each document ID is listed in one column (for example, if the document IDs are 1 to 500). It is expressed as a vector arranged in line. Each element of the vector is set to "1" when the document ID is included in the same cluster document ID list 705, and "0" when the document ID is not included.

In step S802, among the learning data (901), those having the same correct answer document ID (911) are put together as in the document ID vector (Example 2) 902 of the learning data of FIG. For example, there are two documents whose correct document ID is "0028", "L0001" and "L0003". These are summarized as in the first line of 902. Specifically, these two learning data IDs are listed in the learning data ID list 916, and the document ID vector of 901 shows only (1) and none (0) in each ID. Try to show how many there were in total. For example, the value corresponding to the document ID "0005" is "1" in 914, but is "2" in 917. It goes without saying that such simple summing is just an example and there are various calculation methods.

In step S803, the training data of 902 is grouped. The purpose is to determine the training data to use when generating one training model. That is, when some classification information is included in the document, it is possible to collect the same classification information in the document indicated by the correct answer document ID and group the learning data. However, in the present invention, this is possible. Since it is assumed that such classification information is not available or cannot be used, when drilling down as shown in FIG. 6, training data containing similar document groups generates one learning model. It is assumed that it will be used to do so.

As a method of grouping learning models here, there is vector clustering as a well-known technique. In Example 2 of FIG. 9, it is a technique of comparing and clustering 500-dimensional vectors corresponding to the number of documents. There is also a technique for allowing the same vector to be included in a plurality of clusters as duplicate clustering. In any case, it goes without saying that any method may be used as long as it is a technique for dividing these vector groups into clusters.

As a result, even if there is no classification information in the document group in advance as described above, it is possible to obtain the effect that the learning data can be divided into appropriate groups and a plurality of learning data can be generated. The grouped learning model group is illustrated as the group 903 (a, b) of FIG.

The iterative processing of S804 to S806 is a processing for each learning data group grouped in S803.

S805 pays attention to each of the grouped learning data groups (for example, 903a and 903b in FIG. 9), and generates a learning model with the learning data. In the case of rank learning, it can be realized by SVM (support vector machine) or the like. The generated learning model is stored in the learning model storage unit 123.

This completes the description of the generation of the learning model in the present invention according to the flowchart of FIG.

FIG. 10 is an example of a diagram illustrating a data structure when storing a learning model according to an embodiment of the present invention and a similarity calculation of clusters selected at the time of retrieval. First, the learning model stored in the learning model storage unit 123 will be described. In this figure, it is assumed that two learning models are stored.

The main body of the learning model generated by SVM (support vector machine) or the like is stored in the learning model storage unit 123. However, in the prior art, information on under what conditions these learning models are used is not included, and an application (or user) using the learning model should use from a plurality of learning models. Will be selected. However, as a premise of the present invention, since the information corresponding to the classification information is not fixedly prepared in the document group and the cluster of FIG. 5 is dynamically generated, it is possible to search which learning model should be used. There is no choice but to decide at times.

A feature of the present invention is that it includes a linguistic feature 1003 of a learning model that indicates in what circumstances the learning model is used in relation to each learning model.

As an example of the setting data 1005 of the linguistic feature 1003, a case where it is represented by a document ID vector (sum) is shown. The document ID vector (sum) 1006 is simply the sum of the document ID vectors of the training data included in the group 903a of FIG. That is, in the learning data used when this learning model was generated, the tendency of what kind of document and how much appeared in the same cluster as the correct answer document is described. In similar search and document clustering, this indicates one type of linguistic feature, which is the linguistic feature of the learning model in the present invention.

FIG. 11 is an example of a flowchart for explaining the search process according to the embodiment of the present invention. Each step of the flowchart of FIG. 11 is executed by the CPU 201 on the information processing apparatus 100.

In step S1101, the search condition is received from the search user or the application, and in step S1102, the document group that hits the search condition is acquired from the document storage unit 121.

In step S1103, the document group acquired in step S1102 is clustered. An example of this is 501 in FIG. 5, for example, three clusters.

In step S1104, the list of the clusters is presented to the user, and in step S1105, the user selects one from the presented clusters. That is, drill down. It is assumed that the search user selects a cluster called "Major League" from 501 in FIG.

In step S1106, a document ID list included in the cluster selected by the user (“Major League” in the example) is generated as a vector. 1007 (document ID vector included in the cluster automatically generated at the time of search) of FIG. 10 is generated. Next, among the document ID vectors (sum) 1006 of 1001a to 1001b, the one having the highest degree of similarity to the vector of 1007 is specified.

However, even the one with the highest degree of similarity may be inappropriate to be adopted as a learning model. Therefore, in step S1107, when compared with the threshold value stored in the storage unit (not shown) and there is no learning model in which the vector similarity exceeds the threshold value (NO), the learning model is not used and the similarity in step S1102 is performed. The ranking of the search results may be presented to the user as it is. If there is an appropriate learning model, in step S1108, the results of the similar search in step S1102 are reranked and presented to the user as search results in step S1109.

Finally, in step S1110, if one of the search results presented by the user is specified as an appropriate document for the user's own search, it is accepted as a correct answer selection, and step S1111 Is registered in the learning data storage unit 122 as new learning data. This training data will be used when the next training model is generated.

This completes the description of the process of selecting an appropriate learning model from clustering and drilldown using the flowchart of FIG. 11, re-ranking the document group appearing in the cluster, and presenting it to the user. ..
<Embodiment 2>
Other embodiments will be described. In the flowchart of FIG. 8, the learning data having the same correct answer document ID is put together in step S802, but this process does not have to be performed.

In that case, the learning data having the same correct document ID will be included in different training data groups. Even if the correct documents are the same, if the user's search intention is different, the linguistic characteristics of the search conditions are different, and other document IDs included in the same cluster may be completely different. It is not necessary to forcibly combine such learning data into one and use it to generate the same learning data, but by using it to generate different learning models, a learning model that more reflects the user's intention is generated. You can get the effect of being possible.
<Embodiment 3>
In the example of FIG. 10, the setting data 1005 of the linguistic feature 1003 of the learning model is set as the document ID vector (sum) 1006, but it goes without saying that any data can be used as long as it represents the linguistic feature of the cluster. ..

For example, characteristic words (important words, etc.) are extracted by natural language processing from the "question sentence" of the learning model used to generate the corresponding learning model and the text in the document that is the correct answer, and stored in 1005. You may.

In this case, the feature words may be extracted from the documents included in the cluster selected by the search user during the search and compared with 1005.

Also, it does not have to be the word itself. There are well-known technical models. "Tomas Mikolov, Kai Chen, and Jeff Dean, Effective stimation of world vector space, CoRR, Vol.abs / 1301.3781, 2013"
In this technique, words appearing in a large number of documents are learned to be represented as, for example, a 200-dimensional feature vector. Furthermore, the content of the document can be thought of as the sum of those feature vectors. Therefore, the features of the question sentences involved in the generation of the learning model and the document group included in the cluster are expressed as feature vectors, and at the time of search, the feature vector is generated from the document group included in the cluster selected by the user and similar. It is also possible to compare degrees.

This has the effect of making it possible to select learning models that are more semantically similar, not just the similarity of numerical vectors composed of document IDs and the similarity of feature words.

It goes without saying that any feature can be used as long as it is a method for selecting a learning model for optimally re-ranking the list of documents contained in the cluster.

Further, although a document is targeted in this embodiment, it can be applied to various types of data such as images that can be searched, classified, and evaluated as data.

It should be noted that the structure and contents of the various data described above are not limited to this, and it goes without saying that the structure and contents are various depending on the intended use and purpose.

Although some embodiments have been described above, the present invention can take an embodiment as, for example, a system, an apparatus, a method, a computer program, a recording medium, or the like, and specifically, a plurality of devices. It may be applied to a system composed of, or may be applied to a device composed of one device.

Further, the computer program in the present invention is a computer program in which a computer can execute the processing methods of the flowcharts shown in FIGS. 8 and 11, and the storage medium of the present invention can execute the processing methods in FIGS. 8 and 11. Computer programs are stored. The computer program in the present invention may be a computer program for each processing method of the devices shown in FIGS. 8 and 11.

As described above, a computer in which a recording medium on which a computer program that realizes the functions of the above-described embodiment is recorded is supplied to the system or device, and the computer (or CPU or MPU) of the system or device is stored in the recording medium. Needless to say, the object of the present invention can be achieved by reading and executing the program.

In this case, the computer program itself read from the recording medium realizes the novel function of the present invention, and the recording medium storing the computer program constitutes the present invention.

Recording media for supplying computer programs include, for example, flexible disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, CD-Rs, DVD-ROMs, magnetic tapes, non-volatile memory cards, ROMs, EEPROMs, etc. Silicon disks, solid state drives, etc. can be used.

Further, by executing the computer program read by the computer, not only the functions of the above-described embodiment are realized, but also the OS (operating system) or the like running on the computer is activated based on the instructions of the computer program. Needless to say, there are cases where a part or all of the actual processing is performed and the processing realizes the functions of the above-described embodiment.

Further, the computer program read from the recording medium is written to the memory provided in the function expansion board inserted in the computer or the function expansion unit connected to the computer, and then its function is based on the instruction of the computer program code. Needless to say, there are cases where a CPU or the like provided in the expansion board or the function expansion unit performs a part or all of the actual processing, and the functions of the above-described embodiment are realized by the processing.

Further, the present invention may be applied to a system composed of a plurality of devices or a device composed of one device. It goes without saying that the present invention can also be applied when it is achieved by supplying a computer program to a system or device. In this case, by reading the recording medium in which the computer program for achieving the present invention is stored into the system or the device, the system or the device can enjoy the effect of the present invention.

Further, by downloading and reading a computer program for achieving the present invention from a server, database, or the like on the network by a communication program, the system or device can enjoy the effect of the present invention.

It should be noted that all the configurations in which each of the above-described embodiments and modifications thereof are combined are also included in the present invention.

100 Information processing device 101 Search condition reception unit 102 Similar search unit 103 Clustering unit 104 Display unit 105 Learning model selection unit 106 Reranking unit 107 Learning data registration unit 108 Learning model generation unit 109 Generation model determination unit 121 Document storage unit 122 Learning Data storage unit 123 Learning model storage unit

Claims (7)

An information processing device that manages multiple learning models.
A reception means that analyzes and classifies the data searched according to the search conditions, presents the group, and accepts the designation of the group.
It is characterized by comprising a first feature data based on the data classified into the designated group and a selection means for selecting a learning model according to the second feature data associated with each of the learning models. Information processing device. The information processing device according to claim 1, wherein the learning model is a learning model for ranking learning that ranks the searched data. The data is a document, and the first feature data and the second feature data are the linguistic features of the data classified into the designated group and the language of the data used for training the learning model, respectively. The information processing apparatus according to claim 1 or 2, wherein the information processing apparatus represents a specific feature. The first feature data and the second feature data are data showing the relationship between the group and the data classified into the group, respectively, and the relationship between the learning model and the data used for learning the learning model. The information processing apparatus according to any one of claims 1 to 3, wherein the data indicates the above. The information processing apparatus according to any one of claims 1 to 4, further comprising a display control means for displaying and controlling the searched data by the selected learning model. It is a control method of an information processing device that manages multiple learning models.
A reception step in which the reception means analyzes the data searched according to the search conditions, presents the classified group, and accepts the designation of the group.
The selection means includes a selection step of selecting a learning model according to the first feature data based on the data classified into the designated group and the second feature data associated with each of the learning models. A control method for an information processing device characterized by. A program that can be executed by an information processing device that manages multiple learning models.
The information processing device
A reception means that analyzes and classifies the data searched according to the search conditions, presents the group, and accepts the designation of the group.
A program for functioning as a selection means for selecting a learning model according to the first feature data based on the data classified into the designated group and the second feature data associated with each of the learning models.

Images