JP2020161012A - Information processing apparatus, control method and program - Google Patents

Abstract

To provide an information processing apparatus, a control method and a program that can select an appropriate learning model from learning models which are classified to be managed, in order learning or the like in information search.SOLUTION: An information processing apparatus comprises: reception means that receives specification of a classification item; and selection means that selects one category of learning models on the basis of the specified classification item.SELECTED DRAWING: Figure 1

Description

The present invention relates to a document retrieval technique for presenting a document that seems to be appropriate for a specified search condition from a group of documents to be searched.

Conventionally, in order to present appropriate search results to users, the relationship between the search conditions and the terms (morphological analysis, character strings cut out by a certain standard such as N-Gram) included in each document included in the document group is determined. There is a technique to calculate as a statistical value. These techniques are referred to as a similarity search or the like (hereinafter, the technique is collectively referred to as a similarity search in the description of the present invention, and is distinguished from the search by rank learning described later in the present invention).

In addition, the feature amount when the learning data and the document group to be searched are similar is modeled by machine learning, and when a new search condition is specified, the ranking is adjusted based on the learning model to perform a similar search. There is a ranking learning technique that improves the accuracy of.

A large amount of learning data is required for rank learning, but it is difficult to collect the learning data. It is conceivable to collect learning data from the user's search log after starting the operation of similar search as a system, but it may be a burden on the user to evaluate the search results, so it is possible to collect a sufficient amount of logs. could not say it all. Also, before the start of operation, it is limited to learning data created by the developer for testing.

Patent Document 1 is a technique for finding a question (question text of learning data) most similar to an inquiry from a user with respect to a prepared answer (so to speak, a FAQ document group) and returning a corresponding answer. We provide technology to improve topic estimation accuracy even when there are few question sentences.

Specifically, the question sentence of the learning data is expanded by replacing the word appearing in the question sentence of the learning data with the word in the corresponding answer, that is, the number of learning data is increased. In addition, in order to exclude unnatural question sentences from the expanded question sentences, the existence probability of the question sentence is calculated using a stochastic language model, and it is used as training data only when the existence probability exceeds a certain threshold.

JP-A-2017-37588

However, in the technique of Patent Document 1, it is determined whether or not the expanded question sentence is appropriate by using a stochastic language model, but the replaced word is included in the answer prepared in advance. There is a possibility that technical terms or terms specific to an organization are used. In that case, the stochastic language model lacks cases, and the question text may not be expanded appropriately.

Further, the technique of Patent Document 1 aims to enhance the learning effect by expanding the question sentences used as learning data. However, if the number of training data increases, the calculation time required for learning becomes enormous, which may become impractical.

An object of the present invention is to provide a technique capable of selecting an appropriate learning model from the classified and managed learning models in ranking learning and the like in information retrieval.

The present invention is an information processing device that manages a plurality of learning models corresponding to a category determined by a classification item, and is based on a receiving means that accepts the designation of the classification item and the designated classification item. It is characterized by having a selection means for selecting a learning model of a category.

According to the present invention, it is possible to select an appropriate learning model from the classified and managed learning models in ranking learning and the like in information retrieval.

It is a figure which shows an example of the functional structure which concerns on embodiment of this invention. It is a block diagram which shows an example of the hardware configuration applicable to the information processing apparatus 100 which concerns on embodiment of this invention. This is an example of a document to be searched for similarities according to the embodiment of the present invention. This is an example of the classification and the number of documents to be searched according to the embodiment of the present invention. This is an example of a user interface at the time of searching according to the embodiment of the present invention. This is an example of learning data related to the embodiment of the present invention. This is an example of information for determining whether or not to generate a learning model for the category according to the embodiment of the present invention. This is an example of a criterion for determining whether or not to generate a learning model for the category according to the embodiment of the present invention. It is an example of the flowchart explaining the process at the time of learning which concerns on embodiment of this invention. This is an example of a flowchart for explaining one category evaluation process at the time of learning according to the embodiment of the present invention. This is an example of a flowchart for explaining the accuracy evaluation process at the time of learning according to the embodiment of the present invention. This is an example of a storage unit that stores the learning model according to the embodiment of the present invention in association with the category. This is an example of a flowchart for explaining the search process according to the embodiment of the present invention. This is an example of a flowchart for explaining a process of selecting a category at the time of searching according to the embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

In the present invention, the search result of the conventional document is redesignated by machine learning, and the search order is redesignated by using machine learning. This is called ranking learning. In particular, in the present invention, for convenience of explanation, the process of determining the learning model in advance is "generation of the learning model", and the search results based on the search conditions such as the user are redesignated by using the generated learning model. We will call the process "reranking".

FIG. 1 is a diagram showing an example of a functional configuration according to an embodiment of the present invention. This function configuration can be broadly divided into a learning function and a search function.

The learning category determination unit 101 learns from the learning data (user log such as search conditions and selection of correct answer) stored in the learning data storage unit 121 and the information of the search target document stored in the search target document storage unit 123. It is a functional part that determines the document category that generates the model. The criteria for evaluating which category is to be learned is set in the learning execution condition storage unit 122, and is referred to by the learning category determination unit 101. In addition, information associated with each category to be evaluated is stored in the document / learning status storage unit 700.

The learning model generation unit 102 generates a learning model for each category of the learning target determined by the learning category determination unit 101, and stores the learning model in the learning model storage unit 124. At the time of learning, the learning model generation unit 102 calls the similar search unit 103 to search the search target document storage unit 123.

The search condition reception unit 111 is a functional unit for receiving search conditions from a user who uses the search process via the user interface or from another application.

The category acquisition unit 112 refers to the search condition received by the search condition reception unit 111, the learning model storage unit 124, and the document / learning status storage unit 700, and executes re-ranking using the learning model of any category. To decide.

The re-ranking unit 113 passes the search conditions received by the search condition receiving unit 111 to the similar search unit 103, and the similar search unit 103 executes a similar search on the search target document storage unit 123 based on the conditions. .. Further, the re-ranking unit 113 refers to the learning model storage unit 124, and re-ranks the search results using the learning model corresponding to the determined category.

The result presentation unit 114 presents the search result to the user interface or the application that called the search function of the present invention.

However, the category acquisition unit 112 may not be able to determine the category. In that case, the similar search result of the similar search unit 103 is presented as it is without re-ranking.

FIG. 2 is a block diagram showing an example of a hardware configuration applicable to the information processing apparatus 100 according to the embodiment of the present invention.

As shown in FIG. 2, the information processing apparatus 100 includes a CPU (Central Processing Unit) 201, a RAM (Random Access Memory) 202, a ROM (Read Only Memory) 203, an input controller 205, and a video controller 206 via the system bus 204. , Memory controller 207, communication I / F controller 208, etc. are connected.

The CPU 201 comprehensively controls each device and controller connected to the system bus 204.

Further, the ROM 203 or the external memory 211 will be described later, which is necessary for realizing the functions executed by the BIOS (Basic Input / Output System) and the OS (Operating System) which are the control programs of the CPU 201, and each server or each PC. Various programs etc. are stored. In addition, information necessary for carrying out the present invention is stored. The external memory may be a database.

The RAM 202 functions as a main memory, a work area, and the like of the CPU 201. The CPU 201 realizes various operations by loading a program or the like necessary for executing the process from the ROM 203 or the external memory 211 into the RAM 202 and executing the loaded program.

Further, the input controller 205 controls input from a pointing device such as a keyboard (KB) 209 or a mouse (not shown).

The video controller 206 controls the display on a display such as the display 210. The display may be a display such as a liquid crystal display. These are used by the administrator as needed.

The memory controller 207 is an external storage device (hard disk (HD)) for storing boot programs, various applications, font data, user files, edit files, various data, etc., a flexible disk (FD), or a PCMCIA (Personal Computer). Controls access to external memory 211 such as Compact Flash® memory connected via an adapter to the Memory Card International Association card slot.

The communication I / F controller 208 connects and communicates with an external device via the network, and executes communication control processing on the network. For example, communication using TCP / IP (Transmission Control Protocol / Internet Protocol) is possible.

The CPU 201 can display the outline font on the display 210 by executing the outline font development (rasterization) process on the display information area in the RAM 202, for example. Further, the CPU 201 enables a user instruction by a mouse cursor (not shown) or the like on the display 210.

Various programs described later for realizing the present invention are recorded in the external memory 211, and are executed by the CPU 201 by being loaded into the RAM 202 as needed.

FIG. 3 is an example of a document to be searched according to the embodiment of the present invention. As an example, a collection of FAQs (Frequently Asked Questions) used to support software products is described, but the documents to be selected in the present invention include texts such as papers, newspaper articles, company regulations, and posts on SNS. Anything is acceptable as long as it is available, and it is not limited to FAQ.

In the examples described in 300a and 300b, one FAQ includes "FAQ" that uniquely identifies the FAQ, "product type" and "query classification" that classify the contents of the entire FAQ, and further support contents, for example. It consists of fields such as "question" that serves as a guide as to which FAQ should be referred to in response to the user's inquiry of the software product, and "answer" that describes the response, and describes the contents corresponding to each field. Including. Here, the FAQ ID associates a character string of a query with a specific FAQ in the learning data described later, and is not limited as a field name. The names and numbers of other fields are not limited.

FIG. 4 is an example of the classification of search target documents and the number of documents according to the embodiment of the present invention. The set of documents shown in FIG. 3 is stored in the search target document storage unit 123, and the search target document storage unit 123 does not need to be classified administratively, for example, by storing in a different database. It is in a state where it can be classified by fields such as product type and inquiry classification in FIG.

As in the embodiment of the present invention, when the product is classified by two systems, product type and inquiry classification, the table is two-dimensional as shown in FIG. 4, but it goes without saying that any dimension may be used. In addition, these classifications correspond to the narrowing conditions of documents, and these are referred to as categories. It is assumed that the numerical value described in each cell of the table is the number of documents in the document group corresponding to the category narrowed down by the two conditions of product type and inquiry classification. Even if one of them is narrowed down, the corresponding set of documents is set as a category.

First, the case of classifying by product type will be described. The collection of documents is divided into three groups: individuals, corporations, and educational institutions. By specifying one of these three (for example, "for individuals"), the total number of documents in the vertical direction is the number of documents when narrowed down by "product type = for individuals".

On the other hand, a case of classifying by inquiry classification will be described. The set of documents is classified into operating procedures, troubles, product information, ... (Omitted), and license groups. By designating one of these (for example, "license"), the total number of documents in the horizontal direction is 200 documents when narrowed down by "query classification = license". Further narrowing down by product classification, the cells in which the two classifications intersect are 50 documents for "individual", 100 documents for "corporate", and 50 documents for "educational institution".

As in this example, the set of documents narrowed down by "product type = for individuals & inquiry classification = license" is the set of documents narrowed down by either "product type = for individuals" or "inquiry classification = license". It becomes a subset. In the present invention, when there is an inclusion relationship as a set, a large set narrowed down by "product type = for individuals" or "inquiry classification = license" is classified as a higher category, and two conditions "product type = for individuals & inquiry classification =". A small set narrowed down by "license" is sometimes called a subcategory. The entire document is in the top category. Therefore, in this two-dimensional example, three category hierarchies are created, but when narrowed down by more conditions, it is composed of more hierarchies.

In the above explanation, one system is narrowed down by one value such as "query classification = license", but these values are not determined from the table of FIG. 4, and the learning data described in FIG. 6 is used. It is due to. For example, if there is learning data using the OR condition such as "query classification = (product information OR license)", the narrowing condition becomes one category, and the categories are "query classification = product information" and "query classification". = It is a higher category of "License". In this case, it goes without saying that even if the table of FIG. 4 is two-dimensional, the hierarchy may be three-dimensional or more.

FIG. 5 is an example of a user interface at the time of searching according to the embodiment of the present invention. In the embodiment of the present invention, the user interface is exemplified for explaining the method of collecting learning data.

On the search condition input screen 501, the user inputs a question sentence (search condition) (502). It is also possible to specify the narrowing conditions. As described with reference to FIG. 4, the document set to be searched is narrowed down by the narrowing conditions, and the effect of relatively improving the accuracy can be obtained. In this example, the selection list 505 may be displayed by pressing the selection button 504 in the fields of "product type" (503) and "query classification" (506), and may be selected from the list. However, narrowing down is not an essential condition. You may input from the keyboard, and this example does not limit the input method. When the search button 507 is pressed after inputting the search conditions, the search conditions are sent to the search condition reception unit 111 of the information processing apparatus 100, and a search result list (not shown) is displayed.

When the user selects a document whose details are to be viewed from the search result list, the document viewing screen 511 is displayed. Further, the search conditions specified by the user are displayed in the columns 512 and 513.

If the document displayed on the document viewing screen 511 is the information requested by the user, the user can press the button 515 to notify the information processing device 100 and store it in the learning data storage unit 121. The format of the data stored in the learning data storage unit 121 will be described with reference to FIG.

FIG. 6 describes in detail an example of the data format of the learning data storage unit 121, and one line represents one learning data. For example, when the button 515 is pressed on the document viewing screen 511 of FIG. 5, the information of the line indicated by 607 is registered.

The learning data is composed of a question sentence 601 that is a character string input when actually searching, a FAQID 602 (information that identifies a document) that is a correct document as the information requested by the user, and a narrowing condition 603. The narrowing conditions are blank, such as 604 when narrowed down by product type, 605 when narrowed down by product type and inquiry classification, and 606 when all documents are searched without narrowing down.

Further, since FIG. 6 shows only the learning data, the FAQID 602 always has a value, but if the button 515 is not pressed (the correct answer is not presented), it is left blank and the search log that does not become the learning data is displayed. You may register in the same table. Even in that case, it is possible to collect information on the category, such as what narrowing conditions are specified for the information frequently inquired by the user.

The explanation is returned to the document viewing screen 511. The search log may be notified to the information processing apparatus 100 and stored regardless of whether or not the button 515 is pressed.

These screens are just examples. For example, a user interface that allows each of a plurality of search results to be checked on the search result list screen is provided so that learning data can be specified without checking the details on the document viewing screen 511. May be good.

FIG. 7 is an example of information for determining whether or not to generate a learning model for the category according to the embodiment of the present invention. A table created by aggregating the learning data storage units 121 described with reference to FIG. 6 is shown, and is stored in the document / learning situation storage unit 700.

Evaluate each category based on the numbers shown in this table and the learning execution range 802 shown in Fig. 8, and instead of learning all categories, focus on the categories that are considered to have a high operational learning effect. The purpose is to shorten the learning time. After prioritizing, it is conceivable to estimate the learning time from the number of learning data and learn only the categories that can be learned within a certain time (for example, 6 hours from midnight to 6 am). ..

One line in FIG. 7 shows the category shown on the far left (corresponding to the narrowing condition), and the values aggregated for that category are arranged on the right for each item. This numerical value is compared with each threshold value shown in the learning execution condition storage unit 122 of FIG. 8, and it is determined whether or not learning should be performed for the category. In addition, since the illustrated table is partially omitted and does not describe information on all categories, the numerical values in the following explanations are used for calculations even for items not described in this table. ..

First, the category specified by "Product type = For corporations" will be described as an example. The number of documents included in the same category (and subcategories) is 1,500, which is in the range of 200 to 5,000 under the condition of FIG. 8, so "○" indicating that it is better to study is given. Similarly, the number of searches (total of the same category and lower categories) is 20,000, and "○" is given.

The coverage rate of the number of searches is a numerical value indicating the ratio of the number of searches included in the search log "when the learning model is not generated" in the same category "is executed without learning". That is, even if the same category is not learned, if it is further narrowed down by a lower category (for example, 704), there is a corresponding learning model, so that a highly accurate search is possible. However, if you search by "product type = for corporations" without narrowing down to the lower categories, the accuracy will decrease because there is no learning model. If the ratio of the number of searches exceeds the “coverage rate of the number of searches” (30%) in Fig. 8, it is judged that it is better to learn the same category even if all the lower categories have been learned, and “○” is displayed. Give. Next, the amount of training data and the coverage rate of the amount of training data are also determined in the same manner.

Finally, "accuracy (average ranking)" is information based on how many places in the search result list the information desired by the search user is displayed when the search is performed. That is, the user presses the button 515 in FIG. When pressed, how many documents the document was may be stored in a column (not shown) of the learning data storage unit 121 of FIG. 6. That is, how much the learning effect has been the search result since the previous learning. It will be judged for each learning model whether or not to perform new learning based on whether or not it appears as the ranking of.

For example, 703 to 705 in which the correct answer is within the 20th place even in the unlearned state may lower the priority of learning, so "○" is not given. On the other hand, like 701, it is 24th on average when not learning, but if learning improves 13th on average, it is considered that there is a learning effect and "○" is given. ..

If there is a "-" in the same column, there is no accuracy evaluation after learning because it is a category that was not done at the time of the previous learning. The handling in this case is a design matter and there are various implementations. For example, by evaluating the ranking of "unlearned" and the "improvement difference during learning", the number of "○" to be given is 0 to 0. There are two and three steps available.

It is also possible to train a certain percentage of the latest learning data with temporary learning data and the rest as evaluation data with temporary learning data, and actually evaluate whether or not the evaluation data has a learning effect. Good. However, since the purpose of the present invention is to improve the efficiency of learning in the first place, learning only for evaluation is contrary to the purpose. Therefore, as an evaluation result other than accuracy among the above evaluations, the accuracy may be evaluated only when a plurality of categories have the same priority and only some of the categories must be selected.

Another example of determining whether or not to learn will be described. This is the category corresponding to "query classification = license" in line 702. In the same category, the coverage rate of the number of searches is 0%. This is a numerical value when a learning model of a lower category (703 to 705) is generated. That is, although the number of searches is 10,000, all of them correspond to subcategories, and there is no case where the search is performed by narrowing down only "query classification = license". Therefore, "○" is not given because the same category does not need to be learned. However, in the lower categories (703 to 705), for example, the number of documents is small, and the ranking is good even when unlearned, so that "○" is not given in other evaluations and there is a possibility that the documents will not be learned as a result. In that case, the coverage rate of "query classification = license" in line 702 is evaluated again, and if all the subcategories are not learned, the coverage rate is 100%, and "○" is added to this column to give priority. Needs to be reassessed. The process of re-evaluating the priority is determined in step S911 of the flowchart of FIG. That is, if there is a change in the numerical value of FIG. 7 used for the evaluation, the evaluation may be performed again.

Further, although FIG. 8 shows the number of documents, the number of searches, the number of learning data, and the number of cases, it may be a ratio of the total number. For example, immediately after the start of operation in which there are only 1,000 learning data in total, the number of learning data in each category does not satisfy the condition. In that case, it may be calculated as a percentage. Further, even if the ratio of data in a specific category is large by using the number, the number of times, and the ratio together, it may be considered that the threshold value or more as an absolute number is necessary, and learning may not be performed at all.

In any case, it is only one example showing the embodiment of the present invention, and various information may be combined for judgment. Further, in FIG. 7, when the condition is satisfied, "○" is given and it is determined whether or not to learn by the number, but a weight 803 is given to each item or a calculation formula is prepared. The score may be calculated. In that case, for example, the case where the number of training data is 10,000 may be set as the highest score, and the score may become worse as the boundary (minimum 5,000, maximum 20,000) in 802 of FIG. 8 is approached.

This completes the description of FIGS. 7 and 8.

FIG. 9 is an example of a flowchart for explaining the process at the time of learning according to the embodiment of the present invention. Perform learning for categories with high priority based on learning data. Each step of the flowchart of FIG. 9 is executed by the CPU 201 on the information processing apparatus 100.

In step S901, the learning data is read from the learning data storage unit 121. Here, search logs that are not learning data (those for which the correct FAQID is not specified) may be read (to confirm the number of searches, etc.).

In step S902, the table described in the document / learning status storage unit 700 is generated using the learning data (including the search log) read in step S901. However, the column of "number of ○" will be given according to the evaluation in the subsequent steps. The accuracy (average rank) column is also blank at this point when actually learning in FIG. 11, which will be described later.

Steps S903 to S906 are processes that are repeatedly executed for all the rows corresponding to the categories registered in the table described in step S902. At this point, since it has not been evaluated at all, all categories that have even one registration in the learning data (search log) are targeted.

In step S904, one category is taken out from the table and set as the category of interest, and in step S905, the evaluation corresponding to the category is executed (described later in FIG. 10). In the evaluation, the learning execution condition storage unit 122 described with reference to FIG. 8 is referred to.

In step S907, the category to be actually learned is selected according to the number of “◯” assigned to the evaluated category (or the score described with reference to FIG. 7).

In step S908, it is confirmed whether or not the information in FIG. 7 has been changed by selecting the category in step S907. For example, if it is decided not to learn the categories corresponding to 703 to 705, the search count coverage rate and the learning data coverage rate of the category corresponding to 702, which is a higher category common to the three categories, are changed. That is, since none of the lower categories is learned, both coverage rates are 100%, and the learning execution priority of the upper category 702 changes. If the priority has changed (YES), the process returns to step S907 and the learning category is selected again. If it does not change (NO), the process proceeds to step S909.

Step S909 generates a learning model for the selected category and stores the training data as described later in FIG.

FIG. 10 is an example of a flowchart for explaining one category evaluation process at the time of learning according to the embodiment of the present invention. Each step of the flowchart of FIG. 10 is executed by the CPU 201 on the information processing apparatus 100. The categories for evaluation are specified in FIG.

In step S1001, it is determined whether or not the number of searches and the total value described in the corresponding line in FIG. 7 are within the range of the learning execution range 802 specified by the number of documents in FIG. If it is within the range, the process proceeds to step S1002 and "○" is given. If it is not in the range, the process proceeds to step S1003.

In step S1003 and step S1004, "○" is added according to the determination of the number of searches.

In step S1005 and step S1006, "○" is assigned according to the determination of the number of searches and the coverage rate.

In step S1007 and step S1008, "○" is added according to the determination of the amount of training data and the whole.

In step S1009 and step S1010, "○" is assigned according to the determination of the amount of training data and the coverage rate.

In step S1011, the priority of the category to be learned is determined by the number of “◯” given up to the previous step (or the score described in FIGS. 7 and 8).

In step S1012, the time required for learning each category is estimated. Specifically, it is possible to estimate the learning time from the amount of learning data shown in FIG. This estimation may be incorporated in this step as a calculation formula, or the estimated time according to the amount of learning data may be manually prepared in a table or the like and referred to. After estimating the time, if it is specified to study in the time allowed for learning, for example, 6 hours from midnight to 6 am (setting file not shown), the category with the highest priority After the category that exceeds 6 hours in total, it is assumed that it is not a learning target.

In step S1013, it is determined whether or not to execute the accuracy evaluation, and if so, to which category. For example, depending on the number of “○” mentioned above, there is a category in which the judgment of whether or not to learn does not change regardless of the result of the accuracy evaluation. On the other hand, depending on the accuracy evaluation, the learning target tentatively decided in the previous section may be replaced. In that case, the evaluation should be made for the minimum category in which the evaluation may be replaced. Since learning for evaluation itself takes time, the category to be learned for evaluation is determined in consideration of the time. If there is a category for accuracy evaluation, the process proceeds to step S1014. If not, the process of this flowchart is completed, and the process returns to the flowchart of FIG. The explanation of step S1014 will be described later with reference to FIG.

In step S1015 and step S1016, "○" is given according to the unlearned rank of accuracy (average rank).

In step S1017 and step S1018, “◯” is given according to the difference in improvement in accuracy (average rank) during learning.

FIG. 11 is an example of a flowchart for explaining the accuracy evaluation process during learning according to the embodiment of the present invention. Each step of the flowchart of FIG. 11 is executed by the CPU 201 on the information processing apparatus 100.

In step S1101, it is determined whether or not to learn with the latest learning data (learning data storage unit 121). This determination may be set manually, for example, and may be determined in advance as to whether or not to execute learning as an evaluation. A discriminant may be prepared and the information processing apparatus 100 may make a judgment. As an example of the discriminant, the time required for learning for evaluation can be estimated from the amount of learning data, and it can be processed in the possible learning time (for example, the above-mentioned 6 hours) together with the actual learning time after the evaluation is completed. It may be judged by whether or not. If it is determined that learning is to be performed, the process proceeds to step S1101. If it is determined not to do so, the process proceeds to step S1105.

In step S1102, a certain criterion is used for using a part of all the training data as a purpose for generating a learning model for evaluation and another part as a purpose for evaluating the effect of the learning model. Divide by. For example, each may be randomly divided into 50%. Further, the data for learning and evaluation may be reduced depending on the time available for evaluation.

In step S1103, a learning model for evaluation for the category of interest is generated by using the learning data divided into the purposes for learning.

In step S1104, the category of interest is actually searched using the learning data divided into the objectives for evaluation, and the order of correct answers included in the results of the similar search unit 103 and the reranking unit 113 is determined. Acquire and evaluate. These evaluation results are described in the columns of the accuracy (average rank) of FIG. 7 when not learned and when improved during learning.

In step S1105, based on the existing learning data in FIG. 7, it is estimated that the tendency to improve the ranking from the previous learning model does not change significantly even if the latest learning model is used, and the accuracy evaluation result is obtained. is there. That is, although the accuracy is evaluated, the past achievements are used without the learning processing for evaluation and the evaluation processing by searching and re-ranking. In actual operation, when a search condition is input, the re-ranking unit 113 ranks the final search result (if there is a learning model) for the search result in the similar search unit 103. The ranking of "unlearned" and "learned" can be obtained by one search. With the above processing, the flowchart of FIG. 11 is completed, and the process returns to the processing of FIG.

Although not specified in the flowchart, it is not always necessary to use all the learning data. For example, when the information processing apparatus 100 of the present invention is operated for a long period of time, learning data from several years ago remains. However, the learning data may be used in the embodiment of the present invention only for those within one year by updating the user's search conditions and the registered documents. This completes the description of FIG.

FIG. 12 is an example of a storage unit that stores the learning model and the category according to the embodiment of the present invention in association with each other. The learning model 1202 is used in the reranking unit 113 of the search process described later, but the learned category table 1201 is also stored (may be another storage unit) in order to associate with the narrowing condition specified in the search condition.

In the association between the narrowing condition and the category, the order of the field names and the value of the condition described in the narrowing condition may be changed. That is, the learning model of the category corresponding to "Product type = Personal & Inquiry classification = (Product information OR license)" is the order in which "Product type" and "Inquiry classification" are specified in narrowing down the search conditions, and the inquiry classification. If the OR value does not depend on the specified order of "product information" and "license" (even if the order is different) and the conditions are the same, the association can be performed.

This completes the description of the embodiment of the process for generating the learning model in the present invention. Next, the search process using the generated learning model will be described with reference to FIGS. 13 and 14.

FIG. 13 is an example of a flowchart for explaining the search process according to the embodiment of the present invention. FIG. 14 is an example of a flowchart illustrating a process of selecting a category at the time of searching according to the embodiment of the present invention. Each step of the flowchart of FIGS. 13 and 14 is executed by the CPU 201 on the information processing apparatus 100.

In step S1301, the search condition is accepted from the user or from another application through the user interface of FIG. As described with reference to FIG. 5, the search condition specified by the user includes a question sentence and a narrowing condition. If there is no narrowing condition here, it may be assumed that the category "all documents" indicating the set of the entire document corresponds to it.

In step S1302, the narrowing-down condition portion is extracted from the search conditions received in step S1301 and the category is specified. As explained in FIG. 12, the order of description is not essential in narrowing down the search conditions, so it is not considered, and if the actual narrowing down points to the same document group in the processing described later, the same category is supported. Therefore, it is assumed that the learning model 1202 of the corresponding category of FIG. 12 can be acquired. Even if there is no learning model, the narrowing conditions themselves correspond to the categories. The category is referred to as a "starting point category". That is, it is a category explicitly set by the search user.

Step S1303 determines the category to be searched. Details will be described with reference to the flowchart of FIG.

In step S1401, it is determined with reference to FIG. 12 whether or not there is a learning model corresponding to the starting point category. If there is a learning model (YES), the process proceeds to step S1405, and it is specified to re-rank using the learning model of the starting point category, that is, the learning model suitable for the narrowing down specified by the user. If there is no learning model (NO), the process proceeds to step S1402.

In step S1402, among the categories higher than the starting point category, all the categories having the learning model are listed.

When there is no learning model in the upper category, the search result is the same because the search is finally narrowed down by the narrowing condition included in the search condition and the ranking is not adjusted in the reranking unit 113 after that. If there is a learning model, it is presumed that it should be prioritized because it has a learning effect because it is re-ranked by the learning model of the upper category after searching the same document group under the user narrowing condition. Therefore, it is only necessary to list the upper categories that have a learning model.

In step S1403, among the upper categories, the category with the highest accuracy is selected with reference to the accuracy (average rank) of FIG. If there is no superiority or inferiority in a plurality of categories, the one with the smallest number of documents may be selected.

In step S1404, it is determined whether or not there is a selected category. For example, in step S1402, it is conceivable that there is no selected category if there is no upper category in which the learning model is generated in the first place. Even if there is a learning model corresponding to the higher category, since the learning model is generated for the document group for a wider range than the starting category, the document group narrowed down by the narrowing conditions corresponding to the starting category. On the other hand, there is no learning effect, and there is a possibility that the accuracy will decrease. In that case, do not select the upper category. If there is no higher category (NO), the process proceeds to step S1405, and the starting category is searched. However, there is no learning model (processing in this case will be described in step S1305 of FIG. 13). If there is a selected supercategory (YES), it is determined to proceed to step S1406 and rerank using the learning del of the selected superordinate category. Here, the description of the flowchart of FIG. 14 is completed, and the story returns to the place where the process of step S1303 of FIG. 3 is completed.

In step S1304, the search target document storage unit 123 is searched by the similar search unit 103 under the narrowing conditions received in step S1301. For example, even if a learning model corresponding to a higher category is used, it is more accurate for the similar search unit 103 to narrow down by the starting point category with the strongest narrowing condition and search from a small number of documents. .. Even if it is re-ranked using the learning model in a later step, it is better that the accuracy is high in the similar search.

In step S1305, it is determined whether or not the learning model exists. That is, in FIG. 14, when it is decided to re-rank in the starting category, the learning model is stored in the starting category, and when it is decided to be re-ranked in the higher category, the corresponding learning model in the higher category is stored in the learning model storage unit. Search from 124.

In step S1306, the search results acquired by the similar search in step S1304 are re-ranked.

In step S1307, the similar search result of step S1304 and, if re-ranking is performed, the result of re-ranking in step S1306 are presented to the caller. This completes the description of the processing of the flowcharts of FIGS. 13 and 14.

It should be noted that the structure and contents of the various data described above are not limited to this, and it goes without saying that the structure and contents are various depending on the intended use and purpose.

Although some embodiments have been described above, the present invention can take an embodiment as, for example, a system, an apparatus, a method, a computer program, a recording medium, or the like, and specifically, a plurality of devices. It may be applied to a system composed of, or may be applied to a device composed of one device.

Further, the computer program in the present invention is a computer program in which a computer can execute the processing methods of the flowcharts shown in FIGS. 9 to 11, 13 and 14, and the storage medium of the present invention is FIGS. 9 to 11 and FIGS. A computer program in which a computer can execute the processing methods of 13 and 14 is stored. The computer program in the present invention may be a computer program for each processing method of the devices of FIGS. 9 to 11, 13, and 14.

As described above, a computer in which a recording medium on which a computer program that realizes the functions of the above-described embodiment is recorded is supplied to the system or device, and the computer (or CPU or MPU) of the system or device is stored in the recording medium. Needless to say, the object of the present invention can be achieved by reading and executing the program.

In this case, the computer program itself read from the recording medium realizes the novel function of the present invention, and the recording medium storing the computer program constitutes the present invention.

Recording media for supplying computer programs include, for example, flexible disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, CD-Rs, DVD-ROMs, magnetic tapes, non-volatile memory cards, ROMs, EEPROMs, etc. A silicon disk, solid state drive, or the like can be used.

Further, by executing the computer program read by the computer, not only the functions of the above-described embodiment are realized, but also the OS (operating system) or the like running on the computer is activated based on the instructions of the computer program. Needless to say, there are cases where a part or all of the actual processing is performed and the processing realizes the functions of the above-described embodiment.

Further, the computer program read from the recording medium is written to the memory provided in the function expansion board inserted in the computer or the function expansion unit connected to the computer, and then its function is based on the instruction of the computer program code. Needless to say, there are cases where the CPU provided in the expansion board or the function expansion unit performs a part or all of the actual processing, and the processing realizes the functions of the above-described embodiment.

Further, the present invention may be applied to a system composed of a plurality of devices or a device composed of one device. It goes without saying that the present invention can also be applied when it is achieved by supplying a computer program to a system or device. In this case, by reading the recording medium in which the computer program for achieving the present invention is stored into the system or device, the system or device can enjoy the effect of the present invention.

Further, by downloading and reading a computer program for achieving the present invention from a server, database, or the like on the network by a communication program, the system or device can enjoy the effect of the present invention.

It should be noted that all the configurations in which each of the above-described embodiments and modifications thereof are combined are also included in the present invention.

100 Information processing device 101 Category determination unit 102 Learning model generation unit 103 Similar search unit 111 Search condition reception unit 112 Category acquisition unit 113 Reranking unit 114 Result presentation unit 121 Learning data storage unit 122 Learning execution condition storage unit 123 Search target document Storage unit 124 Learning model storage unit 700 Document / learning situation storage unit

Claims (9)

An information processing device that manages multiple learning models corresponding to the categories determined by the classification items.
A reception means that accepts the designation of the above classification items,
An information processing apparatus including a selection means for selecting a learning model of one category based on the designated classification item. The information processing apparatus according to claim 1, wherein the selection means selects the learning model in a category determined by the designated classification item when the learning model exists. The information according to claim 1 or 2, wherein the selection means selects another learning model including the category when the learning model of the category determined by the designated classification item does not exist. Processing equipment. The information processing device according to claim 3, wherein the selection means selects a learning model of one category based on the number of learning data used when creating a learning model for each category. It is equipped with an evaluation means for evaluating the learning effect of the learning model for each category.
The information processing apparatus according to claim 3, wherein the selection means selects a learning model of one category based on the evaluated learning effect. The information processing device searches for a document by a similar search based on a search text, and re-ranks and outputs the searched document using a learning model selected by the selection means. The information processing device according to any one of claims 1 to 5. The information processing device according to claim 6, wherein the information processing device outputs a document searched by the similar search when a learning model of one category is not selected by the selection means. It is a control method of an information processing device that manages a plurality of learning models corresponding to a category determined by a classification item.
A reception step in which the reception means accepts the designation of the classification item,
A control method of an information processing apparatus, wherein the selection means includes a selection step of selecting a learning model of one category based on the designated classification item. It is a program that can be executed in an information processing device that manages multiple learning models corresponding to the categories determined by the classification items.
The information processing device
A reception means that accepts the designation of the above classification items,
A program for functioning as a selection means for selecting a learning model of one category based on the specified classification item.

Images