JP2010250439A - Retrieval system, data generation method, program and recording medium for recording program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a retrieval system which equipped with a system that constructs a large-scale and structured knowledge base using a program. <P>SOLUTION: A retrieval system 1 is configured of: a knowledge base 2 having a knowledge block of a tree structure storing a predetermined word included in a book at the top level, and information related with the predetermined word under the top level; a knowledge base retrieval system 4 for retrieving the knowledge block stored in the knowledge base 2 based on the word input from an input part; and a knowledge base construction system 3 for constructing the knowledge block stored in the knowledge base 2 based on the text data read from a data reading means 30 for reading the text data of the book. The knowledge base construction system 3 generates the knowledge block by executing a natural language processing means 31 and a structured formatting means 32 from the text data read from the data reading means 30, and stores the knowledge block in a knowledge base 2. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a search system for searching and presenting target information based on a search term from a database storing document data of a book, a data generation method for generating document data to be stored in the database, and means associated with data generation The present invention relates to a program for realizing the above functions on a computer and a recording medium on which the program is recorded.

  It is not a search system equipped with a knowledge base that systematically aggregates information or a simple text search to accurately find the target information (knowledge) from a database that stores a huge amount of document data such as books. A question answering system has been developed to answer the questioner's question.

  As an example of such a search system, a system that interactively answers a question from a user, in which the user inputs a question sentence, an input analysis unit that is input by the user and parses the language, and a phrase A knowledge base that stores a knowledge unit that is an explanatory sentence of and a dialogue management unit that matches a question sentence input by a user with the knowledge unit, and the knowledge base includes a phrase such as a proper noun. An interactive help system having a structure having at least one of a knowledge unit that is defined at the top level of a tree structure and a knowledge unit that is not categorized is known. (See Non-Patent Document 1).

  In such an interactive help system, as the number of knowledge units stored in the knowledge base increases, the number of options for answers to questions from users increases, and appropriate answers can be presented, so the answer rate for questions increases. .

Interactive help system of Kyoto University Media Center and automatic reference service system of Kyoto University Library. IPSJ report. Natural Language Processing Study Group Report, Vol. 2000, no. 53, p. 92,2000

  However, in the interactive help system described in Non-Patent Document 1, since a worker extracts a corresponding sentence from text data and creates a knowledge unit, it takes a lot of time and effort to construct a large-scale knowledge base. There is such inconvenience. In addition, since the knowledge units are constructed manually, the number of knowledge units stored in the knowledge base is reduced, and there is an inconvenience that the answer rate to questions is lowered.

  The present invention has been made in view of the above problems, and has as its main object to provide a search system equipped with a system for constructing a large-scale and structured knowledge base by a program.

  A search system according to the present invention includes a knowledge base having a predetermined word contained in a book at a top level and having a tree-structured knowledge block in which information related to the predetermined word is stored below the top level, and an input unit A knowledge base search system for searching a knowledge block stored in the knowledge base based on a word input from the input unit, and a data reading means for reading text data of a book, A knowledge base construction system comprising a knowledge base construction system for constructing a knowledge block stored in the knowledge base based on text data read from the data reading means, wherein the knowledge base construction system comprises the reading Perform morphological analysis on the text data read from the means, and use the result of the morphological analysis. Natural language processing means for performing syntax analysis, and structured appearance processing means for extracting sentences based on the syntax analysis of the natural language processing means and classifying the extracted documents into predetermined categories. (Claim 1).

Thereby, since the knowledge block generation process can be automated, the amount of knowledge blocks to be stored in the knowledge base can be increased, and the response rate to the questions can be improved.
Further, since the knowledge blocks are constructed based on the form and structure of the sentence, it can be classified into categories designated in advance, and the search speed can be improved.

  The knowledge block preferably includes at least one of sentences classified into eight predetermined categories (Claim 2).

  Further, a data generation method for generating the knowledge block from text data, the morphological analysis is performed on the text data, and the syntax analysis is performed using the result of the morphological analysis, and the natural data processing step It is desirable to have a structured appearance processing step of extracting a sentence based on the processing result of the language processing step and classifying it into a predetermined category (claim 3).

  The present invention is a program that causes a computer to function as the natural language processing means and the structured appearance processing means.

  The present invention is a computer-readable recording medium in which the program according to claim 4 is recorded (claim 5).

As described above, according to the present invention, it is possible to increase the amount of knowledge blocks stored in the knowledge base by automating the knowledge block generation process, and to provide a search system with an improved response rate. .
Moreover, since the knowledge block is constructed based on the form and structure of the sentence, it can be classified into categories designated in advance, and the amount of work by the operator can be reduced.

FIG. 1 is a configuration diagram showing the configuration of the search system. FIG. 2 shows the configuration of the knowledge block, and FIG. 2A is a configuration diagram showing the configuration of the knowledge block. (B) is a diagram showing classification of categories of knowledge blocks. FIG. 3 is an image diagram illustrating an example of a user interface. FIG. 4 is a processing flow diagram for generating a knowledge block from text data of a book in the knowledge base construction system 3. FIG. 5 is an image diagram showing an example of execution results of morphological analysis. FIG. 6 is an image diagram illustrating an example of the execution result of syntax analysis. FIG. 7 is an image diagram showing an example of sentence extraction by keyword.

  The search system of the present invention will be described below with reference to the drawings.

  As shown in FIG. 1, a search system 1 according to the present invention includes a knowledge base 2 that stores book document data, a knowledge base search system 4 that searches document data stored in the knowledge base 2, and a knowledge base 2. It comprises a knowledge base construction system 3 for constructing stored document data, and a knowledge base management system 5 for managing these knowledge base retrieval system 4 and knowledge base construction system 3.

  The knowledge base 2 stores document data of books by forming a knowledge block 21 having a tree structure as shown in FIGS. 2 (a) and 2 (b).

  The tree structure of the knowledge block 21 has a predetermined word at the top of the tree structure, and there are, for example, definitions (texts of word meanings and explanations of definitions), methods (texts about techniques and usages), examples (examples). (Text about possible or impossible), signs (text about signs such as “can't do it”), reasons (text about reasons), comparison (text about comparing multiple things), others ( There are tags classified into 8 types of sentences) that cannot be classified into these categories. Hereinafter, the text stored in this tag is used as knowledge.

  The knowledge base 2 storing the knowledge block 21 having such a tree structure preferably employs an XML (Extensible Markup Language) database. In particular, the advantage of tree structure and meta information management can be maximized. It is desirable that the XML document be a native XML database that can store and manipulate XML documents that can be stored in the structure.

The knowledge base construction system 3 includes a data reading means 30 for reading book data in a PDF (Portable Document Format) (registered trademark) format, and text for converting the read book data into text by processing such as OCR (Optical Character Reader). The processing unit 33 includes a natural language processing unit 31 that performs morphological analysis on the text-formatted book data and uses the result of the morphological analysis, and includes a predetermined phrase from the parsed book data. A structured appearance processing means 32 for extracting a sentence and classifying the extracted sentence into the above eight categories to generate the knowledge block 21 is configured.
Here, in the present search system 1, the PDF format is handled as an example of the book data, but is not limited to the PDF format, and can be handled in book data of other formats.

The knowledge base search system 4 is a system that searches the knowledge block 21 stored in the knowledge base 2 and returns an answer. The knowledge base search system 4 includes a user interface 41 for inputting a question from the user. It comprises a natural language processing means 42 for performing analysis and syntax analysis, a search algorithm 43 for searching the knowledge block 21, and an answer sentence generation processing means 44 for creating a response sentence to be displayed to the user. The knowledge base search system 4 is preferably a Java (registered trademark) EE application server that operates on a server and specializes in Web access.
As shown in FIG. 3, the user interface 41 includes an input unit that can input a search sentence on the screen.

  The knowledge base management system 5 is a Web system for managing the operation of the knowledge base construction system 3 and includes an upload processing means 51 that delivers book data to the knowledge base construction system 3.

  The natural language processing means 31, the structured appearance processing means 32, the text processing means 33, the natural language processing means 42, the search algorithm 43, the answer sentence generation processing means 44, the upload processing means 51, the unit knowledge described above or later. The additional processing means 53, the full-text search means 55, and the response rate other analysis means 57 generally cause one or a plurality of computers having a CPU, a ROM, a RAM, and the like to function according to a predetermined program. It is constituted by.

  The program can be recorded on a computer-readable recording medium and provided to the user. The recording medium includes a flexible disk, a CD-ROM, a DVD-ROM, or a recordable type medium such as a hard disk or a semiconductor memory.

  A process for generating the knowledge block 21 from the text data of the book in the search system 1 having the above configuration will be described.

  The upload processing means 51 of the knowledge base management system 5 reads the book data in PDF format, and then performs an upload process for transferring the book data to the knowledge base construction system 3.

  As shown in FIG. 4, the data reading means 30 of the knowledge base construction system 3 receives the PDF data (S101: YES), and stores the PDF data in a server (not shown) (S102).

  Next, when receiving the construction start signal from the knowledge base management system 5 (S103: YES), the text processing means 33 reads PDF data from the server and performs text processing (sentence extraction) processing (104).

  Further, the natural language processing means 31 performs natural language processing for analyzing the structure such as dependency for each word on the text-processed book data (S105).

  Further, the structured appearance organization processing means 32 extracts a sentence including a predetermined word based on the result of the natural language processing, and performs structured appearance organization processing for classifying the sentences into categories (S106). A knowledge block 21 is created and stored in the knowledge base 2 (S107).

  Here, the text conversion processing in step S104 causes noise such as page header, number of pages, program list, etc. to enter in some places just by extracting text from PDF data. is there. Therefore, after text conversion, it is preferable to format the text document as much as possible by a program in order to minimize the trouble in analysis by natural language processing described later.

  As specific text document formatting, when reading the book data of the knowledge base construction system 3, the number of pages and the page header causing the noise are deleted from the received book data, Extract text data for each punctuation by combining sentences that are not connected or line breaks in the middle of the text, and because the knowledge of the annotation itself is knowledge that can be used in the construction of the knowledge base, it is also possible to extract annotations Do.

  In the natural language processing in step S105, first, as shown in FIG. 5, morphological analysis is performed on the text-formatted book data to analyze forms such as nouns and case particles for each word used in the sentence. Do. Next, as shown in FIG. 6, the sentence data indicating the dependency for each clause is analyzed with respect to the text data divided for each reading by the morphological analysis.

Here, as a morphological analysis tool, for example, ChaSen (http://chasen.naist.jp/hiki/ChaSen/) developed at Matsumoto Laboratory of Nara Institute of Science and Technology is available. As a syntax analysis tool, for example, CaboCha according to the GNU Lesser Public License (LGPL) can be used.
In addition, it is desirable to execute the natural language process without performing a complicated process as much as possible in order to increase the processing speed.

  Further, in the structured appearance organization process in step S106, as shown in FIG. 7, first, a sentence including a predetermined word (keyword) is extracted from the text subjected to the morphological analysis by the natural language process, Next, category classification is performed on the extracted sentence, and a knowledge block 21 relating to the keyword is constructed.

  Here, as a keyword for extracting a sentence, for example, an index listed at the end of a book can be used. Since some indexes use expression notation that is inappropriate for use as keywords, it is desirable to exclude and organize them. Specifically, there are cases where parentheses are added to the index, such as “Courier (font name)”, and words are written twice, and extraction is performed in a state that includes a plurality of such words. When executed, there is a possibility that the target sentence in which only one word is described may not be extracted, and thus the sentence is extracted without such parentheses.

The classification of categories is performed by collecting the characteristics of sentences for each category, creating a setting file storing rules for classification in advance by operating the knowledge base management system 5, and referring to the setting file. .
Specifically, the sentence classified into the signs category has a feature including a negative sentence form such as “cannot be done”. A file in which the category relationship is set is created, and the extracted sentences are categorized.

As described above, by automating the generation process of the knowledge block 21, the amount of the knowledge block 21 stored in the knowledge base 2 can be increased, and the response rate can be improved.
Further, since the knowledge block 21 is constructed based on the form and structure of the sentence, it can be classified into categories designated in advance, and the amount of work by the operator can be reduced.

  The search of the knowledge block 21 in this search system 1 is demonstrated.

  When a question sentence is input to the user interface 41 of the knowledge base search system 4, first, the natural language processing means 42 analyzes the structure of keywords, main clauses, and the like that are the subject of the question. Next, when the search algorithm 43 extracts the keyword to be questioned and the category of the question sentence, the search algorithm 43 extracts the same category under the knowledge block of the keyword in the knowledge base 2, and further, the search algorithm 43 extracts words other than the question target keyword (unique) An answer sentence formed by the answer sentence generation processing means 44 is presented to the user interface 41 with respect to an answer having a high degree of coincidence from a tree structure that is a result of syntactic analysis and a noun or an adjective.

  It should be noted that browsing means 52 for displaying the data stored in the knowledge base 2 in the knowledge base management system 5, single knowledge addition processing means 53 for adding only single knowledge individually to the knowledge block 21, and storage means 58 The administrator can add knowledge alone by creating a new knowledge block 21 for a question that could not be answered to the user's question.

  Specifically, the storage unit 58 stores the question when asked about knowledge not stored in the knowledge base 2. On the other hand, the unitary knowledge addition processing unit 53 includes a natural language processing unit 54 and a full-text search unit 55, which searches the knowledge block 21 based on the search sentence input by the administrator and is stored. Add knowledge if there was no knowledge. Further, by having the construction rule simulation processing means 56, the structured reference processing means 32 can update the file referred to when categorizing, and it is possible to eliminate the problems that occurred during the extraction.

  Further, the question history can be managed by providing the knowledge base management system 5 with the answer rate and other analysis means 57 for managing the question history and the answer history as means for managing the knowledge base search system 4. Furthermore, by collecting and tabulating evaluations related to the degree of understanding from the user to the response rate and other analysis means 57, it becomes possible for the administrator to add words to knowledge with low evaluation to make it easier to understand.

  Needless to say, the search system described in this embodiment can also be used in a foreign language other than Japanese.

DESCRIPTION OF SYMBOLS 1 Search system 2 Knowledge base 3 Knowledge base construction system 4 Knowledge base search system 5 Knowledge base management system 21 Knowledge block 30 Data reading means 31 Natural language processing means 32 Structured appearance processing means 33 Text processing means 41 User interface 42 Natural language Processing means 43 Search algorithm 44 Answer sentence generation means 51 Upload means 52 Browse means 53 Single knowledge addition processing means 54 Natural language processing means 55 Preamble search means 56 Construction rule simulation means 57 Answer rate other analysis means 58 Storage means

Claims (5)

A knowledge base having a predetermined word included in a book at a top level and having a tree-structured knowledge block in which information related to the predetermined word is stored below the top level;
A knowledge base search system comprising a user interface having an input unit, and searching for a knowledge block stored in the knowledge base based on a word input from the input unit;
A search system comprising a data base for reading text data of a book, and a knowledge base construction system for constructing a knowledge block stored in the knowledge base based on the text data read from the data read means Because
The knowledge base construction system performs a morphological analysis on the text data read from the reading means, and performs a syntax analysis using a result of the morphological analysis, and a syntax of the natural language processing means A retrieval system comprising: structured text processing means for extracting a sentence based on an analysis and classifying the extracted document into a predetermined category.   The search system according to claim 1, wherein the knowledge block includes at least one of sentences classified into eight predetermined categories.   A data generation method for generating the knowledge block from text data, comprising: a natural language processing step of performing morphological analysis on the text data and performing syntax analysis using a result of the morphological analysis; and the natural language processing And a structured appearance processing step of extracting a sentence based on a processing result of the step and classifying the sentence into a predetermined category.   The search system according to claim 1 or 2, wherein a program causes a computer to function as the natural language processing means and the structured appearance processing means.   The computer-readable recording medium which recorded the program of Claim 4.

Images