JPH03172966A - Similar document retrieving device - Google Patents

Abstract

PURPOSE:To efficiently perform retrieval with a high precision by providing a document input part, an index extracting part, etc., to retrieve a similar document without setting a keyword, a logical formula, or the like and developing indexes as a thesaurus. CONSTITUTION:Document information inputted from the external is converted to code information adapted to this device by a document input part 1, and a sentence inputted from the document input part 1 is divisionally written and is divided into word units by a modification analyzing part 2, and modification analysis of individual words is performed. The degree of resemblance of documents is judged in consideration of degrees of importance of indexes given by an index extracting part 3 and semantic closeness of synonyms developed as a thesaurus and in accordance with the degree of resemblance of modification structures. Thus, similar documents are efficiently retrieved from the data base of a full text with a high precision.

Description

DETAILED DESCRIPTION OF THE INVENTION [Industrial Application Field 1] The present invention relates to a similar document retrieval device that can efficiently and highly accurately retrieve similar documents from a full-text database.

[Prior Art] Various systems have been proposed and implemented with the aim of constructing information retrieval systems. In these systems, the search methods for retrieving the necessary information from the database include entering the document number or document name, searching for keywords assigned to each document, and searching for all characters in the document. has been taken.

For example, in the patent text search system (PATOLIS), the patent number (application number, publication number, publication number, etc.)9
Patent documents can be searched by bibliographic information such as international classification number and applicant. In addition, the purpose of the invention, industrial application field, effect, structure, etc. are registered as free keywords for each patent, and if you enter the combination of the keyword string and logical formula (AND, OR, NOT) that you want to search, You can search for the desired patent document by using .

The last method is called full-text search, which outputs documents that have words that match the keywords entered by the user. In this case, the search results may be output in the order of search or sorted and output in order of the number of keyword matches.

[Problems to be Solved by the Invention] In the prior art described above, even if a large number of keywords and logical expressions are combined in a complex manner, it is not always possible to obtain a desired document. In addition, keyword matching methods include direct keyword matching, logical expressions (AND, OR), etc.

NOT, etc.) and compound conditional expressions (neighborhood condition 2 context condition) are used, but in many cases, the required document is searched during the search, and it is necessary to manually search for the desired document from a large amount of search results. There were problems such as not being able to do so.

This invention was made to solve the above problems, and an object of the present invention is to provide a similar document search device that can efficiently and accurately search for similar documents from a full-text database. .

[Means for Solving the Problems] A similar document retrieval device according to the present invention includes a document input section that allows a sentence to be searched to be inputted directly from the outside, and a document input section that separates the input document and calculates the relationship between clauses from morpheme information. A dependency analysis unit that analyzes dependency relationships, an index extraction unit that determines sentence structure from the dependency analysis results, extracts an index from this sentence structure, and assigns importance, and morpheme information and dependency of the input document. a document storage unit that stores received information and index information;
A Socirus expansion unit that extracts synonyms and synonyms of the index created by the index extraction unit from the thesaurus dictionary, a similar document search unit that calculates the similarity of sentence structures using the index and its synonyms and synonyms as keywords, and a similar document search unit. The apparatus also includes a similar document output section that outputs the documents searched by the section on the screen in descending order of similarity.

[Operation 1] In this invention, since the document to be searched is directly input as input, it is possible to search for similar documents without setting keywords or logical formulas. In addition, the importance of the index assigned by the index extraction part, the synonyms expanded in the thesaurus,
Since the degree of similarity of documents is determined based on the degree of similarity of dependency structures in consideration of the closeness of the meanings of synonyms, it is possible to search for similar documents with higher precision than in the past.

〔Example〕

Examples of the present invention will be described below.

First, the overall configuration of the similar document search device of the present invention will be described.

FIG. 1 is a block diagram showing the configuration of an embodiment of the present invention. 1 is the document input section, which inputs document information (
image information, code information) into code information suitable for this device. Reference numeral 2 denotes a dependency analysis section which separates the sentence input from the document input section 1 and divides it into word units. Furthermore, dependency analysis is performed by adding part-of-speech information, conjugation form, and morpheme information to each word. 3 is an index extraction unit that extracts an index and assigns importance to the extracted index.

The importance of an index is determined from the syntax information obtained from the dependency information of the dependency analysis unit 2. 4 is a document storage unit that stores morpheme information, dependency information, index information, and the like. 5 is a thesaurus expansion section, which contains synonyms of the index extracted by the index extraction section 3;
Extract synonyms from the thesaurus dictionary. 6 is a similar document search unit that calculates the degree of similarity of documents using the index extracted from the index extraction unit 3, the synonyms and synonyms acquired by the thesaurus expansion unit 5, and the dependency relationships from the dependency analysis unit 2. . 7
A similar document output section outputs the documents searched by the similar document search section 6 on the screen in descending order of similarity.

FIG. 2 shows a block diagram of the system configuration of the similar document search device of the present invention. The character recognition unit 10 is in charge of converting image information into code information. As an input device for image information, an image reader 11. A throat scanner 1
2. There is a FAX 13° and a CD-ROM 14. Further, devices for directly inputting code information include a floppy disk 15 and a magnetic tape 16. Further, the user terminal is provided with a keyboard 17 so that the user can input documents directly from the keyboard.

Dependency analysis unit 2 using CPU 20 and RAM 21. Index extraction unit 3. The processes of the thesaurus development section 5 and similar document search section 6 are executed. The DISK 30 includes a document database 10o, a thesaurus dictionary 101. A morphological analysis dictionary 102 is stored. Document databases, dictionaries, etc. are normally stored in the DISK 30, but in order to speed up various processes, the DISK 30 is stored according to the capacity of the RAM 21.
The data is transferred from there to the RAM 21 and used. Similar document search results are displayed on the display 70. The units 10 to 17 and 70 correspond to the document input unit 1 shown in FIG. 1, and documents are directly input from these units. Further, the document storage section 4 in FIG. 1 corresponds to the document database 100.

Details of each process will be described below with reference to FIGS. 1 and 2.

In the document input section 1, both image information and code information can be input. Image information can be input by FAX13.
CD-ROM14. This is done from an image reading device such as Image League 11.

Each device driver cuts out characters from the image information acquired from each device based on document structure information, and transfers the cut out character images to the character recognition unit 10. The character recognition unit 10 performs a process of converting the transferred character image into code information. This process uses, for example, the invention of Miyahara (Miyahara: Character reading system, Japanese Patent Application No. 57-222489). After the code information is converted, it is transferred to the RAM 21 for other processing.

When code information is directly read from the floppy disk 15 or magnetic tape 16, the read information is directly transferred to the RAM 21. Even when code information is input directly from the keyboard 17, it is similarly transferred to the RAM 21.

The dependency analysis unit 2 first performs morphological analysis of the input document. In morphological analysis, morphological analysis dictionary 102
Separate sentences using , and add morphological information. Morphological information is written.

Add reading 9 parts of speech, conjugated forms, meaning category numbers, etc.

Dependency analysis is performed using this morpheme information. A typical technique for dependency analysis is the invention by Inagaki et al. FIG. 3 shows an example of dependency analysis of an input sentence using this dependency analysis method.

If we divide the input sentence ``Input means for inputting kana character strings and homophone selection instruction signals, and...'' into phrases, we can obtain ``Input means for inputting kana character strings and homophone selection instruction signals, and...''. and ``input means''. Next, the dependency relationships of the word groups divided into phrases are determined. In this input sentence, as shown in Figure 3, the "kana character string" and the "homophone selection instruction signal" form a parallel syntax, and both relate to "to input", and the phrase "to input" ” pertains to “input means.” The dependency analysis unit 2 is in charge of the process of determining such dependency relationships between clauses.

The index extraction unit 3 extracts an index from the words divided by morphological analysis and weights the extracted index.

Methods for index extraction include the unnecessary word dictionary method and the controlled word dictionary method. The former is a method in which words that should not be extracted as an index are registered in an unnecessary word dictionary, and words that are not in the unnecessary word dictionary are extracted as an index.

The latter is a method in which words to be used as an index are registered in a controlled word dictionary, and if a word that matches a word in the controlled word dictionary exists in a document, it is output as an index.

The controlled word dictionary method is used for documents for which a controlled word dictionary is prepared, and the unnecessary word dictionary is used for documents for which a controlled word dictionary is not prepared. In the example shown in FIG. 3, since a controlled word dictionary is not prepared, the unnecessary word dictionary method is applied. For this type of document, prepare an unnecessary word dictionary as shown in Figure 4,
Noun-equivalent words and phrases (including proper nouns, pseudonyms, etc.) excluding unnecessary words are extracted as an index.

Index weighting uses sentence structure points calculated from the dependency analysis results. Sentence structure points are determined by the dependencies between clauses and the attributes of the dependencies. A standard value of sentence structure points is given to the clause at the end of a sentence, and the sentence structure point for each clause is the total value of the inter-clause link points of dependencies passed from the clause at the end of the sentence to the target clause.

FIG. 5 shows the correspondence between attached words of bunsetsu and link points between bunsetsu. An example is shown in Figure 6, where in the input sentence "Input means for inputting signals.", "Input means."

” is given the standard value of sentence structure points (0), and the phrase “to input” has a dependency relationship through the particle “no”, so the inter-clause link points are 2, and the sentence structure points are also It becomes 2. Similarly, the clause ``signal'' has an inter-clause link point O, so it is ``for input''.
This is the same sentence structure point 2 as . In the same way, find the sentence structure points for all clauses. The higher the sentence structure point, the lower the importance of the index.

The document storage unit 4 stores document information in a document database 100.
Register. The information to be registered is not just document code information, but also morpheme information given by the dependency analysis section 2 and index information given by the index extraction section 3. FIG. 7 shows an example of an input document, and FIG. 8 shows a state in which input sentences are stored. Each line is divided into clauses, and each clause is given a clause number, 9 clause descriptions, 1 clause reading, 9 parts of speech, a dependency number, an index, etc. When a new document is stored, the index information required by the similar document search unit 6 is also updated. FIG. 9 shows an index information table. The index information table stores an index of a document database and a document number to which the index is assigned. This table is used to quickly obtain matched document numbers.

The thesaurus development section 5 develops the index extracted by the index extraction section 3 into synonyms and synonyms so that the search device can handle various document expressions. FIG. 10 shows an example in which the index is developed using the thesaurus dictionary 101. The words listed at the left end are the index group extracted from the document. The thesaurus dictionary 101 is searched for the index, and if a headword matches the index, synonyms and similar words assigned to the headword are extracted.

In this thesaurus dictionary 101, synonyms are classified into three types. Synonyms 0. Synonym 1. This is synonym 2. These are broadly classified based on the similarity of their meanings. In other words, the synonym O
is defined as a word that is semantically similar to the headword and can be replaced under any conditions.Synonym i!■
, synonym 2. As the number increases, the meaning becomes further away from the headword.

For example, in the case of the index "document", the synonym O is "
Documents” are listed. In this case, "Document" can be used as "Document J" to mean the same meaning. Also, synonyms include "Text", "Document J", etc.
There are ``sentences'', ``sentences j, etc.

Further, the thesaurus dictionary 101 includes an ambiguity determination table, which determines the distinction between words that are the same in notation but have different meanings depending on the field of the document. FIG. 11 shows an ambiguity determination table. The polysemy judgment table consists of information on the spelling, pronunciation, and field of use of the word, and when expanded into a thesaurus, the most semantically correct synonym for the field of the input document.

Output synonyms. For example, rCDJ is a [cash dispenser] used in banking-related fields.
Is it a music-related "compact disc"?
Although it is not clear whether this is the case, in the device of the present invention, when the field of a document is listed in the ambiguity determination table, priority is given to synonyms and synonyms corresponding to that field. If there is no suitable field, extract all synonyms and synonyms.

The similar document search unit 6 performs processing to search for documents similar to the input document. Matching between the input document and the document database 100 is performed in two stages. The first stage is a keyword inclusion rate check, in which it is checked to what extent the keywords developed in the thesaurus of the input document and the keywords in the document database match. Matching is performed at high speed between the index information table of FIG. 9 and the thesaurus-expanded words. The keyword inclusion rate is the value obtained by dividing the number of matched keywords by the number of keywords assigned to each document (duplicated keywords are counted as one).

Perform processing based on keyword inclusion rate. Binary relationships extracted by dependency relationships include the relationship between a noun (phrase) and a verb (phrase), and the relationship between a noun and a noun within a noun phrase. In the case of noun phrases, the last noun in the noun phrase was used to represent the meaning of the noun phrase, and in the case of verb phrases, all the words making up the verb phrase were targeted.

In the input sentence of FIG. 7, for example, noun>verb> In the second stage of matching, the documents hit by the keyword inclusion rate are narrowed down by determining the degree of similarity in sentence structure. Since it would take time to process all documents with two or more matching keywords, the processing level is determined according to the user's needs. Two types of processing levels can be specified: keyword inclusion rate and number of documents.

In the device of the present invention, as a narrowing down method, binary relationships between nouns and verbs are extracted, regardless of the similarity of binary relationships such as dependency relationships. In addition, words of the same rank with high usage frequency are extracted based on the relationship within the noun phrase and the pronunciation of kana characters. Furthermore, when a verb phrase is composed of multiple predicates, the noun (phrase) is assumed to have a binary relationship with the multiple predicates. For example, in the case of the sentence ``convert into a sentence containing kanji and kana and display it...'', the sentence is converted into a sentence containing kanji and kana and is displayed as a noun><verb> a sentence containing kanji and kana.

If these binary relationships exist in documents in the document database, matching points are given.

Matching points change depending on the importance of the index and thesaurus points. Each evaluation point will be explained below.

The importance of the index is obtained by changing the standard weight given by the document storage unit 4, taking into account the sentence structure, according to the matching situation in the first stage. The standard weighting based on the sentence structure given above is a weighting for the general content of the document, and a lighter weight for specialized content that goes deeper into the content of the document. However, search requests include searches in a broad sense (for example, ``I want to collect patents related to kana-kanji conversion devices'') and search requests in a narrow sense (for example, ``I want to collect patents related to homophone selection for kana-kanji conversion devices''). There is a need to satisfy these two aspects at the same time. In the case of the apparatus of the present invention, the weighting of the index is changed so as to be able to respond to a wide range of search requests to a narrow search request.

If the keyword inclusion rate is low in the first stage of matching, it is determined that the content of the current document database does not hold enough data for the input document, and priority is given to searching in a broader sense. conduct. Therefore, the importance of the index is maintained at the standard weighting. If the keyword inclusion rate is high, it is determined that the search is in-depth into specialized content, and the search is executed with the index importance given the greatest weight to deep content.

The thesaurus point represents the semantic distance to the index of synonyms and similar words assigned by the thesaurus development unit 5. The closer the semantic distance to the index, the higher the thesaurus points. That is, the index and synonym 0 are given 3 thesaurus points, synonyms 1 and 2 are given 2 thesaurus points, and synonyms are given 1 thesaurus point.

The evaluation value of each index is the sum of the index importance and thesaurus points, and if the dependency relationships match, the matching point is the sum of the respective evaluation values. The final degree of similarity between documents is a value obtained by multiplying the sum of matching points by the keyword inclusion rate.

FIG. 12 shows an example of finding matching points. The input sentence is ``Convert a kana character string into a kanji-kana mixed sentence,'' and the sentence to check the similarity (matching sentence) is ``Replace the first character with the second character,'' and each Assume that the dependency relationship and index importance are determined as shown in the figure. In this case, looking at each clause, the word "character" matches in the first clause (thesaurus point 3). In the bunsetsu, °゛character゛ is a synonym for ゛kanji°°, so thesaurus point 1゜In the third clause, ゛conversion°
Since the substitution of ° for ゛ is a synonym, it becomes the thesaurus point l. Here, the index importance of the input sentence, the index importance of the matching sentence, and the thesaurus points are all multiplied together to calculate the evaluation value of the index. If the dependency relationships are the same,
The point obtained by multiplying the index evaluation values of the dependent clause and dependent clause is set as the matching point for each dependency relation. For example, the first sentence is converted into a sentence with kanji and kana, and °゛ has a dependency relationship, and the second sentence has a dependency relationship between °° and ``° replacement'' in the second character. Since they match, the matching point for the dependency of this phrase is 1OX10=100 points, which is the sum of the points of each phrase.

Similarly, matching points are found for all dependency relationships, the sum is taken, and the sum is multiplied by the keyword inclusion rate to obtain the document similarity.

Similar document output section 7 displays the search results of similar document search section 6 on display 70 . In this case, the documents are sorted in descending order of similarity, and five documents with high similarity are displayed on the screen so that the contents of each document can be easily understood. 5 on the screen
If all documents cannot be displayed, priority is given to displaying documents with a high degree of similarity.

[Effects of the Invention] As explained above, the present invention has a document input section that directly inputs a document and uses it as code information, and a document input section that separates and writes input character strings, adds morpheme information, and converts the text based on the morpheme information. a dependency analysis unit that determines the dependency relationships between clauses, and an index that determines the sentence structure from the dependency analysis results of the dependency analysis unit, extracts an index from this sentence structure, and assigns the importance of the index. an extraction unit; a document storage unit that stores input documents, dependency analysis results, and index extraction results;
a thesaurus expansion unit that expands the index of the index extraction unit using a thesaurus dictionary; a similar document search unit that determines the degree of similarity between an input document and stored documents based on the degree of similarity of indexes and the degree of similarity of dependency relationships; and a similar document output section that outputs similar documents that have been searched, the user can search for similar documents without having to consider keywords, logical formulas, etc.

Furthermore, when searching for similar documents, a thesaurus expansion of the index makes it possible to perform a complete search. The searched documents are narrowed down using the index based on the search target and are executed with high precision. Furthermore, the two-step search method allows the search time to be shortened.

[Brief explanation of the drawing]

FIG. 1 is a block diagram showing the configuration of an embodiment of the present invention.
Figure 2 is a system configuration block diagram of a similar document search device.
Figure 3 is a diagram explaining an example of dependency analysis, Figure 4 is a diagram explaining an unnecessary word dictionary, Figure 5 is a table diagram showing the correspondence between attached words of dependent clauses and inter-clause link points, and Figure 6 is a diagram explaining a dependency analysis example. Figure 7 is a diagram explaining an example of sentence structure point calculation, Figure 7 is a diagram explaining an example of an input document, Figure 8 is a diagram showing accumulated document information, Figure 9 is an index information table diagram, and Figure 10 is a thesaurus expansion. 11 is an ambiguity determination table, and FIG. 12 is a diagram illustrating an example of matching point calculation. In the figure, 1 is a document input section, 2 is a dependency analysis section, 3 is an index extraction section, 4 is a document storage section, 5 is a thesaurus expansion section, 6 is a similar document search section, 7 is a similar document output section, and 10 is a similar document output section. Character recognition unit, 11 is image league, 12 is hand scanner, 1
3 is a FAX, 14 is a CD-ROM, 15 is a floppy disk, 16 is a magnetic tape, 17 is a keyboard, 2Q is a CPU, 100 is a document database, 101
102 is a thesaurus dictionary, and 102 is a morphological analysis dictionary. Inter-clause link points − Sentence structure points

Claims (1)

[Claims] A document input section that directly inputs a document and uses it as code information, and a dependency that separates the input character string, adds morpheme information, and determines the dependency relationship between clauses based on the morpheme information. an analysis unit, an index extraction unit that determines a sentence structure from the dependency analysis result of the dependency analysis unit, extracts an index from this sentence structure, and assigns the importance of the index; and an input document, and the dependency analysis result. , a document storage unit that stores index extraction results; a thesaurus expansion unit that expands the index of the index extraction unit in a thesaurus dictionary; and a dependency relationship between the similarity between the input document and the stored document and the similarity of the index. a similar document search unit that determines based on the degree of similarity;
A similar document search device comprising: a similar document output unit that outputs searched similar documents.