JPH04182871A - Similar document retrieval device - Google Patents

Abstract

PURPOSE:To efficiently retrieve document which are similar to desired input condition at a high speed by giving an input document concept information on a meaning level and its partial product composite item as indexes, and performing unification retrieval for the indexes by using retrieval condition items in the same form. CONSTITUTION:A document input part 1 converts, for example, an input document 30 into an electronic input document 31 and a document analytic part 10 analyzes this electronic input document 31 and outputs the obtained concept information 32 in specific item form. Then a document retrieval part 12 converts this concept information 32 into retrieval conditions 33. Then the unification retrieval is performed for an index part 8 to obtain the storage address of a document having similar concept information and then obtains a set 34 of similar documents from a document data base 7 through a document data base control part 6. Consequently, the documents which are similar to desired input conditions can be retrieved efficiently at a high speed.

Description

DETAILED DESCRIPTION OF THE INVENTION [Field of Industrial Application] The present invention relates to a similar document search device that searches for documents similar to input conditions from a document database that has been digitized and stored in large quantities.

[Prior Art] Figure 9 is a block diagram of the ``Similar Document Search System Using Dependency Relationships,'' which was presented at the Information Processing Society of Japan's 39th (late 1989) National Conference IG-5. (1) is a document input section that mainly consists of an image league, character recognition section, and natural language input section; (2) is a document processing section that performs syntax analysis and index extraction processing of the input document; and (3) is similar to the input document. a similar document search unit that extracts documents from a document database (7), (4) a word dictionary, (5) a thesaurus dictionary,
(6) is a document database management section, and (7) is a document database.

In the document processing section (2), (2a) is a morphological analysis section, (2b) is a dependency analysis section, and (2c) is an index extraction processing section. Also, in the similar document search section (3), (3a)
is a thesaurus expansion section, and (3b) is a similarity calculation section.

Next, the operation will be explained.

Input documents are stored in a document database (7) via a document database management unit (6) after being subjected to syntactic analysis and index extraction processing in a document processing unit (2). At this time, in the document processing unit (2), the morphological analysis unit (2a) first divides the input document into word units and phrase units using the word dictionary (4), and adds part-of-speech information and semantic category information, and then The modification analysis unit (2b) uses a word dictionary (4) and a thesaurus dictionary (5) to uniquely determine the relationship between the dependency and the usage of a clause in order to reduce ambiguity in Japanese. Next, the index extraction processing unit (2C) extracts, as keywords, words obtained by removing unnecessary words from the noun-equivalent phrases appearing in the document.

<When searching for a document similar to an input document> After processing the input document that is the search condition using the same process as in the above 2 and extracting keywords, the similar document search section (3) first searches the thesaurus expansion section (3a). In order to achieve a search with fewer omissions, keywords are expanded into synonyms and synonyms using a thesaurus dictionary (5)2, and based on these, the document database (7) is searched via the document database management unit (6). However, in this way, a huge number of similar document candidates are output.

Therefore, the similarity calculation unit (3b) first checks the keyword inclusion rate by counting the number of matching keywords between the search condition and each candidate document, and then sets the dependency for candidates with a high keyword inclusion rate. Check the similarity of the two, and if they are similar, perform processing such as further increasing the degree of similarity. instructor,
The similarity of each candidate document is determined using the search method, and those with high similarity are selected as search results.

[Problems to be Solved by the Invention] Conventional searches for similar documents as described in 2 above are performed as described above. First, in order to check the keyword inclusion rate, it is necessary to calculate the number of matching keywords among a huge number of documents. Furthermore, after narrowing down the keywords to those with a high keyword inclusion rate, it is necessary to check the similarity of the relationships between the dependencies, making the process extremely inefficient.

The present invention has been made to solve such problems, and an object of the present invention is to provide a similar document search device that can efficiently and quickly search for documents similar to input conditions.

[Means for Solving the Problems] A similar document search device according to the present invention searches for documents similar to input conditions from a document database that has been digitized and accumulated in large quantities. When storing the input document in a document database, a document analysis unit that forms conceptual information in a predetermined term format by syntactically and semantically analyzing the input document; An index creation unit that is developed as an index and given to the input document, and a search for a predetermined term format created based on the input document or natural language input when searching the document database for documents similar to the input conditions. and a document search unit that performs a unified search for conditions against the index.

[Operations] When searching for similar documents in this invention, first, when input documents are stored, the concept that the document means is expressed as concept information in a predetermined term format, and this is indexed together with its partial compound term. If you want to search for documents similar to the input conditions, create the search conditions in the predetermined term format based on the input documents or natural language input, and apply this to the previous word index. This method searches for similar documents efficiently and quickly by performing a unified search.

[Example] An example of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram of a similar document retrieval device according to an embodiment of the present invention. In the figure, (1) is a document input section, and (10) is a document analysis section, which analyzes the syntax and semantics of input documents. to form conceptual information in a predetermined term format. (11)
is an index creation unit, which develops conceptual information obtained as a result of document analysis and information on its partial compound terms as an index when the document is searched, and assigns it to the input document. (4) is a word dictionary, which includes part-of-speech information, conjugation inflection information, case information (deep path information), semantic feature information, etc. for words. (13) is a grammar rule dictionary, which is a collection of grammar rules expressed in expressions such as "noun phrase - noun particle". (5) is a thesaurus dictionary that describes the relationships between individual concepts, (6) is a document database management unit that stores and manages documents, (7) is a document database that is a collection of stored documents, and (12) is an input Documents similar to the document are retrieved from the document database (7) by the document database management department (
6) is a document retrieval unit that performs a search, and (8) is an index unit that stores an index and the like.

In the document analysis unit (10), (10a) is a morphological analysis unit that extracts words from the input document using a word dictionary (4), associates these words with parts of speech, and furthermore, , and performs processing to obtain information used in syntax analysis described later. (iob) is a syntactic analysis unit, which performs processing to create a structure based on grammar based on the information obtained by the morphological analysis unit (10a) and the grammar rule dictionary (13). (10e) is a semantic analysis section, and a syntax analysis section (1
From the structure created by step 0b), the case relationships are analyzed based on the case information in the word dictionary (4), and the concept of the meaning of the document is extracted.

In addition, in the document search unit (12), (12a) is a thesaurus dictionary expansion unit, which refers to the thesaurus dictionary (5) for words representing each concept extracted by the semantic analysis unit (lOe) to prevent omissions in the search. Expand into synonyms and synonyms. (
12b) is a search condition creation unit that creates a search condition expression according to appropriate conditions in order to prevent the search range from exploding due to thesaurus dictionary expansion. (12c) is a unification search processing unit, which performs unification (
Unification) Search.

Next, the operation will be explained. Figure 2 is a diagram showing the flow of processing for storing input documents in a document database. Convert to
Next, the document analysis unit (10) converts this electronic input text tF(2
1) is parsed and semantically analyzed, and conceptual information (22) in a predetermined term format obtained as a result is output. Next, the index creation unit (11) creates an index (23) according to this conceptual information (22).
) and send it to the document database management department (
6) Output.

FIG. 3 is a diagram showing details of the analysis process of the document analysis unit (10). When an electronic input document (21) as shown is input from the document input unit (1), the morphological analysis unit (10B
, ) performs a morphological analysis process on the electronic input document (2]) using part of speech information, inflection information, etc. in the word dictionary (4), and as a result of the analysis, the morphological information (21a
) is output. Next, the syntactic analysis unit (fob) extracts the morphological information (2) by referring to the grammar rule dictionary (13).
], a), the syntax tree (2
Create l b). Next, the semantic analysis unit (10c) uses the deep path information of the word dictionary (4) to construct the syntax tree (21b).
For example, conceptual information (22) in the form of terms as shown in the figure is obtained. Here, the deep path indicates what role each concept representing the word plays with respect to the verb in the deep structure (the structure of the concept that the sentence attempts to express).

Figure 4 is a diagram showing a part of the word dictionary (4).
The deep paths that can be taken for this "verb" include "action nominative, object nominative, place nominative, instrumental case", etc. Among these, the semantic feature of the action nominative is "animate" or "inanimate". This indicates that superficially (when it appears in a sentence) it appears with the particle ``ga'' or ``wa''. Furthermore, the semantic feature of something that can be a target path is ``state'', which superficially indicates that it appears with the particle ``wo''. Similarly, the diagram shows the deep path of the place path and instrumental case, and although not shown, other cases such as the experienced person case, the source case, the target case, and the time path are also used.

In this way, the semantic analysis unit (10c)
) is a relation, that is, a syntax tree (2l b
), words that fit each case are extracted based on the deep case information and semantic feature information of the word dictionary (4) as described above, and conceptual information (
22).

FIG. 5 is a diagram showing details of the processing of the index creation section (11),
In this index creation unit (11), first, conceptual information in term form (
22). Extract all partial compound terms from 22). If we say this to boat fishing, for example, the conceptual information is f(g (h (x))
, g (y) ), all subcompound terms are f(g (h (x) ) , g
(y) ) g (h (x) ) h (x) g (y). Next, the index creation unit (11) generates conceptual information (22)
All partial compound terms extracted from the input document (20)
is expanded into the index (23) as a key value when searched. This is to enable a search for some concepts included in the concept information (22) and to enable a more detailed search. For example, if you want to search for documents that include the concept "X and Y are connected in series," search for the key value "Connect in series (objective ([X, Y]))" from index (23). It will be a good thing if you do. Of course, the composite ring itself of the conceptual information (22) is also a key value like the partial composite term. Next, the index creation unit (11) creates a table associating key values with storage addresses of input documents in the document database (7) as shown in the figure, and uses this table as an index (23).
shall be.

Here, the compound ring is the same as the compound ring defined in the Prolog language, where compound ring: ·=function operator (argument string), interval number: ·=atom argument sequence: ·=argument 1. Argument 2. .

It is expressed as The argument may be a compound ring. According to this, concept information (22) is a complex ring whose functor is a "concept" and argument 1, and furthermore, this argument is a functor that says "make constant", and its argument is a complex ring, and so on. (
nesting) structure.

FIG. 6 is a diagram showing the flow of processing for searching a document database for a document similar to an input document.
is converted into an electronic input document (31), and then the document analysis unit (10) analyzes this electronic input document (31) and outputs conceptual information (32) in a predetermined term format obtained as a result. . The processing up to this point is the same as the processing when storing the input document (20).

Next, the document search unit (12) converts this conceptual information (32) into search conditions (33), and performs a unified search on the index unit (8) to find documents with similar conceptual information. Obtain the storage address and collect similar documents (34) from the document database (7) via the document database management unit (6).
get.

FIG. 7 is a diagram showing details of the processing of the document search unit (12).
In the figure, first, a thesaurus expansion unit (12a) uses a thesaurus dictionary (5) to expand words appearing in the concept information (32) into synonyms and synonyms, and as a result of this thesaurus expansion, a compound ring (32a) is obtained. This is conceptual information (32
) is replaced with a set of synonyms and synonyms OR-combined. Thesaurus dictionary (5) is related (e.g. synonymous relations, similar relations, superordinate/subordinate relations, etc.)
It associates concepts expressed by words. For example, FIG. 8 shows a part of a thesaurus dictionary (5).

Next, the search condition creation unit (12b)' selects only important or noteworthy partial compound terms from the compound ring (32a) to obtain a search condition expression (32b) as shown. Here we show an example where the object case and the instrumental case are important. In this way, the designation of which case is important can be done by, for example, storing the case in advance as a system constant in the search condition creation unit (12b), or by allowing the user to designate it as appropriate. This is to prevent the number of search condition expressions from exploding due to combinations of words as a result of expanding the search range by expanding the thesaurus dictionary.

Next, the unification search processing unit (12c) executes the search condition expression (33
) of the individual complex rings (Ctl, Cs],.

Cs2. ) performs a unification search for each key value of the index (23) in the index section (8), and obtains the storage location of the document corresponding to the key value for which the unification was successful.

Here, unification is the same as unification in the Pro log language, and is a function that turns two patterns (structures) into one pattern by assigning appropriate values to the variables that appear in them. There is a function to check the equality of two patterns. Unification is said to be successful when the assignment is successful or equiindividuality is confirmed.

For example, the unification of object case (X) and object case (resistance) is successful, as object case =:=object case (equivalence) resistance '3 X (assignment).

In the above embodiment, when searching for similar documents, the document analysis unit (10) analyzes the document serving as the search condition, and the document search unit (12) automatically generates the search condition expression.
For example, when you want to search for documents that include the concept "X and y are connected in series," you can connect them in series (object case (
“X.

yJ)), the user may create the search condition by himself/herself and perform the search.

In addition, if the word corresponding to the target case is unknown, for example,
It is also possible to connect them in series (object case (X)) by using them as logical variables.

Further, in the above embodiment, all partial compound terms are extracted from the concept information (22), but partial compound terms may be extracted under certain conditions.

[Effects of the Invention] As explained above, this invention adds conceptual information at the semantic level and its partial compound terms to an input document as an index, and performs a single search on the index using a search condition term in the same format. Since the search is performed based on the search criteria, documents similar to the desired input conditions can be searched efficiently and quickly.

[Brief explanation of drawings]

FIG. 1 is a block diagram of a similar document search device according to an embodiment of the present invention, FIG. 2 is a diagram showing the flow of processing for storing input documents in a document database, and FIG. 3 is an analysis diagram of the document analysis unit (10). A diagram showing details of the processing, FIG. 4 is a diagram showing a part of the word dictionary (4), FIG. 5 is a diagram showing details of the processing of the index creation unit (11), and FIG. 6 is similar to the input document. FIG. 7 is a diagram showing the process flow of searching a document from a document database; FIG. 7 is a diagram showing details of the processing of the document search unit (12); FIG. 8 is a diagram showing a part of the thesaurus dictionary (5); FIG. The figure is a block diagram of the ``Similar Document Search System Using Dependency Relationships,'' which was presented at the 39th (late 1989) national convention of the Information Processing Society of Japan, IG-5. In the figure, (1) is the document input section, (2) is the document processing section,
(2a) is the morphological analysis part, (2b) is the modification part of the analysis part,
(2C) is an index extraction processing unit, (3) is a similar document search unit,
(3a) is thesaurus expansion part, (3b) is similarity calculation part, (4) is word dictionary, (5) is thesaurus dictionary, (6)
is a document database management section, (7) is a document database, (8) is an index section, (10) is a document analysis section, (10a)
) is the morphological analysis unit, (10b) is the syntactic analysis unit, (10e
) is a semantic analysis unit, (11) is an index creation unit, (12) is a document search unit, (12a) is a thesaurus dictionary development unit, (12
b) is a search condition creation section, (12c) is a unification search processing section, and (13) is a grammar rule dictionary. Note that the same reference numerals in each figure indicate the same or corresponding parts.

Claims (1)

[Claims] In a similar document search device that searches for documents similar to input conditions from a document database that has been digitized and stored in large quantities, when storing input documents in the document database, the input document is A document analysis unit that forms conceptual information in a predetermined term format through syntactic analysis and semantic analysis, and an index creation unit that develops the formed conceptual information and its partial compound term information as an index and adds it to the input document. and a document search that performs a unified search of the predetermined term-format search conditions created based on the input document or natural language input against the index when searching the document database for documents similar to the input conditions. A similar document retrieval device comprising: