JP2009104475A - Similar document retrieval device, and similar document retrieval method and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a similar document retrieval device and a similar document retrieval method and program capable of discovering a document where not only input retrieval character strings coincide but similar tags are attached or a document where a similar keyword is used in retrieving a similar document from a large amount of documents stored in a text format. <P>SOLUTION: In a similar document retrieval device for retrieving documents, with respect to a normalized document to be a document which normalizes words included in an input document, stores words for which a tag is attached to the document, and normalizes words in the document, a tag is generated in the document according to a setting file, an index where a word to be a clue for retrieving a similar document matches a document ID is generated, statistical information to be statistical information on names of attributes included in the document stored in the storage means is generated, and the document is retrieved on the basis of the statistical information. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

The present invention relates to a system and method for retrieving similar documents from a large number of documents stored in a text format.

  Conventionally, various search methods have been proposed for searching a target document from a large amount of natural language documents such as newspaper articles and patent publications. These search methods are roughly classified as follows.

(1) Keyword Search In the keyword search, indexing of individual documents and keywords representing the contents of the documents is performed in advance. In this case, as a method for determining a keyword, automatic keyword extraction by morphological analysis or the like, manual keyword addition, and a combination of these methods are known (see, for example, Patent Document 1 and Patent Document 2).

  However, in this keyword search, only a character string indexed as a keyword can be searched, and the accuracy of automatic keyword extraction by morphological analysis depends on the accuracy of the word / grammar dictionary. There is a problem that a lot of human work is required for maintenance.

(2) Fuzzy search A fuzzy search is a method of searching not only a character string that completely matches a search character string but also a document that includes a partially matching character string.

  For example, the user's memory for the search character string is ambiguous, the search character string may appear with various modifications, and it may be difficult to enumerate all such modifications.

  A conventional typical partial character string designation method uses a regular expression (formal expression). According to this, it is possible to specify an arbitrary character 0 times or more, specify an arbitrary character 1 or more times, line end position, line head position, arbitrary character within a specific character code range, etc. ( (See Patent Document 3).

  However, in these ambiguous searches, it is difficult to specify the degree of ambiguity of the character string to be searched, and there is a problem that the search result includes many character strings that are not desired by the user or are unnatural. is there.

(3) Concept-based search Even if the search character strings do not match, there are cases where it is desired to search for documents containing words having similar meanings or documents belonging to the same category.

  A meaning (concept) of a word that indicates the relationship between the word itself and the attribute of the word is called a concept base, and a method of searching for similar documents using this is known.

  In terms of concept-based construction, dictionary data is often used, but it cannot be applied to new words or proper nouns that are not in the dictionary, and the “meaning used in sentences in a specific field” (context-dependent relationship). There is a problem that it cannot be extracted. Therefore, use of co-occurrence relations in corpora and the like is performed (see Patent Document 4).

However, the concept-based search requires a large amount of documents and words for constructing the concept base, and at the same time, a great deal of time is spent on the processing. In addition, proper nouns, technical terms, buzzwords, etc. are constantly created, so it is necessary to constantly update the concept base. In addition, there is a problem that it is difficult to cope with a case where a search character string is input with notation that is not included in the concept base, hiragana, or notation due to notation fluctuation.
Japanese Patent Laid-Open No. 4-205560 JP-A-4-215181 JP-A-62-221027 JP-A-8-147324

  In any of the above prior arts, the search character string itself must be present in the desired document or a word having a similar meaning must exist.

  However, for example, when a user tries to obtain a ringtone (registered trademark) from WEB, the word “ringtone” is not necessarily included in the site. If "Ringtone" (registered trademark) or "Mobile music distribution service" is described instead of "Ringtone", even if the site is searched by the user, it will not be searched There is.

  In addition, the name of such a service is often a new word that is not listed in the dictionary, and there is a problem that it is difficult to search even if an ambiguous search or a concept base is used.

  That is, there is a problem that it is difficult to comprehensively find a site desired by a user only with a conventional method of comparing characters in a sentence with a search character string.

In the present invention, when a similar document is searched from a large number of documents stored in a text format, not only the input search character string is matched, but also a similar tag or a similar tag is assigned. An object of the present invention is to provide a similar document search apparatus, a similar document search method, and a program capable of finding a document in which a keyword is used.

The present invention relates to a document input means for a user to specify a document and to read the specified document, a document that divides the input document into words, normalizes the divided words, and calculates a feature amount of the document. Analyzing means, normalizing means for normalizing words included in the input document, a setting file storing words to be tagged to the document, and words included in the document For a normalized document, which is a normalized document, a tag generation unit for generating a tag in the document according to the setting file and storing the tag in the storage unit, and a word and a document ID as a clue for searching for similar documents Index generation means for generating a corresponding index, and attribute name statistical information that is statistical information about attribute names included in the document stored in the storage means Statistical information processing means stored in a storage device, document search means for searching for a document based on the attribute name statistical information, and keywords used as search conditions for searching for a predetermined document, the document search means and document analysis Request processing transmission means for transmitting to the means, and document display control means for reading out and displaying the document included in the search result from the document storage means via the communication means. It is a similar document search device.

  According to the present invention, when a similar document is searched from a large number of documents stored in a text format, not only the input search character string matches but also a document with a similar tag, There is an effect that documents in which similar keywords are used can be found.

Thus, according to the present invention, it is possible to present more sites desired by the user with simpler means, and the user's work efficiency and convenience are improved.

  The best mode for carrying out the invention is the following examples.

  FIG. 1 is a diagram showing a similar document search device RS1 that is Embodiment 1 of the present invention.

  The similar document search device RS1 is an embodiment that detects a word group (a document having a similar attribute name configuration) for a certain document designated by the user.

  The similar document search device RS1 includes a server SV1 and a browser B1.

  The server SV1 includes normalization means 10, setting file 20, document generation means 30, index generation means 40, statistical information processing means 50, document search means 60, document analysis means 70, and request processing transmission means. 80, a communication unit 90, a document input unit 100, and a document display control unit 110.

  The normalizing means 10 normalizes words included in the document. That is, the normalizing means 10 normalizes attribute values included in documents stored in the document database DB1. Note that, as the name suggests, a word is a word obtained by dividing a document, and among many words (words), only the word specified in the setting file as an attribute value becomes an attribute value.

  “Normalization” is, for example, adding an attribute value “mail magazine” written in Japanese without omitting an attribute value “mail magazine” written with an abbreviation. In this way, when normalized, it was added instead of replaced. When the normalized word is completely replaced, when it is finally displayed in the browser, the changed word (= normalized word) This is because the displayed document changes to avoid this. Instead of performing the additional recording, the attribute value “Melmaga” written in abbreviated symbols may be replaced with the attribute value “mail magazine” written in Japanese without abbreviations.

  That is, the normalizing means 10 detects an attribute value that is a synonym having a different expression while having the same meaning in the document, and adds the detected synonym. When detecting synonyms, there are the following methods.

  FIG. 2 is a diagram illustrating an example of a method for detecting synonyms using a co-occurrence pattern.

  A method using the co-occurrence pattern shown in FIG. 2 can be employed. By this processing, the word of the document in the document database DB1 is normalized.

  FIG. 3 is a diagram illustrating an example of a document before normalization in the first embodiment.

  FIG. 4 is a diagram illustrating an example of normalization and a tagged document.

  If the normalized document is input and the attribute value “mermaga” is specified to be classified into the attribute name “magazine”, the document generation unit 30 receives “text ID” and “tag” as shown in FIG. Is added, and the generated document is stored in the document database DB1.

  The specification of the attribute value, the attribute name, and the like is described in the setting file 20.

  In the setting file 20, a word to which a tag is added is specified.

  The document generation unit 30 adds a tag to the document according to the setting file 20 and stores the normalized document, which is a document obtained by normalizing words included in the document, in the document storage unit. That is, the document generation unit 30 embeds the document ID and the attribute name as a tag. The tag is enclosed in <> in FIG.

  The index generation means 40 generates an index in which a word that serves as a clue to search for a similar document and a document ID correspond to each other.

  Next, pre-search processing in the similar document search device RS1 in the first embodiment will be described.

  FIG. 5 is a flowchart specifically showing the search pre-processing in the similar document search device RS1 in the first embodiment.

  First, the word is normalized (S1), a document including the normalized word is generated, and the generated document is stored (S2). An index is generated (S3), statistical information is generated (S4), and the process ends.

  Index generation means 40 generates an index. This “index” is a table in which a word included in a document stored in the document database DB1 is associated with a document ID of a document including the word.

  FIG. 6 is a diagram illustrating an example of an index.

  As shown in FIG. 6, in the index, for example, the word “description” is associated with the document ID “001” of the document including the word.

  Further, the statistical information processing means 50 generates statistical information for each attribute name of the setting file 20 based on the document D, and stores it in the attribute value statistical information.

  That is, the statistical information processing means 50 generates and stores statistical information about attribute names included in the document stored in the document storage means.

  FIG. 7 is a diagram illustrating an example of attribute name statistical information.

  In the attribute name statistical information shown in FIG. 7, “magazine” is described as an attribute name. The document ID of the document including this “magazine”, and the attribute name “magazine” is included in the document indicated by the document ID. The number included is information correspondingly described.

  Next, among the processes performed in each part of the block diagram shown in FIG. 1, a process in which the similar document search device RS1 searches for a similar document for a document input from the browser B1 will be described.

  FIG. 8 is a flowchart specifically showing a process in which the similar document search apparatus RS1 searches for a similar document among documents input from the browser B1 among the processes performed in each part of the block diagram shown in FIG. is there.

  The document search means 60 searches for a document based on the attribute name statistical information. That is, the document search unit 60 is a unit that compares attribute name statistical information attached to each document in a document group that satisfies a specified condition, and detects a document having a value within a certain range. The “document group satisfying the specified condition” is a document group including the word specified by the word input unit.

  For example, when a document α is input by the user, the document input unit 100 transmits the document α to the communication unit 90.

  The communication means 90 gives the transmitted document α to the request processing transmission means 80, and the request processing transmission means 80 gives the given document α to the document analysis means 70.

  The document analysis means 70 analyzes the contents of the input document. That is, the document analysis unit 70 normalizes words included in the document α and searches the document for attribute values stored in the setting file 20. That is, the document analysis unit 70 executes two processes, a process of dividing into words and a process of calculating a feature value (tag) for each attribute name. If the attribute value exists, the feature amount is calculated for each attribute name, and the result is returned to the request processing transmission unit 80.

  The request processing transmission unit 80 transmits a keyword used as a search condition for searching for a predetermined document to the document search unit and the document analysis unit.

  The document input unit 100 reads a document designated by the user.

  The document display control unit 110 reads out the document included in the search result from the document storage unit via the communication unit 90 and displays it.

  When calculating the feature amount, there are a method of simply counting the number of appearances, a method of setting and classifying a threshold value, and a method of calculating the ratio of the number of appearances in all words appearing in the document.

  That is, first, a document is input (S11), the document is analyzed, that is, the document is divided into words, the feature amount for each attribute name is calculated (S12), the document is searched (S13), and the search result Is transmitted (S14). The search result is displayed on the browser (S15), and the process ends.

  FIG. 9 is a diagram illustrating an example of feature vectors in the first embodiment.

  The request processing transmission unit 80 gives a processing request to the document search unit 60. The document search unit 60 uses the attribute name statistical information and the document having the same combination of the attribute name and the feature amount or the above combination is similar. The retrieved document ID is retrieved, and the retrieved document ID is returned to the request processing transmission means 80.

  As shown in FIG. 9, a document whose attribute name and feature amount combination is the same as the document α has a feature amount “2” of the attribute name “input” and a feature amount of the attribute name “keys” “ 5 ”, the feature amount of the attribute name“ magazine ”is“ 3 ”, and so on....

  However, when searching for a document based on the attribute name statistical information, it is possible to search not only a document whose combination of the attribute name and its feature quantity is exactly the same as the document α, but also a document similar to the document α. It is desirable to do so. That is, the document similar to the document α has a feature amount for the attribute name within a predetermined range from the feature amount for each attribute name in the document α.

  The standard of the document similar to the document α is, for example, a standard that the document α has a feature amount within ± predetermined% of the feature amount for each attribute name in the document α, or a ± This is a criterion that the feature amount within the predetermined percentage is half or more of the total attribute names in the document α.

  The request processing transmission unit 80 reads all the documents corresponding to the document ID from the document database DB1, and transmits the read documents to the browser via the communication unit 90.

  FIG. 10 is a diagram illustrating a display example of a document displayed by the document display control unit 110.

  As shown in FIG. 10, the document display control unit 110 of the browser extracts a part from each given document, creates a document list, and displays it. A part to be extracted is a place (column, row, format, keyword) designated in advance. When one document is selected by the user, the displayed list is erased and the entire selected document is displayed.

FIG. 11 is a diagram illustrating an example in which the document ID 001 in FIG. 10 is selected and the entire document with the document ID 001 is displayed.

  Example 2 is an example in which a word group specified by a user (that is, a document having a similar attribute name or attribute value configuration) is detected.

  FIG. 12 is a block diagram showing a similar document search device RS2 that is Embodiment 2 of the present invention.

  The similar document search device RS2 is a device in which the document analysis unit 70 is deleted from the similar document search device RS1 and attribute name statistical information A2 is provided instead of the attribute name statistical information A1. A browser B2 is provided instead of the browser B1.

  The similar document search device RS2 includes a normalization unit 10, a setting file 20, a document generation unit 30, an index generation unit 40, a statistical information processing unit 50, a document search unit 60, a request processing transmission unit 80, A communication unit 90, a document input unit 100, and a document display control unit 110 are included.

  The browser B2 includes an attribute name attribute value input unit 120 for inputting an attribute name and an attribute value, and a document display control unit 130.

  FIG. 13 is a flowchart specifically showing pre-search processing in the similar document search device RS2 among the processes performed in the respective parts of the block diagram shown in FIG.

  The processes of S21 to S23 are the same as the processes of S1 to S3 in the similar document search device RS.

  The statistical information processing means 50 generates statistical information for each attribute name stored in the setting file 20 based on the document stored in the document database DB1, and stores it as attribute name statistical information.

  First, the word is normalized (S21), a document including the normalized word is generated, the generated document is stored (S22), and an index is generated (S23). Then, statistical information is generated (S24). finish.

  FIG. 14 is a diagram illustrating a state where one attribute name is assigned to one piece of statistical information.

  Further, one piece of statistical information is information including one or more items in which a document ID is associated with an attribute value included in a document with this ID.

  That is, FIG. 14 pays attention to one attribute name “magazine”, and shows the ID of a document including the attribute name “magazine” and the attribute value included in the document corresponding to the document ID. FIG.

  FIG. 15 shows a specific example of the process in which the similar document search apparatus RS2 searches for a similar document with respect to the attribute name or attribute value input from the browser, among the processes performed in each unit constituting the similar document search apparatus RS2. It is a flowchart shown in FIG.

  The user inputs an attribute name (S31), searches for a document based on the input attribute name (S32), and transmits the processing result of the searched document to browser B2 (S33). The retrieved document is displayed on the browser B2 (S34).

  That is, in the attribute name attribute value input unit 120, when the user inputs “magazine” as the attribute name, the input attribute name “magazine” is transmitted to the communication unit 90. A plurality of attribute names or attribute values may be selected. Also, the attribute name or attribute value may be selected by default.

  FIG. 16 is an explanatory diagram of a method for selecting an attribute name or attribute value on the browser B2.

  The communication means 90 gives the received attribute name to the request processing transmission means 80, the request processing transmission means 80 gives the attribute name to the document search means 60, and the document search means 60 is based on the attribute name statistical information. The document ID of the document including all the input attribute names is searched, and the searched document ID is returned to the request processing transmission unit 80.

Note that the processing of S33 and S34 is the same as the processing of S14 to S15 in the similar document search device RS1.

  The third embodiment of the present invention is an embodiment that detects a word and a word group (documents including attribute names or attribute values and having similar attribute name configurations) specified by the user.

  FIG. 17 is a block diagram showing a similar document search device RS3 that is Embodiment 3 of the present invention.

  The similar document search device RS3 is the same as the similar document search device RS2, but a browser B3 is provided instead of the browser B2. The browser B3 includes an attribute name attribute value input unit 120, a word input unit 140, and a document control unit 150.

  The word input unit 140 is a unit for inputting a specific keyword.

  The pre-search process in the similar document search device RS3 is the same as the process in the second embodiment shown in FIG.

  FIG. 18 shows that the similar document search device RS3 searches for a similar document for words and attribute names input from the browser B3, among the processes performed in each part of the similar document search device RS3 shown in FIG. It is a flowchart which shows a process concretely.

  For example, when the word “Chaku-Uta” is input by the user, the word input unit 140 transmits the word “Chaku-Uta” to the communication unit 90. A plurality of the words may be input.

  The attribute name attribute value input means 120 may be designated when the user designates “magazine” as the attribute name.

  That is, first, a word specified by the user (which is an input word or a user-specified word and different from a word obtained by dividing the document) and an attribute name are input via the word input unit 140 (S41). The document is searched based on the word specified by the user (S42), and the document is searched based on the attribute name (S43). These search results are integrated (S44), the processing result after the integration is transmitted (S45), and displayed on the browser B3 (S46).

  FIG. 19 is an explanatory diagram of selecting an attribute name and inputting a word on the browser B3.

  The communication means 90 gives the transmitted keyword and attribute name α to the request process, and the request process transmission means 80 gives the given keyword and attribute name α to the document search means 60.

  The document retrieval means 60 retrieves the document ID associated with the index for the keyword “Chaku-Uta”, and retrieves the document ID of the document including all the input attribute names based on the attribute name statistical information. Then, the document ID common to both is returned to the request processing transmission means 80.

  The processing from S45 to S46 is the same as the processing from S14 to S15.

  Moreover, the said Example can be grasped | ascertained as invention of a method. That is, in the above embodiment, the user designates a document, a document input step for reading the designated document, a document division step for dividing the inputted document into words, and a word included in the inputted document. About a normalization process for normalizing a document, a target to which a tag is attached to a document, a word storage process for storing a word in a setting file, and a normalized document that is a document obtained by normalizing a word included in the document An index for generating an index in which a tag is generated in a document according to the setting file and stored in a storage device, a word serving as a clue to search for a similar document, and a document ID Generating the attribute name statistical information, which is statistical information about the attribute name included in the document stored in the storage device, and storing the attribute name statistical information in the storage device; Request to transmit information processing step, document search step for searching for document based on attribute name statistical information, and keyword used as search condition for searching predetermined document to document search means and document analysis means A similar document search method comprising: a process transmission step; and a document display control step of reading and displaying a document included in the search result from the storage device via a communication unit.

  The keyword is a word input through the word input unit 140, that is, a word specified by the user.

A program that causes a computer to function may be assumed as each means constituting the similar document search apparatus.

It is a figure which shows similar document search device RS1 which is Example 1 of this invention. It is a figure which shows the example of the method of detecting a synonym using a co-occurrence pattern. 6 is a diagram illustrating an example of a document before normalization in Embodiment 1. FIG. It is a figure which shows the example of normalization and a tag addition document. 6 is a flowchart specifically illustrating search pre-processing in the similar document search device RS1 according to the first embodiment. It is a figure which shows the example of an index. It is a figure which shows the example of attribute name statistical information. It is a flowchart which shows specifically the process in which the similar document search apparatus RS1 searches a similar document among the processes currently performed by each part of the block diagram shown in FIG. 1 about the document input from the browser. 6 is a diagram illustrating an example of a feature vector in Embodiment 1. FIG. It is a figure which shows the example of a display of the document which the document display control part 110 displays. FIG. 11 is a diagram illustrating an example in which a document ID 001 in FIG. 10 is selected and an entire document with a document ID 001 is displayed. It is a block diagram which shows similar document search device RS2 which is Example 2 of this invention. It is a flowchart which shows concretely the search pre-processing in similar document search device RS2 among the processes currently performed by each part of the block diagram shown in FIG. It is a figure which pays attention to one attribute name "magazine" and shows the ID of the document including this attribute name "magazine" and the attribute value included in the document corresponding to this document ID. Among the processes performed in each part constituting the similar document search device RS2, a flowchart specifically showing the process in which the similar document search device RS2 searches for a similar document for the attribute name or attribute value input from the browser. is there. It is explanatory drawing of the method of selecting an attribute name or an attribute value on browser B2. It is a block diagram which shows similar document search device RS3 which is Example 3 of this invention. Of the processes performed in each part of the similar document search device RS3 shown in FIG. 17, the similar document search device RS3 searches for similar documents for words and attribute names input from the browser B3. It is a flowchart shown in FIG. It is explanatory drawing in the case of selecting an attribute name and inputting a word on browser B3.

Explanation of symbols

RS1 ... similar document search device,
10 ... normalization means,
20 ... Setting file,
30 ... Document generation means,
40: Index generation means,
50. Statistical information processing means,
60 ... Document search means,
70: Document analysis means,
80 ... Request processing transmission means,
90 ... communication means,
B1 ... Browser,
100: Document input means,
110 ... Document display control means,
RS2 ... Similar document search device,
B2 ... Browser,
120 ... Attribute name attribute value input means,
130: Document display control means,
RS3 ... Similar document search device,
B3 ... Browser,
140 ... word input means,
150. Document display control means.

Claims (6)

A document input means in which a user designates a document and reads the designated document;
Document analysis means for dividing the input document into words, normalizing the divided words, and calculating document features;
Normalization means for normalizing words contained in the input document;
A configuration file that stores the words to be tagged with the document;
A document generation unit that generates a tag in the document and stores the tag in the storage unit according to the setting file for a normalized document that is a document obtained by normalizing a word included in the document;
Index generation means for generating an index in which a word that is a clue to search for a similar document and a document ID correspond;
Statistical information processing means for generating attribute name statistical information, which is statistical information about attribute names included in a document stored in the storage means, and storing the attribute name statistical information in the storage device;
A document search means for searching for a document based on the attribute name statistical information;
Request processing transmission means for transmitting a keyword used as a search condition for searching for a predetermined document to the document search means and the document analysis means;
A document display control means for reading out and displaying a document included in the search result from the document storage means via a communication means;
A similar document search device characterized by comprising: A normalization means for normalizing words contained in the document;
A configuration file that stores the words to be tagged with the document;
A document generation unit that generates a tag in the document according to the setting file and stores the tag in the document storage unit with respect to a normalized document that is a document obtained by normalizing a word included in the document;
Index generation means for generating an index in which a word that is a clue to search for a similar document and a document ID correspond;
Statistical information processing means for generating attribute name statistical information, which is statistical information about attribute names included in a document stored in the document storage means, and storing the attribute name statistical information in a storage device;
A document search means for searching for a document based on the attribute name statistical information;
Request processing transmission means for transmitting search conditions to the document search means and the document analysis means;
An attribute name attribute value input means for the user to select the displayed attribute name and attribute value;
A document display control means for reading out and displaying a document included in the search result from the document storage means via a communication means;
A similar document search device characterized by comprising: In claim 2,
A similar document search apparatus comprising word input means for inputting a specific keyword. In any one of Claims 1-3,
The document search means is means for comparing attribute name statistical information attached to each document in a document group satisfying a designated condition, and detecting a document having a value within a certain range. Document retrieval device. A document input process in which the user designates a document and reads the designated document;
A document analysis step of dividing the input document into words, normalizing the divided words, and calculating document features;
A normalization step of normalizing words contained in the input document;
A word storing step of storing in a setting file a word to be tagged with a document;
A document generation step of generating a tag in the document and storing it in a storage device according to the setting file for a normalized document that is a document obtained by normalizing words included in the document;
An index generation step of generating an index in which a word serving as a clue to search for a similar document and a document ID correspond;
A statistical information processing step of generating attribute name statistical information, which is statistical information about attribute names included in the document stored in the storage device, and storing the attribute name statistical information in the storage device;
A document search step for searching for a document based on the attribute name statistical information;
A request processing transmission step of transmitting a keyword used as a search condition for searching for a predetermined document to the document search means and the document analysis means;
A document display control step of reading and displaying a document included in the search result from the storage device via a communication means;
A similar document search method characterized by comprising:   A program that causes a computer to function as each means constituting the similar document search device according to claim 1.

Images