JP7193000B2 - Similar document search method, similar document search program, similar document search device, index information creation method, index information creation program, and index information creation device - Google Patents

Description

The embodiments of the present invention relate to a similar document search method, a similar document search program, a similar document search device, an index information creation method, an index information creation program, and an index information creation device.

2. Description of the Related Art Conventionally, as one of natural language processing by a computer, there is search processing for searching for documents similar to an input document from among documents stored in a database. For example, a sample inquiry document and a reply document corresponding to the sample are registered in the database. Then, it is conceivable to build an interactive interface such as a chatbot that retrieves samples similar to the input inquiry text and outputs response texts corresponding to the similar samples.

As a method of retrieving samples similar to the input document from the samples registered in the database, there is a method of evaluating the degree of commonality of words appearing between two documents. For example, the following search methods are conceivable. A plurality of hash functions (sometimes called Min-hash functions) for calculating one hash value from a set of words contained in a document are defined. Each hash function has a correspondence relationship in which different values are associated with different words, and outputs the minimum value of values corresponding to words included in a certain word set as a hash value. A vector listing a plurality of hash values calculated using the plurality of hash functions is generated in advance for the samples registered in the database. Then, vectors are similarly calculated from the set of words contained in the input document and the plurality of hash functions described above, and a vector similar to the vector of the samples registered in the database is searched.

JP 2018-173909 A

However, in the above-described conventional technology, when the input document is short or the expressions used in the document are diverse and the number of expressions that match the sample is small, the degree of commonality of words is evaluated low, resulting in a decrease in search accuracy. be. For example, since the input documents of the chatbot are mainly spoken words, sentences are short, and expressions are diverse, the probability of common words appearing between the input document and the sample is generally low. As a result, the sample retrieval accuracy for the input document tends to be low.

An object of one aspect is to provide a similar document search method, a similar document search program, a similar document search device, an index information creation method, an index information creation program, and an index information creation device that improve the accuracy of searching for similar documents. do.

In one proposal, in the similar document search method, a computer executes the process of generating, the process of calculating, and the process of searching. The generating process is based on a set of words included in the search target document and inter-word information indicating the closeness of the meaning of the words. Generate a hash function that assigns close values to words in descending order of semantics. The calculation process calculates summary information for each of a plurality of search target documents based on the generated hash function, and calculates summary information for the input document based on the generated hash function. In the searching process, a document similar to the input document is searched from a plurality of search target documents based on comparison between the calculated summary information of the search target document and the summary information of the input document.

In one aspect, search accuracy for similar documents is improved.

FIG. 1 is a block diagram of a functional configuration example of an information processing apparatus according to an embodiment; FIG. 2 is an explanatory diagram of an example of a processing flow of the information processing apparatus according to the embodiment; FIG. 3 is a flowchart illustrating an example of hash function generation processing. FIG. 4 is an explanatory diagram for explaining the hash function. FIG. 5 is an explanatory diagram for explaining the degree of similarity between words. FIG. 6 is an explanatory diagram for explaining hash values based on Min hash. FIG. 7 is an explanatory diagram for explaining an overview of the operation of the information processing apparatus according to the embodiment; FIG. 8 is an explanatory diagram for explaining narrowing down of search target documents. FIG. 9 is an explanatory diagram showing a display example of the operation screen. FIG. 10 is a diagram illustrating an example of a computer that executes programs.

Hereinafter, a similar document search method, a similar document search program, a similar document search device, an index information creation method, an index information creation program, and an index information creation device according to embodiments will be described with reference to the drawings. Configurations having the same functions in the embodiments are denoted by the same reference numerals, and overlapping descriptions are omitted. Note that the similar document search method, similar document search program, similar document search device, index information creation method, index information creation program, and index information creation device described in the following embodiments are merely examples, and the embodiments are not limited to these. It is not limited. Moreover, each of the following embodiments may be appropriately combined within a non-contradictory range.

[overview]
This embodiment will exemplify an information processing apparatus that retrieves a document similar to an input document (hereinafter also referred to as an inquiry document) from samples (hereinafter also referred to as search target documents) registered in a database.

In this information processing apparatus, first, preliminary processing is performed, and then search processing is performed. In the pre-processing, a hash value is calculated for each search target document using a hash function, and an index structure such as a tree structure (for example, a search tree) for searching the search target document is created from the calculated hash value.

In the search process, a hash function is used to calculate the hash value of the inquiry document. Next, the information processing device compares the hash value of the inquiry document with the hash value of each search target document, and searches for a similar hash value among the search target documents shown in the index structure. Next, the information processing apparatus determines search target documents having hash values close to the hash value of the inquiry document as similar document search results.

A plurality of hash functions for calculating hash values are defined based on a set of words obtained by extracting words contained in search target documents registered in the database. Specifically, the information processing apparatus uses a hash function where W is a set of words, and h is a set (a plurality of hash functions) of all injections from W to {0, 1, ..., |W|-1}. Generate multiple .

The information processing apparatus performs the following processing on the search target document and the inquiry document, obtains a vector of a plurality of hash values, and obtains summary information summarizing the search target document and the inquiry document with a hash function.
• Randomly select a hash function from h.
・A set of words is obtained by extracting words contained in a document to be searched.
・The hash value is the smallest integer among the integers obtained from each word by the selected h.
・Multiple hash values are obtained by performing random selection of functions multiple times.

Documents that are similar to each other have a high common word appearance ratio (Jacquard coefficient). For a selection of hash functions from h, the probability of matching hash values is equal to the Jacquard coefficient. Therefore, when comparing hash values of documents when searching for similar documents, the Jacquard coefficient can be calculated stochastically from the rate of matching of each element of the vector. It reflects the closeness (similarity) of

Let n be the number of search target documents, m be the number of words (average value) in each search target document, and k be the number of hash functions. Then, the amount of calculation for calculating the vector of the search target document is O (kmn). The amount of calculation for calculating a vector of one query document is O(km), and the calculation cost of neighborhood search using this vector is O(log(n)). K hash functions are randomly generated in advance. If k is O(log(n)), the collision probability of calculating the same hash value from different search target documents can be sufficiently reduced.

However, in verification of similarity based on Hamming distance, the difference between the elements of the vectors is meaningless, and the similarity is 0 when the elements do not match (when there is no common word). Therefore, when common words appear infrequently, the accuracy of similarity search tends to be low.

Therefore, in the present embodiment, the information processing apparatus generates a hash function that assigns a close value in descending order of meaning with reference to a predetermined word based on inter-word distance information indicating the closeness of meaning between words. Specifically, the information processing device extracts words contained in the search target document, collects the words, and randomly selects a reference word. Next, based on the inter-word distance information, the information processing device sorts the words included in the set of words in the order of closest meaning to the reference word, and obtains a unique value in the sort order (for example, an integer that increases in the sort order). numbers). The information processing device generates a plurality of hash functions by repeating the above process.

For example, if the set of words is {cat, rice, . As a result, {cat, cat, . . . rice, food} is obtained. For the set of words sorted in this way, the information processing device assigns integer values in order of sorting, such as {cat=0, cat=1, . . . rice=5, food=6}.

The hash value generated by the hash function generated in this manner is a value corresponding to the distance (also called between words) of the meaning of a given word. Therefore, it is possible to verify the degree of similarity not only by the Hamming distance between hash values but also by the Euclidean distance. Therefore, even if there is no common word and the hash values do not match (if the elements do not match), the degree of similarity can be verified by Euclidean distance, and the accuracy of similarity search can be improved.

[Configuration example]
FIG. 1 is a block diagram of a functional configuration example of an information processing apparatus according to an embodiment; As shown in FIG. 1, the information processing apparatus 100 is an apparatus that performs a process of obtaining a similar document 102 similar to an input inquiry document 101 from search target documents stored in a search target document database 121 . For example, a personal computer or the like can be applied as the information processing apparatus 100 .

The information processing apparatus 100 includes an index generation module 111 that performs preprocessing for creating an index structure, a search module 112 that performs search processing, and a storage unit that stores various data related to processing (search target document database 121, inter-word distance information, etc.). It has a storage unit 122, a hash function storage unit 123, and an index storage unit 124). That is, the information processing device 100 is an example of a similar document search device and an index information creation device.

The search target document database 121 is a database in which search target documents to be searched for the inquiry document 101 are registered. The search target documents in the search target document database 121 may be registered in advance, may be added through interaction with the user using the information processing apparatus 100 through a dialog interface such as a chatbot, or may be added automatically from the network. may be collected at

The inter-word distance information storage unit 122 stores inter-word distance information representing the closeness of meaning (between words) of each word to other words. Specifically, the inter-word distance information includes a function d(d, w) representing the distance (word inter-word) between each word (v) and another word (w). That is, inter-word distance information is an example of inter-word information indicating the closeness of word meanings of words.

Hash function storage unit 123 stores a plurality of different hash functions generated by index generation module 111 . Each hash function has a correspondence relationship that associates each word that can appear in a search target document with a unique integer (in order of meaning close to a given word), accepts a set of words, and outputs one hash value. do. Also, different hash functions have different correspondences.

The index storage unit 124 stores an index structure for searching for search target documents similar to the inquiry document 101 . The index structure is generated by the index generator 132 based on vectors (summary information) calculated from the search target documents using a plurality of hash functions. An example.

For example, a search tree can be applied as the index structure. A search tree includes a plurality of nodes (a leaf node and each node leading up to the leaf node) connected in a tree structure. A leaf node of the search tree indicates a document to be searched. For example, a leaf node includes a vector of a search target document and identification information (for example, document ID) that identifies the search target document. However, leaf nodes do not have to contain vectors.

Two child nodes are connected to each non-leaf node. Each non-leaf node has a threshold for a particular dimension in the vector. If the hash value of the specified dimension in the input vector is greater than or equal to the threshold, proceed to the right child node, and if the hash value of the specified dimension is less than the threshold, proceed to the left child node. By tracing the search tree from the root node to the leaf nodes in this way, it is possible to efficiently search for a search target document that is close to a predetermined vector.

The index generation module 111 has a hash function generation section 131 and an index generation section 132 .

Hash function generation unit 131 generates a plurality of hash functions based on search target documents stored in hash function generation unit 131 and inter-word distance information stored in inquiry reception unit 133, and generates a plurality of hash functions. A hash function is stored in the hash function storage unit 123 .

Specifically, the hash function generator 131 extracts words contained in the search target document, collects the words, and randomly selects a reference word. Next, the hash function generation unit 131 refers to the inter-word distance information in the inter-word distance information storage unit 122, and sorts the words included in the set of words in order of closeness to the reference word. Next, the hash function generating unit 131 generates a hash function by assigning unique values (for example, integer values that increase in sort order) to the words included in the set of words. The hash function generation unit 131 generates a plurality of hash functions by repeating the process of generating the above hash functions.

For example, it is assumed that a function d(d, w) representing the distance (word spacing) between each word (v) and another word (w) is given as inter-word distance information. This function d(d, w) can be created in advance with reference to the degree of similarity between words, the vector representation of words, and the like. Also, the function d(d, w) is as follows.
• 0≤d(v,w) for any v,wεW.
• d(w,w)=0 for any wεW.

As for the hash function, for any u, v∈W, there is w∈W such that h(u)>h(v)⇔d(u, w)>d(v, w).

The hash function generator 131 randomly selects w from W and sorts all vεW in ascending order of d(v, w). Then, the hash function generator 131 assigns integers 0, 1, 2, . . . to the sorted words w, v ₁ , v ₂ . Note that the hash function generation unit 131 may use the value of d(v, w) as it is instead of allocating an integer (assuming there is no duplication).

The index generation unit 132 generates an index structure based on the search target documents stored in the search target document database 121 and the hash function stored in the hash function storage unit 123, and stores the generated index structure in the index storage unit 124. Store.

Specifically, the index generation unit 132 extracts a word set for each search target document, inputs the extracted word set to each of a plurality of hash functions, and generates a hash value vector, that is, summary information of the search target document. calculate. Next, the index generator 132 generates an index structure so that multiple vectors corresponding to multiple search target documents can be efficiently retrieved. For example, the index generation unit 132 generates a search tree by focusing on one dimension in the vector and repeatedly determining the threshold value of the hash value of the dimension so that the set of vectors is divided into two. . At this time, the index generating unit 132 generates intermediate nodes so that leaf nodes of the search tree are associated with a single vector as much as possible.

The search module 112 has an inquiry reception unit 133 , a hash value calculation unit 134 , a search unit 135 and an output unit 136 .

The inquiry reception unit 133 receives the inquiry document 101 . The inquiry receiving unit 133 may receive the inquiry document 101 input as a character string from the user, or may convert a voice signal of an inquiry utterance verbally uttered by the user into a character string. Also, the inquiry receiving unit 133 may receive a character string or an audio signal from another information processing device.

The hash value calculation unit 134 generates a vector corresponding to the inquiry document 101, that is, summary information of the inquiry document 101, based on a plurality of hash functions stored in the hash function storage unit 123. FIG. Specifically, the hash value calculation unit 134 extracts a word set from the inquiry document 101, inputs the extracted word set to each of a plurality of hash functions, and calculates a hash value vector.

Based on the index structure stored in the index storage unit 124 and the vector of the inquiry document 101, the search unit 135 searches for a search target document that is most similar to the inquiry document 101 by neighborhood search. Specifically, the search target document that is most similar to the inquiry document 101 has the largest number of dimensions with matching hash values when the vectors are compared.

For example, the search unit 135 traces the search tree stored in the index storage unit 124 from the root node to the leaf nodes while comparing the hash value of a specific dimension in the vector of the inquiry document 101 with a threshold, Reach a particular leaf node. The search unit 135 selects a search target document corresponding to the reached leaf node.

Note that if two or more search target documents are associated with the reached leaf node, that is, if the number of dimensions with matching hash values is the same and the search tree cannot narrow down the search target document to one, The search unit 135 compares the hash values to find the Euclidean distance. Next, the search unit 135 selects documents with the closest Euclidean distance as the most similar search target documents.

The output unit 136 outputs the retrieved search target document as the similar document 102 . For example, the output unit 136 may display the character string of the similar document 102 on a display or the like, or may convert the similar document 102 into an audio signal and reproduce the audio through a speaker. The output unit 136 may also transmit the character string or audio signal of the similar document 102 to another information processing device.

Also, instead of outputting the similar document 102 , the output unit 136 may perform processing associated in advance in the search target document database 121 with respect to the searched search target document. Specifically, it is assumed that predetermined processing (for example, schedule registration, mail transmission) is registered in the search target document database 121 for each search target document. The output unit 136 reads from the search target document database 121 and executes the process associated with the searched search target document, thereby making it possible to perform the process corresponding to the inquiry document 101 .

[Example of operation]
FIG. 2 is an explanatory diagram of an example of a processing flow of the information processing apparatus according to the embodiment; As shown in FIG. 2, the information processing apparatus 100 performs a preliminary process (S1) for creating an index structure, and a search process (S2) for retrieving and outputting similar documents 102 to a query document 101. FIG.

First, the preprocessing (S1) will be described. In the pre-processing (S1), a search target document is first read from the search target document database 121 and input to the hash function generator 131 (S11).

The hash function generating unit 131 receives the input of the distance between words from the inquiry receiving unit 133 (S13), and generates a plurality of hash functions 123a based on the input search target document and the distance between words (S12 ).

FIG. 3 is a flowchart illustrating an example of hash function generation processing. As shown in FIG. 3, the hash function generator 131 receives an input of a search target document (S31), extracts words (appearing words) contained in the search target document (S32), and generates a set of words (words set) is acquired (S33).

Next, the hash function generation unit 131 generates a plurality of hash functions 123a by repeating the processes of S34 to S39 k times, which is the number of hash functions to be generated.

Specifically, the hash function generation unit 131 randomly selects one word from the word set (S35), receives input of inter-word distance information indicating the distance between words (S36), and other words are referred to from the inter-word distance information (S37). Next, the hash function generator 131 sorts the words in the word set in descending order of distance to the selected word (S38), and assigns an integer value that increases in order of alignment to each aligned word as a hash value (S39). .

Next, the hash function generator 131 outputs a plurality of hash functions 123a obtained by repeating the processes of S34 to S39 k times, and stores them in the index generator 132 (S40).

FIG. 4 is an explanatory diagram for explaining the hash function 123a. As shown in FIG. 4, h ₁ , h ₂ , . . . h2, .

For example, h1 has (cat) as _a reference word, and each word in the word set is assigned an integer value according to the inter-word distance from (cat). Also, _h2 is based on the word (rice), and each word in the word set is assigned an integer value corresponding to the inter-word distance from the word (rice). Also _, h3 is based on a word (cat), and an integer value is assigned to each word in the word set according to the distance between words with respect to (cat). Also, h4 is based _on the word (bait), and an integer value is assigned to each word in the word set according to the distance between words with respect to (bait). In addition, _h5 uses (flower) as a reference word, and an integer value corresponding to the inter-word distance from (flower) is assigned to each word in the word set. Also, h6 is based _on the word (water), and each word in the word set is assigned an integer value corresponding to the inter-word distance to (water).

Here, a case where the hash values of the following documents A to C are obtained by the hash function 123a illustrated in FIG. 4 will be exemplified.
Document A: "Feed the cat"
Document B: "Feed the Nyanko"
Document C: "Watering the flowers"

The word set of document A is {cat, rice}, the word set of document B is {nyanko, food}, and the word set of document C is {flower, water}. Therefore, the hash value H _A of document A by the hash function 123a illustrated in FIG. 4 is H _A =001133. Similarly, the hash value H _B of document B is H _B =110022. Also, the hash value H _C of document C is H _C =424200.

Here, when Hamming distances are calculated by comparing hash values of documents A to C, H _A , H _B : 6, H _A , H _C : 6, H _B , H _C : 6. Also, when comparing the hash values of documents A to C and calculating the Euclidean distance, H _A , H _B : 1, H _A , H _C : 6.9, H _B , H _C : 6.2. In this way, even when it is difficult to verify the degree of similarity using the Hamming distance (when no common words are included), it is possible to verify mutually similar documents (documents A and B in the illustrated example) using the Euclidean distance.

Returning to FIG. 2, following S12, the index generator 132 generates an index structure 124a based on the input search target document and the generated hash function 123a (S14), and stores the generated index structure 124a as an index. Stored in section 124 .

In the search process (S2), the inquiry document 101 received by the inquiry reception unit 133 is input to the hash value calculation unit 134 (S21). Hash value calculation unit 134 generates a plurality of hash values of input inquiry document 101 based on a plurality of hash functions stored in hash function storage unit 123 (S22), and obtains a vector corresponding to inquiry document 101. .

Next, the search unit 135 compares the vector of each search target document in the index structure stored in the index storage unit 124 with the hash value of the vector of the query document 101 (S23), and finds the search target most similar to the query document 101 (S23). Search for documents. The output unit 136 outputs the similar document 102 searched by the search unit 135 (S24).

FIG. 5 is an explanatory diagram for explaining the degree of similarity between words. Specifically, FIG. 5 is a bird's-eye view of the arrangement of the words W1 to W6 in a high-dimensional space indicating the degree of similarity between words. Words W1 to W6 in FIG. 5 each indicate a word contained in the document. Here, the words W1 to W3 are similar to words such as "cat" and form a cluster indicated by a dotted line. Similarly, words W4-W6 are similar for words such as "dog" and form a separate cluster from words W1-W3.

As shown in FIG. 5, in a high-dimensional space of similarity between words, it is difficult to evaluate similarity well with simple projection. For example, an orthogonal projection onto axis A1 brings the values of similar words (words W1-W3 or words W4-W6) closer together. However, in an orthogonal projection onto an axis A2, which is different from axis A1, words that are dissimilar to each other (eg, words W1 and W4) may be closer together than words that are similar to each other (eg, words W1 and W3).

In this embodiment, since a reference word is randomly selected and a hash value based on Min hash is used, projection based on the distance from the reference word (reference point) is used.

FIG. 6 is an explanatory diagram for explaining hash values based on Min hash. Here, the word W1 is "cat", the word W2 is "nyanko", the word W3 is "cat", the word W4 is "dog", the word W5 is "mouse", the word W6 is "dog", and the word W7 is "dog". ” shall be It is also assumed that the reference point is the word W1.

As shown in FIG. 6, based on the word W1 as a reference point, the hash function 123a calculates 1: "cat", 2: "cat", 3: "cat", 4: "dog", 5: " A value of 6: "dog", 7: "dog" is assigned. Here, regarding the distance from the reference point, the closer to the reference point, the higher and lower the degree of similarity is maintained (for example, 1: "cat", 2: "nyanko", 3: "cat", etc.). However, at greater distances from the reference point (eg 5: "mouse", 6: "dog"), dissimilar words may have closer values.

In this embodiment, since the minimum hash value among the hash values of the word set is used (Min hash), it is suitable as a projection capable of expressing the degree of similarity of words while maintaining the degree of similarity.

FIG. 7 is an explanatory diagram illustrating an overview of the operation of the information processing apparatus 100 according to the embodiment. As shown in FIG. 7, in the pre-processing (S1), the information processing apparatus 100 extracts words contained in each search target document 121a of the search target document database 121 based on the similarity between words based on the distance information between words. For the set of , a plurality of hash functions are generated that assign close values in descending order of word meaning based on a predetermined word. Next, the information processing apparatus 100 performs Min Hash conversion on each of the search target documents 121a using the plurality of generated hash functions, generates an index structure for searching the calculated vector, and stores the index structure in the index storage unit 124. Store.

In the search process (S2), the information processing apparatus 100 performs Min Hash conversion on the input inquiry document 101 using the same plurality of hash functions generated in S1. Next, the information processing apparatus 100 compares the hash value vector calculated from the inquiry document 101 with the hash value vector stored in the index storage unit 124 to find the search target document 121a that is most similar to the inquiry document 101. Search for

In the illustrated example, for the query document 101 "I want to coordinate a meeting", the hash values of the word "coordination" in the search target document 121a match "schedule coordination" and "meeting coordination". Therefore, the Hamming distance is close. Here, "meeting" contained in the inquiry document 101 has a closer meaning and a closer Euclidean distance than "schedule". Therefore, for the inquiry document 101 "I want to coordinate a meeting", a similar document 102 "I want to coordinate a meeting" is obtained.

Note that the output unit 136 may output one similar document 102 that is most similar to the searched search target document, or may output a plurality of similar documents 102 whose similarities obtained by the Hamming distance and the Euclidean distance are equal to or greater than a predetermined threshold. similar document 102 may be output.

FIG. 8 is an explanatory diagram for explaining narrowing down of search target documents. Specifically, FIG. 8 is a graph showing an example of the relationship between the similarity threshold and the number of hits of the search target document 121a.

The graph 10 shows the relationship between the similarity threshold between the inquiry document 101 and the search target document 121a and the number of search target documents 121a having a similarity greater than the threshold (the number of hits). The degree of similarity is the sum of the number of dimensions in which the hash values of the query utterance vector and the search target utterance vector coincide with the Euclidean distance between the hash values.

(A) In the method of calculating vectors without considering related words, the similarity between the inquiry document 101 and each search target document 121a is calculated to be low as a whole. Therefore, the relationship between the similarity threshold and the number of hits is shown by curve 11 . In other words, even if the similarity threshold is set low, the number of hits decreases, and similar retrieval target documents 121a are often omitted in retrieval.

(B) In the method of calculating vectors by equating related terms, the degree of similarity between the query document 101 and each search target document 121a is calculated to be high as a whole. Therefore, the relationship between the similarity threshold and the number of hits is shown by curve 12 . That is, even if the similarity threshold is set high, the number of hits increases, making it difficult to efficiently narrow down the similar search target documents 121a.

(C) In the method of the present embodiment, the degree of similarity between the query document 101 and each search target document 121a takes into consideration the Euclidean distance. Therefore, the relationship between the similarity threshold and the number of hits is continuous like curve 13 . As a result, the search target documents 121a similar to the inquiry document 101 can be efficiently narrowed down.

FIG. 9 is an explanatory diagram showing a display example of the operation screen. The operation screen 20 is a screen that accepts various operations from the user by presenting it to the user of the information processing apparatus 100 through an interactive interface such as a chatbot. In the operation screen 20, a display area 21 is an area for displaying processing results and the like, and an input area 22 is an area for inputting documents and the like. For example, the information processing apparatus 100 searches similar documents 102 for the query document 101 input to the input area 22 and outputs the search results to the display area 21 .

For example, the information processing apparatus 100 searches for a similar document 102 using an input document 21a indicating “I want to adjust the schedule” as an inquiry document 101, and displays a search result 21b in the display area 21. FIG. Specifically, schedule registration processing corresponding to "schedule adjustment" obtained as a similar document 102 from among the search target documents 121a is executed for the inquiry document 101 "I want to adjust the schedule".

In the information processing apparatus 100, even if the inquiry document 101 is mainly spoken language, and one sentence is short and has various expressions, like the interactive interface such as the chatbot in the example of FIG. 102 searches can be performed.

[effect]
As described above, the information processing apparatus 100 has the hash function generator 131 , the index generator 132 , the hash value calculator 134 , and the searcher 135 . The hash function generation unit 131 generates a predetermined word Generates a hash function that assigns values that are close to each other in order of meaning. The index generator 132 calculates summary information for each of the search target documents 121a based on the generated hash function. The hash value calculation unit 134 calculates summary information of the input document (inquiry document 101) based on the generated hash function. Based on the comparison between the calculated summary information of the search target document 121a and the summary information of the query document 101, the search unit 135 searches for documents similar to the input document from among the plurality of search target documents 121a.

Therefore, in the information processing apparatus 100, the hash values included in the summary information of the search target document 121a calculated based on the generated hash function and the summary information of the query document 101 are calculated according to the distance of the meaning of a predetermined word. value. Therefore, in the information processing apparatus 100, when comparing the summary information of the search target document 121a and the summary information of the query document 101, for example, not only the Hamming distance between the summary information but also the Euclidean distance is used to determine the similarity between the summary information. The degree of verification can be performed, and the search accuracy for similar documents can be improved.

In addition, each of the index generation unit 132 and the hash value calculation unit 134 calculates the minimum value among the values assigned to each word in the set of words included in the search target document 121a or the query document 101 in the generated hash function. Calculate as a hash value. In this manner, the information processing apparatus 100 can calculate summary information of the search target document 121a or the query document 101 using the Min-hash function.

Further, the hash function generation unit 131 generates a plurality of hash functions by repeating the process of reselecting predetermined words and generating hash functions. Then, each of the index generating unit 132 and the hash value calculating unit 134 summarizes a vector including a plurality of hash values calculated from a set of words included in the search target document 121a or the query document 101 using the generated plurality of hash functions. Calculated as information. As a result, in the information processing apparatus 100, in the comparison between the summary information of the search target document 121a and the summary information of the query document 101, by comparing vectors listing a plurality of hash values, for example, Hamming distance, Euclidean distance, etc. can be obtained to verify the degree of similarity.

In addition, the index generation unit 132 of the information processing apparatus 100 generates index information in which the calculated summary information of each of the plurality of search target documents 121a is associated with the search target document 121a. The search unit 135 compares the summary information of the inquiry document 101 with the summary information associated with the search target document 121a in the index information. By generating index information in advance, the information processing apparatus 100 can quickly search for documents similar to the inquiry document 101 from among the plurality of search target documents 121a.

[others]
It should be noted that each component of each illustrated device does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution and integration of each device is not limited to the one shown in the figure, and all or part of them can be functionally or physically distributed and integrated in arbitrary units according to various loads and usage conditions. Can be integrated and configured.

For example, in the present embodiment, the information processing apparatus 100 having the index generation module 111 and the search module 112 was exemplified. good. That is, the preprocessing (S1) and the search processing (S2) may be performed by different information processing apparatuses.

Various processing functions performed by the information processing apparatus 100 may be executed in whole or in part on a CPU (or a microcomputer such as an MPU or MCU (Micro Controller Unit)). Also, various processing functions may be executed in whole or in part on a program analyzed and executed by a CPU (or a microcomputer such as an MPU or MCU) or on hardware based on wired logic. It goes without saying that it is good. Further, various processing functions performed by the information processing apparatus 100 may be performed in collaboration with a plurality of computers by cloud computing.

By the way, the various processes described in the above embodiments can be realized by executing a prepared program on a computer. Therefore, an example of a computer (hardware) that executes a program having functions similar to those of the above embodiments will be described below. FIG. 10 is a diagram illustrating an example of a computer that executes programs.

As shown in FIG. 10, the computer 1 has a CPU 201 that executes various arithmetic processes, an input device 202 that receives data input, a monitor 203 and a speaker 204 . The computer 1 also has a medium reading device 205 for reading a program or the like from a storage medium, an interface device 206 for connecting with various devices, and a communication device 207 for communicating with an external device by wire or wirelessly. The computer 1 also has a RAM 208 that temporarily stores various information, and a hard disk device 209 . Each unit ( 201 to 209 ) in computer 1 is connected to bus 210 .

The hard disk device 209 stores a program 211 for executing various processes described in the above embodiment. The hard disk device 209 also stores various data 212 (for example, information on the inter-word distance information storage unit 122, the search target document database 121, the hash function storage unit 123, and the index storage unit 124) referenced by the program 211. FIG. The input device 202 receives input of operation information from the operator of the computer 1, for example. The monitor 203 displays, for example, various screens operated by the operator. The interface device 206 is connected with, for example, a printing device. The communication device 207 is connected to a communication network such as a LAN (Local Area Network), and exchanges various information with external devices via the communication network.

The CPU 201 reads out the program 211 stored in the hard disk device 209, develops it in the RAM 208, and executes it, thereby generating a hash function generation unit 131, an index generation unit 132, an inquiry reception unit 133, a hash value calculation unit 134, and a search unit. 135 and output unit 136. Note that the program 211 does not have to be stored in the hard disk device 209 . For example, the computer 1 may read and execute the program 211 stored in a storage medium readable by the computer 1 . Examples of storage media readable by the computer 1 include portable recording media such as CD-ROMs, DVD discs, USB (Universal Serial Bus) memories, semiconductor memories such as flash memories, and hard disk drives. Alternatively, the program 211 may be stored in a device connected to a public line, the Internet, a LAN, etc., and the computer 1 may read and execute the program therefrom.

Reference Signs List 1 Computer 10 Graphs 11 to 13 Curve 20 Operation screen 21 Display area 21a Input document 21b Search result 22 Input area 100 Information processing device 101 Inquiry document 102 Similar document 111 Index generation module 112 Search module 121 Search target document database 121a Search target document 122 Interword distance information storage unit 123 Hash function storage unit 123a Hash function 124 Index storage unit 124a Index structure 131 Hash function generation unit 132 Index Generation unit 133 inquiry reception unit 134 hash value calculation unit 135 search unit 136 output unit 201 CPU
202... Input device 203... Monitor 204... Speaker 205... Medium reading device 206... Interface device 207... Communication device 208... RAM
209 Hard disk device 210 Bus 211 Program 212 Various data A1 to A2 Axis W1 to W7 Word

Claims (9)

Based on a set of words included in the search target document and inter-word information indicating the closeness of meaning of the words, for each word included in the set of words, a predetermined word is used as a reference for the word. Generate a hash function that assigns close values in order of meaning,
calculating summary information for each of the plurality of search target documents based on the generated hash function;
calculating summary information of the input document based on the generated hash function;
searching for a document similar to the input document from among the plurality of search target documents based on a comparison between the calculated summary information of the search target document and the summary information of the input document;
A similar document search method characterized in that processing is executed by a computer. In each of the calculating processes, a minimum value among values assigned to each word in a set of words included in the search target document or the input document in the generated hash function is calculated as a hash value.
2. The similar document search method according to claim 1, characterized by: In the generating process, a plurality of hash functions are generated by repeating the process of reselecting the predetermined word and generating the hash function,
In each of the calculating processes, a vector including a plurality of hash values calculated from a set of words included in the search target document or the input document by the plurality of generated hash functions is calculated as summary information.
3. The similar document retrieval method according to claim 2, characterized by: a computer further executing a process of generating index information in which the calculated summary information of each of the plurality of search target documents and the search target documents are associated with each other;
The searching process compares summary information of the input document with summary information associated with the search target document in the index information.
2. The similar document search method according to claim 1, characterized by: Based on a set of words included in the search target document and inter-word information indicating the closeness of meaning of the words, for each word included in the set of words, a predetermined word is used as a reference for the word. Generate a hash function that assigns close values in order of meaning,
calculating summary information for each of the plurality of search target documents based on the generated hash function;
calculating summary information of the input document based on the generated hash function;
searching for a document similar to the input document from among the plurality of search target documents based on a comparison between the calculated summary information of the search target document and the summary information of the input document;
A similar document search program characterized by causing a computer to execute processing. Based on a set of words included in the search target document and inter-word information indicating the closeness of meaning of the words, for each word included in the set of words, a predetermined word is used as a reference for the word. a hash function generation unit that generates a hash function that assigns values close to each other in order of meaning;
a first calculation unit that calculates summary information of each of the plurality of search target documents based on the generated hash function;
a second calculation unit that calculates summary information of an input document based on the generated hash function;
a search unit that searches for a document similar to the input document from among the plurality of search target documents based on a comparison between the calculated summary information of the search target document and the summary information of the input document;
A similar document retrieval device characterized by comprising: Based on a set of words included in the search target document and inter-word information indicating the closeness of meaning of the words, for each word included in the set of words, a predetermined word is used as a reference for the word. Generate a hash function that assigns close values in order of meaning,
calculating summary information for each of the plurality of search target documents based on the generated hash function;
generating index information for searching summary information of each of the plurality of calculated search target documents;
An index information creation method characterized in that processing is executed by a computer. Based on a set of words included in the search target document and inter-word information indicating the closeness of meaning of the words, for each word included in the set of words, a predetermined word is used as a reference for the word. Generate a hash function that assigns close values in order of meaning,
calculating summary information for each of the plurality of search target documents based on the generated hash function;
generating index information for searching summary information of each of the plurality of calculated search target documents;
An index information creation program characterized by causing a computer to execute processing. Based on a set of words included in the search target document and inter-word information indicating the closeness of meaning of the words, for each word included in the set of words, a predetermined word is used as a reference for the word. a hash function generation unit that generates a hash function that assigns values close to each other in order of meaning;
a calculation unit that calculates summary information for each of the plurality of search target documents based on the generated hash function;
an index information generation unit that generates index information for searching for summary information of each of the plurality of calculated search target documents;
An index information creation device characterized by having:

Images