JPH10269235A - Device and method for similar document retrieval - Google Patents

Abstract

PROBLEM TO BE SOLVED: To retrieve a document by precise similarity calculation by finding the similarity according to the distribution state of other words, which are present nearby a word present in both a comparison source document and a comparison destination document in common. SOLUTION: A similarity calculation part 20b calculates the similarity between the comparison source document and comparison destination document by using a weight coefficient calculated by a weight coefficient determination part 20C. The weight coefficient determination part 20C calculates a weight coefficient large as to a common word in both the comparison source document and comparison destination document according to the total number of words, which are present within a certain distance from the word common to both the comparison source document and comparison destination document, which is decided by a word deciding function and present in both the comparison source document and comparison destination document in common. Then, a document list display part 20f displays a list of document titles on the screen of a display device in order from a document having large similarity together with corresponding similarity information.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a similar document search apparatus and a similar document search method for searching for a document based on the similarity between a comparison source document and a comparison destination document.

[0002]

2. Description of the Related Art In recent years, a large amount of electronic document data has been distributed, and a system for retrieving a document similar to a designated document from a document database for the purpose of automatic classification and the like has been put to practical use. Is coming.

In these systems, when searching for a similar document similar to a designated document (which is called a comparison source document), a comparison source document and each document in a database (which is called a comparison destination document) are searched. Is calculated, and a comparison destination document having a large similarity value is output as a search result.

As a method of calculating the degree of similarity, a method based on a vector space model that increases the degree of similarity as two documents include more common words is generally used. In this method, first, words are extracted from both a comparison source document and a comparison destination document, and vector data is created from each of them by using the word weight as an element.

As a weight for a word, a positive value is used when the word exists, and 0 is set when the word does not exist. As the value of the weight, the frequency of occurrence of each word in the document is used. After calculating vector data for words extracted from both the comparison source document and the comparison destination document, the inner product of them is calculated, and this is used as the similarity.

FIG. 13 shows the concept of a conventional similarity calculation method. As shown in FIG. 13, the comparison source document includes
Words A, B, C, D, E, and F are included. On the other hand, the comparison target document includes the words A, C, E, G, H, and I. In this case, word A, word C, and word E
However, since one exists for each of the comparison source document and the comparison destination document, the similarity for each combination of words is 1, and the similarity between the comparison source document and the comparison destination document is 3.

[0007] In addition, weights can be given according to co-occurrence relationships between words. In this case, a co-occurrence relationship between words is pre-dictionaryed, and a word having a co-occurrence relationship registered in this dictionary is given a large weight to calculate a similarity reflecting presence or absence of a related word. can do.

[0008]

As described above, according to the conventional similarity calculation method, it is necessary to weight each word when calculating the similarity of a document.
When the word A exists in both the comparison source document and the comparison destination document, the value of the product of the appearance frequency of the word A in the comparison source document and the comparison destination document is exclusively used as the weight. However, the appearance of other words near word A that actually characterizes the contents of the document is not taken into account.

Although weights can be given according to co-occurrence relations, co-occurrence relations between words must be made in a dictionary in advance, and if not registered in the dictionary, the appearance of other words should be considered. Will not be done.

[0010] For this reason, it is difficult for the similar document search apparatus using the similarity calculation in the related art to obtain a document search result with sufficient accuracy. The present invention has been made in consideration of the above circumstances, and has high accuracy similarity calculation in consideration of the distribution of other words near the words existing in both the source document and the destination document. It is an object of the present invention to provide a similar document search device and a similar document search method capable of searching for a document by using the same.

[0011]

SUMMARY OF THE INVENTION According to the present invention, there is provided a similar document retrieval apparatus for retrieving a comparison destination document similar to the comparison source document based on the comparison source document. The similarity between the comparison source document and the comparison destination document is obtained based on the distribution state of other words existing in the vicinity of the word that is commonly present, and the document is searched.

Further, according to the present invention, based on a comparison source document,
In a similar document search device that searches for a comparison destination document similar to the comparison source document, a word extraction unit that extracts a word from each of the comparison source document and the comparison destination document, and a word extracted by the word extraction unit Word information storage means for storing word information together with appearance position information in each document; word determination means for extracting common words which are present in common in a comparison source document and a comparison destination document; and the word information storage means For searching for a word that is present within a certain distance from the common word extracted by the word determination means for both the comparison source document and the comparison destination document based on the word information stored by A search unit and a common word in both the comparison source document and the comparison target document according to the total number of words existing within a certain distance from the common word searched by the word search unit. Characterized by comprising a weight calculation unit for calculating the weights for the word.

[0013] Further, a comparison target document similar to the comparison source document is searched by using a method based on a vector space model, wherein the weight calculated by the weight calculation means is used as a weight coefficient, and Term data calculating means for calculating one term data of an inner product between word vectors for the word extracted by the word extracting means, using the weighting factor obtained by the word extracting means, and normalizing the inner product data by using the weighting factor. And an inner product data normalizing means for performing the following.

With such a configuration, in the process of calculating the similarity, the weight of the common word can be set larger as the number of words that are commonly present within a certain distance from the common word is increased. It is possible to calculate the similarity with high accuracy in consideration of not only the numerical information but also the appearance of other words around the word. Furthermore, in the prior art, in order to give a large weight between words having a co-occurrence relation, it was necessary to previously compile the co-occurrence relation into a dictionary.

Further, the weighting factor is a real number of 1 or more, and a comparison is made between each word existing within a certain distance from the common word with respect to both the comparison source document and the comparison destination document. It is characterized in that the smaller the distance from the common word in the original document and the compared document, the larger the value.

In this way, the weight of the common word is set to be larger as the words that appear in both the comparison source document and the comparison target document are more concentrated around the common word in calculating the inner product. , And accurate weighting becomes possible.

[0017]

Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram illustrating a configuration of a similar document search device according to the present embodiment. The similar document retrieval apparatus reads a program recorded on a recording medium such as a magnetic disk, and is realized by a computer whose operation is controlled by the program.

As shown in FIG. 1, the similar document search apparatus according to the present embodiment includes a control device 10, an input device 11, a display device 12, an external storage device 13, a memory 14, and a communication device 15. Are interconnected via

The control device 10 is composed of a CPU, is connected to various hardware devices via a bus, and performs processes such as control of each device and transfer of data between the devices. .

The input device 11 is composed of a keyboard, a mouse, and the like. The input device 11 inputs various data and commands to the device and outputs the data and commands to the control device 10. The display device 12 is composed of a color liquid crystal display and its controller, and displays a document and a search result (title list) under the control of the control device 10.

The external storage device 13 comprises a recording medium such as a hard disk and a controller, and stores document data to be searched and word information data corresponding to each document. The storage format of the document data and word information data stored in the external storage device 13 will be described later (FIG. 3).

The memory 14 is composed of a dynamic RAM, and is provided with a program section for storing programs for the control device 10 to execute various controls and processes, and a buffer section for storing data necessary for the processes. ing.
Detailed configurations of the program section and the buffer section will be described later (FIG. 2).

The communication device 15 is a device for exchanging data with the outside through a communication line under the control of the control device 10, and includes, for example, a LAN line and a LAN controller.

Next, the details of the program section 20 and the buffer section 22 provided in the memory 14 will be described with reference to FIG. As shown in FIG.
Is a similarity calculation unit 20 as a subroutine called by the main processing unit 20a in addition to the main processing unit 20a.
b, a weight coefficient determining unit 20c, a term data adding unit 20d, an inner product data normalizing unit 20e, a document list displaying unit 20f, a document selecting unit 20g, and a document content displaying unit 20h.

The main processing section 20a controls the entire processing and calls and executes each section as a subroutine. The main processing unit 20a includes a word extraction function of extracting words from documents to be processed (a comparison source document and a comparison target document) by, for example, Japanese analysis processing, and the number of appearances of the words extracted by the word extraction function in the document. And a word information storage function of storing word information data together with information indicating an appearance position.

The similarity calculating section 20b includes a weight counting determining section 2
The similarity between the comparison source document and the comparison destination document is calculated using the weight count calculated by 0c. In addition, the similarity calculation unit 20b implements a word determination function of determining whether a word (common word) common to the comparison source document and the comparison destination document exists.

The weighting factor determining unit 20c determines whether or not the comparison source document and the comparison destination document, which are within a certain distance from a word (common word) common to the comparison source document and the comparison destination document, determined by the word determination function. According to the total number of commonly existing words, the weight coefficient for the common word in both the comparison source document and the comparison destination document is calculated to be large. Further, the weight count determining unit 20c calculates and calculates the value of the coefficient so as to be a real number of 1 or more, and calculates the coefficient value for both the comparison source document and the comparison destination document.
The larger the distance between the common word and the common word in each of the comparison source document and the comparison target document within a certain distance from the common word, the larger the value.

The term data adder 20d calculates and obtains one term data of the inner product between the word vectors by using the weight coefficient for the word obtained by the weight count determiner 20c.

The inner product data normalizing section 20e normalizes the inner product data using the weight coefficient obtained by the weight count determining section 20c. The document list display unit 20f stores the content (similarity information) stored in the similarity storage buffer 22d,
Then, referring to the document title data in each document data, a list of the document titles is displayed on the screen of the display device 12 together with the corresponding similarity information in order from the document having the highest similarity.

The document selection section 20g is provided with a document list display section 20.
The user is caused to select one or more document titles from the list of titles displayed on the screen of the display device 12 by using the input device 11 by f.

The document content display section 20h is provided with a document selection section 20.
The content of the document corresponding to the document title selected by g is displayed on the screen of the display device 12. The buffer unit 22 is provided with a comparison source document data storage buffer 22a, a comparison source word information storage buffer 22b, a comparison destination word information storage buffer 22c, a similarity storage buffer 22d, and an area 22e for work variables. .

The comparison source document data storage buffer 22a
It is for storing document data of a comparison source document, and can store document data in a structure shown in FIG. 3 described later.

The comparison source word information storage buffer 22b is used to store words extracted from the document data of the comparison source document together with information indicating the number of appearances and the appearance position in the document.

The comparison destination word information storage buffer 22c is used to store words extracted from the document data of the comparison destination document together with information indicating the number of appearances and the appearance position in the document.

The comparison source word information storage buffer 22b and the comparison destination word information storage buffer 22c store the extracted words,
Information indicating the number of appearances and the appearance position of each word in the document is stored as word information data in a format shown in FIG. 5 described later.

The similarity storage buffer 22d is used to store the similarity of each document with respect to the comparison destination document. In addition, an area 22e (work variable area 22e) for each work variable is secured in the buffer unit 22. The work variable area 22e includes a document ID count variable idDoc, a count variable iSrc, iDst, kSrc, kDst, a similarity storage variable S, and a weight coefficient variable coefSrc, coefDs used in a process described later.
An area for each variable such as t is reserved.

Next, the storage format of document data and word information data stored in the external storage device 13 will be described with reference to FIG. Each document data includes a header section and a text data section, and the header section includes data representing each attribute of the document such as title data, creation date / time data, and creator data. Document data is ID
Are stored in numerical order (0, 1, 2, ..., N-1).
Further, word information data is stored in the document data in association with each other. The word information data stores words (nouns in the present embodiment) extracted by morphological analysis on the corresponding document data (text data). In addition, information indicating the number of appearances and the appearance position in the document is stored for each word.

FIG. 4 shows an example of a document (text).
FIG. 5 shows an example of word information data corresponding to the document shown in FIG. As shown in FIG. 5, in the word information data, the number of appearances of each word in the text and each appearance position are stored.

The word information data may be provided together with the corresponding document data, or may be obtained from the document data of the comparison source document and the comparison target document by the word extraction function and the word information storage function. May be.

Next, the overall operation of the similar document search apparatus according to this embodiment will be described with reference to the flowchart shown in FIG. The similar document search device according to the present embodiment, when a word (common word) common to both the comparison source document and the comparison target document exists, as a weight for the common word, the distribution state of the word existing near the common word Is calculated in consideration of the similarity. FIG. 6 illustrates the concept of the similarity calculation method according to the present embodiment. As shown in FIG. 6, when a word A common to the comparison source document and the comparison destination document exists, if the same word, for example, the words C and E, exists within a certain distance from the word A, It can be determined that the comparison source document and the comparison destination document are similar. The similar document search device according to the present embodiment enables a highly accurate similarity search to be performed by reflecting the distribution state of the words existing in the vicinity of the common word in the weight (weight coefficient) for the word A. Things.

Here, document data and word information data in the format shown in FIG.
It is assumed that a total of N numbers from 0 to N−1 are stored.

The main processing unit 20a stored in the program unit 20 of the memory 14 controls the entire processing.
First, the input device 11 or the communication device 15
The input document data is stored in the comparison source document data storage buffer 22a (step A1). The comparison source document data includes a header portion representing each attribute of the document such as title data, creation date / time data, and creator data, and a text data portion.

Next, the main processing unit 20a performs a morphological analysis on the text data of the comparison source document data stored in the comparison source document data storage buffer 22a by the word extraction function, and identifies the nouns contained in the text. Extract (step A2).

The main processing section 20a creates the comparison source word information data by the word information storage function and stores it in the comparison source word information storage buffer 22b (step A3). The structure of the comparison source word information data is the same as the structure of the word information data previously shown in FIG. 5, and stores the number of appearances of each word and the appearance position information in the text.

Next, in order to sequentially process each of the plurality of comparison target documents, the main processing unit 20a sets the value of the document ID counting variable idDoc indicating the comparison target document to the initial value 0, and sets the work variable It is stored in the area 22e (step A4).

The main processing section 20a reads the word information data corresponding to the document ID counting variable idDoc from the external storage device 13 and stores it in the comparison target word information storage buffer 22c (step A5).

Subsequently, the main processing section 20a activates the similarity calculating section 20b. The similarity calculation unit 20b compares the word information data of the comparison source document stored in the comparison source word information storage buffer 22b with the comparison target word information storage buffer 2b.
With reference to the word information data of the comparison source document stored in 2c, the comparison source document and the document ID counting variable idD
The similarity with the comparison destination document indicated by oc is calculated (step A6).

Hereinafter, the details of the similarity calculation processing by the similarity calculation unit 20b in step A6 will be described with reference to the flowchart shown in FIG. here,
The headword of the i-th word stored in the comparison source document word information data is wordSrc [i], the number of occurrences is nApSrc [i], and the j-th occurrence position is posS
rc [i] [j], the headword of the i-th word stored in the comparison target document word information data is wordDst
[I], the number of appearances is represented by nApDst [i], and the jth appearance position is represented by posDst [i] [j]. The weight coefficient of each word vector of the source document is a real variable coefSrc [iSrc], and the weight coefficient of each word vector of the target document is a real variable coef.
Dst [iDst], the total number of words stored in the comparison source document data is represented by nSrc, and the total number of words stored in the comparison destination word information data is represented by nDst.

First, the similarity calculation unit 20b stores the similarity storage variable S of the comparison target document currently being processed as the initial value 0 in the work variable area 22e (step B1). Next, the similarity calculation unit 20b substitutes 0 for a variable iSrc indicating a word of the comparison source document (step B).
2) the weighting factor co for the word indicated by the variable iSrc
The initial value 1.0 is substituted for efSrc [iSrc] (step B3).

Next, the similarity calculating section 20b substitutes 0 for a variable iDst indicating a word of the comparison destination document (step B).
4), the weighting coefficient co for the word indicated by the variable iDst
The initial value 1.0 is substituted for efDst [iDst] (step B5).

Next, the similarity calculation unit 20b uses the word determination function to execute the word wordSrc [iSr
c] and the word wordDst [iDst] of the compared document
Are compared, that is, whether they are common words (step B6). If equal, step B
When control is transferred to step 7 and the values are not equal, step B
The control moves to 9.

If not equal, iSr here
weighting factor for the word indicated by c and iDst weighting factor c
oefSrc [iSrc] and coefDst [iD
The value of [st] is not updated and is set as 1.0.

On the other hand, in step B7, the weight coefficient determining section 20c is activated. The weight count determination unit 20c determines the word wor which is a common word among the word vectors of the comparison target document.
dSrc [iSrc] and the word wordDst [iD
st], the weighting factor variable coef related to the element corresponding to
Src [iSrc] and coefDst [iDst]
Is updated according to the amount of words that are commonly present in the comparison source document and the comparison destination document within a certain distance from the common word (the weight increases as the number of words increases). The details of the weight coefficient determination process by the weight count determination unit 20c will be described later (the flowchart shown in FIG. 9).

Next, the term data adder 20d is activated. The term data adding unit 20d calculates the weight coefficient variable co determined by the weight coefficient determining process by the weight counting determining unit 20c.
With reference to the value of efSrc [iSrc], one term data of the inner product of the word vector is added to the similarity storage variable S (step B8). Specifically, the term data adder 20
The addition processing at d is performed according to the following equation (1).

[0055]

(Equation 1)

In the equation (1), nApSrc [i
Src] is the number of appearances of the comparison source word, nApDst [iD
[st] is the number of appearances of the comparison target word, which are used as weights in general inner product calculation. The weight coefficient variable coefSrc obtained above is added to these products.
[ISrc], coefDst [iDst].

When the addition process is completed, the process moves to step B9. In step B9, the similarity calculating unit 20b determines
In order to process the next word of the compared document, the variable iD
Add 1 to the value of st. Here, if the value of the variable iDst is smaller than the total number nDst of words in the comparison target document, that is, if the processing for all the words in the comparison target document has not been completed, the process returns to step B5 to repeat the processing.

On the other hand, when the value of the variable iDst is equal to or more than the total number nDst, the processing of each word of the comparison target document is completed for the word indicated by the variable iSrc which has been processed so far in the comparison source document. In order to process the next word in the document, 1 is added to the value of the variable iSrc (step B11). Here, if the value of the variable iSrc is smaller than the total number nSrc of words of the comparison source document, that is, if the processing for all the words of the comparison source document has not been completed, the process returns to step B3 and repeats the processing.

When the value of the variable iSrc is equal to or greater than the total number nSrc, the inner product data normalizing section 20e is activated. The inner product data normalizing unit 20e normalizes the inner product data by dividing the value of the similarity storage variable S by the product of the size of each word vector (step B13). The specific calculation formula (2) used in the processing in the inner product data normalization unit 20e is shown below.

[0060]

(Equation 2)

When the processing by the inner product data normalizing section 20e is completed, the similarity calculation processing in step A6 of FIG. 7 is completed. Next, details of the weighting factor determination process performed by the weighting factor determination unit 20c performed in step B7 of FIG. 8 will be described with reference to a flowchart illustrated in FIG.

In the weighting factor determination process, the word wordSrc of the comparison source document exists in both the comparison source document and the comparison destination document, and in each of the comparison source document and the comparison destination document.
It is checked whether it exists within a certain distance from [iSrc], and based on the number, a weighting coefficient variable coefSrc [iSrc
rc] is reset (initial value is set to 1.0)
And the word wordDst [iDst] of the compared document
It is checked whether it exists within a certain distance from the data, and the value of the weight coefficient variable coefDst [iDst] is reset (set to the initial value 1.0) based on the number.

First, the weight count determining unit 20c calculates the total number sumSrc representing the total number of words satisfying the condition corresponding to the word wordSrc [iSrc] of the comparison source document, and the word wo
A total sumDst representing the total number of words satisfying the condition corresponding to rdDst [iDst] is initialized by substituting 0 for each (steps C1 and C2).

Further, the weight count determining unit 20c initializes a variable kSrc representing a word of the comparison source document by substituting 0 (step C3). Next, the weight count determining unit 20c
The variable iSrc (indicating a common word) is compared with the variable kSrc, and if the values of the two variables are the same, the same word is represented, so that control is transferred to step C14 (step C4). In step C14, the process moves to the next word of the comparison source document by incrementing the value of the variable kSrc. If the value of the incremented variable kSrc is smaller than the total number nSrc of words of the comparison source document, the control is transferred to step C4 (step C15).

On the other hand, if the variable iSrc is different from the variable kSrc, the weight count determining unit 20c initializes the variable kDst by substituting 0 into a variable kDst representing a word of the document to be compared (step C5).

Next, the weight count determining unit 20c sets the variable iD
st (indicating a common word of the comparison target document) is compared with the variable kDst (step C6). As a result, if the values of the two variables are the same, they represent the same word, so control is performed in step C1.
2 (step C6). In step C12, the process is moved to the next word of the comparison target document by incrementing the value of the variable kDst. Incremented variable kD
If the value of st is smaller than the total number nDst of words in the comparison target document, the control is transferred to step C6 (step C13).

On the other hand, if the variable iDst is different from the variable kDst, the weight count determination unit 20c compares the word wordSrc [kSrc] indicated by the variable kSrc with the word wordDst [kDst] of the comparison target document (step S12). C7).

The weight count determining unit 20c calculates two words wo
rdSrc [kSrc] and the word wordDst [kDs
If t] is equal, control is transferred to step C8, and if they are not equal, control is transferred to step C12.

If not equal, the value of the variable kDst is incremented in order to obtain an equal word, and the processing is shifted to the next word of the document to be compared (step C1).
2). If the incremented value of the variable kDst is smaller than the total number nDst of words of the compared document, step C
The control is transferred to 6 (step C13). Below, the word wor
dSrc [kSrc] and the word wordDst [kDs
t] is equalized, or the same processing is repeated until the determination for all words in the comparison target document is completed.

In step C8, the weight count determining section 20c
Is the common word wordSrc [iSr
c] and the same word wordSr that existed in the compared document
It is determined whether or not the distance from c [kSrc], that is, the absolute value of the difference between the appearance positions of the two words is within a certain range (for example, 80). If the number of appearances of the word wordSrc [kSrc] is two or more, it is determined whether the distance from the common word wordSrc [iSrc] is within a certain range based on all occurrence positions. Then, the weight count determination unit 20c calculates the total number of appearance positions countSrc that satisfies the condition for the word represented by the variable kSrc.
Ask for. The specific processing here will be described later with reference to FIG.
This will be described with reference to the flowchart of FIG.

The weight count determining unit 20c adds the total countSrc for the word represented by the variable kSrc obtained in step C8 to the total sumSrc (step C9).

In step C10, the weight count determining unit 20c determines the common word wordDst [i in the compared document.
Dst] and the same word word existing in the comparison source document.
It is determined whether the distance from Dst [kDst], that is, the absolute value of the difference between the appearance positions of the two words is within a certain range (for example, 80). If the number of appearances of the word wordDst [kDst] is two or more, it is determined whether the distance from the common word wordDst [iDst] is within a certain range based on all the appearance positions. Then, the weight count determination unit 20c calculates the total number of appearance positions co that satisfies the condition.
Find untDst. The specific processing here will be described later using the flowchart of FIG.

The weight count determining unit 20c determines in step C10
Then, the total countDst for the word represented by the variable kDst calculated in step (1) is added to the total sumDst (step C11).

Next, the weight count determining unit 20c shifts the control to step C14 to increment the value of the variable kSrc in order to shift the processing to the next word of the comparison source document.
If the incremented value of the variable kSrc is smaller than the total number nSrc of words in the comparison source document, control is transferred to step C4 (step C15), and thereafter, similarly, for the word indicated by the next variable kSrc in the comparison source document Execute the process.

In step C12, the variable kDs
As a result of incrementing the value of t, if the value is not smaller than the total number nDst of words in the comparison target document, the word w
All words word for ordSrc [iSrc]
Since the processing for Dst [iDst] has been completed, the processing moves to step C14 in order to move the processing to the next word of the comparison source document.

In step C14, the variable kSr
If the value of t is not smaller than the total number of words nSrc in the comparison source document as a result of incrementing the value of t, the processing has been completed for all the words wordSrc [iSrc] in the comparison source document, so that step C16 is performed. Transfer processing to

In the processing so far, the word wordS of the comparison source document exists in both the comparison source document and the comparison destination document.
The total number of words sumSrc existing within a certain distance from rc [iSrc] is the variable kSrc (0 to 0) corresponding to the condition.
nSrc-1) is obtained as the sum of the total countSrc obtained from the processing for each word represented by (nSrc-1) (details will be described later with reference to FIG. 10).

Similarly, the word wordD exists in both the comparison source document and the comparison destination document and is included in the comparison destination document.
The total number of words sumDst existing within a certain distance from st [iDst] is the variable kDst corresponding to the condition.
It is obtained as the sum total of the total countDst obtained from the processing (details will be described later with reference to FIG. 11) for each word represented by (0 to nDst-1).

The weight count determining unit 20c refers to the value of the total sumSrc obtained earlier and refers to the word wor of the comparison source document.
Weight coefficient variable coefS related to dSrc [iSrc]
The value of rc [iSrc] is calculated (step C16).
The value of coefSrc [iSrc] is sum to 1
The value obtained by adding the value of the square root of Src is used.

Similarly, the weight count determining unit 20c refers to the value of sumDst obtained earlier and refers to the weight coefficient variable c relating to the word wordDst [iDst] of the comparison target document.
Calculate the value of oefDst [iDst] (step C
17). In the similar document search device according to the present embodiment, c
The value of oefDst [iDst] is sumD to 1
The value obtained by adding the value of the square root of st is used.

By the above processing, the weight count determining unit 20c
Terminates the weighting factor determination process according to. Next, the details of the processing in step C5 shown in FIG. 9 will be described with reference to the flowchart shown in FIG.

The process in step C5 refers to the word wordSrc [iSrc] appearing in the text of the comparison source document and all occurrence positions of the word wordSrc [kSrc], and determines the absolute value of the distance, that is, the difference between the appearance positions. This is a process of calculating the total countSrc that is less than or equal to 80.

First, the weight count determining unit 20c calculates the total number co
After the value of untSrc is initialized to 0 (Step D1), in the loop from Step D2 to Step D10, the value of j1 corresponding to each appearance position of the word wordSrc [iSrc] is changed from 0 to the number of appearances nApSrc [iSrc].
rc], and step D3 inside this loop.
In the loop from to D8, the word wordSrc
The value of j0 corresponding to each appearance position of [kSrc] is changed from 0 to the number of appearances nApSrc [kSrc].

That is, for all combinations of j0 and j1, the appearance position p of the word represented by the variable kSrc
osSrc [kSrc] [j0] and the occurrence position posSrc [iSrc] [j of the word represented by the variable iSrc
1], that is, the absolute value of the difference dSrc (calculation formula (3) shown below) is calculated (step D4).

[0085]

(Equation 3)

Further, when the value of dSrc is smaller than the predetermined distance 80 in the comparison in step D5, the weight count determination unit 20c increments the value of the total countSrc by the processing in step D6.

Next, details of the processing in step C7 shown in FIG. 9 will be described with reference to the flowchart shown in FIG. The processing in step C7 is performed by using the word wordDst [iDs
t] and all occurrence positions of the word wordDst [kDst], and a process of calculating the total countDst where the absolute value of the distance, that is, the difference between the appearance positions becomes 80 or less, is performed.

First, the weight count determining unit 20c calculates the total number co
After the value of untDst is initialized to 0 (step E1), in the loop from step E2 to step E10, the value of j1 corresponding to each appearance position of the word wordDst [iDst] is changed from 0 to the number of appearances nApDst [iD
st], and step E3 inside this loop.
In the loop from step E8 to the word wordDst
The value of j0 corresponding to each appearance position of [kDst] is changed from 0 to the number of appearances nApDst [kDst].

That is, for all combinations of j0 and j1, the appearance position p of the word represented by the variable kDst
osDst [kDst] [j0] and the appearance position posDst [iDst] [j of the word represented by the variable iDst
1], that is, the absolute value of the difference dDst (calculation formula (4) shown below) is calculated (step E4).

[0090]

(Equation 4)

Further, if the value of dDst is smaller than the predetermined distance 80 in the comparison in step E5, the weight count determining unit 20c increments the total countDst value by the processing in step E6.

In the above-described processing shown in the flowcharts of FIGS. 10 and 11, cou is considered by considering only the total number of words existing near the words of the comparison source document and the comparison destination document.
Although the values of ntSrc and countDst are calculated, the similar document search apparatus according to the present embodiment specifies the distances from the words of the comparison source document and the comparison target document of each corresponding word by countSrc and countDst.
Can be reflected in the value of In this case, FIG.
In step D6 of 0, the value of the real type variable countSrc is calculated by the following formula (5).

[0093]

(Equation 5) In response to this, in step E6 of FIG. 11, the value of the real number type variable countDst is calculated by the following equation (6).

[0094]

(Equation 6)

As described above, step A shown in FIG.
After calculating the similarity by the processing of step 6, the obtained similarity is stored in the similarity storage buffer 22d at a position corresponding to the value of the document ID counting variable idDoc (step A7).

Next, the value of the document ID counting variable idDoc is incremented in order to proceed to the processing of the next comparison destination document (step A8). Further, the value of the variable idDoc is compared with the total number N of documents stored in the external storage device 13 (step A9). If the variable idDoc is smaller than the total number N of documents, the processing from step A5 is repeated. If not, control is transferred to step A10.

In step A10, the document list display section 20
f is activated. The document list display unit 20f refers to the content stored in the similarity storage buffer 22d and the title data in each document data, and sorts the title list together with the corresponding similarity information in order from the document having the highest similarity. It is displayed on the screen of the display device 12. FIG. 12 shows the state of the screen after the title list display.

Subsequently, the document selection section 4g is activated. The document selection unit 20g allows the user to select one or more of the titles displayed on the screen using the input device 11 (step A11).

Next, the document content display section 20h is activated. The document content display section 20 displays the content of the document corresponding to the title selected in step A11 on the screen of the display device 12 (step A12).

In this way, when a word (common word) common to both the comparison source document and the comparison destination document exists,
Since the similarity is obtained in consideration of the distribution of words existing near the common word as a weight for the common word and the document is searched, the document can be searched with high accuracy.

The present invention is not limited to the above embodiment. For example, in the addition processing in step B8 of the term data addition unit 20d, the above-described calculation formula (2) may not be used if the value of the weight coefficient variable is reflected in the result.

The calculation formula used in the processing of steps C11 and C12 by the weight coefficient determination unit 20c is
The calculation formula described above may not be used as long as the weight coefficient increases as the value of untSrc or countDst increases.

Further, step C5 in FIG. 7 (FIG. 10),
And C7 (FIG. 11) used a value of 80 when comparing the distances, but this can be set to any other appropriate value. In addition, various modifications can be made without departing from the spirit of the invention.

Further, the method described in the above-described embodiment can be executed by a computer as a program such as a magnetic disk (floppy disk, hard disk, etc.), an optical disk (CD-ROM,
It is also possible to write the data on a recording medium such as a DVD or a semiconductor memory and apply it to various devices, or to transmit it via a communication medium and apply it to various devices. A computer that realizes the present apparatus reads the program recorded on the recording medium, and executes the above-described processing by controlling the operation of the program.

[0105]

As described above in detail, according to the present invention, the similarity with high accuracy can be obtained in consideration of the distribution of other words near the words present in both the source document and the destination document. The document can be searched by the degree calculation.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a configuration of a similar document search device according to an embodiment of the present invention.

FIG. 2 is a diagram showing a detailed configuration of a program unit 20 and a buffer unit 22 provided in a memory 14.

FIG. 3 is a view for explaining a storage format of document data and word information data stored in an external storage device 13;

FIG. 4 is a diagram showing an example of a document (text).

FIG. 5 is a view showing an example of word information data corresponding to the document shown in FIG. 4;

FIG. 6 is a view showing the concept of a similarity calculation method according to the embodiment.

FIG. 7 is a flowchart for explaining the overall operation of the similar document search device according to the embodiment;

FIG. 8 is a flowchart illustrating details of a similarity calculation process performed by the similarity calculation unit 20b.

FIG. 9 is a flowchart illustrating details of a weight coefficient determination process performed by a weight coefficient determination unit 20c.

FIG. 10 is a flowchart for explaining details of a process for obtaining a total number countSrc of appearance positions.

FIG. 11 is a flowchart for explaining details of a process for obtaining a total number countDst of appearance positions.

FIG. 12 is a view for explaining the state of the screen after the title list display.

FIG. 13 is a view showing the concept of a conventional similarity calculation method.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 10 ... Control device 11 ... Input device 12 ... Display device 13 ... External storage device 14 ... Memory 15 ... Communication device 20 ... Program part 20a ... Main processing part 20b ... Similarity calculation part 20c ... Weight count determination part 20d ... Term data addition Unit 20e: inner product data normalizing unit 20f: document list display unit 20g ... document selection unit 20h ... document content display unit 22 ... buffer unit 22a ... comparison source document data storage buffer 22b ... comparison source word information storage buffer 22c ... comparison destination word Information storage buffer 22d ... Similarity storage buffer 22e ... Work variable area

 ──────────────────────────────────────────────────続 き Continued on the front page (72) Inventor Yukio Nakamoto 1381-1, Shinmachi, Ome-shi, Tokyo Toshiba Computer Engineering Co., Ltd. (72) Inventor Takuya Nishina 1381-1, Shinmachi, Ome-shi, Tokyo Toshiba Computer (72) Inventor Naohide Kubota 1381 Shinmachi, Ome-shi, Tokyo Toshiba Computer Engineering Co., Ltd.

Claims (6)

[Claims] 1. A similar document search device for searching a comparison destination document similar to the comparison source document based on the comparison source document, comprising: A similar document search apparatus, wherein a similarity between the comparison source document and the comparison destination document is obtained based on the distribution status of other words present in the vicinity, and the document is searched. 2. A similar document search apparatus for searching for a comparison destination document similar to the comparison source document based on the comparison source document, wherein a word for extracting a word from each of the comparison source document and the comparison destination document is provided. Extraction means; word information storage means for storing the word extracted by the word extraction means together with appearance position information in each document as word information; and a common word common to the comparison source document and the comparison destination document. A word determining unit for extracting a word; and, based on the word information stored by the word information storing unit, a constant from a common word extracted by the word determining unit for both a comparison source document and a comparison destination document. Word search means for searching for a word that is commonly present within a distance of, and according to the total number of words present within a certain distance from the common word searched for by the word search means, Similar document search apparatus characterized by comprising a weight calculation unit for calculating the weights for common words in both of the comparison target document and 較元 document. 3. Searching for a comparison destination document similar to the comparison source document using a method based on a vector space model, wherein the weight calculated by the weight calculation unit is used as a weight coefficient, Using the weighting factor obtained by
Term data calculating means for calculating one term data of an inner product between word vectors for the word extracted by the word extracting means; and inner product data normalizing means for normalizing the inner product data using the weighting coefficient. 3. A similar document search apparatus according to claim 2, wherein: 4. The weighting factor is set to be a real number of 1 or more, and for each word that is present within a certain distance from the common word for both the comparison source document and the comparison destination document,
4. The similar document search device according to claim 3, wherein the smaller the distance from the common word in the comparison source document and the comparison destination document, the larger the value. 5. A similar document search method for searching for a comparison destination document similar to the comparison source document based on the comparison source document, comprising: A similar document method, wherein a similarity between the comparison source document and the comparison destination document is obtained based on a distribution state of other words existing in the vicinity, and a document is searched. 6. A similar document search method for searching for a comparison destination document similar to the comparison source document based on the comparison source document, wherein a word is extracted from each of the comparison source document and the comparison destination document. The extracted words are stored as word information together with the appearance position information in each document, and common words that are common to the comparison source document and the comparison destination document are extracted. Then, for both the comparison source document and the comparison destination document, a word that exists within a certain distance from the extracted common word is searched, and a word that exists within a certain distance from the searched common word is searched. Calculating a weight for a common word in both the comparison source document and the comparison destination document according to the total number of words to be compared, and determining the similarity between the comparison source document and the comparison destination document using the calculated weight. To search for similar documents Law.