JP6740877B2 - Similarity calculation program, similarity calculation method, and similarity calculation device - Google Patents

Description

The present invention relates to a similarity calculation program, a similarity calculation method, and a similarity calculation device.

In recent years, it has been performed to compare documents and extract similar portions. For example, character strings of two sentences are arranged in rows and columns, and when extracting similar portions of character strings based on the similarity between characters, the number of characters that do not match or are skipped consecutively from the end point is limited. , A technique for exhaustively extracting typical local correspondences by resetting and restarting the extraction is known.

Further, there has been proposed a technique of displaying the distribution state of the character strings in the document that match or is similar to the search character string and the degree of matching thereof.

JP 2012-59100 A JP-A-7-146872 JP-A-8-249445

M. J. Fischer and M. S. Paterson:String-matching and other products, Complexity of Computation (Proceedings of the SIAM-AMS Applied Mathematics Symposium, New York, 1973), pp. 113-125, 1974 D. Gusfield: Algorithms on Strings, Trees and Sequences: Computer Science and Computational Biology, Cambridge University Press, 1997 M. J. Atallah et al. :A randomized algorithm for approximate string matching. Algorithmica, 29:468-486, 2001 K. Baba et al. :A Note on Randomized Algorithm for String Matching with Mismatches, Nordic Journal of Computing, 10(1):2-12, 2003 T. Schoenmeyr and D. Yu-Zhang: FFT-based Algorithms for the String Matching with Mismatches Problem, Journal of Algorithms, 57:130-139, 2005

The above-mentioned technique is a technique of calculating the similarity between two documents at positions corresponding to each other, extracting a character string that matches or is similar to the search character string given to the document to be searched. Therefore, when the similarity between documents is calculated for all two documents with respect to all positional deviations, the processing load increases according to the number of character types in the document and the length of the extracted character string due to document breaks. There's a problem.

Therefore, in one aspect, the present invention aims to reduce the amount of calculation when comparing documents by character-by-character comparison.

According to one aspect, a plurality of vectors obtained by vectorizing the plurality of document data are generated for each character or character string included in the plurality of document data, and each vector of the plurality of vectors is subjected to Fourier transform. A product is calculated, and for each of the character or character string, a summation vector is generated by adding the product for each element, and a Fourier transform is used from the summation vector to calculate a correlation value between the plurality of document data. There is provided a similarity calculation program in which a computer performs a process of generating.

Further, as a means for solving the above problems, a similarity calculation method and a similarity calculation device can be used.

By comparing characters, it is possible to reduce the amount of calculation when comparing documents.

It is a figure for explaining position gap. It is a figure which shows the example of the correlation between documents. It is a figure for demonstrating the relationship between calculation and calculation time. It is a figure which shows the conceptual diagram of a calculation method. It is a figure for demonstrating reduction of calculation amount. It is a figure which shows the hardware constitutions of a similarity calculation apparatus. It is a figure which shows the functional structural example of a similarity calculation apparatus. 9 is a flowchart illustrating a first similarity calculation process for obtaining a correlation between two documents. It is a figure which shows the process example in the flowchart of FIG. It is a flowchart for demonstrating the 2nd similarity calculation process which calculates|requires 1 to N correlation. It is a figure which shows the comparison result of the execution time with respect to the number of vocabularies. It is a figure which shows the comparison result of the execution time with respect to a document length. It is a figure which shows the comparison result of the number of vocabularies in a document, and processing time.

Embodiments of the present invention will be described below with reference to the drawings. First, consider the problem of inputting the character strings of two documents and calculating the number of character matches (correlation) for all misregistrations. The "correlation" means the number of matches with the corresponding words in all the positional deviations.

As a related technique for finding a correlation between sentences, there is a fast Fourier transform (FFT)-based algorithm for finding a correlation. FFT stands for Fast Fourier Transform.

In the process of the FFT-based Algorithm, the convolution operation of O(nlogn) time is repeated σ times. Here, σ indicates the number of types of characters or character strings, and n indicates the length of the character string of each sentence.

Next, consider the correlation calculation time. As an example of correlation calculation, a simple calculation that sequentially compares words and a calculation amount using FFT-Based Algorithm are compared.

Basically, the FFT-Based Algorithm is faster than simple calculation when σ≦n can be considered. That is, when σ is constant and small, it can be regarded as a comparison between O(n ² ) and O(nlogn). However, when σ is indefinite and small, the advantage of O(σnlogσ) over O(n ² log σ) is small. That is, when σ is large, it is not assumed, and a constant and small σ is assumed, and the calculation time is treated as O(nlogn) (Non-Patent Documents 1 and 2). Therefore, there is a problem that the number of FFT executions increases.

To solve this problem, it has been proposed to perform approximation using only the result of k iterations (Non-Patent Documents 3, 4, and 5). However, since the convolution operation is treated as one processing unit, the number of times the convolution operation is processed is reduced.

Furthermore, in such an approximate method (Non-patent documents 3, 4, and 5), an approximate value is output instead of an exact value. For a document of length n, the variance of the estimate of the correlation c _i over k iterations is given by (nc _i )/k. It represents the degree to which the approximate values are scattered with respect to the correct value c _i . Large error for large n, not suitable for long documents. Furthermore, the error is large for small c _i . It shows that the accuracy of the estimated value of the small value of the correlation is low. That is, it cannot be applied to the case where the moving average is calculated or the entire correlation is used as a vector for machine learning.

Conventionally, the convolution operation is a general concept in the field of signal processing and the like. In signal processing, data is processed with a data length determined by communication between transmission and reception as a processing unit, so that σ is small.

Also, the convolution operation can use an existing function in a programming language. In such a development environment, detailed analysis of the processing load in the convolution operation has not been done. The length of documents such as academic journals and academic papers is much larger than the data length in communication.

In the present embodiment, when σ is large, further speeding up of processing is realized, and in recent years, there is a demand for obtaining a correlation with respect to a large amount of large σ data in comparison between documents.

Here, the problem in the calculation time of the correlation is to reduce the actual execution time. Specifically, it is to reduce the coefficient part for σnlogσ, that is, the number of times FFT is executed. The inventor noticed that the convolution operation is performed by the processing of the discrete Fourier transform and the inverse discrete Fourier transform, and realized the speedup of the calculation processing of the correlation when handling a large amount of data of large σ. In the following description, a large amount of data with a large σ will be described using a document as an example.

When obtaining the correlation between two documents s and t, the positions of similar portions that are similar in two or more consecutive characters in each character string of the two documents s and t are not necessarily the same position. It is also necessary to consider the case where the positions where similar portions appear are displaced. The position shift considered in the correlation will be described.

First, the notation relating to the correlation calculation between two character strings is defined below.

文書sと文書tの文字は、長さnの文字列全体からなる集合Σⁿの要素として表される。

The characters of document s and document t are represented as elements of a set Σ ⁿ consisting of the entire character string of length n.

文書sと文書tとの相関c(s，t)は、2n-1次元ベクトルで表され、i番目の要素は、

The correlation c(s,t) between the document s and the document t is represented by a 2n-1 dimensional vector, and the i-th element is

で表される。ただし、

It is represented by. However,

について、

about,

のように文字列を数式のように表わすことで、範囲外の形式的な比較のためにダミーの語を付加することを表現する。このようにすることで、位置ずれを考慮して相関を算出できる。

By expressing a character string like a mathematical expression, a dummy word is added for formal comparison outside the range. By doing so, the correlation can be calculated in consideration of the positional deviation.

FIG. 1 is a diagram for explaining the positional deviation. FIG. 1 shows a conceptual diagram in which dummy words that do not belong to the document s or the document t are added before and after the document t in order to consider the positional deviation of the word of the document s with respect to the document t.

The sentence s is shifted word by word, and the correlation with the word of the document t at the shifted position is calculated. A case where the word order is changed in the document s with respect to the document t corresponds to the shift position. By calculating the correlation for each word shift, the correlation c ₁ , c ₂ ,..., C _2n-1 for each shift is obtained.

If it is assumed that the correlation c(s, t) considering all misregistrations corresponds to the number of matching characters in all misregistrations, a simple calculation makes O(n ² ) times of character comparisons. Here, the time of one character comparison depends on log σ.

FIG. 2 is a diagram showing an example of correlation between documents. FIG. 2 shows an example of correlation when the sentence s is the character string “abbacab” and the document t is the character string “ababbac”. In FIG. 2, the dummy word x is blank.

The head of the document s is aligned with the head of the document t', which is a dummy of n-1's before and after the document t, and the number of matching characters is counted by shifting the document s word by word. In this example, when the beginning of the document s is aligned with the beginning of the document t′, there is no character that matches the documents s and t. Therefore, the correlation c ₁ becomes 0. When shifted by one word, the last two characters of the document s and the first two characters of the document t match. Therefore, the correlation c ₂ is 2.

Further, the correlation c ₃ shifted by one word becomes 0. Correlation c ₄ is 3, correlation c ₅ is 1, correlation c ₆ is 2, correlation c ₇ is 3, correlation c ₈ is 1, correlation c ₉ is 5, correlation c ₁₀ is 1, correlation c ₁₁ is 0, The correlation c ₁₂ is 1, and the correlation c ₁₃ is 0.

Here, the calculation of the FFT convolution performed for each type of character or character string will be described. The cyclic convolution r of two n-dimensional vectors u and v is

で表される。ただし、-n+1≦i≦0についてはv_i=v_n+iである。

It is represented by. However, for −n+1≦i≦0, v _i =v _n+i .

Let R, U, V be the discrete Fourier transforms of r, u, v respectively, and o be the element-wise product,

で表される。このことから、rは、uとvとからFFTによりO(nlogn)時間で計算可能である。計算ルートと計算時間との関係を図３で説明する。

It is represented by. From this fact, r can be calculated in O(nlogn) time by FFT from u and v. The relationship between the calculation route and the calculation time will be described with reference to FIG.

FIG. 3 is a diagram for explaining the relationship between calculation and calculation time. In FIG. 3, n represents the length of the sentence. In the case of a calculation for obtaining r from u and v, that is, a calculation for comparing r between the word u and the word v, the calculation time is represented by O(n ² ).

On the other hand, the calculation time of the discrete Fourier transforms U and V of u and v is O(nlogn), and the calculation time of the product R of each element of U and V is O(n). The calculation time of the inverse discrete Fourier transform from R to r is O(nlogn).

Next, an outline of the FFT-based Algorithm will be described. The FFT-based Algorithm digitizes characters and calculates the correlation between strings by vector convolution. By doing so, if one character a in the set Σ of the entire character string is replaced with 1 and the others are replaced with 0, the correlation considering only the matching of the character a can be calculated by the convolution operation, and the correlation can be calculated in O(nlogn) time. It is possible. It takes O(n) time to replace a document with a sequence of numbers. Further, since congruity or non-coincidence can be expressed by multiplication, convolution operation can be applied.

For each type of character or character string that is an element in the set Σ of the entire character string, a convolution operation of O(nlogn) time is performed, and therefore, σ times of the number of character or character string types are repeated. Then, the correlation is calculated by taking the sum for each element of the vector. That is, the correlation c _i at a certain displacement can be obtained.

More specifically, the calculation formula of the FFT-based Algorithm is shown below.

について、φ_aはaを1、それ以外を0に写す関数とし、定義（数３）より、n≦i≦2n-1について、

, Φ _a is a function that maps a to 1 and the others to 0. From the definition (Equation 3), for n≦i≦2n-1,

と表される。ここで、加算の順序を入れ替えて、

Is expressed as Here, change the order of addition,

とする。

And

(U _a,1 , u _a,2 ,..., u _a,2n-1 ) and (va _,1 , v _a,2 ,..., v _a,2n-1 )

である。数１１では、1≦i≦nのときには、片方を反転させ、n≦i≦2n-1のときには、0を埋める。2n-1次元ベクトルとすると、数１０は、

Is. In Expression 11, one is inverted when 1≦i≦n, and zero is filled when n≦i≦2n−1. Assuming a 2n-1 dimensional vector, Equation 10 is

で表される。数１２内の

It is represented by. Within number 12

は巡回畳み込み演算である。

Is a cyclic convolution operation.

When the positional deviation of similar portions is taken into consideration, the convolution operation is repeated for the number of types of characters or character strings (elements in the set Σ of the entire character string), which increases the calculation time.
-For one convolution operation, two FFTs, one vector element-wise product, and one inverse FFT are performed. Here, the calculation time of the inverse FFT is considered to correspond to the calculation time of the FFT.
-For each calculation time, FFT is O(nlogn) time, and product of each element is O(n) time, and FFT is dominant.
・Therefore, it can be seen that FFT is required 3σ times in one calculation of correlation.

The number of repetitions σ of the convolution operation is a logical minimum value with respect to the alphabet size σ.

For the related technology that performs a convolution operation for each type of character or character string and aggregates the results, the inventor aggregates the result before the Fourier transform (inverse FFT), which is the final process of the convolution operation. We have found to reduce the number of Fourier transforms. A modification of the calculation formula focused on by the inventor in order to reduce the number of Fourier transforms will be described.

If r _a is _a circular convolution of u _a and v _a , then the correlation c(s,t) is

のベクトルから得られる。このベクトルを

Is obtained from the vector of. This vector

と表示することにする。そして、fを離散フーリエ変換、R_a=f(r_a)とすると、数１５は、

Will be displayed. Then, when f is a discrete Fourier transform, and R _a =f(r _a ), Equation 15 becomes

と変形できる。数１６の右辺より、ベクトルの要素ごとの加算後に逆FFTを行なえばよいことが分かる。要素ごとの加算後の逆FFTは、１回のFFTと見なせる。

Can be transformed. It can be seen from the right side of Expression 16 that the inverse FFT may be performed after addition for each element of the vector. The inverse FFT after addition for each element can be regarded as one FFT.

FIG. 4 is a diagram showing a conceptual diagram of the calculation method. In FIG. 4, the convolution operation 2p and the calculation part of the simple method that does not perform the convolution operation 2p are the parts in which the process is repeated for each element (character or character string type) of the set Σ. That is, it is repeated σ times. Then, the values are added up for each element of the obtained σ vectors, and the correlation c(s, t) between the document s and the document t is obtained.

In the related art, in the convolution operation 2p, the FFT calculation is performed 2σ times during the discrete Fourier transform and σ times during the inverse discrete Fourier transform, so that a total of 3σ times are performed.

On the other hand, in the present embodiment, in the convolution operation 2p, the vector-by-element addition is performed before the execution of the inverse FFT, and the inverse FFT is performed on the result, so that the number of times the inverse FFT is calculated changes from σ Reduce to once. In this embodiment, the larger σ is, the more effective the effect is, and the correlation between two documents can be acquired in about two-thirds the time as compared with the related art.

Also, when σ is small, it may be necessary to obtain the correlation for each character from the viewpoint of application, but when σ is large, even if the correlation for a specific character is required, it is large for other characters. It is necessary to consider σ, and it can be said that the present embodiment is more suitable in this respect.

The related technique in FIG. 4 and the calculation example of this embodiment are shown in FIG. 5, and the reduction of the calculation amount will be further described. FIG. 5 is a diagram for explaining the reduction of the calculation amount.

FIG. 5 shows an example of calculation until the correlation c(s,t) between the two documents s and t in the case where the sentence s is the character string “abca” and the document t is the character string “abcb”. An example of the vector is shown on the left, and an example of the discrete Fourier transform of the vector is shown on the right.

Convert to a vector for each word type existing in documents s and t. There are three letters a, b and c in documents s and t. First, in each of the documents s and t, for each of the characters a, b, and c, 1 is indicated in the matching portion, and the non-matching portion is converted into a vector indicated by 0. Document s is a vector
u _a =(1,0,0,1),
u _b =(0,1,0,0),
u _c =(0,0,1,0)
And the document t is a vector
v _a =(0,0,0,1),
v _b =(1,0,1,0),
v _c =(0,1,0,0)
Is converted to.

Discrete Fourier transform each of the vectors u _a, u _{b, and} u _c with respect to the document s,

同様に、文書tに関して、ベクトルv_a、v_b、v_cのそれぞれを離散フーリエ変換し、

Similarly, for document t, perform _a discrete Fourier transform of each of the vectors v _a, v _b, v _c ,

を得る。即ち、離散フーリエ変換が６（2×σ）回行われる。

To get That is, the discrete Fourier transform is performed 6 (2×σ) times.

Next, the product R _x (x=a, b, c) of each element of U _x and V _x is obtained. Therefore,

を得る。

To get

In the related art, by performing an inverse discrete Fourier transform on each R _a , R _b , and R _c ,
r _a =(0,0,0,1),
r _b =(1,0,1,0),
r _c =(0,1,0,0)
To get In the related art, the inverse discrete Fourier transform is performed 3(σ) times. After that, the sum for each element is calculated and the correlation value between the two documents s and t
(0, 1, 0, 3, 0, 0, 1)
To get

On the other hand, in the present embodiment, after obtaining the product R _x (x=a, b, c) of U _x and V _x for each element, the sum of each element is calculated without performing the inverse discrete Fourier transform for each element. Ask. Sought

に対して、逆離散フーリエ変換を行なえば、関連技術と同様に、相関値
(0，1，0，3，0，0，1)
を得る。即ち、１回の逆離散フーリエ変換で、２つの文書sと文書tの相関を得ることができる。この例では、σは3であるため、逆離散フーリエ変換に係る処理時間は１／３になる。全体の処理として、関連技術ではFFTの回数が9σ（=6σ+3σ）、即ち、２７回行われたのに対して、本実施例ではFFTの回数が7σ（=6σ+σ）、即ち、２１回に削減される。σ=3の場合で説明したが、σが大きい程、本実施例の効果は大きい。

On the other hand, if the inverse discrete Fourier transform is performed, the correlation value is
(0, 1, 0, 3, 0, 0, 1)
To get That is, the correlation between two documents s and t can be obtained by one inverse discrete Fourier transform. In this example, since σ is 3, the processing time associated with the inverse discrete Fourier transform is 1/3. As a whole process, in the related art, the number of FFTs is 9σ (=6σ+3σ), that is, 27 times, whereas in the present embodiment, the number of FFTs is 7σ (=6σ+σ), that is, It will be reduced to 21 times. Although the case of σ=3 has been described, the larger σ, the greater the effect of this embodiment.

Next, the hardware configuration of the similarity calculation device 100 (FIG. 6) that performs the above-described processing will be described. FIG. 6 is a diagram showing a hardware configuration of the similarity calculation device. 6, the similarity calculation device 100 is an information processing device controlled by a computer, and includes a CPU (Central Processing Unit) 11, a main storage device 12, an auxiliary storage device 13, an input device 14, and a display. It has a device 15, a communication I/F (interface) 17, and a drive device 18, and is connected to the bus B.

The CPU 11 corresponds to a processor that controls the similarity calculation device 100 according to a program stored in the main storage device 12. A RAM (Random Access Memory), a ROM (Read Only Memory), or the like is used for the main storage device 12, and a program executed by the CPU 11, data required for processing by the CPU 11, and data obtained by the processing by the CPU 11 are obtained. Stored or temporarily saved the data etc.

An HDD (Hard Disk Drive) or the like is used as the auxiliary storage device 13, and stores data such as programs for executing various processes. Various processes are realized by loading a part of the program stored in the auxiliary storage device 13 into the main storage device 12 and executing it in the CPU 11.

The input device 14 has a mouse, a keyboard, and the like, and is used by the user to input various information necessary for the processing by the similarity calculation device 100. The display device 15 displays various information required under the control of the CPU 11. The input device 14 and the display device 15 may be a user interface such as an integrated touch panel. The communication I/F 17 communicates via a wired or wireless network. The communication by the communication I/F 17 is not limited to wireless or wired.
A program that implements the processing performed by the similarity calculation device 100 is provided to the similarity calculation device 100 by a storage medium 19 such as a CD-ROM (Compact Disc Read-Only Memory).

The drive device 18 interfaces the storage medium 19 (for example, a CD-ROM or the like) set in the drive device 18 and the similarity calculation device 100.

Further, the storage medium 19 stores a program that implements various processes according to the present embodiment described later, and the program stored in the storage medium 19 is stored in the similarity calculation device 100 via the drive device 18. Installed. The installed program can be executed by the similarity calculation device 100.

The storage medium 19 for storing the program is not limited to the CD-ROM, and is one or more non-transitory, tangible (tangible) tangible computer-readable structures having a data structure. ) Any medium. The computer-readable storage medium may be a DVD (Digital Versatile Disk) disk, a portable recording medium such as a USB memory, or a semiconductor memory such as a flash memory, in addition to the CD-ROM.

The similarity calculation device 100 may be a laptop, a tablet terminal, or the like. In this case, the storage medium 19 is an SD (Secure Digital) memory card or the like, and the drive device 18 interfaces the storage medium 19 set in the drive device 18 and the similarity calculation device 100.

FIG. 7 is a diagram illustrating a functional configuration example of the similarity calculation device. In FIG. 7, the similarity calculation device 100 includes a vector conversion unit 61, a discrete Fourier transform unit 62, an element multiplication unit 63, an element addition unit 64, and an inverse discrete Fourier transform unit 65. The storage unit 130 stores the document data 70, the vector data 71, the converted vector data 72, the product vector data 73, the sum vector 74, the correlation result 79, and the like.

The vector conversion unit 61 converts each of the documents s, t, etc. of the document data 70 stored in the storage unit 130 into a vector. As described above, the conversion method into a vector may be represented by 1 for each character type and 0 for disagreement for each character type. Vector data 71 including the vectors u, v, etc. obtained by the vector conversion unit 61 is stored in the storage unit 130.

The discrete Fourier transform unit 62 performs a discrete Fourier transform on each of the vectors u, v, etc. of the vector data 71 stored in the storage unit 130. The discrete Fourier transform is executed for the number of times (the number of documents)×(the number of types of characters or character strings σ). The transformed vector data 72 including the vectors U and V after the discrete Fourier transform is stored in the storage unit 130.

The element multiplication unit 63 obtains a product (Hadamard product) for each element for all the vectors in the converted vector data 72. The product vector data 73 including the vector R obtained for each of all character types is stored in the storage unit 130.

The element addition unit 64 obtains the sum vector 74 by obtaining the sum for each element of the vector R for each type of character or character string of the product vector data 73. The obtained sum vector 74 is stored in the storage unit 130.

The inverse discrete Fourier transform unit 65 performs an inverse discrete Fourier transform on the sum vector 74 to obtain a correlation result 79 indicating a correlation value regarding the documents s, t and the like. The correlation result 79 is stored in the storage unit 130. Further, the correlation result 79 may be displayed on the display device 15.

The document data 70 includes two or more documents such as documents s and t, and is given by the user. The user may prepare a plurality of documents as target documents for verifying the correlation in advance, and input the specific document (for example, the original document) to the similarity calculation apparatus 100.

The vector data 71 shows the result of vector conversion for each document s, t, etc. of the document data 70. For example, the document s corresponds to the vector u, and the document t corresponds to the vector v. Similar converted vectors are shown for other documents. In the present embodiment, the "document" corresponds to various data files.

The converted vector data 72 includes vectors u, v, etc. for the documents s, t, etc., and each vector u, v, etc. exists for each type of character. The converted vector data 72 includes a number of vectors of (the number of documents)×(the number of character types σ).

The product vector data 73 indicates a vector R indicating a product for each element for each type of character or character string. The product vector data 73 includes a vector R of the number of types of characters or character strings. In the case of 1:N correlation in which a plurality of target documents are used to calculate the correlation with respect to a specific document, a vector R of the number of character types is generated for each target document.

The summation vector 74 indicates a sum for each element with respect to the vector R for each character type. For the correlation between the two documents s and t, a sum vector 74 of 1 is generated. In this embodiment, it is only this sum vector 74 that the inverse discrete Fourier transform is performed. In the case of 1-to-N correlation, since N summed vectors 74 are generated, the inverse discrete Fourier transform is performed N times, but when compared with the related technique that performs the convolution operation, 1/(character or character string It can be reduced to the number of types) times.

The correlation result 79 shows the correlation between documents obtained by performing the inverse Fourier transform on the sum vector 74. One correlation value is shown for the correlation between two documents s and t. In the 1-to-N correlation, since the correlation value with each target document is output, the correlation result 79 shows N correlation values.

In this embodiment, the processing time can be reduced even if the conversion from the character string of the document to the vector is performed by using an arbitrary function. As an example, each element of the output correlation may be represented by the sum of inner products of vectors instead of the number of matching characters. The document is divided into spaces and divided into punctuation marks to create an array whose elements are processing units, and the sum of the inner products of the vectors representing the characteristics of the elements is calculated. The sum of vector inner products can be regarded as a weighted correlation by conversion into a vector representing the similarity of characters (for example, a distributed expression of words learned from document data).

Next, the first similarity calculation process for obtaining the correlation between two documents s and t will be described. FIG. 8 is a flowchart for explaining the first similarity calculation process for obtaining the correlation between two documents.

In FIG. 8, the vector conversion unit 61 reads the documents s and t from the document data 70 (step S301), and converts each document s, t into a vector for each character type (step S302). Document s is converted to vector u and document t is converted to vector v. Vector data 71 including the vector u and the vector v is stored in the storage unit 130.

The discrete Fourier transform unit 62 performs a discrete Fourier transform on the vector u to generate a vector U, and performs a discrete Fourier transform on the vector v to generate a vector V (step S303). The converted vector data 72 including the vector U and the vector V is stored in the storage unit 130.

The element multiplication unit 63 calculates the product of each element of the vector for each type of character or character string (calculates Hadamard product), and acquires the vector R for each type of character (step S304). The vector R is generated by the number of character types. The product vector data 73 including the vector R of the number of character types is stored in the storage unit 130.

The element addition unit 64 adds the elements of the vector R for each type of character or character string to obtain the summed vector 74 (step S305). The added vector 74 is stored in the storage unit 130.

The inverse discrete Fourier transform unit 65 performs an inverse discrete Fourier transform on the sum vector 74 to obtain a correlation result 79 (step S306). The correlation result 79 is stored in the storage unit 130. Further, the correlation result 79 may be displayed on the display device 15.

FIG. 9 is a diagram showing a processing example in the flowchart of FIG. In FIG. 9, it is associated with the step numbers of FIG. The step S302 after the vector conversion unit 61 reads the documents s and t will be described.

In FIG. 9, there are a plurality of character types included in any of the documents s and t, and the character types are represented by a, b,... Z. Alphanumeric characters, Japanese (Hiragana, Katakana, and Kanji) and other characters in other languages correspond to the character types.

Further, this embodiment can be applied by regarding words, phrases, etc. as characters. In this case, before converting the processing unit regarded as a character in each document s, t into a vector, the processing unit is extracted by dividing the processing unit by a space, dividing by a punctuation mark, etc. Just create an array. The processing unit may be each part of speech, or each character or phrase. The processing unit is the type of character or character string in the document. By processing the arrays represented by the processing units as documents s and t, the processing is performed as follows.

The vector conversion unit 61 performs vector conversion on the document s for each of the character types a, b,... _Z to obtain vectors u _a , u _b ,... U _z (step S302s). Further, the vector conversion unit 61 performs vector conversion on the document t for each of the character types a, b,... _Z to obtain vectors v _a , v _b ,... V _z (step S302t). ..

Discrete Fourier transform unit 62, the vector u _a, u _b generated from the document s, for each of the · · · u _z, vector U _a performing discrete Fourier transform, U _b, obtaining · · · U _z (step S303ua, S303ub,..., And S303uz). Further, the discrete Fourier transform unit 62, the vector v _a generated from the document t, v _b, for each of the · · · v _z, to obtain a vector _{V a, V b, ··· V} z by performing a discrete Fourier transform (Steps S303va, S303vb,..., And S303vz).

The element multiplication unit 63 calculates the product of each element of the vector between the documents s and t for each character type, and obtains the vectors R _a , R _b ,... R _z for each character type ( Steps S304a, S305b,..., S304z).

The element adding unit 64 further adds the values of the elements at the same displacement positions in the vectors _Ra , _Rb ,..., _Rz to obtain one sum vector 74 (step S305). Then, the inverse discrete Fourier transform unit 65 performs an inverse discrete Fourier transform on the sum vector 74 to obtain the correlation result indicating the correlation r(s,t) (step S306).

Next, the second similarity calculation process for obtaining the 1-to-N correlation will be described. FIG. 10 is a flowchart for explaining the second similarity calculation process for obtaining the 1:N correlation. In FIG. 10, the correlation between the specific document s and each target document t _{1 to} t _n is calculated.

10, the vector conversion unit 61 reads the specific document s and n pieces of target document t ₁ ~t _n from the document data 70 (step S401), the type of each document s and t ₁ ~t _n characters It is converted into a vector for each (step S402). The specific document s is converted into a vector u, and the target documents t _{1 to} t _n are converted into vectors v _{1 to} v _n , respectively. Vector data 71 including the vector u and the vectors v _{1 to} v _n is stored in the storage unit 130.

Discrete Fourier transform unit 62 generates a vector U by performing a discrete Fourier transform to the vector u, and generates a vector V ₁ ~V _n by performing a discrete Fourier transform on each vector v ₁ to v _n (Step S303). The converted vector data 72 including the vector U and the vectors V _{1 to} V _n is stored in the storage unit 130.

The element multiplication unit 63 selects one from the vectors V _{1 to} V _n related to the target documents t _{1 to} t _n (step S404-1), and selects the vector U related to the specific document s and the selected vector V _i . , The product of each element of the vector is calculated for each character type (hadamard product is calculated), and the vector R _i for each character type is acquired (step S404-2). The vectors R _i are generated by the number of character types. The product vector data 73 including the vector R _i of the number of character types is stored in the storage unit 130.

The element adding unit 64 adds the elements of the vector R _i for each type of character to obtain a summed vector 74 _i (step S405). The added vector 74 _i is stored in the storage unit 130. The inverse discrete Fourier transform unit 65 performs an inverse discrete Fourier transform on the sum vector 74 _i and adds it to the correlation result 79 of the storage unit 130 (step S406).

Then, the second similarity calculation process determines whether or not the unprocessed vector V _i exists in the converted vector data 72 (step S407). If it exists, the second similarity calculation process returns to step S401-1, and the same process as described above is repeated.

On the other hand, if there is not, the second similarity calculation process ends. The correlation result 79 in the storage unit 130, the correlation value r ₁ ~r _n particular document s and the target document t ₁ ~t _n are stored. When there is no unprocessed vector V _i , the correlation result 79 may be displayed on the display device 15 and the second similarity calculation process may be ended.

Here, the range of the actual data size of the document 80 is shown.

For σ,
・Nucleic acid base: 4 to
・Alphabet: 26
・Double-byte character: 65536
・Number of monolingual lexicons (words are treated as characters): Thousands to tens of thousands
For n,
・Academic papers (number of words): tens of thousands to tens of thousands ・Genome sequence: hundreds of thousands to tens of millions ・Sensor data: hundreds of millions or more Correlation between hundreds of academic papers in an example using related technology It takes several days to calculate. The number of papers published by one research institution in the repository is more than tens of thousands. The following is a comparison of experimental results between this example and related art.

FIG. 11 is a diagram showing a comparison result of the execution time with respect to the number of words. In FIG. 11, the horizontal axis represents the number of vocabularies, and the vertical axis represents the execution time (msec). The vocabulary number corresponds to the type of character, and is indicated by σ in the above. This graph shows the execution time when the document length is 1024 words, 2048 words, and 4096 words.

From FIG. 11, in both the present embodiment and the related art, the longer the vocabulary number, the longer the execution time. However, in any case, the execution time of this embodiment is shorter than that of the related art. In particular, when the document length is 4096 words, the execution time of this embodiment is about 20% shorter than the execution time of the related art when the number of vocabularies exceeds 800 words.

FIG. 12 is a diagram showing the comparison result of the execution time with respect to the document length. In FIG. 12, the horizontal axis represents the document length and the vertical axis represents the execution time (msec). This graph shows the execution time when the number of target vocabularies is 256 words, 512 words, and 1024 words.

From FIG. 12, in both the present embodiment and the related art, the longer the document length, the longer the execution time. However, in any case, the execution time of this embodiment is shorter than that of the related art. Particularly, when the vocabulary number is 1024 words, the execution time of this embodiment is about 20% shorter than the execution time of the related art when the document length is around 3500 words.

FIG. 13 is a diagram showing a comparison result between the number of words in a document and the processing time. In FIG. 13, the horizontal axis indicates the document length, the left vertical axis indicates the execution time (msec), and the right vertical axis indicates the vocabulary number. The vocabulary number on the right vertical axis indicates the statistical value of the vocabulary number according to the length of the document.

From FIG. 13, it can be seen that the vocabulary number increases in proportion to the document length. Further, in the present embodiment, when the document length is up to about 2000 words, there is little difference in execution time from the related art. As the document length approaches 4000 words, that is, as σ increases, the difference in execution time reaches about 20%, and it can be seen that the speedup is realized by this embodiment. The speedup of about 20% is due to the fact that the calculation amount required for convolution is suppressed.

From the above experimental results by the inventor, it can be said that the present embodiment shows that the calculation amount is suppressed even when the number of character types and the length of the character string increase.

In the above-described embodiment, the vector transforming unit 61, the discrete Fourier transforming unit 62 and the element multiplying unit 63 correspond to the first transforming unit, the element adding unit 64 corresponds to the adding unit, and the inverse discrete Fourier transforming unit 65 corresponds to the first transforming unit. It corresponds to two conversion units.

The present invention is not limited to the specifically disclosed embodiments, and various modifications and changes can be made without departing from the scope of the claims.

The following supplementary notes will be further disclosed regarding the embodiments including the above-described examples.
(Appendix 1)
Generating a plurality of vectors obtained by vectorizing the plurality of document data for each character or character string included in the plurality of document data,
Calculating a product for each element of the result of Fourier transform for each of the plurality of vectors,
For each of the character or character string, generate a sum vector in which the product is added for each element,
From the sum vector, using the inverse Fourier transform, to generate a correlation value between the plurality of document data,
A similarity calculation program that causes a computer to perform processing.
(Appendix 2)
On the computer,
The plurality of documents are divided into any of words that are separated by a part of speech, a character, a phrase, or a space, and for each divided word, a match or a mismatch with the word at each shift position of the plurality of document data is determined. The similarity calculation program according to note 1, wherein the plurality of vectors are generated as the values of the elements.
(Appendix 3)
The similarity calculation program according to appendix 1 or 2, wherein the element corresponds to a shift position between the plurality of document data.
(Appendix 4)
The similarity calculation program according to any one of appendices 1 to 3, wherein the product is a Hadamard product.
(Appendix 5)
Generating a plurality of vectors obtained by vectorizing the plurality of document data for each character or character string included in the plurality of document data,
Calculating a product for each element of the result of Fourier transform for each of the plurality of vectors,
For each of the character or character string, generate a sum vector in which the product is added for each element,
From the sum vector, using the inverse Fourier transform, to generate a correlation value between the plurality of document data,
A similarity calculation method in which processing is performed by a computer.
(Appendix 6)
A vector conversion unit that generates a plurality of vectors by vectorizing the plurality of document data for each character or character string included in the plurality of document data;
A first conversion unit that calculates a product for each element of the result of Fourier transform for each of the plurality of vectors;
An addition unit that generates a summation vector in which the product is added for each of the elements for each of the character or the character string;
A second conversion unit that generates a correlation value between the plurality of pieces of document data from the combined vector by using a Fourier inverse transform;
A similarity calculation device having.

61 vector conversion unit 62 discrete Fourier transform unit 63 element multiplication unit 64 element addition unit 65 inverse discrete Fourier transform unit 70 document data 71 vector data 72 post-transformation vector data 73 product vector data 74 summation vector 79 correlation result

Claims (5)

Generating a plurality of vectors obtained by vectorizing the plurality of document data for each character or character string included in the plurality of document data,
Calculating a product for each element of the result of Fourier transform for each of the plurality of vectors,
For each of the character or character string, generate a sum vector in which the product is added for each element,
From the sum vector, using the inverse Fourier transform, to generate a correlation value between the plurality of document data,
A similarity calculation program that causes a computer to perform processing. On the computer,
The plurality of documents are divided into any of words that are separated by a part of speech, a character, a phrase, or a space, and for each divided word, a match or a mismatch with the word at each shift position of the plurality of document data is determined. The similarity calculation program according to claim 1, wherein the plurality of vectors are generated as the values of the elements. The similarity calculation program according to claim 1, wherein the element corresponds to a shift position between the plurality of document data. Generating a plurality of vectors obtained by vectorizing the plurality of document data for each character or character string included in the plurality of document data,
Calculating a product for each element of the result of Fourier transform for each of the plurality of vectors,
For each of the character or character string, generate a sum vector in which the product is added for each element,
From the sum vector, using the inverse Fourier transform, to generate a correlation value between the plurality of document data,
A similarity calculation method in which processing is performed by a computer. A vector conversion unit that generates a plurality of vectors by vectorizing the plurality of document data for each character or character string included in the plurality of document data;
A first conversion unit that calculates a product for each element of the result of Fourier transform for each of the plurality of vectors;
An addition unit that generates a summation vector in which the product is added for each of the elements for each of the character or the character string;
A second conversion unit that generates a correlation value between the plurality of pieces of document data from the combined vector by using a Fourier inverse transform;
A similarity calculation device having.

Images