JP2021179932A - Document processing apparatus, document processing method, and program - Google Patents

Abstract

To determine the degree of correspondence between a plurality of documents.SOLUTION: A document processing apparatus comprises: an input unit that receives input of a plurality of documents; a document dividing unit that divides the documents into words being characters or character strings and divides the documents into portions being groups larger than the words; a word correspondence degree calculation unit that calculates a word correspondence degree that is the degree of correspondence between the words in the different documents; a portion correspondence degree calculation unit that calculates a portion correspondence degree that is the degree of correspondence between the portions in the different documents; and a document correspondence degree calculation unit that calculates a document correspondence degree that is the degree of correspondence between the documents based on the degree of correspondence between the words determined by using the word correspondence degree and the portion correspondence degree.SELECTED DRAWING: Figure 1

Description

The present invention relates to a document processing apparatus, a document processing method, and a program.

A technique for searching documents with similar contents is known.

For example, in information retrieval, the distance or similarity of documents is calculated based on the frequency of appearance of words contained in the document. Then, the documents are presented in the order of the calculated distances being close to each other or the degree of similarity being high. In this way, a technique for searching for a document similar to a certain document is known (for example, Non-Patent Document 1 and the like).

Documents contain different elements other than letters or strings (hereinafter referred to as "words"). That is, a document is composed of a collection larger than a word (hereinafter referred to as a "part"). Then, in the conventional technique, the similarity is determined based only on the word. In this way, in the conventional technique, how much correspondence is taken between documents based on the correspondence between words defined independently of the correspondence between parts (hereinafter, correspondence is taken between different documents). The degree is called "correspondence"). That is, in the conventional technique, the degree of correspondence between parts is not considered in the method using the degree of correspondence between words.

The disclosed technique aims to determine the degree of correspondence in multiple documents.

The disclosed technology is an input unit for inputting multiple documents, and
A document division unit that divides the document into characters or words that are character strings, and divides the document into parts that are larger than the words.
A word correspondence calculation unit that calculates a word correspondence, which is a correspondence between the words in different documents, and a word correspondence calculation unit.
A partial correspondence calculation unit that calculates a partial correspondence, which is a correspondence between the parts of different documents, and a partial correspondence calculation unit.
A document processing device including a document correspondence degree calculation unit that calculates a document correspondence degree, which is a correspondence degree between documents, based on the word correspondence degree and the correspondence degree between words defined by using the partial correspondence degree. Is.

It is possible to determine the degree of correspondence in a plurality of documents.

It is a figure which shows the functional structure example. It is a figure which shows the hardware configuration example. It is a figure which shows the whole processing example. It is a figure which shows the example of a document. It is a figure which shows the example of division. It is a figure which shows the calculation result example of the word correspondence degree. It is a figure which shows the calculation result example of a partial correspondence degree. It is a figure which shows the example of the calculation result of the joint probability.

Hereinafter, a specific example will be described with reference to the drawings, and an embodiment of the document processing apparatus according to the present invention will be described.

<Function configuration example>
FIG. 1 is a diagram showing a functional configuration example. For example, the document processing device 100 has a functional configuration including an input unit 101, a document division unit 102, a word correspondence degree calculation unit 103, a partial correspondence degree calculation unit 104, and a document correspondence degree calculation unit 105.

These functions are realized by, for example, the following hardware.

<Hardware configuration example>
FIG. 2 is a diagram showing a hardware configuration example. For example, the document processing device 100 includes information processing including an input device 11, an output device 12, a drive device 13, an auxiliary storage device 14, a memory device 15, an arithmetic processing device 16, and an interface device 17 connected to each other by a bus B. It is a device.

The input device 11 is a device for inputting various information. For example, the input device 11 is a keyboard, a mouse, a pointing device, or the like.

The output device 12 is a device that outputs various information. For example, the output device 12 is a display or the like.

The interface device 17 is a device for transmitting / receiving data to / from an external device via a network or a cable.

The drive device 13 is a device that transmits / receives data to / from the storage medium 18.

The auxiliary storage device 14 stores installed programs, files, data, and the like. For example, the auxiliary storage device 14 is a hard disk or the like.

The memory device 15 is an example of a main storage device. Therefore, when the program is executed, it operates in cooperation with the arithmetic unit, the control unit, and the like, and executes the process.

The hardware configuration is not limited to the configuration shown in the figure. That is, the document processing device 100 may be configured to further have an arithmetic unit, a control device, a storage device, an input device, an output device, or a peripheral device other than those shown in the figure.

For example, the input unit 101 is realized by an input device 11 or the like. Specifically, the input device 11 is operated to store the characters and the like to be input, and the document is input.

The input unit 101 is not limited to the input device 11. For example, the input unit 101 may input a document in the form of text data, document data, or the like input by the interface device 17, the drive device 13, or the like.

In addition, the document may be input in any hardware and data format. Therefore, for example, a document generated by voice recognition processing, conversion processing, or the like may be used.

In the document division unit 102, the word correspondence degree calculation unit 103, the partial correspondence degree calculation unit 104, and the document correspondence degree calculation unit 105, for example, the memory device 15 and the arithmetic processing unit 16 cooperate to execute processing. Realize. Specifically, it is realized by executing the following processing.

<Overall processing example>
FIG. 3 is a diagram showing an example of overall processing. For example, the document processing apparatus 100 executes the following processing.

Document input processing example (step S1)
The input unit 101 inputs a document. The document to be supported may be a document stored in a storage device in advance. That is, the document processing device 100 may be configured to store documents and the like created in the past.

FIG. 4 is a diagram showing an example of a document. Hereinafter, two documents as shown in the figure will be described as an example. In the example shown below, two documents, the first document D1 and the second document D2, are input.

In the first document D1, the four characters "public", "knowledge", "information" and "report" are on the first line, and the four characters "place", "yes", "information" and "report" are on two lines. It is a document composed of eyes and three lines having the four characters "Germany", "self", "information" and "report" in the third line.

The second document D2 contains the four characters "ho", "yes", "information" and "report" on the first line, and the four characters "public", "knowledge", "information" and "report". It is a document composed of two lines having the second line.

Further, the example shown below is an example in which one character is treated as a "word". Further, the following example is an example of treating as a "part" in units of one line. It is assumed that one line is recognized by the line feed code. Although the line feed code is not specified in the figure, the document has the line feed code as data (in the illustrated example, the line feed code is located at the place where the line feed is present). Therefore, the document processing apparatus 100 can recognize one line by the line feed code.

In this example, the line feed code is not treated as a word.

Example of document division processing for dividing a document into words and parts (step S2)
The document division unit 102 divides the input document to generate a word. Further, the document division unit 102 divides the input document to generate a portion.

Specifically, the first document D1 and the second document D2 are divided into words and parts as follows.

FIG. 5 is a diagram showing an example of division. Hereinafter, the result of dividing the first document D1 into words and parts is shown by the vertical axis in the figure, and the result of dividing the second document D2 into words and parts is shown by the horizontal axis in the figure.

Since the first document D1 is composed of three lines, each line is divided into three parts, an eleventh part P11, a twelfth part P12, and a thirteenth part P13.

Since the first document D1 is composed of 12 characters, each character is divided into 12 words of the 101st character W101 to the 112th character W112.

Since the second document D2 is composed of two lines, each line is divided into two parts, a 21st part P21 and a 22nd part P22.

Since the first document D1 is composed of eight characters, it is divided into eight words of the 201st character W201 to the 208th character W208 for each character.

Hereinafter, the case of division as shown in the figure will be described as an example.

Example of word correspondence calculation processing for calculating word correspondence (step S3)
The word correspondence degree calculation unit 103 calculates the word correspondence degree.

The word correspondence is a value indicating the correspondence between words. Specifically, the word correspondence is a value calculated by the following equation (1).

P (wa → wb | ua → ub) = Probability that the word “wa” corresponds to the word “wb” when the part “ua” corresponds to the part “ub” (1)

The above equation (1) can be expressed as the following equation (1-A).

なお、「ｗａ」は、一方の文書（以下、第１文書Ｄ１とする。）が有する単語（この例では、第１０１文字Ｗ１０１乃至第１１２文字Ｗ１１２のいずれかの単語である。）を示す。また、「ｗｂ」は、他方の文書（以下、第２文書Ｄ２とする。）が有する単語（この例では、第２０１文字Ｗ２０１乃至第２０８文字Ｗ２０８のいずれかの単語である。）を示す。上記（１）式の分子は、第１単語「ｗａ」、及び、第２単語「ｗｂ」の類似度を示す。一方で、上記（１）式の分母は、第１単語「ｗａ」と、第２文書Ｄ２が有する各単語「ｗｋ」とのそれぞれの類似度を総計した総計類似度を示す。したがって、単語対応度は、上記（１）式に基づいて、第１単語「ｗａ」、及び、第２単語「ｗｂ」の類似度が、総計類似度に対して占める割合で計算される値である。

In addition, "wa" indicates a word (in this example, any word of the 101st character W101 to the 112th character W112) possessed by one of the documents (hereinafter referred to as the first document D1). Further, "wb" indicates a word (in this example, any word of the 201st character W201 to the 208th character W208) possessed by the other document (hereinafter referred to as the second document D2). The molecule of the above formula (1) shows the similarity between the first word "wa" and the second word "wb". On the other hand, the denominator of the above equation (1) indicates the total similarity between the first word "wa" and the respective words "wk" in the second document D2. Therefore, the word correspondence is a value calculated based on the above equation (1) by the ratio of the similarity between the first word "wa" and the second word "wb" to the total similarity. be.

The part "ua" and the part "ub" indicate the part to which the word "wa" and the word "wb" belong.

Further, "sim" indicates the degree of similarity. That is, in the above equation (1), "sim (wa, wb)" indicates the result of calculating the similarity between the word "wa" and the word "wb". Hereinafter, the case where the words for which the similarity is to be calculated match each other is defined as "1.00", and the case where the words differ from each other is "0.00" (hereinafter, the description in the case of "0.00" is omitted. Therefore, if it is blank, it indicates "0.00".).

Therefore, the above equation (1), that is, the calculation result of the word correspondence degree is, for example, the following result.

FIG. 6 is a diagram showing an example of the calculation result of the word correspondence degree. As shown in the figure, the 103rd character W103 and the 203rd character W203, the 103rd character W103 and the 207th character W207, the 107th character W107 and the 203rd character W203, the 107th character W107 and the 207th character W207, the 111th character W111 and The 203rd character W203, and the 111th character W111 and the 207th character W207 are both words of the character "feeling". That is, since these words match each other, the degree of similarity is "1.00". It should be noted that the similarity and the correspondence are different, and even if the similarity is "1.00", the words may not correspond to each other.

If there is no correspondence even if the degree of similarity is high, for example, in the case of associating the parts "AA" and "A", in the part "AA", there are two words "A", whereas in "A", there are two words. Since there is only one "A", even if "A" and "A" have a similarity degree of "1.00", one of the A's does not correspond.

104th character W104 and 204th character W204, 104th character W104 and 208th character W208, 108th character W108 and 204th character W204, 108th character W108 and 208th character W208, 112th character W112 and 204th character W204, Further, the 112th character W112 and the 208th character W208 are both words of the character "report". That is, since these words match each other, the degree of similarity is "1.00".

The 106th character W106 and the 201st character W201 are both words of the character "Yes". That is, since these words match each other, the degree of similarity is "1.00".

The 101st character W101 and the 205th character W205 are both words of the character "public". That is, since these words match each other, the degree of similarity is "1.00".

The 102nd character W102 and the 206th character W206 are both words of the character "knowledge". That is, since these words match each other, the degree of similarity is "1.00".

Words other than the above are different from each other (for example, in the case of a combination of the 101st character W101 and the 201st character W201). That is, since these words do not match each other, the similarity is "0.00" (in the figure, the similarity "0.00" is represented by a blank).

The degree of similarity of words is not limited to the case where it is calculated based on whether or not the words match. For example, the similarity of words may be determined by a value calculated based on the inner product of the vectors, expressing the words as vectors.

Further, a word vector or the like may be used as the vector. In the word vector, a feature indicating the content of a word is a plurality of numerical values (hereinafter, a value indicating the feature is referred to as a "feature amount"), that is, a value indicated by a vector. Specifically, the word vector is described in the literature (Distributed Repressions of Sentences and Substances Proceedings of The 31st International Conference on Machine Learning (ICML 2014), etc. It is calculated by.

In this method, a word vector is calculated by referring to a large amount of text called a corpus prepared in advance.

Specifically, the vector inner product calculation is performed as follows. <A, b> indicates the inner product of one vector "a" and the other vector "b". Two words having similar meanings, contexts, related fields, etc. have similar features. Therefore, if the words are similar, the vectors of each other are similar. For example, the inner product of two similar vectors (50,10,5) and (48,9,4) is calculated as "50 × 48 + 10 × 9 + 5 × 4 = 2510". On the other hand, the inner product of two dissimilar vectors (50,10,5) and (9,4,48) is calculated as "50 × 48 + 10 × 9 + 5 × 4 = 730".

As described above, in the case of similar words, that is, words having similar meanings, contexts, or related fields, the inner product of the vectors has a higher value than in the case of dissimilarity. When the inner product of the vectors calculated in this way exceeds a preset threshold value, it may be determined that the words are similar to each other.

Example of partial correspondence calculation processing for calculating partial correspondence (step S4)
The partial correspondence degree calculation unit 104 calculates the partial correspondence degree.

The degree of partial correspondence is a value indicating the degree of correspondence between parts. Specifically, the degree of partial correspondence is a value calculated by the following equation (2).

P (ua → ub) = Σsim (wa, wb) / Σsim (wa, wb)
wa in ua all of wa
wb in ub all of wb (2)

The denominator in the above equation (2) is the total degree of similarity in the pair of the word "wa" in the first document D1 and the word "wb" in the second document D2. Therefore, the value of the denominator does not depend on the part "ua" and the part "ub" unlike the value of the numerator. In the molecule of the above formula (2), the similarity between the first word contained in the part "ua" which is an example of the first part and the second word contained in the part "ub" which is an example of the second part. The "first total similarity" is shown. On the other hand, the denominator of the above equation (2) indicates the "second total similarity", which is the sum of the similarity of all the words included in the first document D1 and the second document D2. Therefore, the partial correspondence degree is a value indicating the ratio of the first total similarity to the second total similarity based on the above equation (2).

Specifically, the above equation (2), that is, the calculation result of the degree of partial correspondence is as follows, for example.

FIG. 7 is a diagram showing an example of a calculation result of the degree of partial correspondence. Hereinafter, an example of calculation for each degree of partial correspondence will be described. That is, what kind of calculation is specifically performed based on the above equation (2) will be described separately for the calculation of the numerator and the denominator in the above equation (2).

The partial correspondence between the 11th part P11 and the 21st part P21 is calculated by summing up the numerator in the above equation (2) with the word correspondence (that is, the calculation result shown in FIG. 6). Specifically, in the case of the calculation result of the word correspondence degree shown in FIG. 6, the partial correspondence degree between the 11th part P11 and the 21st part P21 is "1.00" out of the 16 calculation results. Therefore, the numerator of the above formula (2) is calculated as "1.00 × 2 = 2.00".

On the other hand, in the case of the calculation result of the word correspondence degree shown in FIG. 6, the total of all the words in the first document D1 and the second document D2 is "12 (that is, the word possessed by the first document D1). (That is, the number of words possessed by the second document D2) = 96 ". Among the word correspondences, in the example shown in FIG. 6, the calculation result in which each word correspondence is calculated as "1.00" is "15". Therefore, the denominator of the above equation (2) is calculated as "1.00 × 15 = 15.00".

Therefore, the degree of partial correspondence between the 11th part P11 and the 21st part P21, that is, the above equation (2) is calculated as "2.00 / 15.00 = 0.1333 ... ≈0.13". ..

Since the partial correspondence between the 11th part P11 and the 22nd part P22 is calculated as "1.00" in 4 of the 16 calculation results, the numerator of the above formula (2) is "1.00". × 4 = 4.00 ”. Further, the denominator of the above equation (2) is "15.00", which is the same as the degree of partial correspondence between the 11th portion P11 and the 22nd portion P22.

Therefore, the degree of partial correspondence between the 11th part P11 and the 22nd part P22, that is, the above equation (2) is calculated as "4.00 / 15.00 = 0.2666 ... ≈0.27". ..

Since the partial correspondence between the 12th part P12 and the 21st part P21 is calculated as "1.00" in 3 of the 16 calculation results, the numerator of the above formula (2) is "1.00". × 3 = 3.00 ”. Further, the denominator of the above equation (2) is "15.00", which is the same as the degree of partial correspondence between the twelfth part P12 and the 21st part P21.

Therefore, the degree of partial correspondence between the 12th part P12 and the 21st part P21, that is, the above equation (2) is calculated as "3.00 / 15.00 = 0.20".

Since the partial correspondence between the twelfth part P12 and the 22nd part P22 is calculated as "1.00" in two of the 16 calculation results, the numerator of the above formula (2) is "1.00". × 2 = 2.00 ”. Further, the denominator of the above equation (2) is "15.00", which is the same as the degree of partial correspondence between the twelfth portion P12 and the 22nd portion P22.

Therefore, the degree of partial correspondence between the twelfth part P12 and the 22nd part P22, that is, the above equation (2) is calculated as "2.00 / 15.00 = 0.1333 ... ≈0.13". ..

Since the partial correspondence between the 13th part P13 and the 21st part P21 is calculated as "1.00" in two of the 16 calculation results, the numerator of the above formula (2) is "1.00". × 2 = 2.00 ”. Further, the denominator of the above equation (2) is "15.00", which is the same as the degree of partial correspondence between the 13th portion P13 and the 21st portion P21.

Therefore, the degree of partial correspondence between the 13th part P13 and the 21st part P21, that is, the above equation (2) is calculated as "2.00 / 15.00 = 0.1333 ... ≈0.13". ..

Since the partial correspondence between the 13th part P13 and the 22nd part P22 is calculated as "1.00" in two of the 16 calculation results, the numerator of the above formula (2) is "1.00". × 2 = 2.00 ”. Further, the denominator of the above equation (2) is "15.00", which is the same as the degree of partial correspondence between the 13th portion P13 and the 22nd portion P22.

Therefore, the degree of partial correspondence between the 13th part P13 and the 22nd part P22, that is, the above equation (2) is calculated as "2.00 / 15.00 = 0.1333 ... ≈0.13". ..

When the degree of partial correspondence is calculated as described above, the result as shown in FIG. 7 is obtained.

Example of document correspondence calculation processing for calculating document correspondence (step S5)
The document correspondence calculation unit 105 calculates the document correspondence.

The document correspondence is a value indicating the correspondence between documents. To calculate the degree of document compatibility, first, the value calculated by the following formula (3) (hereinafter referred to as "joint probability". The joint probability is described as "P (A, B)". , Indicates the probability that events "A" and "B" will occur at the same time.)

P (wa → wb, ua → ub) = P (wa → wb | ua → ub) × P (ua → ub)
(3)

The first term on the right side of the above equation (3), that is, "P (wa → wb | ua → ub)" is the degree of word correspondence, which is the calculation result of the above equation (1). Further, in this example, it is the calculation result shown in FIG.

Further, the second term on the right side of the above equation (3), that is, "P (ua → ub)" is the degree of partial correspondence, which is the calculation result of the above equation (2). Further, in this example, it is the calculation result shown in FIG.

Therefore, as shown in the above equation (3), the joint probability is a calculation result obtained by multiplying the word correspondence degree and the partial correspondence degree. Specifically, the joint probability is calculated as follows.

FIG. 8 is a diagram showing an example of calculation results of joint probabilities. As shown in the above equation (3), the joint probability shown in the figure is the result calculated by multiplying the word correspondence degree shown in FIG. 6 and the partial correspondence degree shown in FIG. 7.

Specifically, the joint probability of the 103rd character W103 and the 203rd character W203, and the 104th character W104 and the 204th character W204 is calculated as "1.00 × 0.13 = 0.13".

The simultaneous probability of the 101st character W101 and the 205th character W205, the 102nd character W102 and the 206th character W206, the 103rd character W103 and the 207th character W207, and the 104th character W104 and the 208th character W208 is "1.00". × 0.27 = 0.27 ”.

The simultaneous probability of the 106th character W106 and the 202nd character W202, the 107th character W107 and the 203rd character W203, and the 108th character W108 and the 204th character W204 is "1.00 × 0.20 = 0.20". calculate.

The joint probability of the 107th character W107 and the 207th character W207, and the 108th character W108 and the 208th character W208 is calculated as "1.00 × 0.13 = 0.13".

The simultaneous probability of the 111th character W111 and the 203rd character W203, and the 112th character W112 and the 204th character W204 is calculated as "1.00 × 0.13 = 0.13".

The simultaneous probability of the 111th character W111 and the 207th character W207, and the 112th character W112 and the 208th character W208 is calculated as "1.00 × 0.13 = 0.13".

Next, the document correspondence calculation unit 105 calculates the document correspondence as shown in the following equation (4).

Document correspondence of the first document D1 and the second document D2 = Σmax P (wa → wb, ua → ub) / max (the number of words in the first document D1 and the number of words in the second document D2) all of wa
(4)

In the above equation (4), the word "wb" that gives P (wa → wb, ua → ub) to the word "wa" is referred to as "correspondence destination of the word" wa "". Then, in the above equation (4), "max" is a function for extracting the maximum value. Further, in the above equation (4), the value calculated by "max P (wa → wb, ua → ub)" is referred to as "maximum correspondence". Further, the three sets of the word "wa", the word "wb" and the maximum degree of correspondence are referred to as "correspondence between words".

Then, in the calculation in the above equation (4), when the word "wb" possessed by the second document D2 corresponds to a plurality of words possessed by the first document D1, the maximum correspondence degree of each is compared. , Only large values are added (calculation of "Σ" in the above equation (4)). That is, small values are excluded from the addition.

In the example shown in FIG. 8, the corresponding destinations to be added (hereinafter referred to as “addition target C1”) are seven values as shown in the figure. On the other hand, other values are excluded from the addition.

For example, the 111th character W111 in the 13th part P13 is a value excluded from the addition target (hereinafter referred to as “exclusion target C2”). The 111th character W111 has a maximum correspondence degree of "0.13" when the correspondence destination is the 203rd character W203 or the 207th character W207. The reason why the exclusion target C2 is not the target of addition is as follows.

Reason 1) The 203rd character W203 has a larger value than the 107th character W107 and has a maximum correspondence degree of "0.20".

Reason 2) The 207th character W207 has a larger value than the 103rd character W103 and has a maximum correspondence degree of "0.27".

As described above, when the addition target C1 is totaled, the numerator of the above equation (4) is calculated. In this example, the numerator of the above formula (4) (hereinafter referred to as “total maximum correspondence”) is calculated as “0.27 × 4 + 0.20 × 3 = 1.68”. Further, since the number of words in the first document D1 is "12" and the number of words in the second document D2 is "8", the denominator of the above equation (4) (hereinafter referred to as "maximum number of words") is ". 12 "is calculated.

Therefore, based on the above equation (4), the document correspondence of the first document D1 and the second document D2 is "1.68 / 12 = 0.14" by dividing the total maximum correspondence by the maximum number of words. Can be calculated.

In this way, the degree of correspondence between documents containing a plurality of parts calculates the degree of correspondence between words determined by using both the degree of correspondence between words and the degree of correspondence between parts. That is, the document correspondence is calculated based on the correspondence between words determined by using both the word correspondence and the partial correspondence. When the degree of correspondence between documents calculated in this way is used, the degree of correspondence between documents can be calculated with higher accuracy than when only the degree of similarity between words is used. That is, it is easier to find a document with more corresponding contents.

For example, with respect to the second document D2, the "unique information" (that is, the thirteenth part P13) included in the first document D1 does not have a part corresponding to the second document D2. However, in the method of calculating the degree of similarity between words, the degree of correspondence of the thirteenth part P13 is also taken into consideration in calculating the degree of correspondence of the document. Therefore, for example, the first document D1 containing the extra part "unique information" corresponds to the document containing only the eleventh part P11 and the twelfth part P12 (hereinafter referred to as "third document"). It will be calculated if there is.

On the other hand, in the present embodiment, it can be determined that the third document corresponds to the second document D2 rather than the first document D1.

The document is, for example, a contract or the like. Hereinafter, the case where the document is a contract will be described as an example.

In the examination using the contract, the person who conducts the examination determines whether or not the contract to be examined contains the contents necessary for the contract and whether or not the contents unnecessary for the contract are described. And so on. In this way, the examiner judges the validity of the contract. At that time, the examiner may use another contract that has been examined in the past as reference information.

When searching for a document to be used as a reference for such an examination, and when there are a large number of documents examined in the past, that is, a large amount of contracts accumulated for reference, a document that can be referred to is available. It is desirable that it be extracted automatically.

When searching for similar documents based only on the frequency of appearance of words, the contract is composed of multiple parts such as articles and clauses, so if you do not consider the degree of partial correspondence, it is an appropriate document. May not be searched.

In this example, it is desirable that the past contracts to be referred to in the current examination include as many parts as possible corresponding to the characteristic parts in the current examination and do not include the parts other than the characteristics as much as possible. In order to search for such a contract, it is often difficult to search by word-based similarity or the like.

Therefore, the degree of partial correspondence is calculated, and the degree of correspondence of words considering the degree of partial correspondence is used to search for the corresponding document. By doing so, the corresponding document can be searched. That is, in this example, it is possible to accurately search for past contracts whose contents are similar to those of the contract to be examined in the examination.

<About words and parts>
The word is not limited to one letter as shown in the above example. That is, the word may be a "character string" that is a combination of a plurality of characters. For example, when dividing into character strings, the document is divided into predetermined numbers to generate words. The predetermined number may be set in advance or may be optimized by learning. When the document is divided into words, for example, an analysis method such as morphological element analysis is used. In addition, in order to divide into words, a method of dividing into words by extracting n characters from the document by a method such as N-gram may be used.

The portion does not have to be a group of characters in units of one line as shown in the above example. For example, the part may be a phrase, a sentence, a paragraph, or a collection of characters in units of sentences. Then, when the document is divided into parts, the document may be divided into parts for each delimiter such as a kuten. In addition, words and parts may be generally applied to character strings (sequences) other than natural language, such as program source code or DNA sequences.

<Other embodiments>
The characters may include numbers, symbols, and the like.

Machine learning is a technology for making a computer acquire learning ability like a human being, and the computer autonomously generates an algorithm necessary for judgment such as data identification from learning data taken in advance, and a new one. It is a technology that applies this to various data to make predictions. When machine learning is used in the implementation of the present invention, the learning method for machine learning may be any of supervised learning, unsupervised learning, semi-teacher learning, enhanced learning, and deep learning, and further, these methods. A learning method that combines learning methods may be used, and a learning method for machine learning does not matter.

Each function of the embodiment described above can be realized by one or more processing circuits. Here, the "processing circuit" as used herein is a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, or a processor designed to execute each function described above. It shall include devices such as ASIC (Application Specific Integrated Circuit), DSP (Digital Signal Processor), and FPGA (Field Programmable Gate Array) circuit module.

Further, the document processing device 100 of each of the above-described embodiments is, for example, a PJ (Projector: projector), an output device such as a digital signage, a HUD (Head Up Display) device, an industrial machine, a medical device, a network home appliance, and an automobile (Connected). It may be a Car), a notebook PC (Personal Computer), a mobile phone, a tablet terminal, a game machine, a PDA (Personal Digital Assistant), a digital camera, a wearable PC, a desktop PC, or the like.

In addition, all or a part of the whole processing may be described in a computer language and may be realized by a program for causing a computer to execute a document processing method. That is, the program is a computer program for causing a computer or a document processing system using two or more computers to execute each process.

Therefore, when the document processing method is executed based on the program, the arithmetic unit and the control unit of the computer perform arithmetic and control based on the program in order to execute each processing. In addition, the storage device of the computer stores the data used for the processing based on the program in order to execute each processing.

The program can also be recorded and distributed on a computer-readable storage medium. The storage medium is a medium such as a magnetic tape, a flash memory, an optical disk, a magneto-optical disk, or a magnetic disk. In addition, the program can be distributed over telecommunication lines.

The embodiment of the present invention may be realized by a document processing system. Further, the document processing system may execute each processing and storage of data in a redundant, distributed, parallel, virtualized manner, or a combination thereof.

Although the present invention has been described above based on each embodiment, the present invention is not limited to the requirements shown in the above embodiments. With respect to these points, the gist of the present invention can be changed to the extent that the gist of the present invention is not impaired, and can be appropriately determined according to the application form thereof.

100 Document processing device 101 Input unit 102 Document division unit 103 Word correspondence degree calculation unit 104 Partial correspondence degree calculation unit 105 Document correspondence degree calculation unit C1 Addition target C2 Exclusion target D1 First document D2 Second document P11 11th part P12 12th Part P13 13th part P21 21st part P22 22nd part W101 101st character W102 102nd character W103 103rd character W104 104th character W106 106th character W107 107th character W108 108th character W109 109th character W110 110th character W111 111th character W112 112th character W201 201st character W202 202nd character W203 203rd character W204 204th character W205 205th character W206 206th character W207 207th character W208 208th character

Gerard Salton, Michael J. McGill, Introduction to modern information remote, New York, McGraw-Hill, 1983.

Claims (9)

Input section for inputting multiple documents and
A document division unit that divides the document into characters or words that are character strings, and divides the document into parts that are larger than the words.
A word correspondence calculation unit that calculates a word correspondence, which is a correspondence between the words in different documents, and a word correspondence calculation unit.
A partial correspondence calculation unit that calculates a partial correspondence, which is a correspondence between the parts of different documents, and a partial correspondence calculation unit.
A document processing device including a document correspondence degree calculation unit that calculates a document correspondence degree, which is a correspondence degree between documents, based on the word correspondence degree and the correspondence degree between words determined by using the partial correspondence degree. .. The part is a sentence containing the word or a collection of the words line by line.
The document processing apparatus according to claim 1, wherein the document division unit divides the document for each line feed code or delimiter to generate the portion. The character string is a collection of a predetermined number of characters.
The document processing apparatus according to claim 1 or 2, wherein the document dividing unit divides the document into predetermined numbers to generate the word of the character string. In the case of calculating the word correspondence between the first word included in the first document and the second word included in the second document.
The word correspondence is
Claim that the similarity between the first word and the second word is the ratio of the similarity between the first word and all the words contained in the second document to the total similarity. The document processing apparatus according to any one of 1 to 3. In the case of calculating the partial correspondence between the first part included in the first document and the second part included in the second document.
The degree of partial correspondence is
The first document and the second document include the first total similarity, which is the sum of the similarities between the first word included in the first part and the second word included in the second part. The document processing apparatus according to any one of claims 1 to 4, which is a ratio of the similarity of all words to the total similarity of the second total. The similarity is
The document processing apparatus according to claim 4 or 5, wherein the word matches or not, or the word is expressed as a vector and calculated based on the inner product of the vectors. In the case of calculating the document correspondence between the first document and the second document,
The degree of document compatibility is
Among the calculation results obtained by multiplying the word correspondence degree and the partial correspondence degree, the total maximum correspondence degree, which is the total of the maximum correspondence degree which is the maximum value in each word, is included in the first document and the second document. The document processing apparatus according to any one of claims 1 to 6, which is a value calculated by dividing by the maximum number of words, which is the maximum value among the number of words. It is a document processing method performed by a document processing device.
The input procedure for the document processor to input multiple documents,
A document division procedure in which a document processing device divides the document into words that are characters or character strings, and divides the document into parts that are larger than the words.
A word correspondence calculation procedure in which a document processing device calculates a word correspondence, which is a correspondence between the words in different documents, and a word correspondence calculation procedure.
A partial correspondence calculation procedure in which a document processing device calculates a partial correspondence, which is a correspondence between the parts of different documents.
A document correspondence calculation procedure in which the document processing device calculates the document correspondence, which is the correspondence between the documents, based on the word correspondence and the correspondence between words determined by using the partial correspondence. Document processing method including. A program that lets a computer execute a document processing method.
The input procedure for the computer to input multiple documents,
A document division procedure in which a computer divides the document into words that are characters or character strings, and divides the document into parts that are larger than the words.
A word correspondence calculation procedure in which a computer calculates a word correspondence, which is a correspondence between the words in different documents.
A partial correspondence calculation procedure in which a computer calculates a partial correspondence, which is a correspondence between the parts of a different document.
The computer executes a document correspondence calculation procedure for calculating the document correspondence, which is the correspondence between the documents, based on the word correspondence and the correspondence between words determined by using the partial correspondence. Program to make you.

Images