JP2003228571A - Method of counting appearance frequency of character string, and device for using the method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To solve a problem that no effective method for counting the number of documents containing character strings twice time or more is found in the prior art. <P>SOLUTION: A suffix array generating part 12 finds suffixes as to a plurality of document files 26 stored in a document database 24, and generates a suffix array in which an aggregate of all the suffixes is arrayed according to a dictionary order, to be stored as a suffix array file 28 in the document database 24. A character string frequency counting part 14 detects a class in classification of the suffixes, referring to the suffix array file 28, an overlap degree of the character strings is determined in the each class, and a character string frequency with the overlap degree is counted thereby. A document frequency calculating part 16 finds the frequency of the document in which the character strings are appeared k-times or more based on a difference in the character string frequencies with the overlap degree measured by the character string frequency counting part 14. <P>COPYRIGHT: (C)2003,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a technique for counting the frequency of appearance of symbol strings, and more particularly to a method for counting the frequency of appearance of character strings contained in a document, and an apparatus which can use the method.

[0002]

2. Description of the Related Art With the spread of the Internet, various databases are being used in a network environment, and practical applications such as electronic libraries based on document databases are also spreading. In order to efficiently use the database in which a large number of such electronic documents are stored, appropriate keywords are extracted from the stored documents and indexed for easy search, or documents are searched based on the keywords. It is necessary to classify.

In a sticky language such as Japanese, when extracting a keyword, the document is first decomposed into word strings. Morphological analysis is sometimes used to determine word breaks, but it is necessary to improve the accuracy of morphological analysis by expanding the word dictionary and applying complicated grammatical rules to correctly cut out words. There is. Further, in order to extract an appropriate keyword serving as an index, it is necessary to statistically analyze the appearance pattern of words and select useful words that characterize the document.

In such statistical analysis, statistical quantities such as the frequency of appearance of character strings included in a document are used. For example, when a set of documents is given, the total number of appearances cf of a certain character string and the number of documents df containing a certain character string one or more times are used as statistics. Literature Mikio Y
amamoto and Kenneth W. Church, Using Suffix Arrays
to Compute Term Frequency and Document Frequency f
or All Substrings ina Corpus, Computational Lingui
stics, Vol.27: 1, pp.1-30, MIT Press. discloses a method for obtaining the number df of documents containing a certain character string one or more times.

[0005]

When measuring the statistic of such a character string, the amount of calculation becomes a problem. The total number of characters N in the set of documents to be handled (hereinafter referred to as the corpus) is extremely large, and the partial character string is N (N-1) /
Since the number is two, the amount of calculation cannot be processed by an ordinary computer if a simple counting method is used.

The above Mikio Yamamoto and Kenneth W. Ch
In the literature by urch, substrings are classified into a maximum of 2N-1 classes based on the appearance pattern of character strings,
A method of counting document frequency has been proposed. However, this method measures the number of documents in which a certain character string appears one or more times, and cannot be applied to measure the number of documents in which a certain character string appears two or more times. A more meaningful statistic for a string may be obtained from the number of documents in which the string appears more than once, but there is no known efficient way to count this.

The present invention has been made in view of such circumstances, and an object thereof is to provide a counting method for efficiently measuring the appearance frequency of a character string, and a counting device that can use the method. is there.

[0008]

One aspect of the present invention relates to a method of counting the appearance frequency of character strings. This method counts the number of times a character string appears k times (k is a natural number of 2 or more) or more times and the number of times the character string appears k + 1 times or more in a set of documents, and calculates the difference between those times. By asking
The number of documents including the character string k times or more is acquired.

The "document" referred to here may be of any language.
"Document" means a document in which characters or grammars are not always deciphered, such as an old document or a ciphertext, or a sequence of arbitrary symbol sequences such as gene sequences, and means all files in which arbitrary symbol sequences are recorded. It is a broad concept. Therefore, the term “character string” included in a document is not limited to the alphabet of the language, but includes any symbol string forming one symbol system.

The set of partial character strings included in the document is classified into classes such that the partial character strings belonging to the same class have the same number of times that the partial character strings appear in the same document. The number of times may be counted for each class. Using the hierarchical structure of the class, the number of appearances may be counted by adding the number of appearances counted in the lower class to the number of appearances counted in the upper class.

Another aspect of the present invention also relates to a method of counting the appearance frequency of character strings. This method is a set of character strings in the range from a character at a certain position of the document to the end of the document, that is, a set of suffixes, for a corpus including a plurality of documents, and the set is a dictionary order. Generating an aligned suffix array (Suffix Array),
Among the appearances of the given character string x, the degree of duplication is k (k is 2
More occurrences of a natural number) or more strings frequency _cf k (a step of counting the x), wherein _cf number _df k of documents string x occurs more than k times (x) _cf k and (x) _{k +} 1 ( x)).

The suffix array is an array storing the appearance locations in the corpus of suffixes. The degree of duplication is a value defined for each appearance location of the character string x, and the fact that the degree of multiplicity of an appearance location of a character string x is k means that the number of occurrences of the appearance location or less in the suffix array is At the place
In addition, it means that there are k occurrence positions of the character string x belonging to the same document.

The method further includes the step of generating the suffix array for classifying the character strings belonging to the same class so that the character strings appear the same number of times in the same document. The appearance frequency cf _k (x) may be counted for each class. By the step of generating the classification, the hierarchical classification is generated, and the appearance frequency cf
_In the step of counting _k (x), the appearance frequency cf _k (x) counted in the lower class may be added to the appearance frequency counted in the upper class.

The determination of the degree of overlap in the step of counting the appearance frequency cf _k (x) and the detection of the class in the step of generating the classification may be performed at the same time. When a class is defined by a subscript section of a suffix array, in the process of detecting the start and end of the section of a new class, the degree of overlap is determined for each detected class to count the occurrence frequency cf _k (x). You can go.

Yet another aspect of the present invention relates to a counting device. This apparatus includes a counting processing unit that counts the appearance frequency of a character string in a set of documents, and a recording unit that records data regarding the appearance frequency of the character string. The counting processing unit is a set of character strings in a range from a character at a certain position of the document to the end of the document, and the suffix array generation unit generates a suffix array in which the set is arranged in dictionary order, At the same time as detecting the class of suffix array classification, a character string frequency counting unit that counts the frequency of overlapping appearance of the character strings in the class unit by determining the degree of overlap of appearance of the character strings. , And a document frequency calculation unit that calculates the frequency of documents in which the character string appears based on the appearance frequency of the character string.

Each suffix of the suffix array belongs to the same document as the suffix and has a pointer to a suffix immediately before the suffix in the order of arrangement of the suffix array, and the character string frequency counting section is provided with the pointer. The degree of overlap may be determined depending on whether or not the number of times can be sequentially traced.

The character string frequency counting unit classifies the suffix array into a hierarchical class structure and adds the appearance frequency of the character string counted in the lower class to the appearance frequency counted in the upper class. You may.

In the document frequency calculation unit, the character string is k
The frequency of a document that appears (k is a natural number) or more times may be obtained from the difference between the appearance frequency of the character string having the duplication degree of k or more and the appearance frequency of the duplication degree of k + 1 or more.

A keyword extraction unit may be further included for extracting a keyword from the document using the frequency of the document in which the character string appears k times or more (k is a natural number). In particular, the frequency of documents in which a character string appears twice or more may be used.

Yet another aspect of the present invention is a computer program.
Regarding the program. This program contains multiple documents.
From the character at a certain position in the document to the corpus
A set of character strings in the range up to the end of the document
Generate a suffix array in which the sets are ordered lexicographically
And a hierarchical class structure of the suffix array.
The class related to the structure is detected and the class unit
In the appearance of the given character string x, the degree of overlap is k (k
Is a natural number of 2 or more) and the appearance frequency cf of a character string of 2 or more
_k(X) counting step and counting in lower class
Occurrence frequency cf_k(X) is counted in the upper class
The step of adding to the appearance frequency, and the character string x is k times or more.
Number of documents that appear above df_k(X) is added to the character string
The appearance frequency cf in which the degree of overlap is k or more_k(X) and
The appearance frequency cf in which the degree of overlap is k + 1 or more
_{k + 1}A process for obtaining the difference from (x) by a computer
To run.

Yet another aspect of the present invention relates to a method of counting the frequency of appearance of symbol strings. This method is such that, for a set including a plurality of symbol sequences, a set of partial symbol strings included in the sequence is determined by the number of times that a partial symbol string belonging to the same class appears in the same sequence. The number of times an arbitrary symbol string appears k or more (k is a natural number of 2 or more) times while classifying them into the same hierarchical class is counted for each class, and the arbitrary symbol string is k times or more. Determine the number of said sequences included.

It should be noted that any combination of the above constituent elements, and any expression of the present invention converted between a method, an apparatus, a server, a system, a computer program, a recording medium, etc. are also effective as an aspect of the present invention. .

[0023]

BEST MODE FOR CARRYING OUT THE INVENTION Before describing a counting method and a counting apparatus according to an embodiment, the definition of symbols and the concept will be described.

1. The symbol definition tf (d, x) is replaced with the character string x included in the document d.
Is defined as the number of The following two statistics are tf (d,
It is a statistic that can be calculated from x) and is usually used for information retrieval. cf (x): number string x appears in the corpus _{cf (x) = Σ d tf} (d, x) df (x): number df (x) of the document the string x occurs more than once = | {D | tf (d, x) ≧ 1} |

The following statistics used in the present invention are also t
It is obtained from f (d, x). df ₂ (x): number of documents in which character string x appears twice or more df ₂ (x) = | {d | tf (d, x) ≧ 2} | df ₃ (x): character string x occurs 3 times Number of Documents Appearing Above df ₃ (x) = | {d | tf (d, x) ≧ 3} | Generally, df _k (x): number of documents where character string x appears k times or more df _k (x ) = | {D | tf (d, x) ≧ k} |

Here, the main usage of df ₂ (x) is as follows:
Df as "estimated value of conditional probability that a character string is included more than once when the character string is included in the document distribution more than once"
₂ (x) / df (x). As shown in document [6], this conditional probability can be used as an expression of the nature of a word when applied to a word. df ₃ and df _k are extensions of df ₂ and can be used for estimation of similar conditional probabilities and can be used for cross-checking estimates.

2. Suffix Array The Suffix Array is the data structure shown by reference [1]. As shown in the example of Fig. 1, when given text, this data structure is a character string in the range from the character at a certain position of the text to the end of the corpus (this is called a suffix). Is considered, and the set is arranged in lexicographical order. Since each suffix is uniquely determined by the start position, if the body of the text is in the memory, then one can be stored by storing one integer indicating the start position of the character string. In the example of the figure, the suffix "suffix" is added to the text "abcabedcabcde".
The value of suffix [6] is 7 because [6] is “cabcde” and the starting place is the seventh column of the text array.
As described above, if the suffix array is used, the location of an arbitrary substring can be known, but the required storage capacity is O (N) (N order) when the size of the text is N. .

The Suffix Array can be generated by the following routine. / * size: number of characters in the corpus, text: pointer to the beginning of the corpus * / int suffix_compare (struct suffix_struct * x, struct suffix_struct * y) (return strcmp (text + x-> position, text + y->position); / * x-> position, y-> position are pointers to locations corresponding to x and y, respectively * /} for (i = 0; i <size; i ++) {suffix [i] .position = i;} qsort (suffix, size, sizeof (struct suffix_struct), suffix_compare);

When calculating the document frequency, if the document length has an upper limit, the character string in the corpus can be regarded as being separated for each document.
Under this condition, O (NlogN) time is required to create the data structure using the above algorithm.

Generally, when the corpus has no document structure, it is difficult to estimate the upper limit of comparison calculation of character strings by a simple method, but an efficient algorithm for generating a suffix array is known (see [2]),
According to this method, the computational complexity of creating a Suffix Array is O
(NlogN).

With this data structure, an arbitrary character string is given, a binary search can be performed in order to specify the place where the character string appears, and a list of appearance places can be obtained in O (logN) time. The appearance location here is an integer value that corresponds one-to-one with a certain part in the corpus.
Due to the nature of, this also corresponds to a certain suffix.

3. Character String Classification In the counting algorithm according to the embodiment, the character string classification method of the document [3] is used. Character strings x belonging to the same class are classified so that the values of tf (d, x) are the same. This classification is defined using suffix. The suffixes of the Suffix Array are arranged in dictionary order, so the beginning of the character string is often the same as the next suffix. So common [i] is suffix [i] and suffix [i
+1] is the length of the common part from the beginning of the character string. The class definition is shown below.

Here, in order to simplify the description of the definition, j-
In the case of 1 <i, min _{k = i} ^j−1 = ∞. And co
Set mmon [-1] =-1 and common [N] =-1. The larger value of common at the boundary of the interval [i, j] is outgoing (i, j) = max (comm
on [i-1], common [j]), and the smallest common value within the interval [i, j] is inner (i, j) = min _{k = i}
^It is defined as ^j-1 (common [k]).

[Definition] The section [i, j] forming a class means that inner (i, j)> outgoing (i, j).
inner (i, j) is the length of the character string that is the common part in the whole section, and inner (i, j)> outgoing (i, j) is the character string that is the common part in the whole section. Means shorter.

In the example of Suffix Array in FIG. 2, each suffix is
The value of common for ix is shown. In this example, consider whether the intervals [i, j] = [2,2], [i, j] = [1,4], and [i, j] = [1,3] each form a class. .

Outgoing (2,2) = max (common [1], common [2]) = max (6,3) = 6 inner (2,2) = min _{k = 2} ¹ (common [k]) = ∞ outgoing (1,4) = max (common [0], common [4]) = max (2,0) = 2 inner (1,4) = min _{k = 1} ³ (common [k]) = 3 outgoing (1,3) = max (common [0], common [3]) = max (2,6) = 6 inner (1,3) = min _{k = 1} ² (common [k]) = 3 [2,2] is inner (2,2)> outgoing (2,2),
Since the interval [1,4] is inner (1,4)> outgoing (1,4), a class is formed, but the interval [1,3] is inner (1,3) <out
Do not form a class because it is going (1,3). FIG. 3 illustrates the case of the section [1, 4]. When it is determined in this way whether or not a class is formed for each section, a class is formed in the section shown in FIG. The sections [0,0], [1,1], [2,2], [1,2], etc. shown in the figure all form a class, and as will be described later, sections that form a class are hierarchical. Make a structure.

When the interval [i, j] forms a class,
Lengths outgoing (i, j) +1 to inn common to the interval [i, j]
The set of substrings up to er (i, j) is defined as the class of the string corresponding to that section. FIG. 5 shows character strings belonging to the classes of the sections [1, 4], [1, 2], and [1, 1] in the example of FIG. In the interval [1,4], the outgoing value is 2 and the inner value is 3, so the character string "aab"
Is an element of the class. In the interval [1, 2], the outgoing value is 3 and the inner value is 6, so the character string "aab
"b", "aabbcc", and "aabbcc" are class elements. In the interval [1,1], the outgoing value is 6 and in
Since the value of ner is infinite, the character string "aabbcc
“D”, “aabbccdd”, ... (Continued below) are class elements.

Hereinafter, when the appearance pattern of a character string is referred to,
Given a string x and a document d, tf
Refers to the functional form of (d, x). In general, there are N (N-1) / 2 substrings of text of length N, and it is not practical to tabulate all of them, but if they are classified into those having the same appearance pattern, the number is decrease. According to document [3], the number of classes is at most 2N-1, which is a practical size at which a classification table can be actually created and stored.

[Definition of Occurence (C)] For intervals [i, j] defined by class C, set suffix [i], ..., Suf
Let fix [j] be Occurrence (C).

[Property 1] If there is a class C, then for any two elements x and y of C, if an arbitrary document is d, then tf (d, x) = tf (d, y). Proof: When calculating tf (d, x), the set Occurrence (C) of the position where x appears can be obtained for C to which x belongs, and tf (d, x) can be determined from it. If y belongs to the same class as x, since Occurence (C) is the same, tf (d, x) = tf (d, y).

Due to this property, for each class shown in FIG. 5, the number tf of arbitrary elements of each class included in the same document can be obtained as illustrated. For example, for the interval [1, 2], the element "aab" of that class
The number tf of “b”, “aabbcc”, and “aabbcc” included in the same document is equal to the length 2 of this section.

[Property 2] When there is a class C, df (x) = df (y) df ₂ (x) = df ₂ (y) df _n (x) for any two elements x and y of C. = Df _n (y) (n ≧ 3) cf (x) = cf (y) holds.

Proof: From property 1, tf (d, x) = tf
Since (d, y), df (x) = {d | tf (d, x) ≧ 1} = {d | tf
(D, y) ≧ 1} = df (y) df ₂ (x) = {d | tf (d, x) ≧ 2} = {d | t
f (d, y) ≧ 2} = df ₂ (y) df _n (x) = {d | tf (d, x) ≧ n} = {d | t
f (d, y) ≧ n} = df _n (y) cf (x) = Σ _d tf (d, x) = Σ _d tf (d, y)
= Cf (y)

[Property 3] All substrings are at most 2
It can be classified into N-1 classes. There is a proof in Ref. [3], but its outline shows that there are N classes of character strings that appear only once, and no matter how many times they overlap, at most N-1 overlapping classes can be constructed. Prove.

4. The class hierarchical relation section [i1, j1] forms the class C1, and the section [i, j1]
2, j2] form the class C2, and the interval [i1,
j1] is included in the interval [i2, j2], C1
Is defined as a subordinate class of C2. Further, C2 is defined as a class higher than C1.

[Property 4] When the two classes C1 and C2 intersect with each other, either C1 is a higher class of C2 or C1 is a lower class of C2.

Proof The intersection between C1 and C2 means i1 ≦ i2 ≦ j1 ≦ j2 (1) i2 ≦ i1 ≦ j2 ≦ j1 (2) i1 ≦ i2 ≦ j2 ≦ j1 (3) i2 ≦ i1 ≦ j1 ≦ j2 (4).

In the case of (1), it is assumed that i1 <i2. In the interval [i1, j1], max (common [i1-1], common [j1]) <min _{k1 = i1}
^j1-1 (common [k1]), so common [j1] <common [k1] (i1 ≦ k1 ≦ j
1-1). On the other hand, in the section [i2, j2], k1
= I2-1 and k2 = j1 (i2 ≦ k2 ≦ j2-1), there are k1 and k2. Therefore, common [k1] = common [i2-1]> common [k2] = common [j1], and the interval [i2, j2] is max (common [i2-1], common [j2]) <min _{k2 = i2}
^{Since j2-1} (common [k2]) is not satisfied and i1 <i2, the class C2 is not formed, so that there is no intersection between C1 and C2.

In the case of i1 = i2 ≦ j1 ≦ j2, C2 is a superclass of C1 or the same class according to the definition of the class hierarchy.

In the case of (2), it can be proved in the same manner as in (1). Also, in the case of (3), C is defined from the definition of the class hierarchy.
1 is a superclass of C2 or equal,
In the case of (4), C2 is a superclass of C1 or is an equal class.

As described above, when two classes intersect each other, one becomes the upper class or the lower class of the other.

[Property 5] In the Suffix Array, all suffixes form a hierarchical structure by classes. From the proof common [−1] = common [N] = −1, the top class is
This class contains all suffixes. Also, due to the nature 4, unless a class is a partial class of another class, they do not intersect. At this time, the section of the partial class is shorter than that of the upper class.

From the above, the appearance positions of all character strings form a hierarchical structure by the character string class.

[Characteristic 6] For an arbitrary section [i, j], there is a section including a section including [i, j] to form a class. Outgoing (i, i) <∞, inner in interval [i, i]
Since (i, i) = ∞, inner (i, i)> outgoing (i, i), and the interval [i, i] forms the lowest class consisting of one suffix. From Proof Property 5, all suffixes of Suffix Array form a hierarchical structure by classes.

[Symbol] For an arbitrary section [i, j],
Of the intervals forming the class including it, the lowest one is the class determined from [i, j], and Class * ([i,
j]).

The fact that the lowest class including an arbitrary section is uniquely determined is a property required when a counting algorithm with a reduced amount of calculation is constructed, as will be described later.

5. Problems in measurement of document frequency If the document frequency of the character strings belonging to all classes is calculated by a simple method, there is a practical problem in a normal computer. Even if the class size is at most 2N, df (x), df ₂ (x), df
_{As in} the case of measuring ₃ (x), if a set that satisfies the definition of each statistic is created and its size is measured, it takes O (N) time to process each x. x
If there are N, then a total of O (N ² ) time is required. Considering the size of the corpus, this is a process that cannot be executed by an ordinary computer.

By using the Suffix Array, the appearance location of the character string can be specified, and the document frequency for an arbitrary character string can be calculated by counting the number of documents at the specified location. This can be faster than the simple method. However, even this is not practical in the case where many frequently-used character strings are included. Even if the place where the character string appears can be specified, in order to find the number of documents that contain the character string, it is necessary to check for duplication whether or not a certain place exists in the same document. The process of checking for duplication takes time for a character string having many occurrence locations. Calculating the document frequency of all strings also includes calculations for strings that are as frequent as a single character, and such strings are the bottleneck in the calculation.

If the frequency of appearance is simple, the total frequency can be calculated according to the class hierarchy, but the number of documents in which the character string appears cannot be directly collected. For example, consider a corpus as shown in FIG. The character string “abc” appears 6 times in the entire corpus, and the number of documents in which it appears is 4. The character string "abx" appears 7 times in the entire corpus, and the number of documents in which it appears is 5. At this time, if there are only two character patterns “abc” and “abx” following the character string “ab”, the number of appearances cf (ab) of “ab” is as shown in FIG. 6B. , Cf
Calculated by (abc) + cf (abx), which is 13 times. However, the number of documents in which "ab" appears d
f (ab) is incorrect when calculated by df (abc) + df (abx). This is "abc"
This is because the documents in which both "abx" and "abx" appear are counted in duplicate. If the condition for measuring documents is the number of documents that appear twice or more, the change in document frequency due to the upper and lower classes becomes more complicated, and it is difficult to count without avoiding duplication.

6. Duplication degree of appearance place As a preparation for counting the document frequency, the duplication degree of each appearance place of a character string is defined. The purpose of introducing multiplicity is to define a new statistic that can be added during the course of a class hierarchy to measure document frequency, and to obtain document frequency as a function of that statistic. This is because

The appearance location of all character strings is Suffix Arr.
It can be ordered by the number of the array in ay. This order is defined as the suffix order, and this is used to define the degree of overlap of the appearance location of the character string.

[Definition] When the occurrence degree of a certain character string x is k, it means that there are k or less places of appearance of the character string x belonging to the same document in the suffix order or less. I will.

FIGS. 7A and 7B show examples of the degree of overlap.
Abx in suffix order of the Suffix Array shown in FIG.
Below (suffix [3]) are abc (suffix
[0]), abd (suffix [1]), abe (suffix [2]), and abx (suffix [3]). Here, for suffix [3], when the degree of overlap k of the character string “ab” in document # 1 shown in FIG. 7A is calculated, the character strings “abc” and “abd” in document # 1 are obtained. , "Abx" appear, k = 3. Similarly, FIG. 7B shows the result of obtaining the degree of overlap k for other suffixes.

[Property 7] The character string x is t in the document i.
When appearing individually, the degree of overlap is calculated for all t places of appearance, and they are arranged in suffix order to obtain 1, ..., T.

7. When the character string frequency with duplication condition [symbol] x is a character string, the character string frequency with duplication degree is written as cf _k (x), and the document character string frequency with duplication degree is written as tf _k (d, x). [Definition] cf _k (x) is the appearance frequency of a character string whose duplication degree is k or more among the appearances of the character string x in the corpus. [Definition] tf _k (d, x) is the appearance frequency of a character string whose duplication degree is k or more among the appearance of the character string x in the document d.

[Property 8] ∀x, yεC, ∀k; ∀d; tf _k (d, x) = tf _k
(D, y) Property 8 is obvious from the fact that both sides are the number of occurrence Occurrences (C) of C that are included in d and have a degree of overlap of k.

8. Relationship between character string frequency with overlap condition and document frequency Document frequency and character string frequency with overlap condition have the following simple relationship. [Theorem “Relationship between character string frequency and document frequency”] df _k (x) = cf _k (x) −cf _{k + 1} (x)

Proof When tf (d, x) = t, for k ≦ t, tf _k (d, x) -tf _{k + 1} (d, x) = 1 tf (d, x) = t tf _t ( d, x) = 1, tf
_{Since t + 1} (d, x) = 0, ..., For k> t, tf _k (d, x) −tf _{k + 1} (d, x) = 0 cf _k (x) = Σ _d tf _k (d , X), cf _k (x) −cf _{k + 1} (x) = Σ _d (tf _k (d, x) −tf _{k + 1} (d, x)) = | {d | tf (d, x) ≧ k} | = df _k (x)

Examples of actually obtaining cf _k and df _k in a certain corpus are shown in FIGS. 8 (a) and 8 (b). The corpus consists of three documents # 1, # 2, as shown in FIG.
# 3, including the string “ab” for each document,
The appearance frequency of “abc” is tf (1, ab) = 7, tf
(1, abc) = 3, tf (2, ab) = 8, tf
(2, abc) = 5, tf (3, ab) = 6, tf
(3, abc) = 3.

For this corpus, the character string "ab",
Character string frequency cf with duplication of "abc"_kSeeking
The result of calculating the document frequency from the value of
It is shown. cf₈(Ab) = 1, cf₇(Ab) =
3, cf₆From (ab) = 6, df₇(Ab) = cf₇
(Ab) -cf₈(Ab) = 3-1 = 2, df₆(A
b) = cf₆(Ab) -cf₇(Ab) = 6-3 = 3
Become. Also cf₅(Abc) = 1, cf_Four(Abc) =
2, cf_ThreeSince (abc) = 5, df_Four(Abc) = c
f_Four(Abc) -cf₅(Abc) = 2-1 = 1, df
_Three(Abc) = cf _Three(Abc) -cf_Four(Abc) =
5-2 = 3.

[Property 9] When there is a certain class C, cf _k (x) = cf _k (y) holds for any k of its elements x and y. Proof cf _k (x) = Σ _d tf _k (d, x), tf _k (d,
x) = tf _k (d, y), cf _k (x) = Σ _d tf
_{k (d, x) = Σ} d tf k (d, y) = cf k (y)

9. Data structure for duplication determination For each suffix, belong to the same document,
And previo the position of the closest suffix that precedes it in suffix order.
Record it as a us link. If there is no such place, make -1 the previous link. The data structure of this previous link is a memory area whose size is proportional to the size of the corpus.

To determine that the degree of duplication of the appearance location of the character string x is k or more, the previous link is set to k from the appearance location.
Whether or not it can be traced, and if it can be traced, it can be determined by measuring whether the character string still appears at the place after tracing k times.

FIGS. 9A, 9B and 9C are explanatory views of the previous link. 9A and 9B show two documents # 234 and # 235 included in the corpus. FIG. 9C shows the Suffix Array of the corpus. Figure 9
In documents # 234 (a) and (c), suffix [1
2405] 's previous link is shown by the arrow. Similarly, in FIGS. 9B and 9C, a state in which the previous link of suffix [12494] is followed in the document # 235 is indicated by an arrow.

In order to create the data structure of such a previous link, an integer array of the same number as the number of documents is prepared, the document number is obtained for each occurrence of each character string, and the information of the previous link is obtained from the array. And then update the table to the current location.

To create the data structure of the previous link, the same memory area as the number of documents is prepared and the entire corpus is scanned once. This can be generated with the following program / * id_max: number of documents, size: number of corpus characters * / for (i = 0; i <id_max; i ++) {last_suffixes [i] = -1;} for (i = 0; i <size; i ++) {suffix [i] .previous_suffix = last_suffixes [suffix [i] .id]; last_suffixes [suffix [i] .id] = i;}

The duplication determination can be performed by following this previous link k times and determining whether or not the character string is in the same document. In order to reduce the amount of calculation, the counting program according to the embodiment does not simply determine the degree of overlap, but simultaneously determines the degree of overlap and class detection as described below.

10. Algorithm for class detection An outline of an algorithm for class detection is shown below. (1) Suffix Array is checked in order from the smallest suffix order. (2) Find the starting point of the class. (3) Find the place where the class ends. (3-a) Since the classes are nested, the start positions of other classes may be found before the end positions of the class are found. In this case, push that starting location onto the stack. (3-b) If the end location of the class is found, report it and restore the stack.

When a class is obtained by the above algorithm, a class in which the beginning of the desired class can be found and the end of which cannot be found is called a "calculating class". Algorithmically, the class in the stack corresponds to the current “computing class”.

Common [i] is an array having the same size as the character string of the corpus, and has a length at which the character string matches the next suffix in the Suffix Array. The calculation time of this sequence is long when the matching length is long, but in the method of Ref. [2],
Common [i] is calculated at the same time when Suffix Array is calculated. Therefore, the order of calculation amount is not increased.

The class of the character string follows the increase / decrease of common [i]. When common [i] increases, the class currently being calculated is registered as a class that has not been calculated, and it is processed as if a new class started.

On the other hand, when common [i] is decreasing, there are the following two cases. (1) If the current class ends, but there is actually a class that started at the same suffix location as the current class other than the current class, (2) the current class ends and the stack top class When processing must be continued.

In the second case, when the processing of the class in the middle of the calculation of the stack top is continued, the processing is continued from the inspection as to whether or not this class will be terminated immediately.

In order to discover a class, an operation for judging the end of the class is performed for each common [i]. In the second case, the number of iterations is the same as the number of classes being calculated, but the total number of iterations does not exceed the maximum number of classes. Therefore, the maximum number of classes and co
This operation is completed with a calculation amount of O (N) from the number of mmon [i].

The following properties hold for the class currently being calculated. [Property 10] When the character strings starting from the suffix currently being calculated up to the delimiter of the document are classified by the class to which they belong, the class becomes the class currently being calculated.

11. Given that the counting degree k of the character string frequency with a simple duplication condition is given, it is considered to find cf _k (x) in which the degree of duplication is k or more for all the character strings x. The degree of duplication is a function of character string and location, but the degree of duplication of character strings belonging to the same class does not differ. Also, for character strings belonging to the same class, cf _k (x)
Are equal. So prepare counters for each class,
It is also possible to count by performing processing for each suffix. This is called the "simple method". Although the memory area required for this method is O (N), the calculation time becomes a problem.

The counting method is as follows. For a certain place, a set of classes starting from that is obtained, counters are prepared for all classes, it is determined whether or not the degree of overlap is k or more for each class, and It is to add 1.
If this method is simply performed, the number of classes related to one suffix may be large, and thus the amount of calculation of O (Nlog (N)) or less may not be enough. Specifically, this happens for data in which the same character continues for a long time.

12. Counting character string frequency with duplication condition When the character string frequency with duplication degree is calculated by a simple method, all the classes that it consists of are calculated for one suffix, and the counter is updated for all the classes. It must be made. However, you can avoid updating the counter for all classes by using the following properties.

[Property 11] When a certain place is given,
A set of classes corresponding to the first character string of the suffix is required, but the classes have a unique hierarchical relationship. [Property 12] When a place is given, su of that place
If the degree of duplication of a character string of a certain class among the classes corresponding to the leading character string of ffix is k, the degree of duplication of a class higher than the class is k or more.

Due to these two properties, it is possible to add the counters to one for one suffix and one overlap degree k. That is, among the classes with a degree of overlap k or more at a certain suffix, only the counter of the lowest class is added, and when the counting of the lower class is finished, the count value is added to the counter of the upper class. By going
Counts for all classes can be obtained. A counting algorithm that combines the detection of classes and the determination of the degree of duplication utilizing this property will be described below.

13. Class discovery and frequency calculation 13.1 Processing when discovering the beginning of a class The beginning of a class can be discovered by increasing common [i]. At this time, the information of the character string frequency with the degree of duplication currently measured is saved in the stack like the information of other classes, and the character string frequency with the degree of duplication is initialized to 0 and newly counted.

13.2 Fusion of Multiplicity Judgment and Class Selection At some place, the operation of specifying the lowest class among the classes with a multiplicity larger than k can be combined with the multiplicity judgment. Since the determination of the degree of duplication is performed based on whether the section of the location i which has followed the previous link k times and the section of the current location j is included in one document, conversely, a set of classes including the section is obtained, and In the above Clas
You can find s * ([i, j]).

This operation can be performed simultaneously with the detection of the class. This means, "At one place, the degree of overlap is k
Since the section [i, j] defining “larger Class * ([i, j])” has the current location j at the end, it is used that the class is unfinished during the detection.

Specifically, first, the appearance of a character string located after the previous link has been searched k times is obtained. Next, Class * ([i, j]), which is common and is being calculated, is specified from the character string including the appearance location and the first appearance location, and the character string frequency with the degree of duplication of the class is added.

13.3 Processing when the end of a class is found The end of a class can be found by reducing common [i]. At this time, a process of transmitting the count value to the upper class is performed. By adding the count value to the counter of the upper class when the counting of the lower class is finished, as a result, the same value as that of adding to all the classes can be obtained.

14. Example An example of an apparatus for carrying out the counting algorithm described above will now be described. FIG. 10 is a configuration diagram of the counting device 100 according to the embodiment. In terms of hardware, this configuration is the CPU, memory, and other L of any computer.
It can be realized by SI, and in terms of software, it is realized by a program having a counting processing function loaded in a memory, but here, the functional blocks realized by their cooperation are drawn. Therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various forms by only hardware, only software, or a combination thereof.

The counting processing unit 10 uses the document database 24
The plurality of document files 26 stored in is read and the statistical amount regarding the appearance frequency of the character string in the document file 26 is counted. The plurality of document files 26 form a corpus. The counting processing unit 10 includes a SuffixArray generating unit 12, a character string frequency counting unit 14, and a document frequency calculating unit 16.
The Suffix Array generation unit 12 obtains a suffix for each document file 26 and generates a Suffix Array in which a set of all suffixes is arranged in a dictionary order. In addition, the Suffix Array generator 12 uses the common and prev described above for each suffix.
Create an ious link and store it in association with each suffix.
The data regarding the Suffix Array generated in this way is stored in the document database 2 as the Suffix Array file 28.
Stored in 4.

The character string frequency counting unit 14 refers to the Suffix Array file 28, performs class detection of suffix classification and determination of the character string duplication degree for each class, and determines the character string frequency with duplication degree. Count. The document frequency calculation unit 16 obtains the frequency of documents in which a character string appears k times or more based on the difference in the character string frequency with the degree of duplication measured by the character string frequency counting unit 14. The statistics measured by the character string frequency counting unit 14 and the document frequency calculating unit 16 are stored in the document database 24 as the character string appearance frequency data 30.

The inquiry section 18 receives an arbitrary character string designated by the user. In response to a request from the inquiry unit 18, the document frequency calculation unit 16 searches the table of the statistical amount of each class included in the character string appearance frequency data 30 for two minutes,
Outputs the data about the appearance frequency and document frequency for the inquired character string. Inquiry section 18
Provides the user with the statistical data output by the document frequency calculation unit 16.

The keyword extracting section 22 refers to the character string appearance frequency data 30 and extracts keywords that characterize each document file 26. The keyword extraction is performed based on the word break detection and the word appearance pattern using the document frequency df _k as described later, and does not require a word dictionary. The extracted keywords are stored in the keyword index file 32 as an index for searching each document file 26. The search unit 20 receives a character string specified by the user as a keyword, refers to the keyword index file 32, searches for the document file 26 including the specified keyword, and provides the user with the document file 26.

FIG. 11 shows the counting device 10 having the above-mentioned configuration.
It is a flowchart of the counting process performed by 0. The flow of the main part of the counting algorithm will be described with reference to this flowchart. This flowchart is based on Suffix Array
7 shows a flow of class detection and character string frequency counting with duplication performed by the character string frequency counting unit 14 after the generating unit 12 has generated the data structure of the Suffix Array including the previous link.

A variable i indicating the suffix of Suffix Array is initialized to 0 (S10). It is checked whether or not the current value of the variable i is the start position of the new class, that is, the start of the section defining the new class (S12). If it is the start location of the new class (Y of S12), the counter of the character string frequency cf _k with duplication degree of the new class is initialized to 0 (S14). If it is not the start place of the new class (N of S12), step S14 is not performed.

Next, as a counting process, an appropriate class is selected for each degree of duplication according to the degree of duplication, and the counter of the character string frequency cf _k is incremented (S16). The appropriate class mentioned here is the lowest class including the section defined by the current location i and the appearance location j of the character string at the position where the previous link is followed k times.

Next, as post-processing, class end detection is performed. It is checked whether or not the current variable i is the end position of the class, that is, the end of the interval defining the class (S1
8). If it is the end of the class (Y of S18), the character string frequency cf _k with the degree of duplication of the finished class is recorded (S2).
0). Further, the character string frequency cf _k of the finished class is added to the character string frequency counter of the upper class (S2).
2). If the class has not ended (N of S18), steps S20 and S22 are not performed.

In order to check the next suffix, the variable i of the suffix of suffix is incremented by 1 (S24). If the variable i is equal to the size of the array of Suffix Array (S
26 Y), the counting process ends, but if not (N in S26), the counting process is repeated by returning to step S12.

In this way, the character string frequency counting section 14
The suffix is sequentially checked, the class is detected and the degree of duplication is determined at the same time, and the frequency of character strings with the degree of duplication is measured. The details of the processing in each step are as described above.

15. Example of Execution The counting algorithm described above is executed by the program described below. Here, the execution result of this program will be described. The data to be processed as a sample are the following files. One line corresponds to one document. abcabcabc abcd abcde bcde

15.1 First Stage In the first stage, a Suffix Array is created, common is sought, and a previous link is created. The following data is created for the above example. The data is from the beginning, su
The number of the ffix, the number of the document to which the suffix belongs, the number of the suffix that belongs to the same document that appears immediately before, the length of the suffix that immediately follows and the "character string that matches from the beginning", and the character of that suffix. In order. 0 0 -1 0: 1 1 -1 0: 2 2 -1 0: 3 3 -1 0: 4 0 0 3: abc 5 0 4 6: abcabc 6 0 5 3: abcabcabc 7 1 1 4: abcd 8 2 2 0: abcde 9 0 6 2: bc 10 0 9 5: bcabc 11 0 10 2: bcabcabc 12 1 7 3: bcd 13 2 8 4: bcde 14 3 3 0: bcde 15 0 11 1: c 16 0 15 4 : cabc 17 0 16 1: cabcabc 18 1 12 2: cd 19 2 13 3: cde 20 3 14 0: cde 21 1 18 1: d 22 2 19 2: de 23 3 20 0: de 24 2 22 1: e 25 3 23 0: e

15.2 Second Stage In the second stage, class detection and frequency counting are performed. The execution results are shown in Table 1. <Start length shown in the execution result
c0 c1 c2 c3 c4> indicates the class status during the calculation. sta
rt is the suffix number where the section corresponding to the class starts, le
ngth is the length of the longest string in that class.
c0, c1, c2, c3, and c4 are counters of the frequency of character strings having the degree of duplication 1, 2, 3, 4, and 5 belonging to the class, respectively.

The class of the character string is determined by the length of the shortest character string and the length of the longest character string, but in the class in the middle of calculation, there is no character string of the shortest length. The length of the shortest character string belonging to the class is finally obtained from the hierarchical relationship of the classes. Also, the length of the shortest character string cannot be included because it is not known in the middle when looking from the beginning.

Before checking suffix, each time the class under calculation is output in one line. This is the part beginning with "<" in the execution result of Table 1. Next, suffix is output. Each number of suffix is the same as that of the example of the first stage.

Each time one suffix is viewed, the following two processes are performed. (1) For each multiplicity, select an "appropriate class" according to the multiplicity and add the number of appearances. This process is the part that starts with "Incrementing" in the execution result. (2) When a class with two or more sections is found, record it and add the number of appearances to the higher class. This process is the part that begins with "--->Class" in the execution result.

Here, the "appropriate class" is the class of the shortest section including the section determined by the current location (a) and the location (b) where the previous link is traced the number of times of multiplicity. If the previous link is not traced by the multiplicity, it means that there is no class to add.

Classes that have the same beginning and ending locations of the interval are not recorded specially. If it is not recorded, it appears either once or 0 times, which can be judged by O (log (N)).
No record is required.

In the execution results of Table 1, the multiplicity count of the class in the middle of calculation is changed, but the place is two of the inheritance from the result of the end of the class to be added and the upper class. It is one case and limited to those cases.

[Table 1] <S0 L0 0 0 0 0 0>     0 0 -1 0:  -> Incrementing c [0] of Class * [0,0]: (S0, L0) S = "" <S0 L0 1 0 0 0 0>     1 1 -1 0:  -> Incrementing c [0] of Class * [1,1]: (S0, L0) S = "" <S0 L0 2 0 0 0 0>     2 2 -1 0:  -> Incrementing c [0] of Class * [2,2]: (S0, L0) S = "" <S0 L0 3 0 0 0 0>     3 3 -1 0:  -> Incrementing c [0] of Class * [3,3]: (S0, L0) S = "" <S0 L0 4 0 0 0 0>     4 0 0 3: abc  -> Incrementing c [0] of Class * [4,4]: (S4, L3) S = "abc"  -> Incrementing c [1] of Class * [0,4]: (S0, L0) S = "" <S0 L0 4 1 0 0 0> <S4 L3 1 0 0 0 0>     5 0 4 6: abcabc  -> Incrementing c [0] of Class * [5,5]: (S5, L6) S = "abcabc"  -> Incrementing c [1] of Class * [4,5]: (S4, L3) S = "abc"  -> Incrementing c [2] of Class * [0,5]: (S0, L0) S = "" <S0 L0 4 1 1 0 0> <S4 L3 1 1 0 0 0> <S5 L6 1 0 0 0 0>     6 0 5 3: abcabcabc  -> Incrementing c [0] of Class * [6,6]: (S5, L6) S = "abcabc"  -> Incrementing c [1] of Class * [5,6]: (S5, L6) S = "abcabc"  -> Incrementing c [2] of Class * [4,6]: (S4, L3) S = "abc"  -> Incrementing c [3] of Class * [0,6]: (S0, L0) S = ""  ---> Class [5, 6] L = 6 c [0] = 2 c [1] = 1 c [2] = 0 c [3] = 0 c [4] = 0 S = "abcabc" <S0 L0 4 1 1 1 0> <S4 L3 3 2 1 0 0>     7 1 1 4: abcd  -> Incrementing c [0] of Class * [7,7]: (S7, L4) S = "abcd"  -> Incrementing c [1] of Class * [1,7]: (S0, L0) S = "" <S0 L0 4 2 1 1 0> <S4 L3 3 2 1 0 0> <S7 L4 1 0 0 0 0>     8 2 2 0: abcde  -> Incrementing c [0] of Class * [8,8]: (S7, L4) S = "abcd"  -> Incrementing c [1] of Class * [2,8]: (S0, L0) S = ""  ---> Class [7, 8] L = 4 c [0] = 2 c [1] = 0 c [2] = 0 c [3] = 0 c [4] = 0 S = "abcd"  ---> Class [4, 8] L = 3 c [0] = 5 c [1] = 2 c [2] = 1 c [3] = 0 c [4] = 0 S = "abc" <S0 L0 9 5 2 1 0>     9 0 6 2: bc  -> Incrementing c [0] of Class * [9,9]: (S9, L2) S = "bc"  -> Incrementing c [1] of Class * [6,9]: (S0, L0) S = ""  -> Incrementing c [2] of Class * [5,9]: (S0, L0) S = ""  -> Incrementing c [3] of Class * [4,9]: (S0, L0) S = ""  -> Incrementing c [4] of Class * [0,9]: (S0, L0) S = "" <S0 L0 9 6 3 2 1> <S9 L2 1 0 0 0 0>    10 0 9 5: bcabc  -> Incrementing c [0] of Class * [10,10]: (S10, L5) S = "bcabc"  -> Incrementing c [1] of Class * [9,10]: (S9, L2) S = "bc"  -> Incrementing c [2] of Class * [6,10]: (S0, L0) S = ""  -> Incrementing c [3] of Class * [5,10]: (S0, L0) S = ""  -> Incrementing c [4] of Class * [4,10]: (S0, L0) S = "" <S0 L0 9 6 4 3 2> <S9 L2 1 1 0 0 0> <S10 L5 1 0 0 0 0>    11 0 10 2: bcabcabc  -> Incrementing c [0] of Class * [11,11]: (S10, L5) S = "bcabc"  -> Incrementing c [1] of Class * [10,11]: (S10, L5) S = "bcabc"  -> Incrementing c [2] of Class * [9,11]: (S9, L2) S = "bc"  -> Incrementing c [3] of Class * [6,11]: (S0, L0) S = ""  -> Incrementing c [4] of Class * [5,11]: (S0, L0) S = ""  ---> Class [10, 11] L = 5 c [0] = 2 c [1] = 1 c [2] = 0 c [3] = 0 c [4] = 0 S = "bcabc " <S0 L0 9 6 4 4 3> <S9 L2 3 2 1 0 0>    12 1 7 3: bcd  -> Incrementing c [0] of Class * [12,12]: (S12, L3) S = "bcd"  -> Incrementing c [1] of Class * [7,12]: (S0, L0) S = ""  -> Incrementing c [2] of Class * [1,12]: (S0, L0) S = "" <S0 L0 9 7 5 4 3> <S9 L2 3 2 1 0 0> <S12 L3 1 0 0 0 0>    13 2 8 4: bcde  -> Incrementing c [0] of Class * [13,13]: (S13, L4) S = "bcde"  -> Incrementing c [1] of Class * [8,13]: (S0, L0) S = ""  -> Incrementing c [2] of Class * [2,13]: (S0, L0) S = "" <S0 L0 9 8 6 4 3> <S9 L2 3 2 1 0 0> <S12 L3 1 0 0 0 0> <S13 L4 1 0 0 0 0>    14 3 3 0: bcde  -> Incrementing c [0] of Class * [14,14]: (S13, L4) S = "bcde"  -> Incrementing c [1] of Class * [3,14]: (S0, L0) S = ""  ---> Class [13, 14] L = 4 c [0] = 2 c [1] = 0 c [2] = 0 c [3] = 0 c [4] = 0 S = "bcde"  ---> Class [12, 14] L = 3 c [0] = 3 c [1] = 0 c [2] = 0 c [3] = 0 c [4] = 0 S = "bcd"  ---> Class [9, 14] L = 2 c [0] = 6 c [1] = 2 c [2] = 1 c [3] = 0 c [4] = 0 S = "bc" <S0 L0 15 11 7 4 3>    15 0 11 1: c  -> Incrementing c [0] of Class * [15,15]: (S15, L1) S = "c"  -> Incrementing c [1] of Class * [11,15]: (S0, L0) S = ""  -> Incrementing c [2] of Class * [10,15]: (S0, L0) S = ""  -> Incrementing c [3] of Class * [9,15]: (S0, L0) S = ""  -> Incrementing c [4] of Class * [6,15]: (S0, L0) S = "" <S0 L0 15 12 8 5 4> <S15 L1 1 0 0 0 0>    16 0 15 4: cabc  -> Incrementing c [0] of Class * [16,16]: (S16, L4) S = "cabc"  -> Incrementing c [1] of Class * [15,16]: (S15, L1) S = "c"  -> Incrementing c [2] of Class * [11,16]: (S0, L0) S = ""  -> Incrementing c [3] of Class * [10,16]: (S0, L0) S = ""  -> Incrementing c [4] of Class * [9,16]: (S0, L0) S = "" <S0 L0 15 12 9 6 5> <S15 L1 1 1 0 0 0> <S16 L4 1 0 0 0 0>    17 0 16 1: cabcabc  -> Incrementing c [0] of Class * [17,17]: (S16, L4) S = "cabc"  -> Incrementing c [1] of Class * [16,17]: (S16, L4) S = "cabc"  -> Incrementing c [2] of Class * [15,17]: (S15, L1) S = "c"  -> Incrementing c [3] of Class * [11,17]: (S0, L0) S = ""  -> Incrementing c [4] of Class * [10,17]: (S0, L0) S = ""  ---> Class [16, 17] L = 4 c [0] = 2 c [1] = 1 c [2] = 0 c [3] = 0 c [4] = 0 S = "cabc" <S0 L0 15 12 9 7 6> <S15 L1 3 2 1 0 0>    18 1 12 2: cd  -> Incrementing c [0] of Class * [18,18]: (S18, L2) S = "cd"  -> Incrementing c [1] of Class * [12,18]: (S0, L0) S = ""  -> Incrementing c [2] of Class * [7,18]: (S0, L0) S = ""  -> Incrementing c [3] of Class * [1,18]: (S0, L0) S = "" <S0 L0 15 13 10 8 6> <S15 L1 3 2 1 0 0> <S18 L2 1 0 0 0 0>    19 2 13 3: cde  -> Incrementing c [0] of Class * [19,19]: (S19, L3) S = "cde"  -> Incrementing c [1] of Class * [13,19]: (S0, L0) S = ""  -> Incrementing c [2] of Class * [8,19]: (S0, L0) S = ""  -> Incrementing c [3] of Class * [2,19]: (S0, L0) S = "" <S0 L0 15 14 11 9 6> <S15 L1 3 2 1 0 0> <S18 L2 1 0 0 0 0> <S19 L3 1 0 0 0 0>    20 3 14 0: cde  -> Incrementing c [0] of Class * [20,20]: (S19, L3) S = "cde"  -> Incrementing c [1] of Class * [14,20]: (S0, L0) S = ""  -> Incrementing c [2] of Class * [3,20]: (S0, L0) S = ""  ---> Class [19, 20] L = 3 c [0] = 2 c [1] = 0 c [2] = 0 c [3] = 0 c [4] = 0 S = "cde"  ---> Class [18, 20] L = 2 c [0] = 3 c [1] = 0 c [2] = 0 c [3] = 0 c [4] = 0 S = "cd"  ---> Class [15, 20] L = 1 c [0] = 6 c [1] = 2 c [2] = 1 c [3] = 0 c [4] = 0 S = "c" <S0 L0 21 17 13 9 6>    21 1 18 1: d  -> Incrementing c [0] of Class * [21,21]: (S21, L1) S = "d"  -> Incrementing c [1] of Class * [18,21]: (S0, L0) S = ""  -> Incrementing c [2] of Class * [12,21]: (S0, L0) S = ""  -> Incrementing c [3] of Class * [7,21]: (S0, L0) S = ""  -> Incrementing c [4] of Class * [1,21]: (S0, L0) S = "" <S0 L0 21 18 14 10 7> <S21 L1 1 0 0 0 0>    22 2 19 2: de  -> Incrementing c [0] of Class * [22,22]: (S22, L2) S = "de"  -> Incrementing c [1] of Class * [19,22]: (S0, L0) S = ""  -> Incrementing c [2] of Class * [13,22]: (S0, L0) S = ""  -> Incrementing c [3] of Class * [8,22]: (S0, L0) S = ""  -> Incrementing c [4] of Class * [2,22]: (S0, L0) S = "" <S0 L0 21 19 15 11 8> <S21 L1 1 0 0 0 0> <S22 L2 1 0 0 0 0>    23 3 20 0: de  -> Incrementing c [0] of Class * [23,23]: (S22, L2) S = "de"  -> Incrementing c [1] of Class * [20,23]: (S0, L0) S = ""  -> Incrementing c [2] of Class * [14,23]: (S0, L0) S = ""  -> Incrementing c [3] of Class * [3,23]: (S0, L0) S = ""  ---> Class [22, 23] L = 2 c [0] = 2 c [1] = 0 c [2] = 0 c [3] = 0 c [4] = 0 S = "de"  ---> Class [21, 23] L = 1 c [0] = 3 c [1] = 0 c [2] = 0 c [3] = 0 c [4] = 0 S = "d" <S0 L0 24 20 16 12 8>    24 2 22 1: e  -> Incrementing c [0] of Class * [24,24]: (S24, L1) S = "e"  -> Incrementing c [1] of Class * [22,24]: (S0, L0) S = ""  -> Incrementing c [2] of Class * [19,24]: (S0, L0) S = ""  -> Incrementing c [3] of Class * [13,24]: (S0, L0) S = ""  -> Incrementing c [4] of Class * [8,24]: (S0, L0) S = "" <S0 L0 24 21 17 13 9> <S24 L1 1 0 0 0 0>    25 3 23 0: e  -> Incrementing c [0] of Class * [25,25]: (S24, L1) S = "e"  -> Incrementing c [1] of Class * [23,25]: (S0, L0) S = ""  -> Incrementing c [2] of Class * [20,25]: (S0, L0) S = ""  -> Incrementing c [3] of Class * [14,25]: (S0, L0) S = ""  -> Incrementing c [4] of Class * [3,25]: (S0, L0) S = ""  ---> Class [24, 25] L = 1 c [0] = 2 c [1] = 0 c [2] = 0 c [3] = 0 c [4] = 0 S = "e" <S0 L0 26 22 18 14 10>

15.3 Third Stage From this execution result, a class having a value of tf larger than 2 can be obtained. Therefore, the class is sorted with the first key as the first key and the second key as the length. At the same time, the document frequency df _{k is} calculated from the difference in the character string frequency cf _k with the degree of duplication.
Ask for. The results are shown in Table 2. In this example, there are 14 classes in which the value of tf is greater than 2. For each class, the corresponding interval, length, tf, df _k statistic for each class, and the longest character string in the class are shown in order.

This class list does not include a class whose interval is 1. In addition, although the shortest length is not included in the class, this means that when searching for a character string that matches the representative character string of the class from the beginning, the shortest longest length will be included, as described later. This is dealt with by extracting the class information that has a length.

In class sorting, the beginning of the section is set to the first key so that the sections are arranged in approximately alphabetical order.
When the heads of the sections are the same, the shorter length is prioritized, and as a result, the character strings represented by the classes are arranged in the dictionary order.

[Table 2] total = 14 Class [4, 8] L = 3 tf = 5 df1 = 3 df2 = 1 df3 = 1 df4 = 0 S = "abc" Class [5, 6] L = 6 tf = 2 df1 = 1 df2 = 1 df3 = 0 df4 = 0 S = "abcabc" Class [7, 8] L = 4 tf = 2 df1 = 2 df2 = 0 df3 = 0 df4 = 0 S = "abcd" Class [9, 14] L = 2 tf = 6 df1 = 4 df2 = 1 df3 = 1 df4 = 0 S = "bc" Class [10, 11] L = 5 tf = 2 df1 = 1 df2 = 1 df3 = 0 df4 = 0 S = "bcabc" Class [12, 14] L = 3 tf = 3 df1 = 3 df2 = 0 df3 = 0 df4 = 0 S = "bcd" Class [13, 14] L = 4 tf = 2 df1 = 2 df2 = 0 df3 = 0 df4 = 0 S = "bcde" Class [15, 20] L = 1 tf = 6 df1 = 4 df2 = 1 df3 = 1 df4 = 0 S = "c" Class [16, 17] L = 4 tf = 2 df1 = 1 df2 = 1 df3 = 0 df4 = 0 S = "cabc" Class [18, 20] L = 2 tf = 3 df1 = 3 df2 = 0 df3 = 0 df4 = 0 S = "cd" Class [19, 20] L = 3 tf = 2 df1 = 2 df2 = 0 df3 = 0 df4 = 0 S = "cde" Class [21, 23] L = 1 tf = 3 df1 = 3 df2 = 0 df3 = 0 df4 = 0 S = "d" Class [22, 23] L = 2 tf = 2 df1 = 2 df2 = 0 df3 = 0 df4 = 0 S = "de" Class [24, 25] L = 1 tf = 2 df1 = 2 df2 = 0 df3 = 0 df4 = 0 S = "e"

15.4 Processing for Character String Obtaining the values of tf, df ₁ , df ₂ , df ₃ and df ₄ by searching the above-mentioned Table 2 for two for a given arbitrary character string. You can Since this is a binary search and the size of the table is O (N), this processing is O (log (N)).
It ends with the calculation amount of. The output result when a character string is given is shown below. abc --corresponds to Class [4,8] (representative character string) 5 3 1 1 0 abc abcabc --corresponds to Class [5,6] (representative character string) 2 1 1 0 0 abcabc abcd --class [7 , 8] (representative character string) 2 2 0 0 0 abcd abca --Class [5,6] (not representative character string) 2 1 1 0 0 abca abcab --Class [5,6] (Not a representative character string) 2 1 1 0 0 abcab abcabc --Corresponds to Class [5,6] (Representative character string) 2 1 1 0 0 abcabc abcabca --Exists in corpus, not in table 1 1 0 0 0 abcabca abcabcab-not in table, in corpus 1 1 0 0 0 abcabcab abcabcabc-in table, not in corpus 1 1 0 0 0 abcabcabc abcabcabca-not in table, not in corpus 0 0 0 0 0 abcabcabca

16. D for all substrings without classifying calculation amount and memory usage
a method of determining the f _k, even if the classification, without using the hierarchical structure of classes, compared to the method for obtaining the df _k examined independently of each class, the execution time according to the algorithm large Even for a large corpus, the calculation time can be practically processed. In the present algorithm, the entire process is N for the number of characters in the text and kMA is the maximum k to be obtained.
When X is set, after O (NlogN + kMAX × N) preprocessing, the number of documents that appear k times or more for any character string can be obtained by O (logN). Further, the memory usage amount is the capacity of O (N) both in the preprocessing and in the subsequent calculation of the value for an arbitrary character string. As a result of conducting an experiment using a program, which will be described later, which implements this counting algorithm and measuring the execution time, it has been confirmed that the counting can be practically performed.

17. Application to keyword extraction df ₂ / df is used as an estimated value of the probability that a character string appears twice in the document, provided that the character string appears in the document in the probability space of the document. be able to. Reference [6]
Shows that in English, the probability can statistically identify the nature of the word.

An arbitrary character string is converted to IDF (Inverse Document F
Requency) is taken as the potential, and the idea of finding it as the best division point of that potential is given in Ref. [4].
Is shown in. Furthermore, if df _{2 is} also used, the keyword extraction accuracy can be improved. To perform these keyword extraction operations, d for any partial character string
It is necessary to calculate f and df _2, and the operation speed of the system can be improved by using this algorithm.

The measurement of df ₂ has some applications as described above, and among them, the application to the extraction of keywords without using a dictionary is shown.

It is assumed that a corpus including the following document files is given. gakkai-0000000001 CAI for electric circuit exercises and its improvement Describes CAI software and its improvements that support basic electric circuit exercises at universities. The main features of this CAI are that it automatically creates a circuit for a computer according to the level of each learner and inputs the answer by a mathematical expression. In addition, for incorrect answers, a message is displayed to facilitate consideration of the cause, and effective individual learning can be implemented with limited equipment and personnel. Based on the questionnaire of the use results of this CAI by the students of last year and the question and answer at the time of the presentation, we made improvements to make the operation easier and to improve the effect. Electric circuit exercises

Regarding the partial character string of the corpus, tf,
Table 3 shows a part of the results of obtaining N, df, df ₂ , df ₃ , df ₄ , and df ₅ .

[Table 3]   27511 40000 40000 40000 40000 40000 40000 1 20 21 times   14384 40000 5224 3132 2094 1431 938 1 20 22 Circuit       6 40000 3 2 1 0 0 1 20 23 Circuit performance       6 40000 3 2 1 0 0 1 20 24 Circuit exercise   23604 40000 40000 40000 40000 40000 40000 1 21 22 Road       7 40000 4 2 1 0 0 1 21 23 Road performance       7 40000 4 2 1 0 0 1 21 24 Road exercises    3325 40000 1555 702 408 245 162 1 22 23 Performance     192 40000 75 41 25 18 12 1 22 24 Exercise       7 40000 6 1 0 0 0 1 22 25 Exercise       4 40000 4 0 0 0 0 1 22 26 Exercise C       4 40000 4 0 0 0 0 1 22 27 Exercise C A       4 40000 4 0 0 0 0 1 22 28 Exercise C AI    4245 40000 1308 860 640 462 333 1 23 24      29 40000 23 5 1 0 0 1 23 25 Training       5 40000 5 0 0 0 0 1 23 26 Training C       5 40000 5 0 0 0 0 1 23 27 Training CA       5 40000 5 0 0 0 0 1 23 28 Training CA I   For 72462 40000 40000 40000 40000 40000 40000 1 24 25     139 40000 121 16 2 0 0 1 24 26 for C      CA for 51 40000 40 10 1 0 0 1 24 27      27 40000 21 6 0 0 0 1 24 28 CAI   38995 40000 40000 40000 40000 40000 40000 1 25 26 C    3749 40000 1852 837 499 275 142 1 25 27 CA     790 40000 288 186 148 86 43 1 25 28 CAI       9 40000 9 0 0 0 0 1 25 29 With CAI   44832 40000 40000 40000 40000 40000 40000 1 26 27 A    1899 40000 877 445 276 157 76 1 26 28 AI      10 40000 10 0 0 0 0 1 26 29 With AI   44675 40000 40000 40000 40000 40000 40000 1 27 28 I     180 40000 158 16 6 0 0 1 27 29 I and       6 40000 6 0 0 0 0 1 27 30 I and its       5 40000 5 0 0 0 0 1 27 31 I and its  158324 40000 40000 40000 40000 40000 40000 1 28 29 and    2093 40000 1939 141 13 0 0 1 28 30    1789 40000 1669 112 8 0 0 1 28 31 and its      14 40000 13 1 0 0 0 1 28 32 and its modifications       6 40000 5 1 0 0 0 1 28 33 and its modifications Good

In Table 3, for example, the character string "CA
For “I”, df is 288, df ₂ is 186, df
₃ is 148, but regarding the character string "CAI and",
While df is 9, df ₂ and df ₃ are 0. As described above, it can be seen that the character string including word breaks sharply decreases after df ₂ as compared with df. The character string of the document file is divided as follows by detecting the break of a word by utilizing the property that df _{2 and} thereafter are sharply reduced.

Gakkai-000000 / 0001 / electric circuit / exercise / for / C
AI / and / that / improvement // university / etc / de / basic / target / electrical circuit / exercise / support / support / CAI / software / and / the /
Describes / describes improvements. / Book / CAI / Is / Computer / is / issue / title / do / circuit / is / learner / each / person / level /
According to / automatic / target / to / create / do / that / answer / to / mathematical /
It is / input / enter / that / large / large / special / long /. /Also
For /, / wrong / wrong / answer /
/ Na / individual / learning / ga / limit / la / re / ta / equipment / ・ / personnel / in / implement /
Possible / Yes / Yes / No / Distribution / Consideration. / Last year / of / student / by / book / CAI / of / use / result / result / of / questionnaire /, / and
/ Departure / Table / Of / On site / Quality / Suspicion / Etc. / Reference / Consider / In /, / Operation /
/ Easy / and / Effect / Increase / To improve / To improve. / Electrical Circuit // Practice /

In this way, the following keywords are extracted by selecting from the document files that are character-divided, those having a distribution of character strings that are likely to be keywords. Exercise CAI Exercise Support CAI Software C
AI learner automatic message display learning student
CAI exercise

As a result, the keyword can be extracted from the Japanese sentence without any word dictionary. This method is language-independent, and can be applied to undocumented documents such as encrypted documents and documents whose dictionary is not maintained, such as old documents. It can also be applied to the analysis of DNA sequences by applying it to genetic information.

Some other applications will be described. The measurement of df ₂ can be applied to information retrieval. In the bigram information retrieval system, the method of selecting words by the stoplist, which is used in information retrieval by words, is not clear. Therefore, a bigram isolate with _{df 2 / df, df 2 /} df is conceivable applications that select only high bigram. According to document [8], d
Since there is a tendency to select keywords in f ₂ / df, it can be seen that the information retrieval performance is improved by using the keywords.

Application to software tools is also possible. As an application of Suffix Array classification, reference [7] analyzes a class system in a program,
There is a report of a system that measures a peculiar place that does not appear in other places and automatically detects the peculiar place in the program. In this system, only tf is used as the statistic, but by using df and df ₂ , further improvement in performance can be expected as a program measurement tool.

As described above, in the document frequency counting technique shown in this algorithm, the existing Suffix A
Suffix Array is classified into classes using the data structure of rray, and df _k is counted using the classes. Number of documents that are classified into classes and in which a character string appears one or more times d
The method of counting f is known in the literature [3], but in this algorithm, the character string is k times (k is a natural number of 2 or more).
In order to enable efficient measurement of the number of documents df _{k that} appear above, a character string frequency with a degree of duplication is newly defined, and when determining the number of documents in which the character string appears k times or more, Calculation was performed using the difference in row frequency. Moreover, the amount of calculation is suppressed by simultaneously performing the processing of class detection and the determination of the degree of overlap. Furthermore, by using the structure of the nested class hierarchy, the character string frequency with multiplicity obtained by the lower class is added to the character string frequency with multiplicity obtained by the upper class, thereby generating the appearance of the character string. The cumulative frequency can be measured and the calculation amount is suppressed. These techniques have made it possible to count the document frequency in a practical calculation time with respect to the size of the corpus.

The present invention has been described above based on the embodiments. It is understood by those skilled in the art that these embodiments are mere examples, and that various modifications can be made to the combinations of the respective constituent elements and the respective processing processes, and such modifications are also within the scope of the present invention. By the way. Hereinafter, such a modified example will be described.

In the above embodiment, Japanese was used as the execution example of the counting algorithm, but the algorithm is language independent. In the embodiment, the appearance frequency of the character string is measured by taking the document as an example, but the counting algorithm is not limited to the document as the measurement target, and for example, in order to measure the appearance frequency of the gene sequence in the gene information. It is also possible to use. Particularly, in the present invention, a meaningful word can be cut out from the information on the frequency of appearance without using a dictionary. Therefore, when it is used for decoding genetic information, a gene sequence having a specific function can be extracted.

The following is a list of references referred to herein. [1] Manber, Udi and Gene Myer. 1990., Suffix arrays: A new method for on-line string sea
rches, In the first Annual ACM-SIAM Symposium on Di
screte Algorithms, pages 319-327. [2] Hideo Ito, Efficient construction of Suffix Array, Transactions of Information Processing Society of Japan, Vol.41, No.SIG1 (TOD5), pp.31-3
9 (2000). [3] Mikio Yamamoto and Kenneth W. Church, Using Suf
fix Arrays to Compute Term Frequency and Document
Frequency forAll Substrings in a Corpus, Computatio
nal Linguistics, Vol.27: 1, pp.1-30, MIT Press. [4] Tomohiro Ozawa, Mikio Yamamoto, Kyoji Umemura
and Kennth W. Church, Japanese word segmentation us
ing similarity measure for IR, In proceedings of th
e first NTCIR workshop on Research in Japanese tex
tretrieval and term recognition.Tokyo Japan, Augu
st 1999, pp.89-96. [5] Tomohiro Ozawa, Mikio Yamamoto, Eiko Yamamoto, Kyoji Umemura, Word segmentation of search request sentences using information retrieval similarity measure, Proceedings of the Annual Meeting of the Language Processing Society of Japan , March, 1999, p
p.305-308. [6] Kenneth W. Church, Empirical Estimates of Adaptation: The chance of T
wo Noriega's is closeto p / 2 than p 2, Coling2000,
pp.173-179. [7] Hiroyuki Yoshikawa, Toshiro Kijima, Kyoji Umemura, Detection of Defects in Programs Applying n-gram Analysis Method, IPSJ Journal, Vol.9, No.12, pp.3294 -3303. [8] Kyoji Umemura and Kenneth W. Church, Empirical
Term Weighting and Expansion Frequency, Empirical M
ethods in Natural Language Processing and Very Lar
ge Corpora, pp.117-123. [9] Takeyuki Yoshida, Kyouji Umemura, Document Frequency Analysis for Keyword Extraction Metric Language, Vol.23, No.2, pp.65-90

Table 4 shows a list of programs implementing the counting algorithm according to the above-mentioned embodiment.

[Table 4]    1 / * maximum k of df_k to obtain +1,    2 * /    3 #define MAX_C 5    Four    5 / * Document delimiter * /    6 #define SEPARATOR '\ n'    7    8 #include <stdio.h>    9 #include <string.h>   10 #include <stdlib.h>   11 / * Function type to be given to qsort argument * /   12 typedef int (* sortfn) (const void *, const void *);   13   14 #define MESSAGE_FILE stdout   15   16 static char * text; / * Source data of text * /   17   18 static int size; / * number of bytes of text * /   19   20 static int id; / * number of the document being loaded * /   twenty one   22 static int id_max; / * total number of documents * /   twenty three   24 static int common_max; / * Maximum number of characters common to adjacent suffixes Large price * /   twenty five   26   27 struct suffix_struct   28 {int position; / * start of text index * /   29 int common; / * length of common part with the following suffix * /   30 int id; / * Corresponding document number * /   31 int previous_suffix; / * Most of the same documents, suffix Close things * /   32};   33   34 static struct suffix_struct * suffix; / * suffix array * /   35   36 static int * last_suffixes; / * array to create previous suffix  * /   37   38 static FILE * text_file; / * data file * /   39   40 struct pending_struct   41 {int start_suffix;   42 int length;   43 int c [MAX_C];   44};   45   46   47 static struct pending_struct * pendings;   48 static int level;   49   50 static int pending_level (int suffix)   51 {   52 int min, max, mid;   53 min = 0; max = level;   54 while (min + 1 <max) {   55 mid = (min + max) / 2;   56 if (pendings [mid] .start_suffix <= suffix) {   57 min = mid;   58} else {   59 max = mid;   60}   61}   62 if (pendings [max] .start_suffix <= suffix) return max;   63 if (pendings [min] .start_suffix <= suffix) return min;   64 fprintf (stderr, "internal error (pending_level) \ n");   65 exit (2);   66 return -1;   67}   68   69   70 #ifdef DEBUG   71 / * Show status of specified suffix * /   72 static void debug_suffix (int i)   73 {   74 int j;   75 fprintf (MESSAGE_FILE,   76 "% 5d% 3d% 5d% 2d:",   77 i,   78 suffix [i] .id,   79 suffix [i] .previous_suffix,   80 suffix [i] .common);   81 for (j = 0; text [suffix [i] .position + j]! = '\ N'; j ++) {   82 fputc (text [suffix [i] .position + j], MESSAGE_FILE);   83}   84 fputc ('\ n', MESSAGE_FILE);   85 fflush (MESSAGE_FILE);   86}   87   88 / * Display suffix array status * /   89 static void debug_output ()   90 {int i;   91 fprintf (MESSAGE_FILE, "size =% d \ n", size);   92 fprintf (MESSAGE_FILE, "id_max =% d \ n", id_max);   93 fprintf (MESSAGE_FILE, "common_max =% d \ n", common_max);   94 for (i = 0; i <size; i ++) {   95 debug_suffix (i);   96}   97}   98   99 / * Display the status of pending class * /  100 static void debug_pending ()  101 {  102 int i; int j;  103 for (i = 0; i <= level; i ++) {  104 fprintf (MESSAGE_FILE, "<S% d L% d",  105 pendings [i] .start_suffix,  106 pendings [i] .length);  107 for (j = 0; j <MAX_C; j ++) {  108 fprintf (MESSAGE_FILE, "% d", pendings [i] .c [j]);  109}  110 fprintf (MESSAGE_FILE, ">");  111}  112 fprintf (MESSAGE_FILE, "\ n");  113}  114 #endif  115  116 static void error_alloc (void)  117 {  118 fprintf (MESSAGE_FILE,  119 "text% x, suffix% x, last_suffixes% x, classes% x \ n",  120 (int) text,  121 (int) suffix,  122 (int) last_suffixes,  123 (int) pendings);  124 exit (1);  125}  126  127 / * suffix order decision function,  128 breaks at document break  This is a lexical comparison of 129 strings.  130  131 * /  132 static int my_strcmp (char * x1, char * y1)  133 {  134 register unsigned char * x;  135 register unsigned char * y;  136 x = (unsigned char *) x1; y = (unsigned char *) y1;  137 while (* x && * y) {  138 if (* x <* y) return -1;  139 if (* x> * y) return 1;  140 if (* x == '\ n') break;  141 if (* y == '\ n') break;  142 x ++; y ++;  143}  144 return 0;  145}  146  147 static int string_sub (char * s1, char * s2)  148 {  149 while (* s2) {  150 if (* s1! = * S2) {return 0;}  151 s1 ++;  152 s2 ++;  153}  154 return * s1;  155}  156  157 / * A function that determines the order of the strings indicated by suffix  158 If some parts are the same from the beginning, the next character determines the order  The property 159 is not limited to this.  160 * /  161 static int suffix_order (struct suffix_struct * x, struct suffix_st ruct * y)  162 {  163 return my_strcmp (text + x-> position, text + y-> position);  164}  165  166 / * Calculate the number of common characters from the beginning. Function * /  167  168 static int common_length (char * x, char * y)  169 {int i;  170 i = 0;  171 while ((* x == * y) && (* x) && (* x! = '\ N')) {i ++; x ++; y ++;};  172 return i;  173}  174  175 struct class_struct  176 {struct class_struct * next;  177 int first;  178 int last;  179 int length;  180 int c [MAX_C];  181};  182  183 static struct class_struct * class_list = 0;  184 static int class_count = 0;  185  186 static struct class_struct * class_table;  187  188 static void register_class (int first, int last, int length, int c [ ])  189 {int i;  190 struct class_struct * p;  191 #ifdef DEBUG  192 printf ("---> Class [% d,% d] L =% d", first, last, length);  193 for (i = 0; i <MAX_C; i ++) {  194 printf ("c [% d] =% d", i, c [i]);  195}  196 printf ("S = \" ");  197 for (i = 0; i <length; i ++) {putchar (text [suffix [first] .position + i] );}  198 printf ("\" \ n ");  199 #endif  200 p = (struct class_struct *) malloc (sizeof (struct class_struct));  201 if (p == 0) {  202 fprintf (stderr, "register_class \ n");  203 exit (2);  204}  205 p-> first = first;  206 p-> last = last;  207 p-> length = length;  208 for (i = 0; i <MAX_C; i ++) {  209 p-> c [i] = c [i];  210}  211 p-> next = class_list;  212 class_list = p;  213 class_count ++;  214}  215  216 int class_order (struct class_struct * x, struct class_struct * y)  217 {  218 if (x-> first <y-> first) return -1;  219 if (x-> first> y-> first) return 1;  220 if (x-> length <y-> length) return -1;  221 if (x-> length> y-> length) return 1;  222 return 0;  223}  224  225 / * Show location, length, characters, counts for class * /  226 #ifdef DEBUG  227 static void output_class (int first, int last, int length, int c [] )  228 {  229 int i; int j;  230 printf ("Class [% 4d,% 4d] L =% d tf =% d",  231 first,  232 last,  233 length,  234 c [0]);  235 for (j = 1; j <MAX_C; j ++) {  236 printf ("df% d =% d", j, c [j-1] -c [j]);  237}  238 printf ("S = \" ");  239 for (i = 0; i <length; i ++) {putchar (text [suffix [first] .position + i] );}  240 printf ("\" \ n ");  241}  242 #endif  243  244  245 static void make_class_table (void)  246 {struct class_struct * p;  247 int i;  248 class_table = (struct class_struct *)  249 malloc (sizeof (struct class_struct) * class_count) ;  250 if (class_table == 0) {  251 fprintf (stderr, "make_class_table \ n");  252 exit (1);  253}  254 p = class_list;  255 for (i = 0; i <class_count; i ++) {  256 class_table [i] = * p;  257 p = p-> next;  258};  259 qsort (class_table, class_count, sizeof (struct class_struct), (so rtfn) class_order);  260}  261  262 static void clear_class_list (void)  263 {int i;  264 struct class_struct * p;  265 struct class_struct * q;  266 p = class_list;  267 for (i = 0; i <class_count; i ++) {  268 q = p;  269 p = q-> next;  270 q-> next = 0;  271 free (q);  272}  273 class_list = 0;  274}  275  276  277 void df_setup (char * file)  278 {char * p; int i; int j; int ch; int previous;  279 #ifdef DEBUG  280 int ii;  281 #endif  282 text_file = fopen (file, "r");  283 if (text_file == 0) {  284 fprintf (stderr, "File% s not found \ n", file);  285 exit (1);  286}  287  288 / * Calculate the total number of data characters * /  289 size = 0;  290 while (EOF! = (Ch = fgetc (text_file))) {  291 size ++;  292};  293  294 / * Generate the required data area from the total number of characters * /  295 text = (char *) malloc (sizeof (char) * (size + 1));  296 if (text == 0) error_alloc ();  297 suffix = (struct suffix_struct *)  298 malloc (sizeof (struct suffix_struct) * (size + 1));  299 if (suffix == 0) error_alloc ();  300 fseek (text_file, 0, SEEK_SET);  301 p = text;  302 id = 0;  303  304 / * Delimit document at the same time as reading into memory  305 Put on * /  306 id_max = 0;  307 for (i = 0; i <size; i ++) {  308 ch = fgetc (text_file);  309 text [i] = ch;  310 suffix [i] .position = i;  311 suffix [i] .id = id;  312 id_max = id + 1;  313 if (ch == '\ n') {id ++;}  314};  315 text [size] = 0;  316  317  318 / * Routine to create Suffix * /  319 / * The following is an example. * /  320 common_max = 0;  321 for (i = 0; i <size; i ++) {suffix [i] .position = i;}  322 / * sort processing is expected to take up most of the computation time * /  323 qsort (suffix, size, sizeof (struct suffix_struct), (sortfn) suffi x_order);  324 suffix [size] .position = size;  325 for (i = 0; i <size; i ++) {  326 int c;  327 c = common_length (text + suffix [i] .position, text + suffix [i + 1] .po sition);  328 suffix [i] .common = c;  329 if (c> common_max) common_max = c;  330}  331 suffix [size] .common = 0;  332  333  334 / * Generate data structure for duplicate calculations * /  335 last_suffixes = (int *) malloc (id_max * sizeof (int));  336 if (last_suffixes == 0) error_alloc ();  337 for (i = 0; i <id_max; i ++) {last_suffixes [i] = -1;}  338 for (i = 0; i <size; i ++) {  339 suffix [i] .previous_suffix = last_suffixes [suffix [i] .id];  340 last_suffixes [suffix [i] .id] = i;  341}  342  343 #ifdef DEBUG  344 debug_output ();  345 #endif  346  347 / * Extract class structure * /  348  349 pendings = (struct pending_struct *)  350 malloc (sizeof (struct pending_struct) * (common_max + 1));  351 if (pendings == 0) error_alloc ();  352 level = 0;  353 pendings [level] .length = 0;  354 pendings [level] .start_suffix = 0;  355 for (j = 0; j <MAX_C; j ++) {  356 pendings [level] .c [j] = 0;  357}  358 for (i = 0; i <size; i ++) {  359 #ifdef DEBUG  360 debug_pending ();  361 debug_suffix (i);  362 #endif  363 / * pre-processing, check if new class starts * /  364 if (suffix [i] .common> pendings [level] .length) {  365 / * Generate new class from current location. * /  366 / * New class starts with count 0 * /  367 level ++;  368 pendings [level] .start_suffix = i;  369 pendings [level] .length = suffix [i] .common;  370 for (j = 0; j <MAX_C; j ++) {pendings [level] .c [j] = 0;}  371}  372  373 / * Counting, searching and counting the appropriate class for the occurrence of a string * /  374  375 previous = suffix [i] .previous_suffix;  376 pendings [level] .c [0] ++;  377 #ifdef DEBUG  378 printf ("-> Incrementing c [% d] of Class * [% d,% d]: (S% d, L% d) S = \ "",  379 0,  380 i,  381 i,  382 pendings [level] .start_suffix,  383 pendings [level] .length  384);  385 for (ii = 0; ii <pendings [level] .length; ii ++) {  386 putchar (text [suffix [pendings [level] .start_suffix] .position + ii]) ;  387}  388 printf ("\" \ n ");  389 #endif  390 for (j = 1; j <MAX_C; j ++) {  391 int plev;  392 if (previous <0) break;  393 plev = pending_level (previous);  394 pendings [plev] .c [j] ++;  395 #ifdef DEBUG  396 printf ("-> Incrementing c [% d] of Class * [% d,% d]: (S% d, L% d) S = \ "",  397 j,  398 previous,  399 i,  400 pendings [plev] .start_suffix,  401 pendings [plev] .length  402);  403 for (ii = 0; ii <pendings [plev] .length; ii ++) {  404 putchar (text [suffix [pendings [plev] .start_suffix] .position + ii]);    405}  406 printf ("\" \ n ");  407 #endif  408 previous = suffix [previous] .previous_suffix;  409}  410  411 / * Post-processing: end of class detection * /  412 while (suffix [i] .common <pendings [level] .length) {  413 int common = suffix [i] .common;  414 / * When the end of class is found * /  415 register_class (  416 pendings [level] .start_suffix, / * start suffix * /  417 i, / * final suffix * /  418 pendings [level] .length, / * maximum class length * /  419 pendings [level] .c);  420 if (level <= 0) {fprintf (stderr, "internal level \ n"); exit (2 );}  421 if (common> pendings [level-1] .length) {  422 / * There was a class that was not registered as being calculated.  423 The upper class starting from the same place as the class for which the calculation is completed  424 class processing started * /  425 pendings [level] .length = common;  426 / * Inherit count to higher class, but  427 The upper class is currently in the same location as the class being calculated, so  428 No actual operation is required. * /  429  430}  431 if (common <= pendings [level-1] .length) {  432 / * The starting place of the higher class is pending before now thing */  433 for (j = 0; j <MAX_C; j ++) {  434 pendings [level-1] .c [j] + = pendings [level] .c [j];  435}  436 level-;  437}  438 / * After termination processing, check whether it has terminated again. * /  439}  440}  441 #ifdef DEBUG  442 debug_pending ();  443 #endif  444 make_class_table ();  445 clear_class_list ();  446}  447  448  449 static int df_class_string_length (char * s)  450 {int i;  451 i = 0;  452 while (* s ++) i ++;  453 return i;  454}  455  456  457 static int df_class_compare_string (char * x, char * s)  458 {  459 if (string_sub (x, s)! = 0) return 0;  460 return (my_strcmp (x, s));  461}  462  463 static int df_class_compare (int m, char * s)  464 {  465 char * x; int r;  466 x = text + suffix [class_table [m] .first] .position;  467 r = df_class_compare_string (x, s);  468 if (r! = 0) return r;  469 if (class_table [m] .length> df_class_string_length (s)) return 1;  470 if (class_table [m] .length <df_class_string_length (s)) return -1 ;  471 return 0;  472}  473  474  475  476 static int df_class_binary (char * s)  477 {  478 int min; int max; int mid; int cmp;  479 / * Search for a class in cf> = 2 * /  480 min = 0;  481 max = class_count-1;  482 while (min + 1 <max) {  483 mid = (max + min) / 2;  484 cmp = df_class_compare (mid, s);  485 if (cmp <0) {min = mid;} else {max = mid;}  486};  487 if ((string_sub (text + suffix [class_table [min] .first] .position, s) ! = 0) &&  488 (string_sub (text + suffix [class_table [min] .last] .position, s)! = 0)  489) return min;  490 if ((string_sub (text + suffix [class_table [max] .first] .position, s) ! = 0) &&  491 (string_sub (text + suffix [class_table [max] .last] .position, s)! = 0)  492) return max;  493 / * Search whether cf = 1 * /  494 min = 0;  495 max = size-1;  496 while (min + 1 <max) {  497 mid = (max + min) / 2;  498 cmp = df_class_compare_string (text + suffix [mid] .position, s);  499 if (cmp <0) {min = mid;} else {max = mid;}  500}  501 if (string_sub (text + suffix [min] .position, s)! = 0) return -1;  502 if (string_sub (text + suffix [max] .position, s)! = 0) return -1;  503 return -2;  504}  505  506 static int df_class (char * s)  507 {  508 int c;  509 c = df_class_binary (s);  510 #ifdef DOCUMENTATION  511 if (c! = Df_class_simple (s)) {  512 fprintf (stderr, "% d% d% s \ n", c, df_class_simple (s), s);  513}  514 #endif  515 return c;  516}  517  518  519 int cf (char * s)  520 {  521 int c;  522 c = df_class (s);  523 if (c <-1) return 0;  524 if (c <0) return 1;  525 return class_table [c] .c [0];  526}  527  528  529 static int df1 (char * s)  530 {  531 int c;  532 c = df_class (s);  533 if (c <-1) return 0;  534 if (c <0) return 1;  535 return class_table [c] .c [0]-class_table [c] .c [1];  536}  537  538 int dfn (int k, char * s)  539 {int c;  540 if (k> = MAX_C) {  541 fprintf (stderr, "% d: dfn K too large \ n", k);  542}  543 if (k == 1) return df1 (s);  544 c = df_class (s);  545 if (c <0) return 0;  546 return class_table [c] .c [k-1]-class_table [c] .c [k];  547}  548  549 char line [1024];  550  551 int main (int argc, char ** argv)  552 {int i;  553 if (argc! = 2) {  554 fprintf (stderr, "Usage% s filename> output \ n", argv [0]);  555 exit (1);  556}  557 df_setup (argv [1]);  558 #ifdef DEBUG  559 fprintf (stdout, "total =% d \ n", class_count);  560 for (i = 0; i <class_count; i ++) {  561 output_class (class_table [i] .first,  562 class_table [i] .last,  563 class_table [i] .length,  564 class_table [i] .c  565);  566}  567 #endif  568 while (fgets (line, sizeof (line), stdin)) {  569 i = strlen (line);  570 line [i-1] = 0;  571 fprintf (stdout, "% d", cf (line));  572 for (i = 1; i <MAX_C; i ++) {  573 fprintf (stdout, "% d", dfn (i, line));  574}  575 fprintf (stdout, "% s \ n", line);  576}  577 return 0;  578}

[0137]

According to the present invention, the appearance frequency of a character string can be efficiently counted.

[Brief description of drawings]

FIG. 1 Suff used in a counting method according to an embodiment
It is a figure explaining the data structure of ix Array.

FIG. 2 is a diagram showing common values for each suffix of the Suffix Array.

FIG. 3 is a diagram exemplifying sections of Suffix Array forming a class.

FIG. 4 is a diagram showing a hierarchical structure of Suffix Array classes.

5 is a diagram illustrating a partial character string belonging to the class of Suffix Array in FIG.

6A is a diagram showing an example of a corpus including six documents, and FIG. 6B is a diagram showing a character string frequency and a document frequency regarding a character string included in the corpus of FIG. 6A. It is a figure which shows the example of calculation.

FIG. 7A is a diagram showing one of the documents included in the corpus, and FIG. 7B is a diagram showing the degree of overlap of each suffix of the Suffix Array.

8A is a diagram showing a corpus including three documents, and FIG. 8B is a character string frequency with duplication degree and a document frequency regarding a character string included in the corpus of FIG. 8A. It is a figure which shows the example of calculation of.

9 (a), 9 (b), and 9 (c) are explanatory diagrams of previous links.

FIG. 10 is a configuration diagram of a counting device according to an embodiment.

FIG. 11 is a flowchart of a counting process procedure according to the embodiment.

[Explanation of symbols]

10 counting processing unit, 12 Suffix Array generation unit, 1
4 character string frequency counting unit, 16 document frequency calculating unit, 1
8 inquiry section, 20 search section, 22 keyword extraction section, 24 document database, 26 document file, 28 Suffix Array file, 30 character string appearance frequency data, 32 keyword index file, 100 counting device.

Claims (14)

[Claims] 1. The number of times a character string appears k times (k is a natural number of 2 or more) in a set of documents and the character string is k.
A counting device, comprising: a count processing unit that acquires the number of documents in which the character string is included k times or more by counting the number of occurrences of +1 or more times and obtaining the difference between the numbers. 2. The counting processing unit sets a set of partial character strings included in the document so that the partial character strings belonging to the same class have the same number of times of occurrence of the partial character strings. The number of appearances is counted for each class while classifying into classes.
The counting device according to. 3. The counting processing unit counts the number of appearances by adding the number of appearances counted in the lower class to the number of appearances counted in the upper class by utilizing the hierarchical structure of the class. The counting device according to claim 2, wherein: 4. For a corpus containing a plurality of documents,
Range from the character at a certain position in the document to the end of the document
Is a set of character strings of
Generating a suffix array, Among the appearances of the given character string x, the degree of duplication is k (k is 2
Appearance frequency cf of a character string equal to or more than the above natural number cf_k(X)
The step of counting, The number of documents df in which the character string x appears k times or more_k(X)
Cf_k(X) and cf _{k + 1}Process determined by the difference of (x)
A method of counting the appearance frequency of a character string characterized by including
Law. 5. The method further includes the step of generating a classification of the suffix array, wherein the character strings belonging to the same class are classified so that the character strings appear the same number of times in the same document. The appearance frequency c for each class by classification
The counting method according to claim 4, wherein f _k (x) is counted. 6. The step of generating the classifications,
A classification having a hierarchical structure is generated, and the appearance frequency c
The counting method according to claim 5, wherein, in the step of counting f _k (x), the appearance frequency cf _k (x) counted in the lower class is added to the appearance frequency counted in the upper class. . 7. The method according to claim 5, wherein the determination of the degree of overlap in the step of counting the appearance frequency cf _k (x) and the detection of the class in the step of generating the classification are performed at the same time. The counting method described in. 8. A counting processing unit that counts the appearance frequency of a character string in a set of documents, and a recording unit that records data regarding the appearance frequency of the character string, wherein the counting processing unit is the document. A set of character strings in the range from a character at a certain position to the end of the document, the set being a suffix array generating a suffix array in which the set is arranged in a lexicographical order; Simultaneously with the detection, by determining the degree of overlap of the appearance of the character string, a character string frequency counting unit that counts the frequency of appearance of the character strings in duplicate for each class, and the appearance frequency of the character string. A counting device, which includes a document frequency calculating unit that calculates the frequency of the document in which the character string appears. 9. Each suffix of the suffix array belongs to the same document as the suffix and has a pointer to a suffix immediately before the suffix in the order of arrangement of the suffix array, and the character string frequency counting unit 9. The counting device according to claim 8, wherein the degree of overlap is determined by determining whether or not the pointer can be sequentially traced the number of times of the degree of overlap. 10. The character string frequency counting unit classifies the suffix array into a hierarchical class structure, and the appearance frequency of the character string counted in a lower class is counted in an upper class. The counting device according to claim 8 or 9, wherein 11. The document frequency calculation unit determines the frequency of a document in which the character string appears k (k is a natural number) or more times, the appearance frequency of the character string being k or more and the duplication degree. The counting device according to any one of claims 8 to 10, wherein the counting device calculates the difference from the appearance frequency of k + 1 or more. 12. The counting according to claim 8, further comprising a keyword extracting unit that extracts a keyword from the document using the frequency of the document in which the character string appears twice or more. apparatus. 13. A corpus including a plurality of documents,
A step of generating a suffix array in which a set of character strings ranging from a character at a certain position of the document to the end of the document, the set being arranged in lexicographic order; and a hierarchical class structure of the suffix array. The class is detected, and the appearance frequency cf _k (x) of the character string having the degree of duplication k (k is a natural number of 2 or more) or more among the appearances of the given character string x is counted for each class. A step of adding the appearance frequency cf _k (x) counted in the lower class to the appearance frequency counted in the upper class, and the number of documents df _k (x) in which the character string x appears k times or more
And a step of obtaining a difference between the appearance frequency cf _k (x) of which the degree of duplication of the character string is k or more and the appearance frequency cf _{k + 1} (x) of which the degree of duplication is k + 1 or more. A computer program characterized by the above. 14. A set of partial symbol sequences included in the sequence for a set including a plurality of symbol sequences, the number of times that the partial symbol sequences appear in the same sequence for partial symbol sequences belonging to the same class. The number of times that an arbitrary symbol string appears k (k is a natural number of 2 or more) times or more while being classified into hierarchical classes such that A method of counting the frequency of appearance of a symbol string, characterized in that the number of the sequences included is calculated.