Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F40/00—Handling natural language data
  • G06F40/20—Natural language analysis
  • G06F40/268—Morphological analysis
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F40/00—Handling natural language data
  • G06F40/20—Natural language analysis
  • G06F40/279—Recognition of textual entities
  • G06F40/284—Lexical analysis, e.g. tokenisation or collocates
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F40/00—Handling natural language data
  • G06F40/20—Natural language analysis
  • G06F40/279—Recognition of textual entities
  • G06F40/289—Phrasal analysis, e.g. finite state techniques or chunking

Definitions

• the invention relates to a document segmentation method to divide the document into plural blocks by detecting segmentation points where topics of an input document is discontinuous.
• the Hearst method is well known as one of document segmentation methods by detecting topic boundaries (the Hearst method can be seen in "Multi-paragraph segmentation of expository text" by M. A. Hearst in Proceedings of the 32nd Annual Meeting of Association for Computational Linguistics, pp. 9-16, 1994).
• This method first defines certain size windows before and after one of topic boundary candidate points and obtains similarity about what terms appear in each window. If the obtained similarity is high, topic relatedness between the windows is high and the candidate point cannot be a boundary point. If the obtained similarity is low, cohesion between the windows is regarded low and the candidate point can be a boundary point. More specifically, the topic boundary candidate point is moved by regular intervals from the top to the end of the document, similarity value is obtained for each point, and then the point of which the obtained similarity value is minimal can be selected as the topic boundary.
• the Hearst method compares the terms occurring in the windows defined before and after the candidate point in order to detect the topic discontinuity between the windows.
• This method has the following shortcomings.
• the first one is related to the window size. Since the window size is selected very arbitrarily, the distance between the topic boundaries, that is, the length of the topics tend to be longer when the selected window size is large, whereas the length of the topics tends to be shorter when the selected window size is small. Therefore, it is difficult to properly divide such documents containing many topics those of which length are different.
• the second shortcoming is related to the method for detecting topic similarity between the windows before and after the candidate point.
• a conventional method cannot obtain the similarity unless the same terms occur in both windows because it determines the similarity based on the commonality of terms between the windows. In reality, however, when one of a pair of terms that are relevant to each other in the document appears in first window and the other appears in second window, it is considered that the topic similarity exists between the windows. For example, if there is a sentence that has "Giants” and "Matsui” in an article about baseball, “Giants” and “Matsui” can be regarded as relevant terms. Therefore, if "Giants” appears in the first window and "Matsui" appears in the second window, it can be regarded that the topic similarity exists between these windows even if any other common terms are not contained in the windows. The conventional methods cannot detect such similarity because they focus on only the commonality of terms. Thus the conventional method has a problem related to accuracy of the topic similarity.
• a segmentation methods comprises: detecting terms that occur in an input document; segmenting said input document into document segments, each segment being an appropriate sized chunk; generating document segment vectors with as their elements the values related to frequencies of said terms occurring in said document segments; calculating eigenvalues and eigenvectors of a square sum matrix of said document segment vectors; selecting basis vectors consisting a subspace from said eigenvectors to calculate the topic continuity of the document segments; calculating vectors whose m-th element value is the projection values of the each document segment vector onto said m-th basis vector; and determining segmentation points of said document based on the continuity of said projected vectors.
• a segment vector is a vector whose elements take values determined based on frequencies of each of terms occurring in a document segment and represents a concept of the document segment.
• the most natural unit for segmenting a document is a sentence.
• the singular value decomposition process is applied to a set of document segment vectors to expand the set into eigenvalues and eigenvectors that are orthogonal each other. Since each eigenvector is represented by a combination of the terms, it represents a concept by itself. Since eigenvectors are determined inherently to the input document, the concept represented by each eigenvector can be called the eigenconcept. Each eigenvalue can be considered to represent the strength or the energy of the concept represented by the corresponding eigenvector.
• a projection value of a document segment vector onto an eigenvector is the component that the concept represented by the document segment vector has in the corresponding eigenconcept direction, and the squared projection value represents the energy of such component.
• the document segments whose concepts are similar with each other should have the similar values for the components in the direction of each eigenconcept.
• the relatedness of each document segment to each eigenconcept is calculated based on the above-described projection values, then the continuity between each document segment is obtained in terms of the concept relatedness between them, and a point where the continuity is minimal is finally determined as a topic boundary
• the topic continuity is obtained based on the projection values of each document segment vector onto each eigenvector.
• Eigenvectors of a squared sum matrix has such characteristic that coefficients for the terms co-occurring in a document have the same sign and almost same values in the lower-dimensional eigenvectors. Accordingly, as long as one of a pair of terms ("Matsui" in the above example) co-occurring in a document appears in one of the concerned document segment and the other ("Giants” in the above example) appears in the other document segment, the projection values of the respective document segment vectors should have the relatedness even though both of the concerned document segments do not share any term. As the number of the pairs of such terms increases, the projection values become further close together. As a result, if eigenvectors are correctly selected or properly weighted, the similarity between the document segments can be correctly obtained and also a topic boundary can be properly determined even though the same topic does not continue long.
• Figure 1 is a functional block diagram for one embodiment of the invention.
• FIG. 2 is a functional block diagram for another embodiment of the invention.
• Figure 1 schematically illustrates functional blocks of one embodiment of the invention. This embodiment may be implemented using a general- purpose computer as its hardware for executing some programs incorporating inventive features.
• Figure 1 shows a term detection block 11 , a morphological analysis block
• the term detection block 11 detects words and symbol sequences such as numerals from an input document. Words and symbol sequences are generally referred to as "terms" hereinafter. In case of English documents, it is easy to detect terms because in the English orthography terms are always written separately with blanks.
• the morphological analysis block 12 then performs a morphological analysis such as assigning a part of speech to each of the terms. Then, the document segmentation block 13 segments the document into document segments.
• the most fundamental segmentation unit for the document segmentation is a sentence.
• it is easy to segment a document into sentences because a sentence usually ends with a period followed by at least one space.
• Other segmentation methods may be used; for example, several sentences may be combined to form a document segment in a manner that all segments should contain the same number of the sentences, or a document may be segmented into segments in a manner that every segment should contain the same number of terms regardless of any boundary between sentences.
• the document segment vector generation block 14 calculates the number of the dimensions for vectors to be generated from the terms that occur in the entire document and determines correspondences between each dimension and each term. It should be noted that it is not necessary to assign vector elements to all kinds of terms occurring in the document. For example, using a result of the process of assigning parts of speech, it is possible to use only the terms which have been judged as a noun or a verb in the process to generate document segment vector. Then, the document segment vector generation block 14 obtains kinds of the terms occurring in each document segment and occurrence frequencies of each kind of terms, then determines values of corresponding vector elements to generate document segment vectors. Conventional methods are also employed to give a weight to each element value.
• a document is composed of N document segments, in which k kinds of terms occur.
• s n represents the frequency value of terms s k in the n-th document segment or its weighted values, and T transpose.
• the eigenvalue and eigenvector calculation block 16 calculates eigenvalues and eigenvectors of the matrix A.
• ⁇ m and ⁇ m be the m-th eigenvector and eigenvalue of A, respectively.
• ⁇ -i represents the most common direction throughout all sentence segment vectors because the sum of the squared projections of all sentence vectors onto ⁇ . is greater than that onto any other vector. Since ⁇ -i is the sum of the squared projections itself, it can be considered to represent the strength or energy of the concept that ⁇ represents.
• ⁇ 2 is the axis that gives the largest sum of the squared projections under the constraint that ⁇ 2 is orthogonal to ⁇ -i.
• the eigenvectors obtained as described above are used as the basis vectors for a subspace approximating a set of document segment vectors. If L eigenvectors are used, the dimension number of the subspace becomes L, which means that the concept of the input document has been expanded to L eigenconcepts that are orthogonal each other.
• the subspace decision block 17 decides a value of L specifically. Given matrix A whose rank is R, the document would have R eigenconcepts since R eigenvectors can be obtained from the matrix A.
• the subspace represents the central concept by a combination of L eigenconcepts among R eigenconcepts (namely R-L eigenconcepts are supposed to be discarded).
• the basis vectors of the subspace comprises L eigenvectors.
• the following expression may represent how much the central concept occupies relative to the original concept. This expression can be used as a criterion for actually determining the value of L.
• the document segment vector projection block 18 calculates a projection value of each document segment vector onto the subspace.
• p n (Pni,Pn2, " ,PnL.) T be a vector of s n after the projection.
• the projection value p nrn onto the m-th basis vector can be given by the following equation:
• the relatedness calculation block 19 calculates relatedness between each document segment and respective eigenconcept, or basis vector.
• the following is one of methods for calculating such relatedness.
• An average and a standard deviation of the projection values onto the m-th basis vector for all document segments can be obtained by the following equation: p m : Average of the projection values onto the m-th basis vector for all document segments ⁇ m : Standard deviation of the projection values onto the m-th basis vector for all document segments
• TCnm be a variable indicating a positive relatedness of the n-th document segment with the m-th eigenconcept (basis vector)
• v nn be a variable indicating a negative relatedness of the n-th document segment with the m-th eigenconcept (basis vector).
• a and b are predetermined threshold values.
• the former two terms in the equation above will take a value of 1 when the positive or negative relatedness starts from the n-th document segment and the latter two terms will take a value of 1 when the positive or negative relatedness ends at the (n-1)th document segment.
• the topic boundary decision block 20 calculates h(n) for each n by following equation and obtains the maximal point of h(n):
• the maximal point of h(n) is the point where the sum of the variations of the relatedness with L eigenconcepts (basis vector) becomes a maximum locally, so the head of the corresponding document segment is decided to be a topic boundary.
• the document is segmented at topic boundaries.
• projection values of the document segment vectors onto the basis vector have been used in order to calculate the relatedness between the document segments and the eigenconcept, squared projection values can be alternatively used to obtain the same result.
• p n m is given by the following equation instead of the equation 3:
• Figure 2 schematically illustrates functional blocks of another embodiment of the invention. This embodiment may be implemented by using a general-purpose computer as its hardware for executing some programs incorporating inventive features.
• Figure 2 shows a term detection block 11 , a morphological analysis block 12, a document segmentation block 13, a document segment vector generation block 14, a square sum matrix calculation block 15, an eigenvalue and eigenvector calculation block 16, a subspace decision block 17, a document segment vector projection block 18, a topic continuity calculation block 29 and a minimum continuity point detection block 30. From block 11 to block 18 are same blocks as described in Figure 1.
• the topic continuity calculation block 29 calculates projection vectors for all segment vectors between c document segments before the boundary point candidate and c document segments after the boundary point candidate onto the subspace to obtain the topic continuity between the document segments.
• the number of the document segments before or after the boundary point candidate may not reach C. In this case, since all the documents existing before or after the point should be used, C represents the number of the documents existing there.
• Equation 9 is an average of the cosign similarity values for all combinations between the project vectors of C document segments immediately before and after the n-th document segment.
• the similarity based on the project vectors between the shorter document segments may have too great influence on the h(n).
• Equation 10 is intended to decrease such influence.
• the h(n) will take larger value if the topic continues before and after of the boundary point and it will take smaller value if the topic does discontinue there.
• the minimum continuity point detection block 30 obtains the minimal point of h(n).
• the minimal point of h(n) represents the minimal point of the continuity, that is, the maximal point of the discontinuity. Accordingly, the end of the corresponding document segment can be decided to be a topic boundary. Thus the document may be segmented at the topic boundary.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Health & Medical Sciences (AREA)
• Artificial Intelligence (AREA)
• Audiology, Speech & Language Pathology (AREA)
• Computational Linguistics (AREA)
• General Health & Medical Sciences (AREA)
• Physics & Mathematics (AREA)
• General Engineering & Computer Science (AREA)
• General Physics & Mathematics (AREA)
• Machine Translation (AREA)

Priority Applications (1)

Applications Claiming Priority (2)

Publications (1)

Family

ID=18846658

Family Applications (1)

Country Status (2)

Cited By (3)

Families Citing this family (2)

Citations (3)

Family Cites Families (1)

• 2000
  • 2000-12-12 JP JP2000378015A patent/JP2002197083A/ja active Pending
• 2001
  • 2001-11-16 WO PCT/US2001/043534 patent/WO2002048951A1/en active Application Filing

Patent Citations (3)

Also Published As

Similar Documents

Legal Events