JP2009181301A - Expression template generating system, its method, and its program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an expression template generating system capable of precisely generating a template in which typical expressions are sorted from a plurality of similar documents of natural language. <P>SOLUTION: The expression template generating system 1 includes: a route table-generating means 31 which generates a route table holding routes of the same edit distance from a cell of a table of dynamic planning method; a route reversely-searching means 32 for generating a plurality of new alignment rows by holding candidates of identical score when reversely searching a route in the route table; and a directional graph generating means 41 which converts the plurality of alignment rows to a directional graph with an element as an apex, for generating an impression template. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an expression template generation apparatus that generates an expression template obtained by extracting typical expressions from a plurality of series data (text documents), a method thereof, and a program thereof.

  Conventionally, there is known a method for classifying various documents by aligning similar documents in a natural language document and extracting typical expressions from the similar documents (for example, Patent Document 1, Non-Patent Documents 1 and 2).

  According to these methods, for every two documents included in a plurality of similar document groups, for example, a document is aligned by dynamic programming (see Non-Patent Document 3), and a common alignment is obtained from the plurality of document groups. By extracting the data, an expression template indicating a typical expression of the document group is generated. Thus, according to the conventional method, an expression template is generated for each group of similar documents, so that the documents can be classified according to the expressions included in the documents. That is, according to the conventional method, by learning the expression template in advance, it is possible to classify a newly input document based on the degree of similarity with the expression template.

Note that the above-described method of aligning two or more series data is generally called multiple alignment. This multiple alignment is an algorithm in which the alignment of two series data is extended to a large number of series data. Usually, the alignment of two series data can obtain | require an exact solution using a two-dimensional table by a dynamic programming (refer nonpatent literature 3). However, when the input series data is M-dimensional, an M-dimensional table is required to solve the problem by dynamic programming, and the calculation amount becomes enormous, which is not realistic. Thus, in multiple alignment, an approximate solution is obtained by recursively repeating the alignment of two series of data.
JP 2006-113746 A R. Barzilay and L. Lee: “Bootstrapping lexical choice via multiple-sequence alignment”, Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 164-171 (2002). R. Barzilay and L. Lee: “Learning to paraphrase: An unsupervised approach using multiple-sequence alignment”, HLT-NAACL 2003: Main Proceedings, pp. 16-23 (2003). Colmen, Riserson, Ribert, "Algorithm Introduction", Chapter 16, "Dynamic Programming", Modern Science

  Since the conventional multiple alignment recursively repeats the alignment of two series of data, the series of alignment data selected by one alignment affects the subsequent alignment.

  In general, when a natural language document is aligned, there are a plurality of alignment columns with similar degrees of matching. In this case, even if the degrees of similarity match, there is no guarantee that all are optimally aligned. Therefore, when one alignment column is selected as in the prior art, the optimal alignment column is not always selected. As described above, in the conventional multiple alignment, alignment series data that is not necessarily optimal is selected, and therefore, a recursively repeated solution results in an inappropriate alignment as a whole.

  The present invention has been made to solve the above problems, and accurately generates an expression template obtained by extracting a typical expression from a plurality of series data (text documents) similar in natural language. It is an object of the present invention to provide an expression template generation apparatus, a method thereof, and a program thereof capable of performing

  The present invention was created to achieve the above object, and first, the expression template generation device according to claim 1 is a typical expression that appears in a plurality of sequence data composed of a plurality of elements of a natural language. An expression template generation device that extracts and generates an expression template, comprising: an alignment cost calculation means; an alignment cost storage means; a route table generation means; a route reverse search means; and a directed graph generation means; did.

  In such a configuration, the expression template generation device calculates an alignment cost, which is an editing distance indicating the degree of similarity for each element, for each combination of all two sets of a plurality of series data by the alignment cost calculation unit, and the series data Is stored in the alignment cost storage means in association with the alignment cost for each of the two combinations of the alignment column. As a result, initialization data for performing alignment is stored in the alignment cost storage means. Here, the edit distance is a numerical value indicating how much the two series data are different, and is used to transform one series data into the other series data by inserting, deleting, or replacing elements. It is a numerical value in which the number of times is regarded as a distance.

  Then, the expression template generation device uses the route table generation means to move the motions in which the routes having the same edit distance are held in the cells of the two-dimensional array table in order from the alignment row having the lower alignment cost, that is, the higher similarity degree. Generate a path table for static programming. Thereby, the route table is not limited to one route when the editing distance is the same, and a plurality of routes are held. Then, the expression template generation device generates a plurality of new alignment sequences by performing a reverse search of the route in the route table generated by the route table generation unit by the route reverse search unit. As a result, the expression template generation device can generate an alignment sequence retaining ambiguity in multiple alignment.

  Then, the expression template generation device converts the plurality of alignment sequences generated by the path reverse search means into a directed graph having elements as vertices by the directed graph generation means, and generates the expression template. In this way, the expression template generation device can extract a typical expression appearing in the sequence data by aligning the plurality of sequence data, and can make a template.

  The expression template generation device according to claim 2 is the expression template generation device according to claim 1, wherein the path reverse search means generates a plurality of new alignment sequences based on the edit distance. The configuration further includes alignment number limiting means for calculating the alignment cost of the alignment string and limiting the number of alignment strings based on the alignment cost calculated by the path reverse search means.

  In such a configuration, the expression template generation device leaves only a low alignment cost according to a predetermined criterion from a plurality of alignment sequences searched by the path reverse search means. As a result, in a dynamic programming method in which processing is performed recursively, even if a plurality of routes are searched, the increase can be suppressed.

  Furthermore, the expression template generation device according to claim 3 is configured such that the expression template generation device according to claim 1 or 2 further includes a directed graph restriction unit.

  In such a configuration, the expression template generation device has at least one of the frequency of use of the edges constituting the effective graph of the expression template, the number of vertices, or the number of edges in the expression template generated by the directed graph generation means by the directed graph restriction means. Pruning a directed graph with one as a reference. As a result, it is possible to delete a phrase having a low appearance frequency as an expression from the directed graph.

  The expression template generation method according to claim 4 is an expression template generation method for extracting a typical expression appearing in a plurality of series data composed of a plurality of elements of a natural language and generating the expression as an expression template, wherein alignment is performed. The procedure includes a cost calculation step, a route table generation step, a route reverse search step, and a directed graph generation step.

  In such a procedure, the expression template generation method calculates an alignment cost, which is an edit distance indicating a degree of similarity for each element, for each combination of all two sets of a plurality of series data in the alignment cost calculation step. Is stored in the alignment cost storage means in association with the alignment cost for each of the two combinations of the alignment column.

  In the expression template generation method, in a path table generation step, a dynamic programming path table is generated in which the paths having the same edit distance are held in the cells of the two-dimensional array table in order from the alignment column having the lowest alignment cost. . The expression template generation method generates a plurality of new alignment sequences by performing a reverse search of the route in the route table generated in the route table generation step in the route reverse search step. In the expression template generation method, in the directed graph generation step, the plurality of alignment sequences generated in the path reverse search step are converted into directed graphs having elements as vertices, and generated as expression templates.

  An expression template generation program according to claim 5 extracts a typical expression appearing in a plurality of sequence data composed of a plurality of elements of a natural language and generates a computer as an alignment cost for generating an expression template. Means, route table generation means, route reverse search means, and directed graph generation means.

  In such a configuration, the expression template generation program calculates an alignment cost, which is an edit distance indicating the degree of similarity for each element, for every two combinations of a plurality of series data by the alignment cost calculation means, and the series data Is stored in the alignment cost storage means in association with the alignment cost for each of the two combinations of the alignment column.

  Then, the expression template generation program generates a path table for dynamic programming in which the path having the same edit distance is held in the cells of the two-dimensional array table in order from the alignment column having the lowest alignment cost by the path table generation means. . The expression template generation program generates a plurality of new alignment sequences by performing a reverse search of the route in the route table generated by the route table generation unit by the route reverse search unit. Then, the expression template generation program converts the plurality of alignment sequences generated by the path reverse search means into a directed graph having elements as vertices by the directed graph generation means, and generates the expression template.

The present invention has the following excellent effects.
According to the invention of claim 1, claim 4 or claim 5, in the dynamic programming table, the route held in the cell is not limited to one, but the route of the same edit distance (cost) is selected. Since the plurality is held, the optimum alignment row is not deleted. Therefore, according to the present invention, even a slightly different expression can be reflected in the expression template without being deleted as in the prior art.

  According to the invention described in claim 2, since the number of alignment columns generated by dynamic programming is limited by the alignment cost, the number of alignment columns generated recursively can be suppressed and the calculation cost can be suppressed. it can.

  According to the invention described in claim 3, by performing pruning of the directed graph, it is possible to remove an expression that appears less frequently in a plurality of series data. Thus, the present invention can extract only more typical expressions and generate an expression template with high accuracy for classifying expressions.

[Overview of document classification system]
First, an overview of a document classification determination system including an expression template generation device according to an embodiment of the present invention will be described with reference to FIG. FIG. 1 is a block diagram showing a schematic configuration of a document classification discrimination system. The document classification determination system S learns typical expressions appearing in the text document group as expression templates from similar text document groups for each predetermined classification item, and based on the expression template that is the learning result, A classification item to which a newly input text document belongs is determined. Here, the document classification determination system S includes an expression template generation device 1, an expression template storage device 2, and a classification determination device 3.

  The expression template generating apparatus 1 generates a typical expression appearing in a text document as an expression template from a text document describing a similar opinion or impression in which an object to describe the opinion or impression is specified in advance. . For example, the expression template generation device 1 classifies the opinions of readers and viewers for books, movies, broadcasts, etc., in advance, by hand, such as “I was moved”, “I was made to think”, “I was surprised”, An expression template is generated for each classification item for a plurality of similar text documents.

  As shown in FIG. 2, the expression template generated by the expression template generation apparatus 1 is represented by a graph structure in which “vertex” is a morpheme and “edge” is a connection of vertices. In the example of FIG. 2, four expressions “I thought it was wonderful”, “I thought it was wonderful”, “I thought it was amazing”, and “I thought it was amazing” are classified into a certain category (for example, “Impressed” It is shown that it has been generated for the classification item “done”. The expression template generated by the expression template generation device 1 is stored in the expression template storage device 2 for each classification item.

Here, the expression template generation device 1 generates expression templates B _{1 to} B _N corresponding to the text document groups A _{1 to} A _N from the similar text document groups A _{1 to} A _N , and the expression template storage device 2 memorize. For example, B _{1 to} B _N are expression templates corresponding to “impressed”, “made to think”, “unexpected”, and the like.

  The expression template storage device 2 stores the expression template generated by the expression template generation device 1 as a learning result for each classification item, and is a general storage device such as a hard disk.

  The classification discriminating device 3 discriminates to which classification item the newly input text document belongs, based on the expression template stored in the expression template storage device 2. If the input text document includes an expression template expression stored in the expression template storage device 2, the classification determination apparatus 3 may, for example, add “ When the expression “I thought” is included, the text document is determined to be a document belonging to the classification item “I was impressed” based on the expression template of FIG. 2.

  As described above, the document classification determination system S performs the learning process of the expression template for each classification item by the expression template generation device 1, and performs the determination process of a new text document by the classification determination device 3. Hereinafter, the configuration and operation of the expression template generation device 1 will be described in detail.

[Configuration of Expression Template Generation Device]
First, with reference to FIG. 3, the structure of the expression template production | generation apparatus which concerns on embodiment of this invention is demonstrated. FIG. 3 is a block diagram showing the overall configuration of the expression template generation device according to the embodiment of the present invention.

  Here, the expression template generation device 1 includes an alignment cost calculation unit 10, an alignment cost storage unit 20, an alignment sequence generation unit 30, and a template generation unit 40. Note that the text document (series data) input to the expression template generation device 1 is preliminarily divided into elements for each morpheme.

  The alignment cost calculation means 10 calculates an alignment cost, which is an edit distance indicating the degree of similarity for each element, for every two combinations of a plurality of text documents. The alignment cost calculation unit 10 stores an alignment cost list in which the two sets of text documents are associated with each other in the order of the calculated alignment cost, in the alignment cost storage unit 20.

Here, the set of elements of the M-series series data (text document) is Ω ^(M) , the set of empty elements (_) is {_}, i, i ′, i ″ are the order of the elements of the series data (empty elements If the subscript represents (except for), the sequence data {W ⁽¹⁾ }, {W ⁽²⁾ } and the alignment sequence {a ^(j) } are expressed by the following equation (1).

Further, when a function that removes empty elements while maintaining the order of elements from series data including empty elements is defined as _{f_omit} , the series data {W ^(j) } is expressed by the following equation (2).

ここで、２系列データ（２組のテキスト文書）｛Ｗ^（１）｝，｛Ｗ^（２）｝のアライメントコストは、各要素の編集距離の和であって、以下の（３）式の関数ｆ_ｃｏｓｔにより求められる値である。

Here, the alignment cost of the two series data (two sets of text documents) {W ⁽¹⁾ }, {W ⁽²⁾ } is the sum of the edit distances of the respective elements, and is a function of the following equation (3): This is a value obtained by f _cost .

  Edit is an edit distance based on whether or not two elements match, and is defined in advance by the following equation (4).

  Thus, the alignment cost calculation means 10 calculates the degree of similarity between two sets of text documents as the alignment cost.

The alignment cost storage means 20 stores the alignment costs of every two sets of text documents calculated by the alignment cost calculation means 10 as a list of alignment costs associated with the text document pairs. Storage device. This alignment cost storage means 20, for example, as shown in FIG. 4, a text document pairs, should the alignment cost is stored as the alignment cost list T _C which associates tabulated. The alignment cost list T _C is sorted in order of the alignment cost small.

  The alignment sequence generation unit 30 recursively creates one or more new alignment sequences by dynamic programming in order from two sequences of alignment sequences (text documents) having a low alignment cost stored in the alignment cost storage unit 20. Is to be generated.

Here, the alignment sequence generation unit 30 performs alignment as new series data by removing empty elements from the alignment sequence stored in the alignment cost storage unit 20. That is, the alignment sequence {a ^(j) } corresponding to the sequence data W ^{(1) corresponding} to {a, b, a, d} and the sequence data W ⁽²⁾ corresponding to {a, c, d} has the following (5 ), The alignment sequence generating means 30 performs alignment using the alignment sequence {W} from which empty elements are removed as shown in the following equation (6).

The alignment sequence generation means 30 uses the following equation (7) as an alignment target to be read out from the alignment cost storage means 20 as {W ⁽¹⁾ }, {W ⁽²⁾ }. All alignment columns with the lowest cost are generated.

  As described above, the alignment sequence generation means 30 does not limit the alignment sequence that minimizes the alignment cost to one, and therefore can generate an alignment sequence that retains ambiguity. Here, the alignment sequence generating means 30 includes a route table generating means 31, a route reverse searching means 32, and an alignment number limiting means 33 in order to obtain the expression (7) by dynamic programming. Yes.

  The path table generation unit 31 is a dynamic programming path table in which the paths having the same edit distance are held in the cells of the two-dimensional array table in order from the alignment row having the lowest alignment cost stored in the alignment cost storage unit 20. Is generated. Here, the path table generation unit 31 selects an alignment column in the two text documents having the lowest alignment cost from the alignment cost storage unit 20, and sets the two alignment element columns included in the alignment column as two axes. Generate a routing table for dynamic programming. The route table for dynamic programming generated by the route table generating unit 31 is used by the route reverse search unit 32.

  The route reverse search means 32 generates an alignment sequence by performing a reverse search (trace back) on the route in the dynamic programming route table generated by the route table generation means 31. The route reverse search means 32 calculates the alignment cost in the route by sequentially adding the edit distance when performing the reverse search of the route.

  Here, with reference to FIG. 5 and FIG. 6 (refer to FIG. 3 as appropriate), the processing performed by the route table generation unit 31 and the processing performed by the route reverse search unit 32 will be specifically described. FIG. 5 is an explanatory diagram for explaining a method of generating a route table of dynamic programming performed by the route table generating means. FIG. 6 is an explanatory diagram for explaining a method for generating an alignment sequence from a route table of the dynamic programming method performed by the route reverse search means.

As shown in FIG. 5, the path table generating means 31 generates a path table T _DP having a two-dimensional array having two alignment element strings (alignment A, alignment B) as two axes on a memory (not shown). To do. Then, the route table generation means 31 sequentially selects the route and the edit distance at that time from the upper left cell of the route table T _DP indicating the head of each alignment to the lower right cell of the route table T _DP indicating the tail of each alignment. (Cost) is stored on the memory as array data of a two-dimensional array.

For example, in the cell C ₁ of FIG. 5, the path from the C ₂ (downward arrow) includes the “i” -th series of the alignment A and the “j−1” -th series of the alignment B, and the alignment B in the series. Is associated with the edit distance when the “j” -th element of the is inserted. Also, the route (right direction arrow) from the C _3, and "i-1" th and "j" to th sequence alignment B of alignment A, the "i" th element of alignment A to the series Associate the edit distance with the series when inserted.

In addition, when the “i” -th element of alignment A and the “j” -th element of alignment B coincide with the path from C ₄ (arrow pointing downward to the right), the “i−1” -th element of alignment A and The edit distance between the sequence up to the “j−1” th sequence of the alignment B and the sequence when the “i” th element of the alignment A (the “j” th element of the alignment A) is inserted into the sequence is associated. On the other hand, if the “i” -th element of alignment A and the “j” -th element of alignment B do not match, the sequence from “i−1” -th alignment A to “j-1” -th alignment B; The edit distance between the series inserted by replacing the “i” -th element of alignment A and the “j” -th element of alignment A is associated with the series.

  Note that the route table generation unit 31 stores only the optimal (minimum cost) route when the edit distances (costs) of the routes (downward arrow, rightward arrow, and downward right arrow in FIG. 5) are different. In other words, when the editing distance is the same, all the routes are stored.

Then, as shown in FIG. 6, the route reverse search means 32 starts from the cell _{CE at} the end point (lower right) of the route table in the dynamic programming route table T _DP generated by the route table generation means 31. based on the route stored in association with successively the cell, it searches for a route to the cell C _S of the start point of the path table (top left). As shown in FIG. 6, when the same cost is associated with the cell C ₁ by two routes, the route reverse search means 32 generates an alignment string composed of the two routes. As a result, a plurality of alignment rows having the same cost are generated without being limited to one.
Returning to FIG. 3, the description of the configuration of the expression template generation device 1 will be continued.

  The alignment number limiting unit 33 limits the number of alignment columns to be generated based on a predetermined criterion for the alignment columns generated by the path reverse search unit 32. Here, the alignment number limiting unit 33 limits the number of alignment columns to be generated for each of the two series of alignment columns processed by the alignment column generating unit 30. As a result, the number of paths in the dynamic programming method can be limited, and an exponential increase in the number of paths associated with recursive processing can be prevented. For example, the alignment number limiting means 33 can limit the number of paths according to at least one of the following four criteria (reference A1 to reference A4).

<Standard A1 Restriction due to alignment cost>
The alignment number limiting means 33 outputs only the alignment sequence that minimizes the alignment cost of the two series data obtained by the path reverse search means 32. That is, of the plurality of alignment sequences generated by the route reverse search means 32, only the one that minimizes the total alignment cost of the route is left and the others are deleted.

<Standard A2 Limitation by the number of empty elements>
The alignment number limiting means 33 leaves only the alignment string with the smallest number of empty elements among the plurality of alignment strings generated by the path reverse search means 32 and deletes the others.

<Standard A3 Limit by absolute number of optimum routes>
Alignment number limiting means 33 randomly selects an absolute number of predetermined paths from among a plurality of alignment sequences generated by path reverse search means 32 and deletes the others.

<Limitation by absolute number of reference A4 elements>
The alignment number limiting unit 33 accumulates the number of elements in the plurality of alignment columns generated by the path reverse search unit 32, and when the number of elements exceeds a predetermined absolute number, the subsequent alignment columns are deleted.

  In this way, the alignment sequence generation means 30 sequentially reverses the path sequence search means in order to limit the number of alignment sequences when generating one or more new alignment sequences by dynamic programming in order from the two series of alignment sequences. Even when 32 generates an alignment sequence of a plurality of paths, an exponential increase in the number of paths can be prevented.

Note that, when a new alignment string is generated from two series of alignment strings, the alignment string generating unit 30 deletes data relating to the one series of alignment columns stored in the alignment cost storage unit 20. For example, in FIG. 4, a text document pair ^{{W (1), W (} 2)} was ^{aligned, W (1)} relating to data ^{{W (1), W (} 2)}, {W (1), W ⁽³⁾ }, ... are deleted. The alignment sequence to be deleted may be either one of the two series of alignment columns.

  Then, from the alignment sequence of all the two series, a new alignment sequence is generated, that is, until the alignment sequence stored in the alignment cost storage unit 20 is only one sequence, the alignment sequence is sequentially generated, It outputs to the template production | generation means 40.

  The template generation unit 40 converts the plurality of alignment sequences generated by the alignment sequence generation unit 30 into a directed graph having elements as vertices, and generates an expression template. Here, the template generation unit 40 includes a directed graph generation unit 41 and a directed graph restriction unit 42.

  The directed graph generation means 41 converts a plurality of alignment columns into a directed graph having vertices as elements constituting the alignment columns. Here, the directed graph generation means 41 defines an edge having directionality with an element as a vertex (node), an element that appears first in the alignment sequence as a start point, and an element that appears next as an end point. Thereby, the directed graph generation means 41 can generate an expression template standardized by the directed graph from the alignment sequence.

  Here, when the directed graph generation unit 41 shares the same side in the alignment sequence, the directed graph generation unit 41 assigns the frequency to the number shared by the side. As a result, the frequency of connection of elements can be quantified.

  The directed graph restriction unit 42 selects an optimum directed graph (expression template) from the directed graph generated for each alignment sequence generated by the directed graph generation unit 41 based on a predetermined criterion. For example, the directed graph restriction unit 42 selects a directed graph according to any of the following <reference B1 to reference B3>.

<Selection based on frequency of reference B1 side>
The directed graph restriction unit 42 selects, from the directed graphs generated by the directed graph generation unit 41, a directed graph in which the number of edges whose frequency given to an edge of the directed graph is equal to or less than a predetermined frequency is minimized. As a result, an expression template having a higher use frequency can be generated.

<Selection based on the number of vertices in reference B2>
The directed graph restriction unit 42 selects a directed graph having the minimum number of vertices of the effective graph from the directed graphs generated by the directed graph generation unit 41. This can reduce the possibility that the same element is another vertex.

<Standard 3 Selection by number of sides>
The directed graph restriction unit 42 selects a directed graph that minimizes the number of sides of the effective graph from the directed graphs generated by the directed graph generation unit 41. As a result, like the reference B2, the possibility that the same element is a different vertex can be reduced.

  By configuring the expression template generation device 1 as described above, in multiple alignment using dynamic programming, a plurality of optimum paths can be input to the subsequent alignment. Alignment is performed recursively by the alignment sequence that has. As a result, the expression template generation device 1 can perform alignment without removing an appropriate solution in the middle of multiple alignment. Therefore, the expression template generation apparatus 1 accurately generates an expression template indicating a typical expression from a plurality of text documents. be able to.

  In addition, since the expression template generation device 1 limits the number of paths in dynamic programming in multiple alignment, it can prevent an exponential increase in the number of paths and suppress the amount of calculation.

  As mentioned above, although the structure of the expression template production | generation apparatus 1 was demonstrated, this invention is not limited to this. Here, it is assumed that the text document to be input is divided in advance for each morpheme, but if a text document that is not divided for each morpheme is input, the reading of the morpheme, the part of speech, the utilization type, etc. are stored. Referring to the morpheme dictionary (not shown), a morpheme analysis unit (not shown) for performing morpheme analysis is further provided, and a text document divided into morphemes is input to the alignment cost calculation unit 10. It is good.

  The expression template generation device 1 can be realized by a general computer having a CPU and a memory (not shown). At this time, the expression template generation apparatus 1 operates according to an expression template generation program that causes a computer to function as each of the above-described means.

[Operation of expression template generator]
Next, the operation of the present invention will be described with reference to FIGS. FIG. 7 is a flowchart showing the overall operation of the expression template generation apparatus according to the embodiment of the present invention. FIG. 8 is a flowchart showing the alignment operation in the alignment sequence generation means of the expression template generation apparatus according to the embodiment of the present invention.

(Overall operation)
First, the overall operation of the expression template generation device 1 will be described with reference to FIG.

  First, the expression template generation device 1 uses the alignment cost calculation unit 10 to calculate the degree of similarity between text documents based on the edit distance of the text documents for each of the two sets of the plurality of text documents. Is calculated (step S1).

Then, the expression template generation device 1 generates an alignment cost list in which the two sets of text documents and the alignment costs are associated with each other in the order of the alignment costs generated in step S1 by the alignment cost calculation unit 10, and the alignment cost storage unit 20 (Step S2). As a result, a list of alignment costs is generated for every two combinations ( _M C ₂ types) of the M-sequence text document.

  After that, the expression template generation device 1 performs alignment by dynamic programming in order from the two series of text documents having the lowest alignment cost stored in the alignment cost storage unit 20 by the alignment column generation unit 30. A new alignment sequence is generated (step S3). The operation in step S3 will be described later with reference to FIG.

  After the operation of step S3, the expression template generation device 1 deletes one of the two sequences of aligned data from the alignment cost storage unit 20 by the alignment column generation unit 30 (step S4). As a result, the alignment sequence generation means 30 deletes one of the series data that has been aligned from the target of the dynamic programming.

  And the expression template production | generation apparatus 1 determines whether the remainder of the data of the alignment cost list memorize | stored in the alignment cost memory | storage means 20 became one (step S5).

If there are two or more remaining data in the alignment cost list (No in step S5), the expression template generation device 1 returns to step S3 and performs alignment for the next two alignment strings with the lowest alignment cost. .
On the other hand, when the remaining data of the alignment cost list becomes one (Yes in step S5), the expression template generation device 1 is generated by the directed graph generation unit 41 of the template generation unit 40 by the operations of steps S3 to S4. A directed graph with the elements as vertices is generated from the plurality of alignment sequences (step S6).

  After that, the expression template generation device 1 selects an optimum directed graph from the directed graph generated in step S6 by the directed graph restriction unit 42 based on a predetermined criterion and outputs it as an expression template (step S7).

  With the above operation, the expression template generating apparatus 1 can generate a typical expression as an expression template from a plurality of similar text documents.

(Alignment operation)
Next, referring to FIG. 8 (refer to FIG. 3 as appropriate for the configuration), the alignment operation in the alignment sequence generation means of the expression template generation device 1 will be described. This operation corresponds to the operation in step S3 described in FIG.

  First, the expression template generation apparatus 1 selects an alignment column in the two text documents with the lowest alignment cost from the alignment cost storage unit 20 by the route table generation unit 31 of the alignment column generation unit 30 and selects the alignment column as the alignment column. A dynamic programming path table is generated with two included alignment element strings as two axes (step S31). At this time, in the cell of the dynamic programming route table, the alignment sequence generation means 30 supports only the optimum route (minimum cost) for the cell when the edit distance (cost) of each route is different. If the edit distance is the same, all the routes having the same edit distance are stored in association with the cell. As a result, there are one or more routes through the cell during traceback.

  And the expression template production | generation apparatus 1 produces | generates a some alignment row | line | column by carrying out reverse search (trace back) of the route table produced | generated by step S31 by the route reverse search means 32 (step S32).

  And the expression template production | generation apparatus 1 restrict | limits the number of alignment row | line | columns produced | generated by the alignment number restriction | limiting means 33 based on the predetermined reference | standard about the alignment row | line | column produced | generated by step S32 (step S33).

  With the above operation, the expression template generation device 1 prevents the increase in the number of exponential paths even in the case of holding a plurality of optimum paths in the dynamic programming, and is expressed as a typical expression. An alignment sequence that retains the representation can be generated.

[Comparison with conventional methods]
Finally, with reference to FIG. 9, a comparison result between the conventional multiple alignment and the multiple alignment in the expression template generating apparatus 1 according to the embodiment of the present invention will be described. Here, an opinion on a certain broadcast program is input as a text document, and a comparison is made by generating an expression template.

FIG. 9 is a diagram showing an expression template represented by a directed graph, where (a) shows an expression template generated by the conventional technique, and (b) shows an expression template generated by the technique of the present invention. In FIG. 9, an element (morpheme) corresponds to a vertex, and a vertex connection corresponds to an edge. The numbers attached to the edges indicate how many opinions have passed through this connection.
When the expressions included in the expression template shown in FIG. 9 are represented in a table format, a plurality of expressions are extracted as shown in [Table 1] below.

  As shown in [Table 1], according to the multiple alignment of the present invention, “wow” and “great” appearing at the end of the sentence are extracted, but these are not extracted in the conventional method. This is probably because the conventional method has a single route stored in a cell in the dynamic programming route table, and the route has been deleted. As described above, the expression template generation apparatus according to the embodiment of the present invention can appropriately extract expressions that could not be extracted by the conventional method and generate them as expression templates.

It is a block diagram which shows schematic structure of a document classification discrimination system. It is a figure which shows an example of the expression template which the expression template production | generation apparatus which concerns on embodiment of this invention produces | generates. It is a block diagram which shows the whole structure of the expression template production | generation apparatus which concerns on embodiment of this invention. It is a structure figure which shows the data structure of the alignment cost list memorize | stored in an alignment cost memory | storage means. It is explanatory drawing for demonstrating the method of producing | generating the route table of the dynamic programming which a route table production | generation means performs. It is explanatory drawing for demonstrating the method of producing | generating an alignment row | line from the route table of the dynamic programming which a route reverse search means performs. It is a flowchart which shows the whole operation | movement of the expression template production | generation apparatus which concerns on embodiment of this invention. It is a flowchart which shows the alignment operation | movement in the alignment row | line | column production | generation means of the expression template production | generation apparatus which concerns on embodiment of this invention. It is a figure which shows the expression template represented with the directed graph, Comprising: (a) is the expression template produced | generated by the conventional method, (b) has shown the expression template produced | generated by the method of this invention.

Explanation of symbols

S Document classification determination system 1 Expression template generation device 2 Expression template storage device 3 Classification determination device 10 Alignment cost calculation means 20 Alignment cost storage means 30 Alignment string generation means 31 Route table generation means 32 Path reverse search means 33 Alignment number restriction means 40 Template generation means 41 Directed graph generation means 42 Directed graph restriction means

Claims (5)

An expression template generation device that extracts typical expressions appearing in a plurality of series data composed of a plurality of elements of a natural language and generates them as an expression template,
An alignment cost calculating means for calculating an alignment cost that is an edit distance indicating a degree of similarity for each element for every two sets of combinations of the plurality of series data;
Alignment cost storage means for storing the alignment cost in association with each other for each of two combinations of the series data as an alignment column;
A path table generating unit that generates a path table for dynamic programming in which cells having the same edit distance are held in the cells of the two-dimensional array table in order from the alignment row having the lowest alignment cost stored in the alignment cost storage unit. When,
In the route table generated by the route table generating means, by performing a reverse search of the route, a route reverse search means for generating a plurality of new alignment sequences;
A plurality of alignment sequences generated by the path reverse search means, converted to a directed graph having the element as a vertex, and generated as the expression template;
An expression template generation device comprising: When the path reverse search means generates the plurality of new alignment sequences, based on the editing distance, to calculate the alignment cost of the alignment sequence,
The expression template generation device according to claim 1, further comprising an alignment number limiting unit that limits the number of the alignment columns based on the alignment cost calculated by the path reverse search unit.   Directional graph restriction for pruning the directed graph based on at least one of the use frequency, the number of vertices, or the number of edges constituting the effective graph of the expression template in the expression template generated by the directed graph generation means The expression template generating apparatus according to claim 1, further comprising means. An expression template generation method for extracting a typical expression appearing in a plurality of series data composed of a plurality of elements of a natural language and generating as an expression template,
The alignment cost is calculated for every two combinations of the plurality of series data by an alignment cost calculation means for calculating an alignment cost that is an edit distance indicating the degree of similarity for each element, and the series data is aligned. Alignment cost calculation step for associating and storing the alignment cost in the alignment cost storage means for each of the two combinations in a row;
From the alignment sequence with low alignment cost stored in the alignment cost storage means, the path table generation means for generating a dynamic programming path table that holds the paths having the same edit distance in the cells of the two-dimensional array table. In order, a route table generating step for generating the route table;
A route reverse search step for generating a plurality of new alignment sequences by performing a reverse search of the route in the route table generated in the route table generation step by a route reverse search means for reverse searching the route table; ,
A directed graph generation step of converting a plurality of alignment sequences generated in the path reverse search step into the directed graph by the directed graph generation means for generating a directed graph having the elements as vertices, and generating the representation template,
A method for generating an expression template characterized by comprising: In order to extract typical expressions appearing in a plurality of series data composed of a plurality of elements of natural language and generate them as expression templates,
For every two sets of combinations of the plurality of series data, an alignment cost that is an edit distance indicating the degree of similarity for each element is calculated, and for each of the two sets of combinations using the series data as an alignment column, Alignment cost calculation means for associating alignment costs and storing them in alignment cost storage means,
A path table generating unit that generates a path table for dynamic programming in which a path having the same edit distance is held in a cell of a two-dimensional array table in order from an alignment row having a low alignment cost stored in the alignment cost storage unit. ,
In the route table generated by the route table generating means, reverse route search means for generating a plurality of new alignment sequences by performing reverse search of the route,
A directed graph generation unit that converts a plurality of alignment sequences generated by the path reverse search unit into a directed graph having the element as a vertex, and generates the expression template,
An expression template generation program characterized by functioning as

Images