JP7432898B2 - Parameter optimization device, parameter optimization method, and program - Google Patents

Description

The present invention belongs to the field of natural language processing in which language is automatically processed using a computer, and in particular to rhetorical structure analysis technology that represents a document as a tree structure (rhetorical structure tree) with EDU (Elementary Discourse Units) as leaves. It is related.

Various rhetorical structure analyzers have been proposed that automatically construct a rhetorical structure tree by analyzing the rhetorical structure of a document. Many rhetorical structure analyzers are implemented using neural networks that use human-prepared ground truth data, that is, documents annotated with rhetorical structure, as training data.

For example, in the technology disclosed in Non-Patent Document 1, the parameters of a network that estimates a tree structure by dividing a span into two, the parameters of a network that assigns nuclearity to adjacent spans, and relationship labels are determined using ground truth data. I'm learning. Note that nuclear labeling results in the problem of classifying two divided spans into one of three ways: "NS", "SN", and "NN", and relational labeling results in "Elaboration, The problem comes down to classifying the image into one of 18 types of labels, such as ``Background.''

Kobayashi Naoki, Tsutomu Hirao, Hidetaka Kamigaito, Manabu Okumura, Masaaki Nagata: Top-Down RST Parsing Utilizing Granularity Levels in Documents, AAAI-2020, pp. 8099-8106, (2020)

Rhetorical structure annotation costs a lot of money, so it is difficult to prepare a large amount of rhetorical structure annotated data, and at present the largest amount of data is only 385 documents. Therefore, many rhetorical structure analysis techniques face the problem of data sparsity, especially the performance of relational labeling, which has to solve an 18-class classification problem.

The present invention has been made in view of the above points, and it is an object of the present invention to provide a technique for realizing a rhetorical structure analyzer with good performance without using a large amount of rhetorical structure annotated data.

According to the disclosed technology, a pseudo-correct data generation unit that generates, as pseudo-correct data, a subtree common to a plurality of rhetorical structure trees obtained from unlabeled data by a plurality of rhetorical structure analyzers;
a pre-learning unit that optimizes parameters of a rhetorical structure analyzer using a neural network using the pseudo-correct data generated by the pseudo-correct data generating unit;
A parameter optimization device is provided, comprising: a fine-tuning unit that fine-tunes the parameters optimized by the pre-learning unit using correct data.

According to the disclosed technique, a technique is provided for realizing a rhetorical structure analyzer with good performance without using a large amount of rhetorical structure annotated data.

FIG. 3 is a diagram illustrating an example of a rhetorical structure tree. FIG. 2 is a configuration diagram of a parameter optimization device. FIG. 3 is a diagram for explaining a pseudo-correct data generation section. FIG. 2 is a diagram showing Algorithm 1. 3 is a diagram for explaining an example of the operation of algorithm 1. FIG. FIG. 3 is a diagram for explaining a pre-learning section. It is a diagram showing an example of the hardware configuration of the device.

DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention (this embodiment) will be described below with reference to the drawings. The embodiments described below are merely examples, and embodiments to which the present invention is applied are not limited to the following embodiments.

(About rhetorical structure trees)
First, an example of a rhetorical structure tree targeted in this embodiment will be explained. A rhetorical structure tree is obtained by recursively repeating the operation of connecting a series of EDUs (hereinafter referred to as spans), which are the smallest basic units of discourse that make up the tree, through rhetorical relationships to form larger spans. It's a tree.

A leaf of a tree is a unit of EDU (equivalent to a node), and a node of the tree is given a nuclearity label of the span it dominates. One of the two spans (sibling spans) that will be combined will become a core containing important information, and the other will become a satellite that supplements it. In exceptional cases, both sides may be the core. Tree branches are given relational labels that represent the rhetorical relationships between spans.

Figure 1 shows an example of a rhetorical structure tree. In the figure, e ₁ to e ₇ are EDUs, S/N is the nuclearity label of the span (N is the nucleus, S is the satellite), Condition, Elaboration, etc. are the relationship labels between the sibling spans. When the nuclearity of the sibling spans is a combination of S and N, the relationship label is given to the span on the S side, and when it is N and N, it is given to both spans. Condition and Elaboration are given to the combination of S and N, and List and Same-Unit are given to the combination of N and N.

(Summary of embodiment)
In this embodiment, the parameter optimization device 100 first analyzes a large amount of unlabeled data using a plurality of existing rhetorical structure analyzers, and then analyzes data with various labels while maintaining a certain degree of frequency. Build a set. Next, common subtrees among the plurality of rhetorical structure trees obtained from the plurality of rhetorical structure analyzers are extracted as pseudo-correct data. Then, after pre-learning the parameters of a rhetorical structure analyzer based on a neural network using the pseudo-correct data, the parameters are fine-tuned using the correct data (data that has been manually annotated).

By configuring a rhetorical structure analyzer based on a neural network using the parameters obtained through the above processing, the manually annotated data can be used only for fine tuning, thereby eliminating the need to use a large amount of manually annotated data. , it is possible to realize a rhetorical structure analyzer with good performance.

(Device configuration example, operation overview)
FIG. 2 shows a configuration example of the parameter optimization device 100 in this embodiment. As shown in FIG. 2, the parameter optimization device 100 includes a pseudo-correct data generation section 110, a pre-learning section 120, and a fine tuning section 130.

The parameter optimization device 100 may be implemented using one computer or multiple computers. Furthermore, some or all of the functions of the parameter optimization device 100 may be implemented in a virtual machine on the cloud. The pseudo-correct data generation unit 110, the pre-learning unit 120, and the fine-tuning unit 130 may be respectively referred to as a pseudo-correct data generation device, a pre-learning device, and a fine-tuning device.

FIG. 2 also shows the flow of processing. As shown in FIG. 2, the pseudo-correct data generation unit 110 generates pseudo-correct data from unlabeled data, and the pre-learning unit 120 uses the pseudo-correct data to learn parameters of a rhetorical structure analyzer based on a neural network. . The fine tuning unit 130 uses the parameters learned by the pre-learning unit 120 as initial values and learns the parameters again using the correct data (manually annotated data).

The processing contents of each part will be explained in detail below.

(Pseudo-correct data generation unit 110)
The pseudo-correct data generation unit 110 generates a common subtree as a consensus tree from the results of analyzing unlabeled data using a plurality of rhetorical structure analyzers, and passes it to the pre-learning unit 120 as pseudo-correct data.

The pseudo-correct data generation unit 110 will be described with reference to FIG. 3. As shown in FIG. 3, the pseudo-correct data generation unit 110 includes n rhetorical structure analyzers and a consensus tree extraction unit 115.

Regarding the n rhetorical structure analyzers, n different rhetorical structure analyzers may be used, or n pieces of the same rhetorical structure analyzer with modified hyperparameters may be used.

As shown in FIG. 3, each of the n rhetorical structure analyzers independently analyzes input unlabeled data and generates a rhetorical structure tree for each document. This generates n rhetorical structure trees.

Next, the consensus tree extraction unit 115 extracts common subtrees among the n rhetorical structure trees and passes them to the pre-learning unit 120 as pseudo-correct answer data.

<Processing of the consensus tree extraction unit 115>
The processing contents of the consensus tree extraction unit 115 will be explained in detail. FIG. 4 shows Algorithm 1 for extracting common subtrees among n rhetorical structure trees. Algorithm 1 corresponds to a program executed by the consensus tree extraction unit 115.

The consensus tree extraction unit 115 that executes Algorithm 1 receives as input (trees) n different rhetorical structure trees for a certain document, and outputs subtrees (subtrees) common to these as a consensus tree.

The function AGREEMENT shown in the third line is a function that determines whether the subtree with the node span as the vertex is a consensus tree. If span is a leaf, AGREEMENT returns true (lines 4 and 5).

Otherwise, the function AGREEMENT makes the value of S _c true when the frequency of the node span is n (lines 7 and 8), and false otherwise (line 10). Note that the frequency is the number of corresponding nodes span in the target n rhetorical structure trees.

Then, the values of AGREEMENT for the left child node and right child node of node span are stored in S _l and S _r , respectively (11th line and 12th line). When S _c , S _l , and S _r are all true, AGREEMENT returns true; otherwise, it returns false (lines 13 to 17).

The function FINDROO on the second line (and lines 18 to 33) generates a subtree by adding nodes (spans) where S is True, and outputs it.

An agreement tree is obtained by sequentially applying the function AGREEMENT to any one of the n rhetorical structure trees in a depth-first search starting from its root node. However, in order to control the size of the subtree, constraints of a minimum value l _min and a maximum value l _max are introduced for the number of leaves (lines 21 to 26).

Referring to FIG. 5, an example of the operation of the consensus tree extraction unit 115 that operates according to Algorithm 1 in the case of n=2 will be described. First, the frequency of nodes is counted for the two trees shown on the left side of FIG. 5, taking into account the labels. Note that each node represents a span, that is, the start and end index of the EDU that it controls.

On the left side of FIG. 5, the nodes that are common between the two rhetorical structure trees are the underlined nodes. The rhetorical structure tree at the top left of FIG. 5 is taken out and Algorithm 1 is applied. On the right side of FIG. 5, S _c , S _l , S _r , and S at each node are shown when the upper left rhetorical structure tree is taken out and Algorithm 1 is applied.

AGREEMENT(1,10) sets S _c (1,10) to True because the frequency of span (1,10) is 2, and the left child (1,4) and right child (5,10 ) to apply AGREEMENT.

For each, by recursively applying AGREEMENT and determining S _c , we arrive at a leaf of the rhetorical structure tree (a span with equal start and end indices), so the value of S is set to True. In other words, in Algorithm 1, it is determined recursively whether there is a span that appears in common in a plurality of rhetorical structure trees. Then, the value of S of each node is determined based on the already set S _c and the values of S _l and S _r determined from the left and right nodes. The arrow line from bottom to top in the rhetorical structure tree on the right side of FIG. 5 shows how the value of S for each node is determined in sequence. For a certain node, a subtree in which S of that node and S of all nodes below that node are True becomes a consensus tree with that node as the vertex.

As shown on the right side of FIG. 5, in this example, two subtrees are extracted as a consensus tree, one having the span (1, 4) as the vertex and the other having the span (5, 7) as the vertex. For example, S=False for span (5, 10) and span (8, 10), so the subtree with these as vertices does not become a consensus tree. However, only the largest consensus tree in an inclusive relationship is extracted.

(About the pre-learning section 120)
Next, a configuration example and processing contents of the pre-learning section 120 will be explained. The pre-learning unit 120 receives the pseudo-correct data from the pseudo-correct data generating unit 110, and learns the parameters of the rhetorical structure analyzer based on the neural network model. In this embodiment, although any neural rhetorical structure analyzer may be used, parameter optimization of a rhetorical structure analyzer based on a neural network model will be explained using the technique disclosed in Non-Patent Document 1 as an example. do. It is assumed that the parameters are randomly initialized in the pre-learning.

FIG. 6 shows a configuration example of the pre-learning section 120. As shown in FIG. 6, the pre-learning unit 120 includes a tree structure estimation unit 121, a label estimation unit 122, and a parameter optimization unit 123. The processing contents of each part will be explained below.

<Tree structure estimation unit 121>
The tree structure estimation unit 121 estimates the tree structure by estimating the division points of the span. For any span (a series of EDUs consisting of the i-th EDU to the j-th EDU), the score s _split (i; j; k) at which the span is divided by the k-th EDU is given by the following formula.

ここで、Ｗ_ｕは重み行列であり、ｖ_ｌ（添字ｌはＬの小文字）とｖ_ｒはそれぞれ分割された左右のスパンに対する重みベクトルである。ｈ_ｉ：ｋとｈ_{ｋ＋１：ｊ}は以下で定義される。

Here, W _u is a weight matrix, and v _l (subscript l is a lowercase letter L) and v _r are weight vectors for the divided left and right spans, respectively. h _i:k and h _k+1:j are defined below.

h _i:k = MLP _left (u _i:k ), h _k+1:j = MLP _right (u _k+1:j )
MLP _* in the above formula represents a multilayer perceptron. The vector representation of the span u _i:j is obtained by inputting the word vector into the LSTM. As shown in equation (2) below, the span is divided by k that maximizes equation (1) below.

＜ラベル推定部１２２＞
ラベル推定部１２２は、木構造推定部１２１が決定したスパンの分割点ｋに対し、分割した２つのスパンに対する核性、修辞関係ラベルを予測する。予測のスコアは以下の式で与えられる。

<Label estimation unit 122>
The label estimating unit 122 predicts the core nature and rhetorical relationship labels for the two divided spans at the span dividing point k determined by the tree structure estimating unit 121. The prediction score is given by the following formula:

Ｗ_ｌ（添字ｌはＬの小文字）は重み行列であり、ｕ_１：ｉ；ｕ_ｊ：ｎはそれぞれｉ番目のＥＤＵの左側のスパンのベクトル表現、ｊ番目のＥＤＵの右側のスパンのベクトル表現である。最終的に、以下の式（４）で式（３）を最大にするラベルが与えられる。

W _l (the subscript l is a lowercase letter L) is a weight matrix, and u _1:i ; u _j:n are the vector representations of the left span of the i-th EDU and the vector representation of the right span of the j-th EDU, respectively. It is. Finally, the label that maximizes equation (3) is given by equation (4) below.

Ｌは、ラベル集合であり核性ラベルを付与する場合には３種のラベルからなる集合｛Ｎ－Ｓ、Ｓ－Ｎ、Ｎ－Ｎ｝であり、修辞ラベルを付与する場合には１８種のラベルからなる集合｛Ｅｌａｂｏｒａｔｉｏｎ，Ｃｏｎｄｉｔｉｏｎ，......｝である。なお、Ｗ_ｌとＭＬＰは核性ラベルを与える場合と修辞ラベルを与える場合とで独立に最適化される。

L is a label set, which is a set of three types of labels {NS, SN, N-N} when a nuclear label is assigned, and a set of 18 types when a rhetorical label is assigned. A set of labels {Elaboration, Condition, ...}. Note that W _l and MLP are independently optimized for the case of giving a nuclear label and the case of giving a rhetorical label.

<Parameter optimization unit 123>
The parameter optimization unit 123 minimizes all the parameters to be learned, that is, the parameters of W _u , W _l , v _r , v _l , LSTM, and MLP by the sum of two loss functions defined below. get it by doing that. Note that k ^* and l ^* (l is a lowercase letter L) are the division positions and labels of the correct answer (here, pseudo-correct data), respectively.

損失関数を最小化する演算については、誤差逆伝搬法等の既存手法を用いて行うことができる。そして、パラメタ最適化部１２３は、最適化したパラメタをファインチューニング部１３０へ渡す。

The calculation to minimize the loss function can be performed using existing methods such as error backpropagation. Then, the parameter optimization unit 123 passes the optimized parameters to the fine tuning unit 130.

<Fine tuning section 130>
The fine tuning unit 130 re-optimizes the parameters of the neural rhetorical structure analyzer using the correct data (annotated data created manually) with the parameters optimized by the pre-learning unit 120 as initial values. The fine tuning unit 130 outputs optimized parameters.

The configuration of the fine tuning section 130 is the same as the configuration of the pre-learning section 120 shown in FIG. However, the fine tuning unit 130 uses the parameters optimized by the pre-learning unit 120 as the initial values of the parameters, and uses manually created annotated data as the correct data, rather than pseudo-correct data. is different from the pre-learning section 120.

(Example of device hardware configuration)
The parameter optimization device 100, the pseudo-correct data generation unit 110, the pre-learning unit 120, and the fine-tuning unit 130 (collectively referred to as the “device”) are all implemented by, for example, a computer, which will be described in this embodiment. This can be achieved by executing a program that describes the processing contents.

The above program can be recorded on a computer-readable recording medium (such as a portable memory) and can be stored or distributed. It is also possible to provide the above program through a network such as the Internet or e-mail.

FIG. 7 is a diagram showing an example of the hardware configuration of the computer. The computer in FIG. 7 includes a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, etc., which are interconnected by a bus BS.

A program for realizing processing by the computer is provided, for example, by a recording medium 1001 such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily need to be installed from the recording medium 1001, and may be downloaded from another computer via a network. The auxiliary storage device 1002 stores installed programs as well as necessary files, data, and the like.

The memory device 1003 reads the program from the auxiliary storage device 1002 and stores it when there is an instruction to start the program. The CPU 1004 implements functions related to the device according to programs stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network. A display device 1006 displays a GUI (Graphical User Interface) or the like based on a program. The input device 1007 is composed of a keyboard, a mouse, buttons, a touch panel, or the like, and is used to input various operation instructions. An output device 1008 outputs the calculation result.

(Effects of embodiment)
As explained above, in this embodiment, a common subtree from the results of analyzing unlabeled data using a plurality of rhetorical structure analyzers is used as pseudo-correct data, and the pseudo-correct data is used to analyze the neural rhetorical structure analyzer. We decided to optimize the parameters of the neural rhetorical structure analyzer by optimizing the parameters through pre-learning, and using the parameters as initial values and fine-tuning using the ground truth data.

As a result, better parameters can be obtained than when optimization is performed using only correct data (which cannot be prepared in large quantities) from random initial values, and the performance of the neural rhetorical structure analyzer is improved. In other words, it is possible to realize a rhetorical structure analyzer with good performance without using a large amount of correct answer data that has been annotated with rhetorical structure.

(Summary of embodiments)
This specification discloses at least a parameter optimization device, a parameter optimization method, and a program described in each of the following sections.
(Section 1)
a pseudo-correct data generation unit that generates, as pseudo-correct data, a subtree common to the plurality of rhetorical structure trees obtained from the unlabeled data by the plurality of rhetorical structure analyzers;
a pre-learning unit that optimizes parameters of a rhetorical structure analyzer using a neural network using the pseudo-correct data generated by the pseudo-correct data generating unit;
A parameter optimization device comprising: a fine-tuning unit that fine-tunes the parameters optimized by the pre-learning unit using correct data.
(Section 2)
Parameter optimization according to item 1, wherein the pseudo-correct data generation unit generates the pseudo-correct data by recursively determining whether there is a span that appears in common in the plurality of rhetorical structure trees. Device.
(Section 3)
A parameter optimization method executed by a parameter optimization device,
a pseudo-correct data generation step of generating, as pseudo-correct data, a subtree common to the plurality of rhetorical structure trees obtained from the unlabeled data by the plurality of rhetorical structure analyzers;
a pre-learning step of optimizing parameters of a rhetorical structure analyzer using a neural network using the pseudo-correct data generated in the pseudo-correct data generation step;
A parameter optimization method comprising: a fine-tuning step of fine-tuning the parameters optimized in the pre-learning step using correct data.
(Section 4)
A program for causing a computer to function as each part of the parameter optimization device according to item 1 or 2.

Although the present embodiment has been described above, the present invention is not limited to such specific embodiment, and various modifications and changes can be made within the scope of the gist of the present invention as described in the claims. It is possible.

100 Parameter optimization device 110 Pseudo-correct data generation section 115 Consensus tree extraction section 120 Pre-learning section 121 Tree structure estimation section 122 Label estimation section 123 Parameter optimization section 130 Fine tuning section 1000 Drive device 1001 Recording medium 1002 Auxiliary storage device 1003 Memory Device 1004 CPU
1005 Interface device 1006 Display device 1007 Input device 1008 Output device

Claims (4)

a pseudo-correct data generation unit that generates, as pseudo-correct data, a subtree common to the plurality of rhetorical structure trees obtained from the unlabeled data by the plurality of rhetorical structure analyzers;
a pre-learning unit that optimizes parameters of a rhetorical structure analyzer using a neural network using the pseudo-correct data generated by the pseudo-correct data generating unit;
A parameter optimization device comprising: a fine-tuning unit that fine-tunes the parameters optimized by the pre-learning unit using correct data. Parameter optimization according to claim 1, wherein the pseudo-correct data generation unit generates the pseudo-correct data by recursively determining whether there is a span that appears in common in the plurality of rhetorical structure trees. Device. A parameter optimization method executed by a parameter optimization device, comprising:
a pseudo-correct data generation step of generating, as pseudo-correct data, a subtree common to the plurality of rhetorical structure trees obtained from the unlabeled data by the plurality of rhetorical structure analyzers;
a pre-learning step of optimizing parameters of a rhetorical structure analyzer using a neural network using the pseudo-correct data generated in the pseudo-correct data generation step;
A parameter optimization method comprising: a fine-tuning step of fine-tuning the parameters optimized in the pre-learning step using correct data. A program for causing a computer to function as each part of the parameter optimization device according to claim 1 or 2.

Images