JP7457332B2 - Tree structure estimation device, parameter learning device, tree structure estimation method, parameter learning method, and program - Google Patents

Description

The present invention relates to the fields of computer vision, which automatically processes video using a computer, and natural language processing, which automatically processes text, and in particular to a technology that divides video into events (scenes), gives them captions, and represents the relationships between them as a tree structure.

In the field of natural language processing, discourse structure analysis technology has been developed that represents an entire document as a tree structure. In particular, technology has been developed that represents a document as a tree structure based on rhetorical structure theory.

Rhetorical structure is a tree representation of the topic structure of a text, and this structure exists not only in text but also in videos.

In other words, videos can be represented as a tree in which leaves represent event segments (scenes) and their captions, nodes represent the nuclear roles of spans (scene sequences), and edges represent rhetorical relationships between spans. However, unlike in the case of text, the leaves of the tree are video segments and caption sentences, so there is no need to consider the structure within the sentences.

In order to obtain such a structure, in the technique disclosed in Non-Patent Document 1, a caption sentence is generated for a video, and a conventional rhetorical structure analysis technique is applied to the obtained caption to obtain a tree structure. Then, caption sentences and video frames are associated using LSTM.

Arjun R Akula, Song-Chun Zhu: Visual Discourse Parsing, ,CVPR 2019 Workshop on Language and Vision, (2019)

In the prior art, rhetorical structure trees are constructed using only textual information. However, when representing a video as a rhetorical structure tree, the text does not necessarily cover the entire video section corresponding to the scene, so it is not possible to simply use the text to determine the structure and relationships between scenes. Not necessarily enough. In particular, when determining a tree structure, the similarity between scenes is an important factor, but it is often difficult to accurately capture the similarity using text alone.

The present invention has been made in view of the above points, and it is an object of the present invention to provide a technique for generating a rhetorical structure tree that appropriately reflects the similarity between scenes from a video.

According to the disclosed technology, a feature extraction unit extracts, from an input video, a video vector that is a feature amount of the video and a text vector that is a feature amount of a text attached to the video;
a vector synthesis unit that generates a composite vector by synthesizing the video vector and the text vector;
A tree structure estimating device is provided, comprising: a tree structure representing a rhetorical structure tree of the video and a tree structure estimator that estimates a label from the composite vector.

According to the disclosed technology, a technology for generating a rhetorical structure tree that appropriately reflects the similarity between scenes from a video is provided.

FIG. 2 is a diagram showing an example of a rhetorical structure tree. FIG. 3 is a diagram showing an example of a rhetorical structure tree applied to a video. FIG. 1 is a configuration diagram of a video discourse structure analysis device. FIG. 3 is a configuration diagram of a data input section. FIG. 3 is a configuration diagram of a parameter learning section. FIG. 3 is a diagram showing an outline of the operation of a parameter learning section. It is a block diagram of a tree structure estimation part. It is a diagram showing an example of the hardware configuration of the device.

DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention (this embodiment) will be described below with reference to the drawings. The embodiments described below are merely examples, and embodiments to which the present invention is applied are not limited to the following embodiments.

(About rhetorical structure trees)
First, an example of a conventional rhetorical structure tree will be explained. In rhetorical structure theory, a document is expressed as a binary tree (rhetorical structure tree). A rhetorical structure tree is obtained by recursively repeating the operation of connecting a series of EDUs (hereinafter referred to as spans), which are the smallest basic units of discourse that make up the tree, through rhetorical relationships to form larger spans. It's a tree.

A leaf of a tree is a unit of EDU (corresponding to a node), and a node of the tree is given a nuclearity label of the span it dominates. One of the two spans (sibling spans) that will be combined will become a core containing important information, and the other will become a satellite that supplements it. In exceptional cases, both sides may be core. Tree branches are given relational labels that represent the rhetorical relationships between spans. Eighteen types of relational labels representing rhetorical relations are defined.

Figure 1 shows an example of a rhetorical structure tree. In the figure, e ₁ to e ₇ are EDUs, S/N is the nuclearity label of the span (N is the nucleus, S is the satellite), Condition, Elaboration, etc. are the relationship labels between the sibling spans. When the nuclearity of the sibling spans is a combination of S and N, the relationship label is given to the span on the S side, and when it is N and N, it is given to both spans. Condition and Elaboration are given to the combination of S and N, and List and Same-Unit are given to the combination of N and N.

(Overview of the embodiment)
As mentioned above, rhetorical structures exist not only in texts but also in videos, and videos can be represented as trees in which leaves represent events (scenes) and their captions, nodes represent the nuclear roles of spans (scene sequences), and edges represent the rhetorical relationships between spans. For example, the rhetorical structure of a video and the relationships between spans can be represented by a rhetorical structure tree in which the leaves (e) in the rhetorical structure tree shown in Figure 1 are replaced with tuples of scenes and captions.

FIG. 2 shows an example in which a video is represented as a rhetorical structure tree using a tree structure and labels. In the example shown in FIG. 2, the numbers in "[" and "]" represent the start and end times of the video represented by the scene, and c represents the caption text corresponding to the scene.

As mentioned above, the conventional technology disclosed in Non-Patent Document 1 uses only text to determine the structure and relationships between scenes, and therefore often fails to capture the similarities between scenes well. Therefore, in this embodiment, the rhetorical structure and relationship estimation of scenes is performed using not only text information but also image information, making it possible to generate a rhetorical structure tree that appropriately reflects the similarities between scenes.

(Device configuration example, operation overview)
FIG. 3 shows a configuration example of the video discourse structure analysis device 100 in this embodiment. As shown in FIG. 3, the video discourse structure analysis device 100 includes a data input section 110, a tree structure estimation section 120, a parameter learning section 130, and a data output section 140.

The video discourse structure analysis device 100 may be implemented on one computer or on multiple computers. Furthermore, some or all of the functions of the video discourse structure analysis device 100 may be implemented in a virtual machine on the cloud. The data input unit 110, the tree structure estimation unit 120, the parameter learning unit 130, and the data output unit 140 may be implemented as separate devices. It can also be called a device. Furthermore, "data input unit 110 + tree structure estimation unit 120 + data output unit 140" may be referred to as a tree structure estimation device.

The process flow is also shown in FIG. 3. As shown in FIG. 3, the data input unit 110 receives a video, generates a span vector from the video, and passes the generated span vector to the tree structure estimation unit 120.

The parameter learning unit 130 receives the video with the annotations shown in FIG. 2, that is, the annotations for the scene (video section time and caption) and the annotation for the rhetorical structure (tree structure and labels), and performs annotation based on the annotated video. , learns parameters for rhetorical tree structure estimation, coreity label estimation, and relational label estimation using a neural network, and passes the learned parameters to the tree structure estimation unit 120. The tree structure and labels given as annotations are used as correct answer data during parameter learning.

The tree structure estimation unit 120 receives the span vector from the data input unit 11, receives the parameters from the parameter learning unit 130, uses these to estimate scene division points and scene labels, and estimates the scene division points, and passes the scene label to the data output unit 140.

The data output unit 140 receives the scene segmentation points and labels from the tree structure estimation unit 120, and outputs the tree using, for example, an S-expression.

In this embodiment, it is assumed that the data input section 110, the tree structure estimation section 120, the parameter learning section 130, and the data output section 140 are all configured by a neural network. The configuration and processing contents of each part will be explained in detail below.

(Data Input Unit 110)
The data input unit 110 receives as input a video for which a labeled tree is to be generated, and passes a span vector corresponding to a scene sequence to the tree structure estimation device 120. Fig. 4 shows the functional configuration of the data input unit 110. As shown in Fig. 4, the data input unit 110 includes a caption generation unit 111 and a span vector generation unit 112. The processing contents of each unit are as follows.

<Caption generation unit 111>
The caption generation unit 111 uses video caption technology to identify each scene in the input video and generates a caption for the identified scene. In other words, it gives the start and end times and captions for each scene. Note that existing technology may be used for video captioning.

<Span vector generation unit 112>
The span vector generation section 112 has the same video feature extraction section, text feature extraction section, and vector synthesis section as those included in the parameter learning section 130, which will be described later. Details of the video feature extraction section, text feature extraction section, and vector synthesis section will be explained when the parameter learning section 130 is explained. The parameters optimized by the parameter learning unit 130 are used for the parameters of the video feature extraction unit, text feature extraction unit, and vector synthesis unit.

The start time and end time of the scene generated by the caption generation unit 111 and the caption text for the scene are input to each of the video feature extraction unit and the text feature extraction unit. The feature vectors output by each of the video feature extraction section and the text feature extraction section are passed to the vector synthesis section, and the vector synthesis section generates a span vector.

(Parameter Learning Unit 130)
Fig. 5 shows the functional configuration of the parameter learning unit 130. As shown in Fig. 5, the parameter learning unit 130 has a feature extraction unit 131, a vector synthesis unit 134, a tree structure estimation processing unit 135, a label estimation unit 136, and a parameter optimization unit 137. The feature extraction unit 131 has a video feature extraction unit 132 and a text feature extraction unit 133.

FIG. 6 shows an outline of the operation of the parameter learning section 130. The annotated video is provided to each of the video feature extraction unit 132 and the text feature extraction unit 133. An annotated video is annotated with data about scenes in the video (start/end times and captions), division points of spans (scene series), coreity labels of spans, and labels of rhetorical relationships between spans. This is a video.

The video vector output from the video feature extraction section 132 and the text vector output from the text feature extraction section 133 are input to the vector synthesis section 134, and the vector synthesis section 134 synthesizes them to generate a span vector. Based on the span vector, the tree structure estimation processing unit 135, label estimation unit 136, and parameter optimization unit 137 output parameters for estimating the tree structure, coreness, and relationship in the neural network. The processing contents of each part will be explained in detail below.

<Video feature extraction unit 132>
The video feature extraction unit 132 extracts video vectors corresponding to each scene from the annotated video using vectors for frames in the video (for example, feature vectors for each frame obtained by a method such as C3D or I3D) and LSTM. It is generated by

For example, if the start time of a certain scene is 0:10 and the end time is 1:00, vectors corresponding to all frames dominated by that section are input to the forward LSTM and backward LSTM.

When a scene is from 0:10 to 1:00, the number of frames included in that section is n, and the vector (feature vector obtained by C3D or the like) corresponding to the j-th frame is denoted as _vj . The forward LSTM is denoted as ^→ _LSTMf , and the backward LSTM is denoted as ^← _LSTMf . For convenience of description in this specification, LSTMs with an arrow above them are written as " ^→ _LSTMf " and " ^← _LSTMf " in the text of this specification. Similarly, " ^→ " and " ^← " are used for other characters.

Here, using the forward and backward hidden states for the j-th frame, the hidden state h ^v _j of the j-th frame is expressed by the following equation. Note that [ ^→ h ^v _j ; ^← h ^v _j ] indicates the connection of ^→ h ^v _j and ^← h ^v _j .

そして、シーンに対応する動画区間のベクトルをＶ＝［ｈ^ｖ _１；ｈ^ｖ _ｎ］とする。動画特徴抽出部１３２は、各シーンに対する動画ベクトルを出力する。なお、［ｈ^ｖ _１；ｈ^ｖ _ｎ］は、ｈ^ｖ _１とｈ^ｖ _ｎの連結を示す。

Then, let the vector of the video section corresponding to the scene be V=[h ^v ₁ ; h ^v _n ]. The video feature extraction unit 132 outputs a video vector for each scene. Note that [h ^v ₁ ; h ^v _n ] indicates a connection between h ^v ₁ and h ^v _n .

<Text feature extraction unit 133>
The text feature extraction unit 133 generates a text vector corresponding to the scene caption from the annotated video using the word embedding vector included in the sentence and LSTM.

The text feature extraction unit 133 obtains embedding vectors for all words included in the caption sentence, and then inputs them to the forward and backward LSTMs. Similar to the video feature extraction unit 132, the hidden state h ^w _j of the j-th word is expressed by the following equation using the hidden state by forward LSTM and the hidden state by backward LSTM.

そして、ｋ単語からなる文全体のベクトル表現をＳ＝［ｈ^ｗ _１；ｈ^ｗ _ｋ］とする。なお、［ｈ^ｗ _１；ｈ^ｗ _ｋ］は、ｈ^ｗ _１とｈ^ｗ _ｋの連結を示す。テキスト特徴抽出部１３３は、各シーンのキャプションについてのテキストベクトルを出力する。なお、式（２）におけるｗは単語埋め込みベクトルである。

Then, let S=[h ^w ₁ ; h ^w _k ] be the vector representation of the entire sentence consisting of k words. Note that [h ^w ₁ ; h ^w _k ] indicates the connection of h ^w ₁ and h ^w _k . The text feature extraction unit 133 outputs a text vector for the caption of each scene. Note that w in equation (2) is a word embedding vector.

<Vector synthesis unit 134>
The vector synthesis unit 134 first synthesizes, for each scene, a video vector V for the scene and a text vector S corresponding to its caption to generate a scene vector. Now, for the hidden state h ^w _j of the j-th word of the caption corresponding to the scene, the video vector V, and the text vector S, a new hidden state h ^{' w} _j of the j-th word is calculated as follows using a selective gate. Defined by the formula.

Ｗ_Ｓ、Ｕ_Ｓ、Ｕ_Ｖは重み行列であり、ｂ_Ｓはバイアスベクトルである。σはシグモイド関数を表す。〇の中に・を記載した記号はアダマール積を表す。そして、式（３）を用いて、シーンｉのシーンベクトルを以下の式（４）で定義する。

_Ws , _Us , and _Uv are weight matrices, and _bs is a bias vector. σ represents a sigmoid function. The symbol with a dot inside a circle represents the Hadamard product. Using equation (3), the scene vector of scene i is defined by the following equation (4).

次にこれを前向き、後ろ向きＬＳＴＭへ入力し以下の隠れ状態を得る。

Next, input this to the forward and backward LSTMs to obtain the following hidden state.

上記の式におけるｆ、ｂは、前向き、後ろ向きＬＳＴＭの隠れ状態を表す。最終的に、ｍ番目のシーンからｎ番目までのシーン系列に対応するスパンベクトルを以下の式で定義する。

f and b in the above equation represent the hidden states of the forward and backward LSTMs. Finally, the span vector corresponding to the scene series from the m-th scene to the n-th scene is defined by the following equation.

なお、シーンベクトルについては上記の手順をふまず、単純にＶとＳを結合するだけでも良い。つまり、以下の式を用いても良い。スパンベクトルを得る手続については上記と同様であり、式（５）、式（６）により得られる。

Note that for the scene vector, it is also possible to simply combine V and S without following the above procedure. In other words, the following formula may be used. The procedure for obtaining the span vector is the same as above, and can be obtained using equations (5) and (6).

＜木構造推定処理部１３５＞
木構造推定処理部１３５は、スパンの分割点を推定することで木構造を推定する。任意のスパン（ｉ番目のシーンからｊ番目のシーンからなるシーンの系列）に対しｋ番目のシーンでスパンが分割されるスコアｓ_{ｓｐｌｉｔ}（ｉ；ｊ；ｋ）を以下の式で与える。

<Tree structure estimation processing unit 135>
The tree structure estimation processing unit 135 estimates the tree structure by estimating the division points of the span. A score s _split (i; j; k) at which the span is divided at the k-th scene for an arbitrary span (a sequence of scenes consisting of the i-th scene to the j-th scene) is given by the following formula.

ここで、Ｗ_ｕは重み行列であり、ｖ_ｌ（添字ｌはＬの小文字）とｖ_ｒはそれぞれ分割された左右のスパンに対する重みベクトルである。ｈ_ｉ：ｋとｈ_{ｋ＋１：ｊ}は以下で定義される。

where _Wu is a weight matrix, _vl (the subscript l is a lowercase L) and _vr are weight vectors for the divided left and right spans, respectively, _{and hi:k} and _hk+1:j are defined as follows:

h _i:k = MLP _left (u _i:k ), h _k+1:j = MLP _right (u _k+1:j ),
MLP _* stands for multilayer perceptron. The span vector u _i:j is a vector obtained by the vector synthesis unit 134. The span is divided by k, which maximizes equation (8), as shown in the equation below.

＜ラベル推定部１３６＞
ラベル推定部１３６は、木構造推定処理部１３６が決定したスパンの分割点ｋに対し、分割した２つのスパンに対する核性ラベル、修辞関係ラベルを予測する。予測のスコアは以下の式で与えられる。

<Label estimation unit 136>
The label estimating unit 136 predicts a core label and a rhetorical relationship label for the two divided spans at the span dividing point k determined by the tree structure estimation processing unit 136. The prediction score is given by the following formula:

上記の式におけるＷ_ｌ（添字ｌはＬの小文字）は重み行列であり、ｕ_１：ｉ；ｕ_ｊ：ｎはそれぞれｉ番目のシーンの左側のスパンベクトル、ｊ番目のシーンの右側のスパンベクトルである。最終的に、以下の式で式（１０）を最大にするラベルを与える。

In the above equation, W _l (the subscript l is a lowercase letter L) is a weight matrix, and u _1:i ; u _j:n are the left span vector of the i-th scene and the right span vector of the j-th scene, respectively. It is. Finally, the following formula gives a label that maximizes formula (10).

Ｌは、ラベル集合であり核性ラベルを付与する場合には３種のラベルからなる集合｛Ｎ－Ｓ，Ｓ－Ｎ，Ｎ－Ｎ｝となり、修辞関係ラベルを付与する場合には以下の１８種のラベルからなる集合となる。ラベル：ｅｌａｂｏｒａｔｉｏｎ、ｊｏｉｎｔ、ａｔｔｒｉｂｕｔｉｏｎ、ｓａｍｅ－ｕｎｉｔ、ｃｏｎｔｒａｓｔ、ｔｅｍｐｏｒａｌ、ｂａｃｋｇｒｏｕｎｄ、ｅｘｐｌａｎａｔｉｏｎ、ｃａｕｓｅ、ｅｖａｌｕａｔｉｏｎ、ｃｏｎｄｉｔｉｏｎ、ｅｎａｂｌｅｍｅｎｔ、ｔｏｐｉｃ－ｃｏｍｍｅｎｔ、ｃｏｍｐａｒｉｓｏｎ、ｓｕｍｍａｒｙ、ｍａｎｎｅｒ－ｍｅａｎｓ、ｔｏｐｉｃ－ｃｈａｎｇｅ、ｔｅｘｕａｌ－ｏｒｇａｎｉｚａｔｉｏｎ。

L is a label set, and when nuclear labels are assigned, it becomes a set of three types of labels {N-S, SN, NN}, and when rhetorical relation labels are assigned, it becomes a set of the following 18 types of labels: Labels: elaboration, joint, attribute, same-unit, contrast, temporal, background, explanation, cause, evaluation, condition, enablement, topical-comment, comparison, summary, manner-means, topical-change, textual-organization.

Note that W _l and MLP are optimized independently in the case of giving a nuclear label and the case of giving a rhetorical label.

<Parameter Optimization Unit 137>
The parameter optimization unit 137 obtains all parameters to be learned, i.e., _Ws , _US , _UV , _Wu , _Wl , _vr , _vl , LSTM, and MLP parameters, by minimizing the sum of two loss functions defined below. Note that k ^* and l ^* (l is a lowercase L) are the correct division position and label, respectively. The correct division position and label can be obtained from the annotation of the input annotated video. The calculation to minimize the loss function can be performed using an existing method such as backpropagation.

（木構造推定部１２０）
木構造推定部１２０は、パラメタ学習部１３０が出力するパラメタとデータ入力部１１２が出力するスパンベクトルを用いて木構造を推定する。図７に、木構造推定部１２０の機能構成を示す。図７に示すように、木構造推定部１２０は、木構造推定処理部１２１とラベル推定部１２２を備える。以下、各部について説明する。

(Tree structure estimation unit 120)
The tree structure estimating unit 120 estimates a tree structure using the parameters output by the parameter learning unit 130 and the span vectors output by the data input unit 112. FIG. 7 shows the functional configuration of the tree structure estimation section 120. As shown in FIG. 7, the tree structure estimation section 120 includes a tree structure estimation processing section 121 and a label estimation section 122. Each part will be explained below.

<Tree structure estimation processing unit 121>
The tree structure estimation processing section 121 is the same as the tree structure estimation processing section 135 in the parameter learning section 130. The tree structure estimation processing unit 121 receives as input a span vector corresponding to the entire video as an initial state, and obtains a tree structure by recursively dividing this into two. When the number of scenes is m, dividing points are determined by equation (9) by setting i=1; j=m in equation (8) using the parameters determined by the parameter learning unit 130. Repeat this recursively.

For example, assuming that the target video has the tree structure shown in Figure 2, it is first divided into spans c ₁ to c ₃ and spans c ₄ to c ₆ , and each divided span is It is divided as shown.

<Label estimation unit 122>
The label estimation unit 122 is also the same as the label estimation unit 136 of the parameter learning unit 130. The label estimation unit 122 receives the two span vectors divided into two by the tree structure estimation processing unit 121, and estimates a core label and a relational label, respectively. When estimating a nuclear label, it is classified into one of NS, SN, and N-N, and when estimating a rhetorical label, it is classified into one of 18 types of labels.

(Data output unit 140)
The data output unit 140 summarizes the span dividing points estimated by the tree structure estimation processing unit 121 and the label information of the span outputted by the label estimation unit 122, and outputs it as a labeled tree, for example, as an S expression.

(Example of device hardware configuration)
The video discourse structure analysis device 100 includes a data input unit 110, a tree structure estimation unit 120, a parameter learning unit 130, a data output unit 140, a data input device, a tree structure estimation device, a parameter learning device, and a data output device (these are collectively referred to as (hereinafter referred to as "device") can be realized, for example, by causing a computer to execute a program that describes the processing contents described in this embodiment.

The above program can be recorded on a computer-readable recording medium (such as a portable memory) and can be stored or distributed. It is also possible to provide the above program through a network such as the Internet or e-mail.

Figure 8 is a diagram showing an example of the hardware configuration of the computer. The computer in Figure 8 has a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, etc., all of which are interconnected by a bus BS.

A program for realizing processing by the computer is provided, for example, by a recording medium 1001 such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily need to be installed from the recording medium 1001, and may be downloaded from another computer via a network. The auxiliary storage device 1002 stores installed programs as well as necessary files, data, and the like.

The memory device 1003 reads the program from the auxiliary storage device 1002 and stores it when there is an instruction to start the program. The CPU 1004 implements functions related to the device according to programs stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network. A display device 1006 displays a GUI (Graphical User Interface) or the like based on a program. The input device 1007 is composed of a keyboard, a mouse, buttons, a touch panel, or the like, and is used to input various operation instructions. An output device 1008 outputs the calculation result.

(Effects of the embodiment)
As described above, in this embodiment, a video is input and a rhetorical structure tree can be output in which scenes are represented as leaves and relationships between scenes are represented as kernels and relationship labels. In particular, since the rhetorical structure and relationship inference of scenes is performed not only on the basis of text information but also on the basis of image information, a rhetorical structure tree that appropriately reflects the similarity between scenes can be generated from a video.

(Summary of embodiments)
This specification discloses at least the following items: a tree structure estimation device, a parameter learning device, a tree structure estimation method, a parameter learning method, and a program.
(Section 1)
a feature extraction unit that extracts, from an input video, a video vector that is a feature of the video and a text vector that is a feature of a text attached to the video;
a vector synthesis unit that generates a composite vector by synthesizing the video vector and the text vector;
A tree structure estimating device comprising: a tree structure estimation unit that estimates a tree structure representing a rhetorical structure tree of the video and a label from the composite vector.
(Section 2)
The leaves of the rhetorical structure tree correspond to scenes of the video and captions for the scenes, the labels of nodes correspond to the core nature of scene sequences, and the labels of edges correspond to rhetorical relationships between scene sequences. tree structure estimation device.
(Section 3)
The tree structure estimating device according to claim 1 or 2, wherein the feature extraction unit creates a video vector corresponding to the scene by inputting the feature vectors of each frame constituting the scene into LSTM.
(Section 4)
a feature extraction unit that extracts, from an input video, a video vector that is a feature of the video and a text vector that is a feature of a text attached to the video;
a vector synthesis unit that generates a composite vector by synthesizing the video vector and the text vector;
a tree structure estimation unit that estimates a tree structure and a label representing a rhetorical structure tree of the video based on the composite vector;
Parameter optimization for optimizing parameters of a neural network corresponding to the tree structure estimator using the tree structure and the label estimated by the tree structure estimator, and correct answer data representing the rhetorical structure tree of the video. A parameter learning device comprising a conversion section and.
(Section 5)
A tree structure estimation method executed by a tree structure estimation device, comprising:
a feature extraction step of extracting a video vector that is a feature amount of the video and a text vector that is a feature amount of a text attached to the video from the input video;
a vector compositing step of composing the video vector and the text vector to generate a composite vector;
A tree structure estimation method comprising: estimating a tree structure representing a rhetorical structure tree of the video and a label from the composite vector.
(Section 6)
A parameter learning method executed by a parameter learning device, comprising:
a feature extraction step of extracting a video vector that is a feature amount of the video and a text vector that is a feature amount of a text attached to the video from the input video;
a vector compositing step of composing the video vector and the text vector to generate a composite vector;
a tree structure estimation step of estimating a tree structure and a label representing a rhetorical structure tree of the video based on the composite vector;
parameter optimization for optimizing the parameters of a neural network that executes the tree structure estimation step using the tree structure and the label estimated in the tree structure estimation step, and correct answer data representing the rhetorical structure tree of the video; A parameter learning method comprising a transformation step and .
(Section 7)
A program for causing a computer to function as each part of the tree structure estimating device according to any one of items 1 to 3.
(Section 8)
A program for causing a computer to function as each part of the parameter learning device according to item 4.

Although the present embodiment has been described above, the present invention is not limited to such specific embodiment, and various modifications and changes can be made within the scope of the gist of the present invention as described in the claims. It is possible.

100 Video discourse structure analysis device 110 Data input unit 111 Caption generation unit 112 Span vector generation unit 120 Tree structure estimation unit 121 Tree structure estimation processing unit 122 Label estimation unit 130 Parameter learning unit 131 Feature amount extraction unit 132 Video feature extraction unit 133 Text Feature extraction unit 134 Vector synthesis unit 135 Tree structure estimation processing unit 136 Label estimation unit 137 Parameter optimization unit 140 Data output unit 1000 Drive device 1001 Recording medium 1002 Auxiliary storage device 1003 Memory device 1004 CPU
1005 Interface device 1006 Display device 1007 Input device 1008 Output device

Claims (8)

A feature extraction unit that extracts, from an input video, a video vector that is a feature of the video and a text vector that is a feature of text added to the video;
a vector synthesis unit that synthesizes the video vector and the text vector to generate a synthetic vector;
a tree structure estimation unit that estimates a tree structure representing a rhetorical structure tree of the video and a label from the composite vector. 2. A leaf of the rhetorical structure tree corresponds to a scene of the video and a caption for the scene, a node label corresponds to a core of a scene sequence, and an edge label corresponds to a rhetorical relationship between scene sequences. tree structure estimation device. The tree structure estimation device according to claim 1 or 2, wherein the feature extraction unit creates a video vector corresponding to the scene by inputting feature vectors of each frame constituting the scene into LSTM. a feature extraction unit that extracts, from an input video, a video vector that is a feature of the video and a text vector that is a feature of a text attached to the video;
a vector synthesis unit that generates a composite vector by synthesizing the video vector and the text vector;
a tree structure estimation unit that estimates a tree structure and a label representing a rhetorical structure tree of the video based on the composite vector;
Parameter optimization for optimizing parameters of a neural network corresponding to the tree structure estimator using the tree structure and the label estimated by the tree structure estimator, and correct answer data representing the rhetorical structure tree of the video. A parameter learning device comprising a conversion section and. A tree structure estimation method executed by a tree structure estimation device, comprising:
A feature extraction step of extracting from an input video a video vector that is a feature of the video and a text vector that is a feature of text added to the video;
a vector synthesis step of synthesizing the video vector and the text vector to generate a synthetic vector;
and a tree structure estimation step of estimating a tree structure and a label representing a rhetorical structure tree of the video from the composite vector. A parameter learning method executed by a parameter learning device, comprising:
a feature extraction step of extracting a video vector that is a feature amount of the video and a text vector that is a feature amount of a text attached to the video from the input video;
a vector compositing step of composing the video vector and the text vector to generate a composite vector;
a tree structure estimation step of estimating a tree structure and a label representing a rhetorical structure tree of the video based on the composite vector;
parameter optimization for optimizing the parameters of a neural network that executes the tree structure estimation step using the tree structure and the label estimated in the tree structure estimation step, and correct answer data representing the rhetorical structure tree of the video; A parameter learning method comprising a transformation step and . A program for causing a computer to function as each part of the tree structure estimating device according to any one of claims 1 to 3. A program for causing a computer to function as each part of the parameter learning device according to claim 4.

Images