JP2023103966A - Method and device for constructing transformer model for video story question answering - Google Patents

Abstract

To provide a device, a method and a program for constructing a transformer model for video story question answering, which learn a video story by considering the context of prior and subsequent video clips included in video data.SOLUTION: The device comprises: an input-output unit for receiving video data, which includes a plurality of consecutive video clips, and question data for video question answering, and outputting the result from video story question answering; a storage unit in which a program and data for performing the video story question answering are stored; and a control unit that includes at least one processor and constructs a transformer model for the video story question answering by executing the program. The control unit learns a video story from the video data including the plurality of consecutive video clips by considering the context of neighboring prior and subsequent video clips according to chronological order.SELECTED DRAWING: Figure 3

Description

Patent Act Article 30, Paragraph 2 application filed Dec. 20, 2021 Korea Software Congress 2021 (Proceedings of the 2021 Korean Software Comprehensive Conference of the Korean Institute of Information Science and Technology) "270. Multimodal context for video Q&A Transformers (pages 801-803)”

Embodiments disclosed herein relate to an apparatus and method for constructing a transformer model for video story question answering, and more particularly, to constructing a transformer model for video story question answering. An apparatus and method for building a transformer model for story learning video story question answering.

This research is a part of the innovative growth engine project project of the Ministry of Science, Technology and Information Communication and the Institute of Information and Communication Planning and Evaluation (IITP). -005)” and “(SW Star Love) Cognitive Agent SW Development for Daily Life Learning Base (IITP-2015-0-00310-007)” of the SW Computing Industry Source Technology Development Project. .

Recently, thanks to the development of deep learning techniques for vision and natural language processing, interest in multimodal data has increased, and many forms of tasks that measure understanding of video data have attracted attention.

Among various forms of tasks, Video Question Answering measures video comprehension ability by the accuracy of five-choice objective questions in natural language form. In particular, in order to solve video question answering, it is necessary to learn complex correlations of various data appearing in multimodal videos and search for core information about given question answers.

Recently, in order to solve this problem, a large-scale learning model based on a transformer has been introduced. Large-scale dictionary learning for videos has shown considerable performance on a variety of tasks for evaluating video comprehension and is built on models that have performed well in natural language processing. However, according to the prior art, in the case of a transformer that performs video story question answering, the computational cost increases exponentially as the length of the video increases. However, there was a problem that it was possible to process only

On the other hand, the above-mentioned background art is technical information that the inventor has for the derivation of the present invention or acquired during the derivation process of the present invention, and is not necessarily a publicly known technology disclosed to the general public before the filing of the present invention. I cannot say that it is.

Korean Patent Publication No. 10-2020-0144417

Embodiments disclosed herein provide an apparatus and method for building a transformer model for video story question answering that learns video stories by considering the context before and after video clips contained in the video data. It has a purpose.

Other objects and advantages of the present invention can be comprehended by the following description and will be made clearer by an embodiment. Also, it will be readily understood that the objects and advantages of the present invention can be achieved by means and combinations thereof as set forth in the appended claims.

As a technical means for achieving the above technical problems, an apparatus for constructing a transformer model for video story question answering includes video data including a plurality of continuous video clips and question data for video question answering. an input/output unit for receiving and outputting the result of the video story question answering; a storage unit for storing programs and data for performing the video story question answering; and at least one processor for executing the program. a controller for constructing a transformer model for video story question answering, wherein the controller extracts contexts of preceding and succeeding video clips adjacent to each other in temporal order from the video data including the plurality of consecutive video clips. is characterized by learning a video story in consideration of

According to another embodiment, a method of constructing a transformer model for video story question answering performed by an apparatus for constructing a transformer model for video story question answering comprises video data comprising a plurality of consecutive video clips; and receiving question data for video question answering; and learning a video story from the video data including the plurality of consecutive video clips by considering contexts of preceding and succeeding video clips adjacent to each other in temporal order. including.

According to yet another embodiment, the recording medium is a computer-readable recording medium having recorded thereon a program for executing the method of constructing a transformer model for video story question answering. A method for constructing a transformer model for video story question answering performed by a device for constructing a transformer model for video story question answering includes video data including a plurality of continuous video clips and a question for video question answering. receiving data; and learning a video story from the video data including the plurality of consecutive video clips considering the context of preceding and succeeding video clips that are adjacent to each other in temporal order.

According to yet another embodiment, a computer program is executed by an apparatus for building a transformer model for video story question answering and a recording medium for performing the method for building a transformer model for video story question answering. A computer program recorded in A method for constructing a transformer model for video story question answering performed by a device for constructing a transformer model for video story question answering includes video data including a plurality of continuous video clips and a question for video question answering. receiving data; and learning a video story from the video data including the plurality of consecutive video clips considering the context of preceding and succeeding video clips that are adjacent to each other in temporal order.

According to any one of the above-described problem-solving means, a large computational cost is incurred in constructing a transformer considering the context before and after the video clip included in the video data and performing the video story question answering. Although it is not, it has the effect of being able to process long videos effectively.

According to another of the above-mentioned problem-solving means, by constructing a transformer that considers the context before and after the video clip included in the video data, not only the video story question answering but also the subsequent video It has the effect of being able to be used in various fields such as scene prediction and causal relationship inference.

The effects obtained in the disclosed embodiments are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those having ordinary knowledge in the technical field to which the embodiments disclosed in the following description belong. It would be possible.

The accompanying drawings below illustrate the preferred embodiments disclosed in this specification, and serve to make the technical ideas disclosed in this specification more comprehensible together with the specific contents for carrying out the invention. Therefore, the contents disclosed in this specification should not be construed as being limited only to the matters described in the drawings.

It is a figure for demonstrating the transformer model by a prior art. FIG. 4 is a diagram for explaining a transformer model according to one embodiment; 1 is a functional block diagram of an apparatus for building a transformer model for video story question answering according to one embodiment; FIG. 4 is a flow chart illustrating a method of building a transformer model for video story question answering according to one embodiment;

Various exemplary embodiments are described in detail below with reference to the accompanying drawings. The embodiments described below can be modified and implemented in various different forms. In order to more clearly describe the features of the embodiments, detailed descriptions of matters widely known to those of ordinary skill in the technical field to which the following embodiments belong are omitted. In the drawings, parts that are not related to the description of the embodiments are omitted, and similar parts are given similar reference numerals throughout the specification.

Throughout the specification, when it is said that a configuration is connected to another configuration, this includes not only cases in which it is directly connected but also cases in which it is connected with another configuration in between. . Further, when a configuration includes another configuration, it does not mean that the other configuration is excluded, and that the other configuration can be included, unless otherwise specified.

An embodiment will be described in detail below with reference to the accompanying drawings.

FIG. 1 is a diagram for explaining a transformer model according to the prior art.

FIG. 1 shows a transformer model according to the prior art, showing the structure of a transformer model used for video representation learning. Here, the transformer model can be Vanilla Transformer. On the other hand, the transformer shown in FIG. 1 can be set such that the encoder 100 is separated into layers for all video frames. Here, in the transformer shown in FIG. 1, the encoders 100 separated for each section _S1 , _S2 , _S3 may be temporal transformers. Video story question answering can be performed using the transformer model shown in _{FIG . 1.} The transformer as shown in FIG _. Since the context of the video clip included in the video data to be processed is not considered, if the length of the video increases, the calculation cost increases geometrically, so the length is short. Used only for video story question answering. Therefore, there is a need for transformers that can process long videos more effectively, and transformers have been built that take into account the context before and after video clips contained in the video data. A transformer that considers the context before and after a video clip included in video data according to one embodiment will be described in more detail below with reference to FIGS. 2 and 3. FIG.

FIG. 2 is a diagram for explaining a transformer model according to one embodiment.

FIG. 2 shows a transformer model according to one embodiment, showing the structure of the transformer model used for video representation learning. Here, the Transformer Model may be a Contextual Transformer. On the other hand, the transformer shown in FIG. 2 can be set such that the encoder 200 is separated into layers for all video frames. The transformer shown in FIG. 2 learns the video story by considering the context before and after the video clip included in the video data input to the encoder 200 separated for each section _S1 , _S2 , and _S3 . By doing so, the number of video clips in the preceding and succeeding sections that can be considered can be changed as the hierarchy becomes higher. Here, a video clip can mean a short recorded moving picture. For example, two sections can be considered in the sections _S1 and _S3 of the second layer, and three sections can be considered in the section _S2 of the second layer. Here, the video data may include multiple continuous video clips, and the video clip may include multiple visual tokens and text tokens. On the other hand, in the contextual transformer shown in FIG. 2, the separate encoders 200 for each interval S ₁ , S ₂ , S ₃ can be cross-modal transformers, where the cross-modal transformer described above is , a visual token and a text token corresponding to each interval S ₁ , S ₂ , S ₃ can be received as input. The transformer shown in FIG. 2 considers the context before and after the video clip included in the video data input to the encoder 100 separated for each of the sections _S1 , _S2 , and _S3 . so that long videos can be processed effectively without incurring a large computational cost. In addition, by constructing a transformer that considers the context before and after the video clip included in the video data, it can be applied not only to video story question answering, but also to various fields such as predicting the next scene of the video and inferring causality. There are effects that can be used.

On the other hand, the transformer of FIG. 2 described above can be built by the apparatus for building a transformer model for video story question answering shown in FIG.

FIG. 3 is a functional block diagram of an apparatus for building a transformer model for video story question answering according to one embodiment.

Referring to FIG. 3, an apparatus 300 for building a transformer model for video story question answering according to one embodiment includes an input/output unit 310, a storage unit 320, and a control unit 330. FIG.

The input/output unit 310 includes an input unit for receiving input from a user and an output for displaying information such as the result of performing a task or the state of the apparatus 300 for constructing a transformer model for video story question answering. and That is, the input/output unit 310 receives video data including a plurality of continuous video clips and question data for video question answering, and outputs the result of the video story question answering. Here, the video clip may include multiple visual tokens and text tokens.

The storage unit 320 is configured to store files and programs, and may be configured with various types of memory. In particular, the storage unit 320 can store data and programs that enable the control unit 330 to build a transformer model for video story question answering according to an algorithm presented below.

The control unit 330 includes at least one processor such as a CPU, GPU, Arduino, etc., and can control the overall operation of the apparatus 300 for constructing a transformer model for video story question answering. That is, the control unit 330 can control other components included in the apparatus 300 for building a transformer model for video story question answering so as to perform video story question answering. The control unit 330 may execute the program stored in the storage unit 320 to perform operations for constructing a transformer model for video story question answering according to an algorithm presented below. The method by which the control unit 330 performs operations for constructing a transformer model for video story question answering will be described later.

Hereinafter, a process of performing a method of constructing a transformer model for video story question answering according to an embodiment by causing the control unit 330 to execute a program stored in the storage unit 320 will be described in detail.

According to one embodiment, the controller 330 can construct a transformer model for video story question answering using Equation 2 above.

The control unit 330 receives visual tokens and text tokens included in each video clip for each video clip corresponding to a preset section through each separated encoder, and receives the visual tokens and text tokens included in each video clip, and the lower layers of adjacent video clips. The video storage can be trained by computing a hidden representation of the (lower layer) and utilizing the computed hidden representation to compute a contextual representation of the video data. . Here, the controller 330 can learn the temporal order using a masked modality model (MMM) for each video clip. The mask modality model (MMM) is a token-based masking technique proposed in an existing model, the Masked Language Model, by masking the entire token of a predetermined interval. ). A masked modality model (MMM) allows one modality to generate from another modality, while the encoder too easily generates masked tokens from surrounding tokens. can be prevented, and the alignment between modalities can be learned. Here, modalities can be video, text, and the like. Therefore, by performing the above-described learning using the contextual transformer according to one embodiment, it is possible to predict the content of a segment (for example, video data separated by sections) based on the context before and after it, so that natural You can learn the flow of the story.

On the other hand, a masked modality model can be learned through negative contrastive learning. Here, a masked modality model can be expressed as Equation 3 below.

FIG. 4 is a flowchart illustrating a method of building a transformer model for video story question answering according to one embodiment.

The method of building a transformer model for video story question answering according to the embodiment shown in FIG. 4 is processed chronologically by the apparatus 100 for building a transformer model for video story question answering shown in FIGS. including the step of Therefore, although omitted below, the above description of the apparatus 100 for constructing a transformer model for video story question answering shown in FIGS. 2 and 3 is based on the embodiment shown in FIG. It can also be applied to the method of building a transformer model for video story question answering.

Referring to FIG. 4, the apparatus 100 for building a transformer model for video story question answering can receive video data including a plurality of continuous video clips and question data for video question answering (S410). . Here, the video clip may include multiple visual tokens and text tokens.

The apparatus 100 for constructing a transformer model for video story question answering considers the context of the preceding and succeeding video clips adjacent to each other in temporal order from the video data including a plurality of continuous video clips received in step S410. A video story can be learned (S420). The apparatus 100 for constructing a transformer model for video story question answering inputs visual tokens and text tokens included in each video clip for each video clip corresponding to a preset section through separate encoders. , calculating hidden representations of the lower layers of the adjacent video clips before and after each other, and utilizing the calculated hidden representations of the video data considering the context of the preceding and following The video storage can be trained by computing representations. Here, the apparatus 100 for constructing a transformer model for video story question answering can learn temporal order using a masked modality model for each video clip. The mask modality model (MMM) is a token-based masking technique proposed in an existing model, the Masked Language Model, by masking the entire token of a predetermined interval. masking). A masked modality model (MMM) allows one modality to be generated from another modality, while the encoder too easily generates masked tokens from surrounding tokens. can be prevented, and the alignment between modalities can be learned. On the other hand, a masked modality model can be learned through negative contrastive learning. Here, the masked modality model can be expressed as Equation 3 above.

The term 'unit' used in the above embodiments means a hardware component such as software or FPGA (field programmable gate array) or ASIC, and 'unit' plays a certain role. However, 'something' is not meant to be limited to software or hardware. The 'section' can also be configured to reside on an addressable storage medium and can be configured to run on one or more processors. Thus, by way of example, 'section' refers to components such as software components, object-oriented software components, class components and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program proprietary code, Includes drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables.

The functionality provided within a component and 'section' may be combined with a smaller number of components and 'section' or separated from additional components and 'section'.

In addition, the components and 'units' can also be embodied to play one or more CPUs in the device or secure multimedia card.

Meanwhile, the method of constructing a transformer model for video story question answering according to one embodiment described herein is also embodied in the form of a computer-readable medium storing computer-executable instructions and data. can Here, instructions and data can be stored in the form of program code, and when executed by a processor, can generate predetermined program modules to perform predetermined operations. Also, computer-readable media can be any available media that can be accessed by the computer and includes both volatile and nonvolatile media, removable and non-removable media. Also, the computer-readable medium may be a computer storage medium. Computer storage media are volatile and non-volatile, separable and non-separable embodied by any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Any of the media can be included. For example, the computer storage medium may be magnetic storage media such as HDD and SSD, optical storage media such as CD, DVD and Blu-ray discs, or memory contained in a server accessible via a network.

The method of constructing a transformer model for video story question answering according to one embodiment described herein may also be embodied in a computer program (or computer program product) including computer-executable instructions. can. A computer program includes a programmable machine instruction language processed by a processor, and is embodied in a high-level programming language, an object-oriented programming language, an assembly language, a machine language, or the like. can Also, the computer program can be recorded on any type of computer-readable recording medium (eg, memory, hard disk, magnetic/optical medium, SSD (Solid-State Drive), etc.).

Therefore, a method of building a transformer model for video story question answering according to one embodiment described herein can be implemented by a computer program as described above being executed by a computing device. A computing device may include at least a portion of a processor, a memory, a storage device, a high speed interface connecting to the memory and the high speed expansion port, and a low speed interface connecting to the low speed bus and the storage device. . Each such component is connected to each other using various buses and can be mounted on a common motherboard or attached in any other suitable manner.

Here, the processor can process instructions within the computing device. Such commands may be stored in a memory or storage device to display graphic information for providing a GUI (Graphic User Interface) on an external input and output device such as a display connected to a high-speed interface. can have a command word As another example, multiple processors and/or multiple buses may suitably be used with multiple memories and memory configurations. Also, the processor may be embodied in a chipset consisting of chips containing multiple independent analog and/or digital processors.

The memory also stores information within the computing device. As an example, the memory can consist of volatile memory units or collections thereof. As another example, the memory may consist of non-volatile memory units or collections thereof. The memory may also be other forms of computer-readable media, such as magnetic or optical disks.

And the storage device can provide a large amount of storage space for the computing device. The storage device may be a computer-readable medium or a configuration that includes such a medium, and may include, for example, a device in a SAN (Storage Area Network) or other configuration, such as a floppy disk device, hard disk device, It may be an optical disk drive, or a tape drive, flash memory, or other similar semiconductor memory device or array of devices.

The above-described embodiments are for illustrative purposes only, and those skilled in the art to which the above-described embodiments belong may make other specific modifications without changing the technical ideas or essential features of the above-described embodiments. It will be understood that it can be easily transformed into any suitable form. Accordingly, the above-described embodiments are to be understood as illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a distributed manner, and likewise, components described as distributed may also be implemented in a combined form.

The scope of protection sought by this specification is determined by the claims which follow rather than the foregoing detailed description, and any modifications or variations derived from the meaning and scope of the claims and their equivalents. must be construed to include

300 Apparatus for constructing transformer model for video story question answering 310 input/output unit 320 storage unit 330 control unit

Claims (10)

an input/output unit for receiving video data including a plurality of continuous video clips and question data for video question answering, and for outputting a video story question answering result;
a storage unit for storing programs and data for performing video story Q&A;
a controller comprising at least one processor for building a transformer model for video story question answering by executing the program;
The control unit
A transformer model for video story question answering, wherein a video story is learned from the video data including the plurality of continuous video clips, taking into consideration the contexts of preceding and succeeding video clips adjacent to each other in temporal order. equipment for building The video clip is
including a plurality of visual tokens and text tokens;
The control unit
Visual tokens and text tokens included in each video clip corresponding to a preset section through each separated encoder are received as input, and the lower hierarchy of the adjacent video clips is received. layer), and utilizing the calculated hidden representation to calculate a representation of the video data considering the context before and after the video storage is learned. The apparatus for building a transformer model for video story question answering according to claim 1. The control unit
2. The apparatus of claim 1, wherein temporal order is learned using a masked modality model for each video clip. . 4. The apparatus of claim 3, wherein the Masked Modality Model is learned by Negative Contrastive Learning. A method for building a transformer model for video story question answering performed by a device for building a transformer model for video story question answering, comprising:
receiving video data including a plurality of consecutive video clips and query data for video question answering;
and learning a video story from the video data including the plurality of consecutive video clips by considering the context of the preceding and following video clips adjacent to each other in temporal order. how to build. The video clip is
including a plurality of visual tokens and text tokens;
The step of learning the video story comprises:
Visual tokens and text tokens included in each video clip corresponding to a preset section through each separated encoder are received as input, and the lower layer of the adjacent video clips is received. ) and training the video storage by computing a contextual representation of the video data utilizing the computed hidden representation. The method of building a transformer model for video story question answering of claim 5, wherein: The step of learning the video story comprises:
6. The transformer model for video story question answering of claim 5, further comprising learning a temporal order using a masked modality model for each video clip. how to build. 8. The method of building a transformer model for video story question answering of claim 7, wherein the Masked Modality Model is learned by Negative Contrastive Learning. A computer-readable recording medium recording a program for performing the method of claim 5. A computer program recorded on a recording medium to be executed by an apparatus for building a transformer model for video story question answering and to perform the method of claim 5.