JP2023005038A - Moving image summarization device, moving image summarization method, and program - Google Patents

Abstract

To provide a moving image summarization device, a moving image summarization method, and a program for summarizing a moving image which are highly versatile not limited to a specific genre.SOLUTION: In a system in which a moving image summarization device and a terminal are connected to a network, a moving image summarization device 100 for generating a moving image summary includes: a voice-to-text conversion unit 120 that acquires text by performing voice recognition on voice contained in the moving image; a text summarization unit 130 that summarizes sentences obtained from the text into a plurality of sentences; and a summary moving image generation unit 150 that acquires time sections each corresponding to each sentence in the plurality of sentences, extracts a partial moving image corresponding to each time section from the moving image, and combines the extracted partial moving images to generate a summary moving image.SELECTED DRAWING: Figure 2

Description

The present invention relates to techniques for summarizing moving images.

2. Description of the Related Art There has been a demand for grasping the contents of a long moving image (which may be called a video) in a short time, and various moving image summarization techniques have been proposed (for example, Patent Documents 1 to 3).

As an example, in a situation in which excellent presentations are selected from a large number of presentation videos, it is difficult to watch each video over time. By using video summarization technology, it is possible to grasp the contents of individual videos in a short time and to perform efficient evaluation.

JP 2010-039877 A JP 2011-061263 A JP 2015-099958 A

In conventional video summarization techniques, generally, features of images related to a specific genre are extracted from a video, and the features are used to create a summary of the video. However, there are many genres in moving images, and there is a possibility that new genres and expressions will be created in the future.

Therefore, with the video summarization technology in the prior art, videos that can be summarized are limited to those of a specific genre, and there is a problem that videos cannot be general-purposely summarized.

The present invention has been made in view of the above points, and an object of the present invention is to provide a highly versatile technique for summarizing moving images that is not limited to a specific genre.

According to the disclosed technology, a moving image summarization device for creating a moving image summary,
a speech-to-text conversion unit that acquires text by performing speech recognition on speech contained in the moving image;
a sentence summarization unit that summarizes sentences obtained from the text into a plurality of sentences;
Generating a summary video by obtaining a time segment corresponding to each sentence in the plurality of sentences, extracting partial videos corresponding to each time segment from the video, and combining the extracted partial videos to generate a summary video. A video summarization device is provided comprising:

According to the disclosed technique, it is possible to realize a highly versatile video summarization technique that is not limited to a specific genre.

1 is an overall configuration diagram of a system according to an embodiment of the present invention; FIG. 1 is a functional configuration diagram of a video summarizing device; FIG. FIG. 4 is a sequence diagram for explaining the operation of the system; FIG. 4 is a diagram showing an example of a display screen; FIG. 4 is a diagram showing an example of a display screen; FIG. 4 is a diagram for explaining an example of a method for extracting a moving image segment; FIG. It is a figure which shows the hardware configuration example of an apparatus.

An embodiment (this embodiment) of the present invention will be described below with reference to the drawings. The embodiments described below are merely examples, and the embodiments to which the present invention is applied are not limited to the embodiments described below.

In the following description, unless otherwise specified, a "moving image" is a moving image with sound. In the video with sound, the sound and the video are synchronized. In addition, any format may be used for including sound in moving images. For example, it may be of a format in which moving images and audio are provided as separate files.

(System configuration example)
FIG. 1 shows an example of the overall configuration of a system according to this embodiment. As shown in FIG. 1, this system has a configuration in which a moving picture summarizing device 100 and a terminal 200 are connected to a network 300 .

The moving picture summarizing device 100 is a device for summarizing moving pictures using the technology according to the present invention. A terminal 200 is a general terminal such as a smartphone or a PC. Network 300 is, for example, the Internet. Network 300 may be a small network such as a LAN.

(Configuration example of video summarizing device 100)
FIG. 2 shows an example of the functional configuration of the video summarizing device 100. As shown in FIG. As shown in FIG. 2 , the video summarizing device 100 has a video data acquisition unit 110 , a speech-to-text processing unit 120 , a sentence summarization unit 130 , a video section extraction unit 140 , a summary video generation unit 150 and a data storage unit 160 . Note that the function of the moving image section extracting section 140 may be included in the digest moving image generating section 150 . The function outline of each part is as follows.

The moving image data acquisition unit 110 acquires moving image data from the terminal 200 or the like. The speech-to-text processing unit 120 converts the speech in the moving image into sentences. The text summarizing section 130 summarizes the text converted from the speech by the speech-to-text processing section 120 . The video segment extraction unit 140 extracts a time segment of the video based on the time position of the summary sentence (extracted sentence) obtained by the speech-to-text processing unit 120 . The digest video generating section 150 generates a digest video using the video of the time segment extracted by the video segment extraction section 140 . The data storage unit 160 stores various data.

(Example of system operation)
Next, an operation example of the system will be described along the procedure of the sequence chart of FIG. Here, it is assumed that the terminal 200 holds various moving image data (which may be called moving image files). In addition, each moving image contains the sound of human speech.

Assuming that the user of the terminal 200 has viewed a summary of a certain moving image, the user designates a moving image for which the summary is desired to be viewed on the terminal 200 in S101.

At S<b>102 , the terminal 200 uploads data of the specified moving image to the moving image summary device 100 . The moving image data acquisition unit 110 of the moving image summarizing device 100 receives the moving image data and stores it in the data storage unit 160 .

The speech-to-text processing unit 120 reads the moving image data from the data storage unit 160, acquires speech (in this case, human speech) from the data, executes speech recognition, and converts the speech into text (S103, S104).

In S105, the speech-to-text processing unit 120 generates a sentence consisting of a plurality of sentences from the text converted from the speech. This sentence may be generated in the form of a list of sentences. In S106, the speech-to-text processing unit 120 acquires time information (segment, length) of each sentence. The generated sentences (a list of sentences) are stored in the data storage unit 160 together with the time information of each sentence. Note that the technology itself for generating sentences from voice by speech recognition is an existing technology.

In S<b>107 , the speech-to-text processing unit 120 transmits a list of generated sentences (with time information of each sentence) to the terminal 200 . At S108, a list of sentences is displayed on the terminal 200. FIG.

FIG. 4 shows an example of a screen displayed on the terminal 200 in S108. In this example, the time length of the target moving image and a list of sentences obtained by speech recognition are displayed on the screen along with the time (time). A desired length (length of time) of the summarized video can be specified by specifying the "digest time length". In addition, the start of summarization can be instructed by pressing the "start summarization" button.

Note that the desired length of the digest video may be specified by the number of sentences. For example, if there are 100 sentences in total, by specifying "20" as the number of sentences, a summary video for 20 sentences can be created.

Alternatively, the length of the desired digest video may be specified as a ratio of the length of the digest to the total length (referred to as the digest rate). For example, when "1/6" is specified, a 10-minute summary video is generated from a 60-minute video, and a 1-minute video is generated from a 6-minute video.

Alternatively, a digest video may be generated with a predetermined digest rate without specifying the desired length of the digest video.

In S109 of FIG. 3, the user designates a desired length of time for a summary on the terminal 200 and instructs to start creating a summary. At S110, the time length is transmitted from the terminal 200 to the moving picture summarizing device 100 together with the command to create a summary. The text summarizing unit 130 of the video summarizing device 100 receives the summarizing command and the length of time.

In S111, the sentence summarization unit 130 reads out the sentences (list of sentences) obtained by the speech-to-text processing unit 120 and the time information from the data storage unit 160, and summarizes the sentences so as to have the specified length of time. create. The text summary itself can be realized with existing technology.

Any existing technique for summarizing a sentence may be used, but in the present embodiment, as an example, the sentence is summarized by extracting a plurality of sentences from the sentence. For example, it is assumed that 60 sentences from sentence 1 to sentence 60 are included in the entire sentence. Each sentence is associated with a time segment (start time, end time, length of time) in the moving image.

For example, if the specified time length is 10 minutes, the sentence summarization unit 130 may perform a sentence such as "sentence 1 (time length 1 minute)" so that the total time length of the sentences included in the summary is 10 minutes. , Sentence 20 (time length 1 minute), Sentence 21 (time length 2 minutes), Sentence 53 (time length 3 minutes), Sentence 54 (time length 1 minute), Sentence 60 (time length 2 minutes). , to extract multiple sentences that are considered important. The extracted sentence and information on the time interval are stored in the data storage unit 160 .

As described above, it is also possible to specify the number of sentences or the rate of summarization as the length of the summary instead of the length of time. If the number of sentences is specified, extract only the specified number of sentences. When a summarization rate is specified, sentences are extracted so that the length of time corresponds to the summarization rate.

In the example of FIG. 3, after S111, the process proceeds to S112. However, proceeding to S112 after S111 is an example. may In other words, a plurality of sentences (a list of sentences) as a summary result may be output to the user.

In this case, the screen shown in FIG. 5 is displayed on the terminal 200, for example. As shown in FIG. 5, a text summary is added to the screen of FIG. On this screen, the user can specify sentences in a time segment that he/she wants to add to the digest video from all the sentences. You can also specify sentences in a time period that you want to delete from the summary video.

When the above sentence addition is specified, the video summarizing device 100 generates a summarized video from a plurality of sentences obtained by adding sentences specified by the user to the plurality of sentences obtained as a result of summarization. be able to. In addition, when the above sentence deletion is designated, the video summarization device 100 deletes sentences specified by the user from the plurality of sentences obtained as a result of summarization, and then extracts a summary video from a plurality of sentences. can be generated.

In S112, the moving image segment extraction unit 140 acquires information on the corresponding time segment for each sentence extracted in S111.

For example, in the above example, (start time: 0 minutes 0 seconds, end time: 1 minute 0 seconds) is obtained for sentence 1, and for sentence 20, for example, (start time: 20 minutes 30 seconds seconds, end time: 21 minutes and 30 seconds). The same is true for other sentences.

In S113, the video segment extraction unit 140 extracts the video segment corresponding to the time segment (video corresponding to the time position of the time segment, which may be called a partial video) for each of the time segments acquired in S112. Extract from the video of

For example, using the above example, the video 1 in the time interval corresponding to sentence 1 (start time: 0 minutes 0 seconds, end time: 1 minute 0 seconds) is extracted from the video of 60 minutes in total, and sentence 20 is extracted. (start time: 20 minutes and 30 seconds, end time: 21 minutes and 30 seconds). The same is true for other time intervals.

Fig. 6 shows an image of video extraction. As shown in FIG. 6, a moving image of a section corresponding to the time section of each extracted sentence is extracted.

In S114 of FIG. 3, the summarized moving image generation unit 150 generates a summarized moving image by combining the moving images extracted in S113. For example, in S113, if video 1, video 20, video 21, video 53, video 54, and video 60 are extracted, "video 1 + video 20 + video 21 + video 53 + video 54 + video 60", which is a combination of these videos, is the summary video. generated.

In S<b>115 , the digest video generating unit 150 transmits the generated digest video to the terminal 200 . At S116, the digest video is displayed on the terminal 200, and the user views the digest video.

(Other examples)
The sequence shown in FIG. 3 is an example, and processing may be performed in a procedure different from the sequence shown in FIG. For example, the processes of S107 to S110 may not be performed. In this case, a predetermined summarization rate may be used, or the length of time for summarization, etc. may be specified from the terminal 200 to the video summarization apparatus 100 in S101 and S102.

Moreover, the configuration shown in FIG. 1 is also an example, and the configuration is not limited to this configuration. For example, the terminal 200 may include the function of the moving picture summarizing device 100 . In this case, terminal 200 can generate and display a summary video from the original video within its own terminal. A terminal including the functions of the video summarizing device 100 may be called a "video summarizing device". When the function of the moving picture summarizing device 100 is included in the terminal 200, transmitting information to the terminal 200 in FIG. 3 corresponds to displaying information on the display of the terminal 200. FIG.

Also, in the examples described so far, text is obtained by speech recognition from the voice contained in the video, and the summary video is generated using the text. A summary video may be generated using the text without using speech recognition.

In this case, for example, moving image data (moving image file) includes moving images and text. The video and text are synchronized by time stamps, etc., but they are separated, so the text can be obtained without analyzing the video.

For example, the speech-to-text processing unit 120 generates a sentence consisting of a plurality of sentences from the text. This sentence is generated in the form of a list of sentences, and each sentence is attached with information on the time interval in the moving image corresponding to that sentence. The processing after this processing is the same as the processing described so far. Even when using the text contained in the moving image, the image of moving image extraction is the same as that shown in FIG.

(Hardware configuration example)
Both the moving picture summarizing apparatus 100 and the terminal 200 can be implemented by, for example, causing a computer to execute a program describing the processing details described in this embodiment. The computer may be a physical machine or a virtual machine on the cloud. The video summarizing device 100 and the terminal 200 are collectively referred to as "devices".

That is, the device can be realized by executing a program corresponding to the processing performed by the device using hardware resources such as a CPU and memory built into the computer. The above program can be recorded in a computer-readable recording medium (portable memory, etc.), saved, or distributed. It is also possible to provide the above program through a network such as the Internet or e-mail.

FIG. 7 is a diagram showing a hardware configuration example of the computer in this embodiment. The computer of FIG. 7 has a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, etc., which are connected to each other via a bus B. .

A program for realizing processing by the computer is provided by a recording medium 1001 such as a CD-ROM or a memory card, for example. When the recording medium 1001 storing the program is set in the drive device 1000 , the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000 . However, the program does not necessarily need to be installed from the recording medium 1001, and may be downloaded from another computer via the network. The auxiliary storage device 1002 stores installed programs, as well as necessary files and data.

The memory device 1003 reads and stores the program from the auxiliary storage device 1002 when a program activation instruction is received. The CPU 1004 implements functions related to the device according to programs stored in the memory device 1003 .

The interface device 1005 is used as an interface for connecting to the network. A display device 1006 displays a program-based GUI (Graphical User Interface) or the like. An input device 1007 is composed of a keyboard, a mouse, buttons, a touch panel, or the like, and is used to input various operational instructions. The output device 1008 outputs the calculation result.

(Effect of Embodiment)
According to the technology according to the present embodiment, it is possible to summarize a moving image by using the voice or text contained in the moving image. Summarization techniques can be implemented. In addition, there are the following effects as more specific effects.

Demand for video content is increasing with the full-scale introduction of 5G. In addition, almost all events such as seminars and trainings due to the corona wreck have been held online, and the number of video recordings of these events has increased dramatically. In such a situation, the technology according to the present embodiment can automatically generate a summary video, and the generated summary video can be easily viewed on a smartphone, tablet, or the like in a spare time. In addition, this technology is highly desired in the world because it is very effective as a promotion to encourage viewing of video recordings of seminars and lectures and videos of teaching materials for training.

(Summary of embodiment)
This specification discloses at least the moving image summarizing device, moving image summarizing method, and program described below.
(Section 1)
A video summarizing device for creating a video summary,
a speech-to-text conversion unit that acquires text by performing speech recognition on speech contained in the moving image;
a sentence summarization unit that summarizes sentences obtained from the text into a plurality of sentences;
Generating a summary video by obtaining a time segment corresponding to each sentence in the plurality of sentences, extracting partial videos corresponding to each time segment from the video, and combining the extracted partial videos to generate a summary video. A moving picture summarization device comprising a section and .
(Section 2)
A video summarizing device for creating a video summary,
a sentence summarizing unit that summarizes sentences obtained from the text included in the moving image into a plurality of sentences;
Generating a summary video by obtaining a time segment corresponding to each sentence in the plurality of sentences, extracting partial videos corresponding to each time segment from the video, and combining the extracted partial videos to generate a summary video. A moving picture summarization device comprising a section and .
(Section 3)
The text summarization unit executes the summarization based on the length of time specified by the user, the number of sentences specified by the user, the summarization rate specified by the user, or a predetermined summarization rate. Or the moving picture summarization device according to item 2.
(Section 4)
The sentence summarization unit outputs the plurality of sentences, which are the results of summarizing the sentences, to the user;
The summary video generation unit performs an addition process of adding a sentence specified by the user to the plurality of sentences, or a deletion process of deleting the sentence specified by the user from the plurality of sentences, and performs the addition process or 3. The moving image summarizing device according to any one of items 1 to 3, wherein a summarized moving image is generated from the plurality of sentences subjected to the deletion processing.
(Section 5)
A video summarizing method executed by a video summarizing device that creates a video summary,
a speech-to-text conversion step of obtaining text by performing speech recognition on speech contained in the moving image;
a sentence summarization step of summarizing sentences obtained from the text into a plurality of sentences;
Generating a summary video by obtaining a time segment corresponding to each sentence in the plurality of sentences, extracting partial videos corresponding to each time segment from the video, and combining the extracted partial videos to generate a summary video. A video summarization method comprising steps and .
(Section 6)
A video summarizing method executed by a video summarizing device that creates a video summary,
a sentence summarization step of summarizing sentences obtained from the text included in the moving image into a plurality of sentences;
Generating a summary video by obtaining a time segment corresponding to each sentence in the plurality of sentences, extracting partial videos corresponding to each time segment from the video, and combining the extracted partial videos to generate a summary video. A video summarization method comprising steps and .
(Section 7)
A program for causing a computer to function as each part of the moving picture summarizing device according to any one of items 1 to 4.

Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and changes can be made within the scope of the gist of the present invention described in the claims. It is possible.

100 Video summary device 110 Video data acquisition unit 120 Speech text processing unit 130 Text summarization unit 140 Video section extraction unit 150 Summary video generation unit 160 Data storage unit 1000 Drive device 1002 Auxiliary storage device 1003 Memory device 1004 CPU
1005 interface device 1006 display device 1007 input device 1008 output device

Claims (7)

A video summarizing device for creating a video summary,
a speech-to-text conversion unit that acquires text by performing speech recognition on speech contained in the moving image;
a sentence summarization unit that summarizes sentences obtained from the text into a plurality of sentences;
Generating a summary video by obtaining a time segment corresponding to each sentence in the plurality of sentences, extracting partial videos corresponding to each time segment from the video, and combining the extracted partial videos to generate a summary video. A moving picture summarization device comprising a section and . A video summarizing device for creating a video summary,
a sentence summarizing unit that summarizes sentences obtained from the text included in the moving image into a plurality of sentences;
Generating a summary video by obtaining a time segment corresponding to each sentence in the plurality of sentences, extracting partial videos corresponding to each time segment from the video, and combining the extracted partial videos to generate a summary video. A moving picture summarization device comprising a section and . 2. The text summarization unit performs the summarization based on a user-specified time length, a user-specified number of sentences, a user-specified summarization rate, or a predetermined summarization rate. 3. The moving image summarization device according to 2. The sentence summarization unit outputs the plurality of sentences, which are the results of summarizing the sentences, to the user;
The summary video generation unit performs an addition process of adding a sentence specified by the user to the plurality of sentences, or a deletion process of deleting the sentence specified by the user from the plurality of sentences, and performs the addition process or 4. The moving picture summarizing device according to any one of claims 1 to 3, wherein a summarized moving picture is generated from the plurality of deleted sentences. A video summarizing method executed by a video summarizing device that creates a video summary,
a speech-to-text conversion step of obtaining text by performing speech recognition on speech contained in the moving image;
a sentence summarization step of summarizing sentences obtained from the text into a plurality of sentences;
Generating a summary video by obtaining a time segment corresponding to each sentence in the plurality of sentences, extracting partial videos corresponding to each time segment from the video, and combining the extracted partial videos to generate a summary video. A video summarization method comprising steps and . A video summarizing method executed by a video summarizing device that creates a video summary,
a sentence summarization step of summarizing sentences obtained from the text included in the moving image into a plurality of sentences;
Generating a summary video by obtaining a time segment corresponding to each sentence in the plurality of sentences, extracting partial videos corresponding to each time segment from the video, and combining the extracted partial videos to generate a summary video. A video summarization method comprising steps and . A program for causing a computer to function as each unit in the moving picture summarizing device according to any one of claims 1 to 4.

Images