KR20210115401A - System and method for intelligent scene split based on deeplearning - Google Patents

Abstract

A deep learning-based intelligent scene segmentation method and a system thereof are disclosed. According to one aspect of the present invention, the deep learning-based intelligent scene segmentation method includes the steps of: generating, by a system, corresponding frame texts for each of a plurality of frame images included in a video to be edited; detecting, by the system, adjacent frame texts having significant semantic changes by analyzing the frame texts; and determining, by the system, a scene transition boundary in the video to be edited based on the adjacent frame texts detected by the system.

Description

Deep learning-based intelligent scene split method and its system {System and method for intelligent scene split based on deeplearning}

The present invention relates to a video editing method and system using deep learning-based image captioning. More particularly, it relates to a system and method capable of performing image captioning on an image frame included in a video and effectively editing a video based on text generated as a result of performing image captioning.

It also relates to a system and method capable of automatically segmenting scenes in a video based on text.

The utilization of video (movie) is very high, and many attempts are being made to produce video contents according to one-person media, media commerce, and SNS.

However, in order to complete the video content, it is necessary to edit the recorded video, which requires relatively much time, know-how, and cost for such editing.

In general, for conventional video editing, unnecessary frames or cuts should be deleted or only necessary cuts should be selected. In this case, a process of editing while watching each frame was required.

It may be necessary to have a technical idea to effectively perform such editing based on text corresponding to frames included in the video.

In addition, by detecting a meaningful scene change boundary based on text, there is an effect of detecting a scene change in the case where the change of the conventional image characteristic is not large.

Laid-Open Patent Publication No. 10-2011-0062567 (Method and apparatus for summarizing video contents using video clipping)

An object of the present invention is to provide a method and system for editing a video based on text corresponding to each of a plurality of frames included in the video.

Another object of the present invention is to provide a method and system capable of detecting a meaningful scene boundary that is not detected by image features by determining the scene boundary through text.

According to one aspect of the present invention, a deep learning-based intelligent scene segmentation method comprises: generating, by a system, corresponding frame texts for each of a plurality of frame images included in a video to be edited; the system analyzing the frame texts and detecting adjacent frame texts having significant semantic changes by doing so, and determining a scene change boundary in the video to be edited based on the adjacent frame texts detected by the system.

.

According to an embodiment of the present invention, since video editing through text is possible, selection of a frame or searching for a scene can be performed effectively in a very short time, thereby greatly saving resources used for editing.

In addition, by determining the scene boundary through text, there is an effect of detecting a meaningful scene boundary that is not detected by image features.

In order to more fully understand the drawings cited in the Detailed Description, a brief description of each drawing is provided.
1 is a diagram illustrating a schematic flow of a video editing method using deep learning-based image captioning according to an embodiment of the present invention.
2 is a diagram illustrating a schematic flow of an intelligent scene segmentation method based on deep learning according to an embodiment of the present invention.

Since the present invention can apply various transformations and can have various embodiments, specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and it should be understood to include all modifications, equivalents and substitutes included in the spirit and scope of the present invention. In describing the present invention, if it is determined that a detailed description of a related known technology may obscure the gist of the present invention, the detailed description thereof will be omitted.

Terms such as first, second, etc. may be used to describe various elements, but the elements should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise.

In the present specification, terms such as "comprise" or "have" are intended to designate that the features, numbers, steps, operations, components, parts, or combinations thereof described in the specification exist, and one or more other It is to be understood that this does not preclude the possibility of addition or presence of features or numbers, steps, operations, components, parts, or combinations thereof.

In addition, in the present specification, when any one component 'transmits' data to another component, the component may directly transmit the data to the other component, or through at least one other component. This means that the data may be transmitted to the other component. Conversely, when one component 'directly transmits' data to another component, it means that the data is transmitted from the component to the other component without passing through the other component.

1 is a diagram illustrating a schematic flow of a video editing method using deep learning-based image captioning according to an embodiment of the present invention.

Referring to FIG. 1 , a video system (hereinafter, referred to as a system) using deep learning-based image captioning according to an embodiment of the present invention may receive an editing target video to be edited.

The system may then select a plurality of frame images included in the video to be edited.

The selected frame images may be whole frame images included in the video to be edited, or may be part of them. For example, only key frames may be selected, or they may be selected at a predetermined interval (one for every 10 frames).

The system can then generate frame texts for each selected frame image.

For example, if a total of 100 frame images are selected, the system may generate frame texts corresponding to each of the 100 frame images.

Each of the generated frame text may be text that can describe corresponding frame images.

To this end, the system may include a deep learning-based image captioning module. The image captioning module may be a learned configuration to generate text corresponding to the image for the image.

Such an image captioning module may be implemented in various ways, and since it is already known, a detailed description thereof will be omitted.

Once the frame texts are generated, the system can perform video editing using the generated frame texts.

In order to perform video editing, it may be necessary to select a frame to be included in or deleted from the video after editing, and the selection of such a frame may be performed based on text using the generated frame texts.

For example, the system may receive a selection condition. The selection condition may be text for selecting a frame.

For example, when the system receives a selection condition of "two people", the system includes the text "two people" among the frame texts as it is, or a frame having the meaning of two people through semantic search You can search for text. The selection condition is not necessarily one, but may be performed through logical operation.

For example, when a selection condition of "two people" and "mountain" is input, frame texts (ie, a frame image) in which two people and a mountain exist may be selected.

For such a search, sematic search, not keyword search, may be utilized through a language model trained for NLP (Natural Language Processing).

When the frame texts retrieved by the system, that is, the target frame texts are determined, the frame image corresponding to the target frame text, ie, the target frame image, may be a frame image that the user wants to select.

That is, frame images in which two people exist may be selected.

Then, according to the user's request, an editing command such as deleting the selected target frame images or leaving only the selected frame images may be input through a predetermined method (such as a text command or a predetermined UI).

A selection of a frame desired by a user can be very effectively selected based on text using frame text in various ways, and video editing using the frame text can be effectively performed.

For example, if you want to leave only frame images in which only dogs exist in a video in which dogs and cats are photographed, after inputting “dog” as a selection condition, an edit command is input that leaves only the specified target frame images and deletes the remaining frames. An edited video consisting only of frames in which a dog appears can be obtained.

Various types of editing commands (such as selection, deletion, or order adjustment) may be possible.

On the other hand, the system can automatically detect a scene transition boundary using frame texts.

In general, in order to find a scene boundary in the related art, a method using changes in image characteristics (eg, color information, shading, etc.) is used.

However, even consecutive shots that are not well detected in such a change in image characteristics may be significantly changed by image captioning.

That is, a change in the content of an image detected by image captioning may be different from a scene boundary detected by a change in a conventional image feature.

Therefore, according to the technical idea of the present invention, it is possible to detect a scene boundary by detecting a case where there is a significant degree of significant change between frame texts generated by image captioning.

2 is a diagram illustrating a schematic flow of a video editing method using deep learning-based image captioning according to an embodiment of the present invention.

Referring to FIG. 2 , for the text-based scene segmentation method, it is necessary to well determine which object or action is detected in a deep learning model for image captioning.

For example, the image captioning module detects the human behavior and detects a change in human behavior (eg, sitting and picking up an object) as a scene boundary, although the color or shading of the image does not change significantly. It needs to be learned to express this in text.

Then, the system may generate frame texts corresponding to each of the plurality of frame images from the input video to be edited.

In addition, by analyzing the frame texts, it is possible to detect adjacent frame texts having a meaningful change in meaning. Cases where there is a significant change in meaning may be predetermined, for example, such as the appearance of a new object or performance of a new action, or a case where there is a change of more than a preset value by calculating a change in the similarity of frame texts may be detected. .

The system can then determine a scene transition boundary in the video to be edited based on the detected adjacent frame texts. For example, the middle of the detected in-point frame texts may be determined as a scene boundary.

On the other hand, the method according to the embodiment of the present invention may be implemented in the form of a computer-readable program command and stored in a computer-readable recording medium, and the control program and the target program according to the embodiment of the present invention are also implemented in the computer. It may be stored in a readable recording medium. The computer-readable recording medium includes all types of recording devices in which data readable by a computer system is stored.

The program instructions recorded on the recording medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the software field.

Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and floppy disks and hardware devices specially configured to store and execute program instructions, such as magneto-optical media and ROM, RAM, flash memory, and the like. In addition, the computer-readable recording medium is distributed in a computer system connected through a network, so that the computer-readable code can be stored and executed in a distributed manner.

Examples of the program instruction include not only machine code such as generated by a compiler, but also a device for electronically processing information using an interpreter or the like, for example, a high-level language code that can be executed by a computer.

The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

The above description of the present invention is for illustration, and those of ordinary skill in the art to which the present invention pertains can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a distributed manner, and likewise components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present invention. .

Claims (3)

generating, by the system, corresponding frame texts for each of a plurality of frame images included in the video to be edited;
detecting, by the system, adjacent frame texts having significant semantic changes by analyzing the frame texts;
and determining a scene transition boundary in the video to be edited based on adjacent frame texts detected by the system.
A computer program installed in a data processing apparatus and stored in a computer-readable recording medium to perform the method according to claim 1.
As an intelligent scene segmentation system based on deep learning,
processor; and
a memory for storing a computer program executed by the processor;
The computer program, when executed by the processor, a deep learning-based intelligent scene segmentation system to perform the method according to claim 1.

Images