KR102350457B1 - System and method for image searching using image captioning based on deep learning - Google Patents

Abstract

A video stream processing method and system through deep learning-based image captioning are disclosed. According to an aspect of the present invention, a method for processing a video stream through deep learning-based image captioning includes: dividing, by a system, a video to be edited into a plurality of shots, by the system, at least one shot for each of the divided shots generating text; receiving, by the system, a search condition text for image search; and selecting, by the system, matching shot text corresponding to the search condition text, and extracting a matching image corresponding to the selected matching shot text. and generating, by the system, at least one shot text for each of the divided shots, determining at least one selection frame from among a plurality of frames included in each of the divided shots; and generating the shot text through image captioning corresponding to the determined selection frame.

Description

Video stream processing method and system through image captioning based on deep learning {System and method for image searching using image captioning based on deep learning}

The present invention relates to a video editing method and system using deep learning-based image captioning. More specifically, image captioning is performed on at least one frame included in the video, and the video can be effectively processed (eg, edited, searched, etc.) based on the text generated as a result of performing the image captioning. It relates to a system and method.

The utilization of video (movie) is very high, and many attempts are being made to produce video content according to one-person media, media commerce, and SNS.

However, in order to complete the video content, it is necessary to edit the recorded video, and such editing requires relatively much time, know-how, and cost.

In general, for conventional video editing, unnecessary frames or cuts should be deleted or only necessary cuts should be selected, and in this case, a process of editing while watching each frame was required.

It may be necessary to have a technical idea that can effectively perform such editing based on text corresponding to the frame included in the video. Also, there may be a need for a technical idea that allows users to search for and provide only images they want even within a specific video using this.

Laid-Open Patent Publication No. 10-2011-0062567 (Method and apparatus for summarizing video content using video clipping)

SUMMARY OF THE INVENTION An object of the present invention is to provide a method and system by which a user can search for or edit only a desired image portion based on text corresponding to visual information of each of at least one frame included in a video.

In addition, it is to provide a method and system that can be utilized in a platform for sharing and selling images, since users can easily search for desired images within a video through image captioning.

According to an aspect of the present invention, a method for processing a video stream through deep learning-based image captioning includes: dividing, by a system, a video to be edited into a plurality of shots, by the system, at least one shot for each of the divided shots generating text; receiving, by the system, a search condition text for image search; and selecting, by the system, matching shot text corresponding to the search condition text, and extracting a matching image corresponding to the selected matching shot text. and generating, by the system, at least one shot text for each of the divided shots, determining at least one selection frame from among a plurality of frames included in each of the divided shots; and generating the shot text through image captioning corresponding to the determined selection frame.

The determining of at least one selected frame from among the plurality of frames included in each of the divided shots may include determining a moving picture including a plurality of consecutive frames included in the divided shots as the at least one selected frame. In addition, the generating of the shot text through image captioning corresponding to the determined selection frame may include generating the shot text through moving image captioning for the moving image.

In the video stream processing method through the deep learning-based image captioning, the system generates a first edited video in which the matching image is deleted from the editing target video, or the system processes the remaining frames except for the matching image. It may include generating the deleted second edited video.

In the determining of at least one selection frame from among the plurality of frames included in each of the divided shots, a frame including a predetermined object among the divided shots may be determined as the selection frame.

The step of determining a frame including a predetermined object from among the divided shots as the selection frame includes, when the object is included in a central frame corresponding to the center of the divided shot, including the central frame in the selection frame and, when the object is not included in the central frame, searching for frames including the object in the order of adjacent frames from the central frame and including the first searched frame in the selection frame.

The generating of the shot text through image captioning corresponding to the determined selected frame may include generating the shot text as a result of video captioning for a video that is a plurality of consecutive frames including the selected frame. may include the step of

The above method may be implemented by a computer program installed in the data processing apparatus.

According to another aspect, a system for a video stream processing method through deep learning-based image captioning for implementing the technical idea of the present invention includes a processor and a memory for storing a computer program executed by the processor, The processor drives the program, divides the video to be edited into a plurality of shots, generates at least one shot text for each of the divided shots, receives a search condition text, and receives a search condition text corresponding to the search condition text. Selecting matching shot text, extracting a matching shot or matching frame corresponding to the selected matching shot text, determining at least one selection frame from among a plurality of frames included in each of the divided shots, and corresponding to the determined selection frame The shot text is generated through video captioning.

According to an embodiment of the present invention, text representing visual information of each of the divided images in a video is generated, and a video search or edit is possible using the generated text, so that a very efficient image processing (image search or editing) is possible. etc.) can be performed.

In order to more fully understand the drawings cited in the Detailed Description, a brief description of each drawing is provided.
1 is a diagram for explaining the outline of a video stream processing method through deep learning-based image captioning according to an embodiment of the present invention.
2 is a diagram showing a schematic configuration of a system for performing a video stream processing method through deep learning-based image captioning according to an embodiment of the present invention.
3 is a diagram for explaining the concept of a video stream processing method through deep learning-based image captioning according to an embodiment of the present invention.
4 is a diagram for explaining the concept of determining a selection frame according to an embodiment of the present invention.
5 and 6 are diagrams for explaining image processing according to an embodiment of the present invention.

Since the present invention can apply various transformations and can have various embodiments, specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention. In describing the present invention, if it is determined that a detailed description of a related known technology may obscure the gist of the present invention, the detailed description thereof will be omitted.

Terms such as first, second, etc. may be used to describe various elements, but the elements should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise.

In this specification, terms such as "include" or "have" are intended to designate that the features, numbers, steps, operations, components, parts, or combinations thereof described in the specification exist, and one or more other It should be understood that this does not preclude the possibility of addition or presence of features or numbers, steps, operations, components, parts, or combinations thereof.

In addition, in the present specification, when any one component 'transmits' data to another component, the component may directly transmit the data to the other component or through at least one other component. This means that the data may be transmitted to the other component. Conversely, when one component 'directly transmits' data to another component, it means that the data is transmitted from the component to the other component without passing through the other component.

1 is a diagram for explaining the outline of a video stream processing method through deep learning-based image captioning according to an embodiment of the present invention.

Referring to FIG. 1 , a predetermined system 100 may be provided for a video stream processing method through deep learning-based image captioning according to an embodiment of the present invention.

The system 100 may receive an editing target video to be edited.

Then, according to the spirit of the present invention, the system 100 may generate text related to the video to be edited.

The text may be text that can describe or express visual information of the video to be edited.

Such text may be generated through a deep learning model that generates text representing visual information expressed in still images for all frames constituting the editing target video, that is, for each still image.

However, in some embodiments, it may be inefficient to generate text representing visual information for every frame of the video to be edited. This is because there may be little difference in visual information in a significant number of consecutive frames.

Accordingly, according to the technical idea of the present invention, the system 100 divides the video to be edited based on a predetermined criterion, and for each of the divided images, that is, each of the shots, one or Multiple texts can be created. Such text will be defined as shot text in this specification.

The system 100 may generate one or a plurality of shot texts for each shot.

In addition, the system 100 may provide a function and/or UI through which the user can easily select a desired scene (at least one frame) and/or a desired shot based on the generated shot text. When a desired scene/shot is easily selected by the user, the user can very efficiently search for a desired scene/shot from the editing target video or edit the editing target video using the desired search result.

The system 100 may perform image captioning to generate one or a plurality of shot texts for each shot. The image captioning may include still image captioning and/or video captioning.

The system 100 may select one or a plurality of frames (still images) for each shot.

In addition, visual information of a still image for each selected frame may be generated as text, ie, captioned.

A technical idea for generating visual information from a still image as text has been known. For such still image captioning, at least one deep learning model may be used. For example, a first deep learning model (eg, InceptionV3) for detecting at least one object from a still image, and a second deep learning model (eg, InceptionV3) for generating text using features extracted from the first deep learning model , RNN, LSTM, etc.) is used to perform captioning for each still image, and as a result, a still image, that is, text that can describe the frame for each frame can be generated.

The system 100 may determine one or a plurality of frames for each shot as a selection frame, and perform captioning for each selected frame to generate text describing the shot, that is, shot text. When a plurality of frames are selected, it may mean that a plurality of texts may be generated to describe any one shot.

Meanwhile, according to the technical idea of the present invention, the system 100 may use one or a plurality of frames included in the shot to describe the shot, that is, text for describing a still image, but if necessary, the system 100 may use a still image. In some cases, it is difficult to sufficiently explain the shot with only the video.

For example, when any one frame is selected from a specific shot and visual information of the frame is described as text through a captioning model, and a relatively short video including the selected frame from the specific shot, that is, a plurality of consecutive Generating text based on frames can have very different results.

For example, in any one still image, the text 'People are simply opening their mouths' may be generated. ' can be explained.

That is, it may be possible to generate more meaningful information, ie, text that can more accurately describe a corresponding shot, in the case of captioning a moving image including the still image compared to captioning the visual information of the still image.

Therefore, the system 100 according to the technical idea of the present invention may generate text by captioning the moving picture itself included in the shot when generating a text for describing the shot for each shot.

A technical idea for generating, ie, captioning, text for describing a relatively short video has also been known. For example, the SlowFast deep learning model released by Facebook AI Research can generate text that can explain relatively short videos. According to the technical idea of the present invention, in addition to this SlowFast model, an average expert in the technical field of the present invention can easily infer that various deep learning models can be designed, learned, and utilized to explain the text that describes the video. .

Meanwhile, a task of dividing the video to be edited into a plurality of shots has also been known. In general, shot boundary detection implements a predetermined engine for detecting a point at which visual information is converted/transformed to a significant degree, through which shot boundary detection can be performed.

Such shot boundary detection is typically implemented to detect a point at which a camera take or a camera property (direction change, zoom in/out, etc.) is significantly converted. According to the technical idea of the present invention, the camera property Even if this conversion is not performed, a deep learning model trained to detect a point where visual information is significantly converted, that is, a shot boundary, may be used to perform shot segmentation.

For example, a number of learning data, that is, a point at which the visual information is determined to be significantly transformed in the video is provided with labeled data, and the point at which the visual information is significantly converted using a deep learning model that has learned the plurality of learning data A convolutional neural network (CNN) capable of detecting may be implemented. When implementing such a deep learning model, an average expert in the technical field of the present invention will be able to easily infer that which point to detect as a shot boundary may vary depending on which point the labeling of the training data is performed.

Depending on how a configuration (eg, a deep learning model) for shot division is implemented, the length of each of the shots into which the video to be edited is divided may vary, and the number of selected frames may be determined differently for each shot. For example, for a relatively long shot, a plurality of frames for describing the corresponding shot may be selected, and for a relatively short shot, only one frame for describing the corresponding shot may be selected.

In addition, a plurality of shot texts may be generated for each shot, and in this case, the plurality of shot texts may be caption texts of different still images. It is a captioning text, and the remaining part may be a video captioning text.

Of course, according to an embodiment, the shot text may be only the captioning text of a still image or only the captioning text of a moving image.

In any case, the system 100 may segment the video to be edited and generate text that can better describe each segmented video.

Then, the user may search for only a part desired by the user through text in the video to be edited, and edit the video to be edited by using the search result if necessary.

For example, when the system 100 receives a search condition of “two people”, the system 100 includes the text “two people” among the shot texts as it is, or two people through semantic search. A shot text having the meaning of a name, that is, a matching shot text can be searched for. Of course, such a search condition is not necessarily one, but may be performed through a logical operation.

For example, when a selection condition of "two people" and "mountain" is input, shot text with two people and a mountain may be selected as the matching shot text.

For such a search, sematic search, not keyword search, may be utilized through a language model trained for NLP (Natural Language Processing).

Then, the system 100 may extract a matching image corresponding to the matching shot text. The matching image may be a shot corresponding to the matching shot text, or a still image (frame) or moving image corresponding to the matching shot text.

For example, in the above example, a matching shot in which two people exist, a specific frame image included in the matching shot and captioned, or a video included in the matching shot and captioned may be extracted as a matching image.

Then, according to the user's request, an editing command such as editing the editing target video by deleting the selected matching image or leaving only the matching image may be input through a predetermined method (such as a text command or a predetermined UI).

The selection of the matching image desired by the user can be very effectively selected based on the text by using the shot text in various ways, and video editing using this can be effectively performed.

For example, if you want to leave only matching images in which only dogs exist in a video of a dog and a cat, input "dog" as a selection condition, and then enter an edit command that deletes the remaining frames while leaving only the specified matching image. An edited video composed only of appearing frames can be obtained.

Various types of editing commands (selection, deletion, or order adjustment, etc.) may be possible.

Through this process, the user can very efficiently perform image processing using text in which visual information of the video to be edited is expressed.

Meanwhile, when generating shot text for describing each shot, it is preferable that the system 100 generates shot text that can representatively describe the shot.

And if the shot division is performed by detecting a point at which the visual information is changed significantly, the shot may be divided whenever the visual information is switched. In this case, the description for representing each shot is the central point of the shot. (For example, it may be preferable to perform based on a frame corresponding to the middle or the frame closest to the middle based on the number of frames).

As described above, the selection frame for simply describing the shot may be determined based on the position of the frame, but according to an exemplary embodiment, the selection frame needs to be determined based on whether a predetermined object is included in the corresponding frame.

For example, when image processing (search and/or editing) is performed based on human actions or numbers, it is preferable that a frame including a person for each shot is selected as a selection frame and text describing the frame is generated. can do. Only then, text-based image search and/or editing can be efficiently performed based on the object desired by the user.

Accordingly, the system 100 can restrictively determine a frame including a predetermined object from among the divided shots as a selection frame, and perform captioning on the determined selection frame to generate shot text describing the shot. have.

In order to determine the frame including the predetermined object as the selection frame, it should be possible to determine whether the object is included in the still image, and for this, a deep learning model for object detection may be provided in the system 100 is of course Deep learning models for object detection are well known. An average expert in the technical field of the present invention will be able to easily infer that a CNN model is usually used, and that deep learning models of R-CNN, Yolo, InceptionV3, and Resnet series can be utilized.

Meanwhile, not only whether a predetermined object is included, but also the location of the frame may be important in generating a description representing a corresponding shot. Therefore, it may be necessary to determine the selection frame in consideration of both of these two factors.

To this end, the system 100 controls the center frame when the object is included in the center frame corresponding to the center of each shot (eg, the mid-middle frame based on the frame position or the closest frame when there is no mid-middle frame). may be included in the selection frame. That is, the central frame is included in at least a frame for describing the shot, and some additional frames may be further selected as selection frames.

If the central frame does not include the object, the system 100 searches for frames including the object in the order of adjacent frames from the central frame, and includes the first searched frame in the selection frame. have.

Through this, shot text that can representatively describe the shot while satisfying both the frame position and whether the object of interest is included can be generated.

In addition, when video captioning is performed to generate the corresponding shot text, it may be preferable that the action (movement, state change, etc.) of the object is also captioned based on the object desired by the user.

For example, if a user wants to search for or edit a video based on a specific object (eg, a person, an animal, etc.), it is preferable that the action of the specific object is included not only in still image captioning but also in text generated as a result of video captioning. can

To this end, the system 100 performs video captioning on a video that is a plurality of consecutive frames of a predetermined number (including a predetermined length) while including the selection frame determined to include an object, and as a result, You can create shot text. A plurality of frames may be selected so that the selection frame is located in the middle of the moving picture, or the plurality of frames may be selected so that the selection frame is located at a predetermined position before or after the moving picture.

After all, according to the technical idea of the present invention, the system 100 generates text that well represents the visual information of the video to be edited, and such text can be generated for each divided shot, and the user wants to edit the text using the text. A matching image that is a part of the video may be extracted, and various image processing such as editing may be performed using this.

The configuration of the system 100 for implementing this technical idea may be as shown in FIG. 2 .

2 is a diagram showing a schematic configuration of a system for performing a video stream processing method through deep learning-based image captioning according to an embodiment of the present invention.

Referring to FIG. 2 , in order to implement a video stream processing method through image captioning based on deep learning according to the technical idea of the present invention, a system 100 according to the technical idea of the present invention is provided.

The system 100 may include a memory 120 in which a program for implementing the technical idea of the present invention is stored, and a processor 110 for executing the program stored in the memory 120 .

An average expert in the technical field of the present invention easily understands that the function and/or operation performed by the system 100 in this specification can be performed by driving the program stored in the memory 120 by the processor 110 . can be inferred.

An average expert in the art of the present invention will be able to easily infer that the processor 110 may be named in various names, such as a CPU or a mobile processor, depending on the implementation example of the system 100 . In addition, the system 100 may be implemented by organically combining a plurality of physical devices. In this case, at least one processor 110 is provided for each physical device to realize the system 100 of the present invention. An average expert in the art of the present invention can easily infer.

The memory 120 stores the program and may be implemented as any type of storage device that the processor can access to drive the program. Also, depending on the hardware implementation, the memory 120 may be implemented as a plurality of storage devices instead of any one storage device. Also, the memory 120 may include a temporary memory as well as a main memory. In addition, it may be implemented as a volatile memory or a non-volatile memory, and may be defined to include all types of information storage means implemented so that the program can be stored and driven by the processor.

The system 100 may be implemented as a web server connected to a web that the user's terminal can access according to an embodiment, may be installed in the user's terminal, or may be implemented in various ways, and perform the functions defined in this specification. It may be defined to include any type of data processing device that can be used.

In addition, various peripheral devices (peripheral device 1 130 to peripheral device N 130 - 1) may be further provided according to an embodiment of the system 100 . For example, an average expert in the art can easily infer that a keyboard, monitor, graphic card, communication device, etc. may be further included in the system 100 as peripheral devices.

3 is a diagram for explaining the concept of a video stream processing method through deep learning-based image captioning according to an embodiment of the present invention.

Referring to FIG. 3 , the system 100 may receive a video stream, that is, a video to be edited 10 .

Then, the system 100 may divide the video to be edited 10 into a plurality of shots S1, S2, S3, ..., Sn-1, Sn.

As described above, a deep learning model for shot boundary detection may be provided for such shot classification.

Then, the system 100 provides shot texts (st1, st2, st3, stn-) that can representatively describe the corresponding shot for each shot (S1, S2, S3, ..., Sn-1, Sn). 1, stn) can be created.

Then, the system 100 may receive a search condition as text and detect a matching shot text corresponding to the received search condition. For example, the matching shot text may be st2, st3, stn, and then the system 100 as a matching image, the matching shot (S2, S3, Sn) corresponding to the matching shot text (st2, st3, stn) or the matching A selected frame (still image) or selected frames (movie) included in the shots S2, S3, and Sn may be extracted as a matching image.

4 is a diagram for explaining the concept of determining a selection frame according to an embodiment of the present invention.

Referring to FIG. 4 , for a predetermined shot (eg, Si), a selection frame may need to be determined in order to generate shot text for describing the shot (eg, Si).

To this end, the system 100 may simply determine the center frame (eg, f4) as the selection frame in the shot (eg, Si).

However, according to an embodiment, the system 100 may determine the central frame (eg, f4) as a selection frame according to whether a predetermined object is detected in the central frame (eg, f4), or may not. have.

For example, when a predetermined object is included in the central frame (eg, f4) (eg, when an object is detected), the system 100 may determine the central frame (eg, f4) as a selection frame. If the object is not included in the central frame (eg, f4), the system 100 determines that the nearest frame (eg, f3, f5, etc., f3 and f5) of the central frame (eg, f4) is It may be determined in advance), it may be sequentially determined whether the object is included, and a frame determined as including the object first may be included in the selection frame.

Through this, a frame having a position close to the center while describing the shot based on the object desired by the user may be determined as the selection frame.

On the other hand, even when the shot text is captioned based on a moving picture, continuous frames as many as a predetermined number (or time) are selected as a moving picture simply from the central frame (eg, f4), and the moving picture is captured and shot Text may be generated.

However, as described above, when the selection frame including the object is determined, a plurality of frames including the determined selection frame are specified as a moving picture to be captioned, and the specified moving picture is captioned to obtain a shot text describing the shot. may be created.

5 and 6 are diagrams for explaining image processing according to an embodiment of the present invention.

As such, when the shot text is determined and the matching image is determined, the system 100 makes it easy to edit the target video 10 based on the matching image (FIGS. 5 and 6 describe a case where the matching shot is a matching image). can be edited.

For example, when a user wants to edit a matching image (eg, S2, S3, Sn) to delete the matching image and such a command is input, as shown in FIG. 5 , the system 100 displays the matching image (eg, S2, S3, S3). , Sn) may generate an image deleted from the editing target video 10 as an edited video.

For example, if the user leaves only the matching image (eg, S2, S3, Sn) and wants to create an edited video based on it, the system 100 as shown in FIG. 6 provides the matching image (eg, S2, S2, You can create an edited video that leaves only S3, Sn).

Of course, the user may further perform editing actions such as additionally inserting images or changing the order of the edited video.

As a result, according to the technical idea of the present invention, there is an effect that, for a video, text capable of explaining the visual information of the video can be generated and image processing can be performed very effectively by using the text.

On the other hand, the method according to the embodiment of the present invention may be implemented in the form of a computer-readable program command and stored in a computer-readable recording medium, and the control program and the target program according to the embodiment of the present invention are also implemented in the computer. It may be stored in a readable recording medium. The computer-readable recording medium includes all types of recording devices in which data readable by a computer system is stored.

The program instructions recorded on the recording medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the software field.

Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and floppy disks, and hardware devices specially configured to store and execute program instructions, such as magneto-optical media and ROM, RAM, flash memory, and the like. In addition, the computer-readable recording medium is distributed in network-connected computer systems, and computer-readable codes can be stored and executed in a distributed manner.

Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by an apparatus for electronically processing information using an interpreter or the like, for example, a computer.

The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

The above description of the present invention is for illustration, and those of ordinary skill in the art to which the present invention pertains can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a distributed manner, and likewise components described as distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included in the scope of the present invention. .

Claims (8)

dividing, by the system, the video to be edited into a plurality of shots;
generating, by the system, at least one shot text for each of the divided shots;
receiving, by the system, a search condition text for image search; and
selecting, by the system, matching shot text corresponding to the search condition text, and extracting a matching image corresponding to the selected matching shot text;
generating, by the system, at least one shot text for each of the divided shots,
determining at least one selection frame from among a plurality of frames included in each of the divided shots; and
generating the shot text through image captioning corresponding to the determined selection frame;
The step of determining at least one selection frame from among a plurality of frames included in each of the divided shots,
determining, as the at least one selected frame, a moving picture including a plurality of consecutive frames included in the divided shots,
The step of generating the shot text through image captioning corresponding to the determined selected frame comprises:
A video stream processing method through deep learning-based video captioning, comprising generating the shot text through video captioning for the video.
delete In a video stream processing method through video captioning based on deep learning,
dividing, by the system, the video to be edited into a plurality of shots;
generating, by the system, at least one shot text for each of the divided shots;
receiving, by the system, a search condition text for image search; and
selecting, by the system, matching shot text corresponding to the search condition text, and extracting a matching image corresponding to the selected matching shot text;
generating, by the system, at least one shot text for each of the divided shots,
determining at least one selection frame from among a plurality of frames included in each of the divided shots; and
generating the shot text through image captioning corresponding to the determined selection frame;
The video stream processing method through the deep learning-based image captioning,
generating, by the system, a first edited video in which the matching image is deleted from the editing target video; or
Video stream processing method through deep learning-based image captioning further comprising the step of the system generating a second edited video in which frames other than the matching image are deleted.
delete dividing, by the system, the video to be edited into a plurality of shots;
generating, by the system, at least one shot text for each of the divided shots;
receiving, by the system, a search condition text for image search; and
selecting, by the system, matching shot text corresponding to the search condition text, and extracting a matching image corresponding to the selected matching shot text;
generating, by the system, at least one shot text for each of the divided shots,
determining at least one selection frame from among a plurality of frames included in each of the divided shots; and
generating the shot text through image captioning corresponding to the determined selection frame;
The step of determining at least one selection frame from among a plurality of frames included in each of the divided shots,
determining a frame including a predetermined object from among the divided shots as the selection frame,
Determining a frame including a predetermined object from among the divided shots as the selection frame includes:
When the object is included in the center frame corresponding to the center of the divided shot, the center frame is included in the selection frame, and when the object is not included in the center frame, the order of adjacent frames from the center frame A video stream processing method through deep learning-based image captioning, comprising the step of searching for a frame including the object as described above and including the first searched frame in the selected frame.
dividing, by the system, the video to be edited into a plurality of shots;
generating, by the system, at least one shot text for each of the divided shots;
receiving, by the system, a search condition text for image search; and
selecting, by the system, matching shot text corresponding to the search condition text, and extracting a matching image corresponding to the selected matching shot text;
generating, by the system, at least one shot text for each of the divided shots,
determining at least one selection frame from among a plurality of frames included in each of the divided shots; and
generating the shot text through image captioning corresponding to the determined selection frame;
The step of generating the shot text through image captioning corresponding to the determined selected frame comprises:
A video stream processing method through deep learning-based image captioning, comprising generating the shot text as a result of video captioning for a video that is a plurality of consecutive frames of a predetermined number including the selection frame.
A computer program installed in a data processing device and stored in a computer-readable recording medium to perform the method according to any one of claims 1, 3, 5 or 6.
A system for a video stream processing method through deep learning-based image captioning, comprising:
processor; and
a memory for storing a computer program executed by the processor;
The processor drives the program,
A video to be edited is divided into a plurality of shots, at least one shot text is generated for each of the divided shots, a search condition text is input, a matching shot text corresponding to the search condition text is selected, and the selected matching Extracting a matching shot or matching frame corresponding to the shot text,
A system for determining a moving picture including a plurality of consecutive frames included in the divided shots as at least one selected frame, and generating the shot text through moving picture captioning for the moving picture.

Images